The minimal custom scheduler

[brigand]

🚀 Feature Proposal

There should be a configurable function that accepts a list of test suites to execute, and this code lets Jest know when a given test suite can be executed. Jest will give back a promise of the test suite completing.

This is the minimal custom scheduler, and is sufficient for my needs.

Motivation

Custom scheduling of tests has been brought up in several issues, but without any concrete proposals for how it would work. The issues were resolved with "probably won't fix", which is why I'm opening an issue before working on a PR.

The core requirements for this are integration and e2e test suites where some tests need to run to completion before others can start, but most tests can run concurrently (up to the number of workers).

Example

This is the minimal example of an implementation of this custom function, which works for both concurrent execution (with any number of workers) and run-in-band, as jest can freely delay execution of the actual test suite.

type Test = { path: string, run: () => Promise<unknown> }

module.exports = (tests: Array<Test>): Promise<Array<unknown>> => {
  return Promise.all(tests.map(test => test.run()));
}

My actual implementation with dependencies between test suites

// Provides a scheduler for jest test execution based on the conventions of this project.
// Note: currently one file in node_modules/jest-runtime is patched in a postinstall script
// to teach it to defer to this file for scheduling.
const fs = require('fs');

// Test files declare dependencies using the syntax `// @depend signup login`
// where the names correspond to output of `nameForPath`
function readParseDeps(path) {
  const lines = fs.readFileSync(path, 'utf-8').split('\n');

  const deps = new Set();
  for (const line of lines) {
    const m = line.match(/\/\/[ \t]*@depend?[ \t]+(.*)/);
    if (m) {
      for (const dep of m[1].split(/[^a-zA-Z0-9_-]+/).filter(Boolean)) {
        deps.add(dep);
      }
    }
  }
  return [...deps];
}

function assertString(value, message = '') {
  if (typeof value !== 'string') {
    throw new Error(`Expected value to be a string.${message ? '\n' : ''}${message}`);
  }

  return value;
}

// Represents execution stage of a single test suite
const Stage = {
  Initial: () => 'Initial',
  Running: () => 'Running',
  Complete: () => 'Complete',

  is: (a, b) => assertString((a && a.stage) || a) === b,
  isInitial: (value) => Stage.is(value, Stage.Initial()),
  isRunning: (value) => Stage.is(value, Stage.Running()),
  isComplete: (value) => Stage.is(value, Stage.Complete()),
};

// e.g. `nameForPath('/home/some-user/project/tests/signup.test.js') === 'signup'`
function nameForPath(path) {
  return path.replace(/^.*[/\\]/, '').replace(/(\.test)?\.js$/, '');
}

// Represents a single test
class Item {
  constructor({ path, run }) {
    this.stage = Stage.Initial();
    this.path = path;
    this.name = nameForPath(path);
    this._run = run;
    this.deps = readParseDeps(path);
  }

  canRun(items /* : Array<Item>*/) {
    for (const dep of this.deps) {
      const depItem = items.find((x) => x.name === dep);
      if (!depItem) {
        throw new Error(`Depends on unknown test ${JSON.stringify(dep)}`);
      }

      if (!Stage.isComplete(depItem)) {
        return false;
      }
    }

    return true;
  }

  async tryRun(items) {
    if (this.promise) {
      return this.promise.then(() => undefined);
    } else if (!this.canRun(items)) {
      return Promise.resolve();
    } else {
      this.stage = Stage.Running();
      this.promise = this._run().then((result) => {
        this.stage = Stage.Complete();
        return result;
      });

      return this.promise.then(() => undefined);
    }
  }

  toJSON() {
    return { name: this.name, stage: this.stage };
  }
}

// This is the main implementation that's responsible for scheduling test execution.
// ref: src/nm_patch/jest-runner/build/index.js L186
// called with Array<{path: string, run: () => Promise<unknown>}>
async function runInternal(testItems) {
  const items = testItems.map((ti) => new Item(ti));

  async function runItem(item) {
    await item.tryRun(items);

    // The test has completed, so poll other tests that depend on this one. They may
    // or may not be ready to execute. This recurses, so if the dependent tests run
    // they'll notify tests that depend on them, and so on until complete or deadlock.
    if (Stage.isComplete(item)) {
      const candidates = items.filter(
        (x) => Stage.isInitial(x) && x.deps.includes(item.name),
      );
      await Promise.all(candidates.map(runItem));
    }
  }

  // Do our initial runItem calls, of which only tests with no dependencies will
  // actually execute immediately.
  await Promise.all(items.map(runItem)).catch((error) => {
    console.error(`jest.config.run awaiting tryRun calls`, error);
    throw error;
  });

  // If we've reached this, tests may have passed/failed, or we deadlocked.
  // Let's prepare for the deadlock case.
  const byStage = {
    initial: items.filter((x) => Stage.isInitial(x)),
    running: items.filter((x) => Stage.isRunning(x)),
    complete: items.filter((x) => Stage.isComplete(x)),

    // special
    unfinished: items.filter((x) => !Stage.isComplete(x)),
    blocked: items.filter((x) => !Stage.isComplete(x) && !x.canRun()),
  };

  // Detect a deadlock and throw an error if it happens
  if (
    byStage.blocked.length &&
    byStage.blocked.length === byStage.unfinished.length
  ) {
    const statusMsg = Object.keys(byStage)
      .map((stage) => `${stage}: ${byStage[stage].map((x) => x.name)}`)
      .join('\n');
    const initial = byStage.blocked
      .map((x) => ` - ${x.name}: depends on ${x.deps}`)
      .join('\n');
    throw new Error(
      `Test run has deadlocked.\n${statusMsg}\nBlocked tests:\n${initial}`,
    );
  }

  // Normal test run completed, so just return the test results to jest.
  return Promise.all(items.map((x) => x.promise));
}

module.exports = async function run(testItems) {
  try {
    return await runInternal(testItems);
  } catch (error) {
    // Note: this catch does not handle test failure – only internal errors in this module.
    console.error(`jest.config.run.js: error running tests`, error);
    throw error;
  }
};

Pitch

I switched my integration tests from a custom script to jest a few months back, but due to not being able to control the scheduling of tests, I had to always run each test suite sequentially (and ended up essentially running it as a single large test suite, which means no output until the end).

When I revisited the problem recently and read through jest's source, I found this little bit of code and experimented with changing it to support the above API.

Surprisingly, it worked very well and in a few hours I had my integration tests running in parallel, with constraints on the ordering. I'm aware that a PR will be more work.

In both of my code examples, as well as the original jest source code (specifically in _createParallelTestRun), it just asks all of the tests to start as soon as it can. This allows jest to retain control over when tests concretely execute (for run-in-band, or when there are more suites than workers). The user wants to tell jest when a test run can start, and wants jest to tell it when the test run completes. That's the full contract.

This is a small API that's very flexible and allows jest to be used in situations where it currently struggles to fit.

[jeysal]

Thanks a lot for the detailed write-up, this seems more well-thought-out than what I've seen so far. Still, I'm sort of worried about introducing a completely alternative code path to serve such an API into jest-runner. I'm wondering if we could also serve this use case by somehow making it easier for you to extend jest-runner to do this and then use it as a custom runner, which is already possible through config. Replacing a module seems cleaner than branching into different code paths each used by a fraction of users, we've recently allowed people such customization options with testSequencer - except in this case you'd still want to use most functionality from jest-runner, not completely replace it.
cc @SimenB @scotthovestadt

[kryten87]

Thanks for the docs on testSequencer - that's almost what I need.

Is there any way to specify certain tests should be run "in band"?

I have tests for a bunch of Express routes which start up the server & shut it down, but I'm running into problems because Jest is trying to run then asynchronously. I know I can use the --runInBand flag to run all my tests synchronously, but for most of my code it doesn't matter.

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions]

This issue was closed because it has been stalled for 7 days with no activity. Please open a new issue if the issue is still relevant, linking to this one.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.