[DO NOT MERGE] WIP: run tests concurrently; see #2839 #4198

boneskull · 2020-03-03T01:21:06Z

~~This is experimental, untested, and broken. I'm opening it to track progress.~~

~~This is no longer experimental, broken, nor untested, but it does need to be split into separate PRs. This PR will not get merged, but I will reference the new PRs here.~~

NEW PRS:

This PR has been split into many separate ones:

Everything not marked optional is required to land this one way or another. Please follow the individual PRs for further discussion!

The two PRs in boldface are the most impactful.

It does, however, achieve a 50% speedup over our unit tests.

Things that need to happen:

Determine the bounds for what we can expect to work. I do not expect we can run our entire test suite this way.
If it's clear something should work but does not, we might need to swap out the worker threads for child processes. It shouldn't be too terribly difficult to create an adapter to allow operation either way. Worker threads will be more performant, if they can prove themselves widely suitable. However, there's stuff you can't do with worker threads--and nyc might not work with them (need to test this).
Decide whether to ship both worker and child-process implementations. If we can pull it off with a slick adapter, I think we should. Probably default to "worker thread mode" unless it's just too hinky.
Once we feel confident in the implementation, run LoopBack's test suites in parallel, and maybe TypeScript's. See if we can uncover any other problems.
Tests -- our own test runs should prove this out pretty well, but will still need some basic integration & unit tests. Check it works with ESM. Windows.
Documentation. Probably necessitates its own section in the docs, and will need a clear explanation of the limitations. Need to check generated API docs as well.
Verify browser build doesn't ship this stuff.
Use with "watch" mode. "Only rerunning changed code" and other enhancements to watch mode are out-of-scope.

Future, once this ships:

Party!! 🎉 🎈
The abstraction we're using, workerpool, purports to work in a browser. Can this work in a browser for Mocha?

boneskull · 2020-03-05T00:27:41Z

status:

Ensuring the event data (Test, Suite and Hook instances) can be communicated to the main process via the structured clone algorithm. This needs to be done as to not break our contract with reporters, since they expect these objects; these objects must have the correct prototype and references.

boneskull · 2020-03-05T18:06:08Z

update: (de-)serialization is too slow; when doing this, the parallel runs are slower against our unit test suite than serial.

the most performant solution would be to restrict what reporters can expect; e.g., not emit an entire Test object, but rather one that's a duck-typed subset. This would need to be documented, and should be considered a breaking change to the API. We'd still need implementations of certain functions known to be used, like fullTitle(), to avoid completely breaking third-party reporters.

boneskull · 2020-03-12T18:38:03Z

update: I've decided to abandon workers for now.

The main problem with them is that worker threads are not currently debuggable. This makes it both difficult to implement, and will be hostile to any user trying to debug tests. See nodejs/node#26609.

boneskull · 2020-03-12T20:14:18Z

A potential "gotcha" with these tests is that the duration of an individual test seems to have more variability if that test performs I/O (depending on the number of concurrent jobs). If you are running, say, 7 jobs in parallel, and you have 8 cores, there's not necessarily any "extra" cores for libuv's thread pool, so the I/O sits for longer waiting to get processed.

but, good news: using child processes (instead of workers), our Node.js integration tests are passing. When run with four (4) concurrent jobs, the execution time is cut from 2m23s to 46s.

There are some rough edges, and I suspect certain reporters will not work very well with this (instead of seeing the result of each test as it finishes, the results of each file are buffered and dumped at once). Also, I haven't yet written tests specifically intended to run in parallel, which should surface some issues with certain options (--allowUncaught, --delay & --retries are dubious).

boneskull · 2020-03-12T20:16:30Z

unbuffered output would make sense for certain reporters (dot, min, tap) but not the default (spec) because it'd be impossible to tell what was going on due to the indenting. unbuffered output seems like a good candidate for a future PR

boneskull · 2020-03-12T20:19:35Z

from what I can tell, there isn't a noticeable speedup when using child processes vs worker threads.

I'm not sure if running mocha vs _mocha with concurrent jobs makes a significant different in performance.

maybe we should add some benchmark tests (but not in this PR)

coveralls · 2020-03-12T22:46:58Z

Coverage increased (+0.2%) to 93.247% when pulling 0b13c92 on boneskull/issue/2839-concurrent into 38d579a on master.

boneskull · 2020-03-27T17:24:05Z

we need to increase the global default timeout from 200ms. running tasks concurrently in CI means that there's much more variability around how long any given test will take (see AppVeyor builds)

boneskull · 2020-03-27T18:47:57Z

thoughts on how --file should work:

--file causes test files to be loaded in a specific order. The difference between mocha --file a.spec.js "*.spec.js" and mocha "*.spec.js" is that we are guaranteed that a.spec.js will always be executed first; otherwise the order in which files are executed is dependent on either glob or your shell's wildcard expansion.

When running tests with --parallel, we have even fewer guarantees of the order in which files are executed; the execution time of an individual file takes also becomes a variable.

Why would we care about running files in order in the first place? Why is --file a thing?

This is because in a.spec.js, above, we might have top-level hooks; hooks that are attached to the root suite. A hook attached to the root suite will execute across all subsequently-loaded files. This means you can do some "global" setup in a "before each" or "before all" hook. If your suites themselves do not depend on async behavior to programmatically generate--but all your tests need async setup--this is a more appropriate solution than using --delay.

The current implementation of --parallel does not have a single root suite. Each file has its own root suite. This breaks the above use case of --file.

To solve this:

Any test file specified by --file will not run in parallel. These must run in serial.
I'll look into sharing (or cloning, or otherwise reproducing) the root suite in its state after the above serial run.

This surfaces a couple holes in our integration tests. Note that they all currently pass on Travis CI!

boneskull · 2020-03-27T19:22:45Z

after looking at this a bit, I don't think there's a way to make --file work in a way that's akin to what we have now (in a way that I'd want to implement).

instead of --file, if someone needs this sort of behavior, you're going to need some boilerplate:

// root-hooks.js

beforeEach(async function() {
  await somethingAsync();
});

// test.spec.js
require('./root-hooks');

describe('my thing', function() {

});

boneskull · 2020-03-27T21:02:52Z

it's truly annoying that it's not easy (I don't know how to do it!) to re-use a Mocha instance by removing files and adding new ones after a call to mocha.run() has completed. I'm not entirely sure what the problem is, but something certainly breaks.

giltayar · 2020-03-28T06:01:31Z

@boneskull a thought: when running in parallel, and with --watch, you will probably remove the need for clearing the module cache. If this becomes the way to go, even for watch with non-parallel execution, then that solves the ES module cache problem (the problem is that there is no mechanism to clear the ES module cache).

juergba · 2020-03-28T15:59:42Z

The root-suite before/after hooks problematic is not only limited to the --file option. Root hooks can be declared in any of the test files, independently of its index. Since you don't know the content of those hooks, you can't decide wether to execute them once per test run or once for each sub-process.
@bajtos already mentioned this problem.

boneskull · 2020-03-29T19:29:08Z

OK. Since:

The behavior of root-level hooks are dependent upon the order in which test files are run, and
--parallel will run test files in a nondeterministic order,
We cannot support --file

We can support root-level hooks still, but as this PR is written, they will only be applicable to the tests within the file in which they are defined.

boneskull · 2020-03-31T00:40:07Z

@giltayar Good insight. That'd make getting --watch running with this more valuable. I'll investigate further.

OK, so this is looking pretty solid. Next steps:

look at watch mode
write docs

This is going to be a pretty significant thing to review...

boneskull · 2020-04-17T22:53:52Z

Note that we're likely to see this message in CI:

(node:6291) Warning: (Mocha) not enough CPU cores available (2) to run multiple jobs; avoid --parallel on this machine

This is true; we only have one core for the main process, and another for a single worker process. This makes things slightly slower, but I think it's important to run our test suite in parallel mode. That said, we should also not ignore serial mode, so I think a serial run of the unit tests as a new build job is appropriate.

* test helper improvements - enables `RawResult` (the result of `invokeMocha()` or `invokeMochaAsync()`) to check the exit code via `to have code` assertion - add passed/failing/pending assertions for `RawRunResult` (the result of `runMocha()` or `runMochaAsync()`) - expose `getSummary()`, which can be used with a `RawResult` (when not failing) - reorganize the module a bit - create `runMochaAsync()` and `runMochaJSONAsync()` which are like `runMocha()` and `runMochaJSON()` except return `Promise`s - better trapping of JSON parse errors - better default handling of `STDERR` output in subprocesses (print it instead of suppress it!) - do not let the `DEBUG` env variable reach subprocesses _unless it was explicitly supplied_ - add an easily copy-paste-able `command` prop to summary - add some missing docstrings Ref: #4198 * increase timeout in watch test for CI the same code should be in PR #4240

> (this PR depends on most other PRs linked to #4198, so they should be merged first; documentation will be in another PR) This PR adds support for running test files in parallel via `--parallel`. For many cases, this should "just work." When the `--parallel` flag is supplied, Mocha will swap out the default `Runner` (`lib/runner.js`) for `BufferedRunner` (`lib/buffered-runner.js`). `BufferedRunner` _extends_ `Runner`. `BufferedRunner#run()` is the main point of extension. Instead of executing the tests in serial, it will create a pool of worker processes (not worker _threads_) based on the maximum job count (`--jobs`; defaults to `<number of CPU cores> - 1`). Both `BufferedRunner` and the `worker` module consume the abstraction layer, [workerpool](https://npm.im/workerpool). `BufferedRunner#run()` does _not_ load the test files, unlike `Runner#run()`. Instead, it has a list of test files, and puts these into an async queue. The `EVENT_RUN_BEGIN` event is then emitted. As files enter the queue, `BufferedRunner#run()` tells `workerpool` to execute the `run()` function of the pool. `workerpool` then launches as many worker processes are needed--up to the maximum--and executes the `run()` function with a single filepath and any options for a `Mocha` instance. The first time `lib/worker.js` is invoked, it will "bootstrap" itself, by handling `--require`'d modules and validating the UI. Note that _reporter validation_ does not occur. Once bootstrapped, it instantiate `Mocha`, add the single file, swap any reporter out for the `Buffered` reporter (`lib/reporters/buffered.js`) then execute `Mocha#run()`, which invokes `Runner#run()`. The `Buffered` reporter listens for events emitting from the `Runner` instance, like a reporter usually does. But instead of outputting to the console, it buffers the events in a queue. Once the file has completed running, the queue is drained: the events collected are (trivially) serialized for transmission back to the main process. `BufferedRunner#run()` receives the list of events, trivially _deserializes_ them, and re-emits the events to whatever the chosen reporter is (e.g., the `spec` reporter). In this way, the reporters don't know that the tests were run in parallel. Practically, the user will see reporter output in "chunks" instead of the "stream" of results they usually expect. This method ensures that while the test files run in a nondeterministic order, the reporter output will be deterministic for any given test file. Once the result (the queue of events) has been returned to the main process, the worker process stays open, but waits for further instruction. If there are more files in `BufferedRunner#run()`'s queue, `workerpool` will instruct the worker to take the next file from the list, and so on, and so forth. When all files have executed, the pool terminates, the `EVENT_RUN_END` event is emitted, and the reporter handles it. > (this section is pasted from the documentation with minimal edits) Due to the nature of the following reporters, they cannot work when running tests in parallel: - `markdown` - `progress` - `json-stream` These reporters expect Mocha to know _how many tests it plans to run_ before execution. This information is unavailable in parallel mode, as test files are loaded only when they are about to be run. In serial mode, tests results will "stream" as they occur. In parallel mode, reporter output is _buffered_; reporting will occur after each file is completed. In practice, the reporter output will appear in "chunks" (but will otherwise be identical). In parallel mode, we have no guarantees about the order in which test files will be run--or what process runs them--as it depends on the execution times of the test files. Because of this, the following options _cannot be used_ in parallel mode: - `--file` - `--sort` - `--delay` Because running tests in parallel mode uses more system resources at once, the OS may take extra time to schedule and complete some operations. For this reason, test timeouts may need to be increased either globally or otherwise. When used with `--bail` (or `this.bail()`) to exit after the first failure, it's likely other tests will be running at the same time. Mocha must shut down its worker processes before exiting. Likewise, subprocesses may throw uncaught exceptions. When used with `--allow-uncaught`, Mocha will "bubble" this exception to the main process, but still must shut down its processes. > _NOTE: This only applies to test files run parallel mode_. A root-level hook is a hook in a test file which is _not defined_ within a suite. An example using the `bdd` interface: ```js // test/setup.js beforeEach(function() { doMySetup(); }); afterEach(function() { doMyTeardown(); }); ``` When run (in the default "serial" mode) via `mocha --file "./test/setup.js" "./test/**/*.spec.js"`, `setup.js` will be executed _first_, and install the two hooks shown above for every test found in `./test/**/*.spec.js`. **When Mocha runs in parallel mode, test files do not share the same process.** Consequently, a root-level hook defined in test file _A_ won't be present in test file _B_. There are a (minimum of) two workarounds for this: 1. `require('./setup.js')` or `import './setup.js'` at the top of every test file. Best avoided for those averse to boilerplate. 1. _Recommended_: Define root-level hooks in a required file, using the new (also as of VERSION) Root Hook Plugin system. Parallel mode is only available in Node.js. If you find your tests don't work properly when run with `--parallel`, either shrug and move on, or use this handy-dandy checklist to get things working: - ✅ Ensure you are using a supported reporter. - ✅ Ensure you are not using other unsupported flags. - ✅ Double-check your config file; options set in config files will be merged with any command-line option. - ✅ Look for root-level hooks in your tests. Move them into a root hook plugin. - ✅ Do any assertion, mock, or other test libraries you're consuming use root hooks? They may need to be migrated for compatibility with parallel mode. - ✅ If tests are unexpectedly timing out, you may need to increase the default test timeout (via `--timeout`) - ✅ Ensure your tests do not depend on being run in a specific order. - ✅ Ensure your tests clean up after themselves; remove temp files, handles, sockets, etc. Don't try to share state or resources between test files. Some types of tests are _not_ so well-suited to run in parallel. For example, extremely timing-sensitive tests, or tests which make I/O requests to a limited pool of resources (such as opening ports, or automating browser windows, hitting a test DB, or remote server, etc.). Free-tier cloud CI services may not provide a suitable multi-core container or VM for their build agents. Regarding expected performance gains in CI: your mileage may vary. It may help to use a conditional in a `.mocharc.js` to check for `process.env.CI`, and adjust the job count as appropriate. It's unlikely (but not impossible) to see a performance gain from a job count _greater than_ the number of available CPU cores. That said, _play around with the job count_--there's no one-size-fits all, and the unique characteristics of your tests will determine the optimal number of jobs; it may even be that fewer is faster! - updated signal handling in `bin/mocha` to a) better work with Windows, and b) work properly with `--parallel` to avoid leaving zombie workers - docstrings in `lib/cli/collect-files.js` - refactors in `lib/cli/run-helpers.js` and `lib/cli/watch-run.js`. We now have four methods: - `watchRun()` - serial + watch - `singleRun()` - serial - `parallelWatchRun()` - parallel + watch - `parallelRun()` - parallel - `lib/cli/run.js` and `lib/cli/run-option-metadata.js`: additions for new options and checks for incompatibility - add `lib/reporters/buffered.js` (`Buffered`); this reporter is _not_ re-exported in `Mocha.reporters`, since it should only be invoked internally. - tweak `landing` reporter to avoid referencing `Runner#total`, which is incompatible with parallel mode. It didn't need to do so in the first place! - the `tap` reporter now outputs the plan at the _end_ instead of at the beginning (avoiding a call to `Runner#grepTotal()`, which is incompatible with parallel mode). This is within spec, so should not be a breaking change. - add `lib/buffered-runner.js` (`BufferedRunner`); subclass of `Runner` which overrides the `run()` method. - There's a little custom finite state machine in here. didn't want to pull in a third-party module, but we should consider doing so if we use FSM's elsewhere. - when `DEBUG=mocha:parallel*` is in the env, this module will output statistics about the worker pool every 5s - the `run()` method looks a little weird because I wanted to use `async/await`, but the method it is overriding (`Runner#run`) is _not_ `async` - traps `SIGINT` to gracefully terminate the pool - pulls in [promise.allsettled](https://npm.im/promise.allsettled) polyfill to handle workers that may have rejected with uncaught exceptions - "bail" support is best-effort. - the `ABORTING` state is only for interruption via `SIGINT` or if `allowUncaught` is true and we get an uncaught exception - `Hook`, `Suite`, `Test`: add a `serialize()` method. This pulls out the most relevant information about the object for transmission over IPC. It's called by worker processes for each event received by its `Runner`; event arguments (e.g., `test` or `suite`) are serialized in this manner. Note that this _limits what reporters have access to_, which may break compatibility with third-party reporters that may use information that is missing from the serialized object. As those cases arise, we can add more information to the serialized objects (in some cases). The `$$` convention tells the _deserializer_ to turn the property into a function which returns the passed value, e.g., `test.fullTitle()`. - `lib/mocha.js`: - refactor `Mocha#reporter` for nicer parameter & variable names - rename `loadAsync` to `lazyLoadFiles`, which is more descriptive, IMO. It's a private property, so should not be a breaking change. - Constructor will dynamically choose the appropriate `Runner` - `lib/runner.js`: `BufferedRunner` needs the options from `Mocha#options`, so I updated the parent method to define the parameter. It is unused here. - add `lib/serializer.js`: on the worker process side, manages event queue serialization; manages deserialization of the event queue in the main process. - I spent a long time trying to get this working. We need to account for things like `Error` instances, with their stack traces, since those can be event arguments (e.g., `EVENT_TEST_FAIL` sends both a `Test` and the `Error`). It's impossible to serialize circular (self-referential) objects, so we need to account for those as well. - Not super happy with the deserialization algorithm, since it's recursive, but it shouldn't be too much of an issue because the serializer won't output circular structures. - Attempted to avoid prototype pollution issues - Much of this works by mutating objects, mostly because it can be more performant. The code can be changed to be "more immutable", as that's likely to be easier to understand, if it doesn't impact performance too much. We're serializing potentially very large arrays of stuff. - The `__type` prop is a hint for the deserializer. This convention allows us to re-expand plain objects back into `Error` instances, for example. You can't send an `Error` instance over IPC! - add `lib/worker.js`: - registers its `run()` function with `workerpool` to be called by main process - if `DEBUG=mocha:parallel*` is set, will output information (on an interval) about long-running test files - afaik the only way `run()` can reject is if `allowUncaught` is true or serialization fails - any user-supplied `reporter` value is replaced with the `Buffered` reporter. thus, reporters are not validated. - the worker uses `Runner`, like usual. - tests: - see `test/integration/options/parallel.spec.js` for the interesting stuff - upgrade `unexpected` for "to have readonly property" assertion - upgrade `unexpected-eventemitter` for support async function support - integration test helpers allow Mocha's developers to use `--bail` and `--parallel`, but will default to `--no-bail` and `--no-parallel`. - etc: - update `.eslintrc.yml` for new Node-only files - increase default timeout to `1000` (also seen in another PR) and use `parallel` mode by default in `.mocharc.yml` - run node unit tests _in serial_ as sort of a smoke test, as otherwise all our tests would be run in parallel - karma, browserify: ignore files for parallel support - force color output in CI. this is nice on travis, but ugly on appveyor. either way, it's easier to read than having no color Ref: #4198

(documentation will be in another PR) Adds "root hook plugins", a system to define root hooks via files loaded with `--require`. This enables root hooks to work in parallel mode. Because parallel mode runs files in a non-deterministic order, and files do not share a `Mocha` instance, it is not possible to share these hooks with other test files. This change also works well with third-party libraries for Mocha which need the behavior; these can now be trivially consumed by adding `--require` or `require: 'some-library'` in Mocha's config file. The way it works is: 1. When a file is loaded via `--require`, we check to see if that file exports a property named `mochaHooks` (can be multiple files). 1. If it does, we save a reference to the property. 1. After Yargs' validation phase, we use async middleware to execute root hook plugin functions--or if they are objects, just collect them--and we flatten all hooks found into four buckets corresponding to the four hook types. 1. Once `Mocha` is instantiated, if it is given a `rootHooks` option, those hooks are applied to the root suite. This works with parallel tests because we can save a reference to the flattened hooks in each worker process, and a new `Mocha` instance is created with them for each test file. * * * Tangential: - Because a root hook plugin can be defined as an `async` function, I noticed that `utils.type()` does not return `function` for async functions; it returns `asyncfunction`. I've added a (Node-specific, for now) test for this. - `handleRequires` is now `async`, since it will need to be anyway to support ESM and calls to `import()`. - fixed incorrect call to `fs.existsSync()` Ref: #4198

> (this PR depends on most other PRs linked to #4198, so they should be merged first; documentation will be in another PR) This PR adds support for running test files in parallel via `--parallel`. For many cases, this should "just work." When the `--parallel` flag is supplied, Mocha will swap out the default `Runner` (`lib/runner.js`) for `BufferedRunner` (`lib/buffered-runner.js`). `BufferedRunner` _extends_ `Runner`. `BufferedRunner#run()` is the main point of extension. Instead of executing the tests in serial, it will create a pool of worker processes (not worker _threads_) based on the maximum job count (`--jobs`; defaults to `<number of CPU cores> - 1`). Both `BufferedRunner` and the `worker` module consume the abstraction layer, [workerpool](https://npm.im/workerpool). `BufferedRunner#run()` does _not_ load the test files, unlike `Runner#run()`. Instead, it has a list of test files, and puts these into an async queue. The `EVENT_RUN_BEGIN` event is then emitted. As files enter the queue, `BufferedRunner#run()` tells `workerpool` to execute the `run()` function of the pool. `workerpool` then launches as many worker processes are needed--up to the maximum--and executes the `run()` function with a single filepath and any options for a `Mocha` instance. The first time `lib/worker.js` is invoked, it will "bootstrap" itself, by handling `--require`'d modules and validating the UI. Note that _reporter validation_ does not occur. Once bootstrapped, it instantiate `Mocha`, add the single file, swap any reporter out for the `Buffered` reporter (`lib/reporters/buffered.js`) then execute `Mocha#run()`, which invokes `Runner#run()`. The `Buffered` reporter listens for events emitting from the `Runner` instance, like a reporter usually does. But instead of outputting to the console, it buffers the events in a queue. Once the file has completed running, the queue is drained: the events collected are (trivially) serialized for transmission back to the main process. `BufferedRunner#run()` receives the list of events, trivially _deserializes_ them, and re-emits the events to whatever the chosen reporter is (e.g., the `spec` reporter). In this way, the reporters don't know that the tests were run in parallel. Practically, the user will see reporter output in "chunks" instead of the "stream" of results they usually expect. This method ensures that while the test files run in a nondeterministic order, the reporter output will be deterministic for any given test file. Once the result (the queue of events) has been returned to the main process, the worker process stays open, but waits for further instruction. If there are more files in `BufferedRunner#run()`'s queue, `workerpool` will instruct the worker to take the next file from the list, and so on, and so forth. When all files have executed, the pool terminates, the `EVENT_RUN_END` event is emitted, and the reporter handles it. > (this section is pasted from the documentation with minimal edits) Due to the nature of the following reporters, they cannot work when running tests in parallel: - `markdown` - `progress` - `json-stream` These reporters expect Mocha to know _how many tests it plans to run_ before execution. This information is unavailable in parallel mode, as test files are loaded only when they are about to be run. In serial mode, tests results will "stream" as they occur. In parallel mode, reporter output is _buffered_; reporting will occur after each file is completed. In practice, the reporter output will appear in "chunks" (but will otherwise be identical). In parallel mode, we have no guarantees about the order in which test files will be run--or what process runs them--as it depends on the execution times of the test files. Because of this, the following options _cannot be used_ in parallel mode: - `--file` - `--sort` - `--delay` Because running tests in parallel mode uses more system resources at once, the OS may take extra time to schedule and complete some operations. For this reason, test timeouts may need to be increased either globally or otherwise. When used with `--bail` (or `this.bail()`) to exit after the first failure, it's likely other tests will be running at the same time. Mocha must shut down its worker processes before exiting. Likewise, subprocesses may throw uncaught exceptions. When used with `--allow-uncaught`, Mocha will "bubble" this exception to the main process, but still must shut down its processes. > _NOTE: This only applies to test files run parallel mode_. A root-level hook is a hook in a test file which is _not defined_ within a suite. An example using the `bdd` interface: ```js // test/setup.js beforeEach(function() { doMySetup(); }); afterEach(function() { doMyTeardown(); }); ``` When run (in the default "serial" mode) via `mocha --file "./test/setup.js" "./test/**/*.spec.js"`, `setup.js` will be executed _first_, and install the two hooks shown above for every test found in `./test/**/*.spec.js`. **When Mocha runs in parallel mode, test files do not share the same process.** Consequently, a root-level hook defined in test file _A_ won't be present in test file _B_. There are a (minimum of) two workarounds for this: 1. `require('./setup.js')` or `import './setup.js'` at the top of every test file. Best avoided for those averse to boilerplate. 1. _Recommended_: Define root-level hooks in a required file, using the new (also as of VERSION) Root Hook Plugin system. Parallel mode is only available in Node.js. If you find your tests don't work properly when run with `--parallel`, either shrug and move on, or use this handy-dandy checklist to get things working: - ✅ Ensure you are using a supported reporter. - ✅ Ensure you are not using other unsupported flags. - ✅ Double-check your config file; options set in config files will be merged with any command-line option. - ✅ Look for root-level hooks in your tests. Move them into a root hook plugin. - ✅ Do any assertion, mock, or other test libraries you're consuming use root hooks? They may need to be migrated for compatibility with parallel mode. - ✅ If tests are unexpectedly timing out, you may need to increase the default test timeout (via `--timeout`) - ✅ Ensure your tests do not depend on being run in a specific order. - ✅ Ensure your tests clean up after themselves; remove temp files, handles, sockets, etc. Don't try to share state or resources between test files. Some types of tests are _not_ so well-suited to run in parallel. For example, extremely timing-sensitive tests, or tests which make I/O requests to a limited pool of resources (such as opening ports, or automating browser windows, hitting a test DB, or remote server, etc.). Free-tier cloud CI services may not provide a suitable multi-core container or VM for their build agents. Regarding expected performance gains in CI: your mileage may vary. It may help to use a conditional in a `.mocharc.js` to check for `process.env.CI`, and adjust the job count as appropriate. It's unlikely (but not impossible) to see a performance gain from a job count _greater than_ the number of available CPU cores. That said, _play around with the job count_--there's no one-size-fits all, and the unique characteristics of your tests will determine the optimal number of jobs; it may even be that fewer is faster! - updated signal handling in `bin/mocha` to a) better work with Windows, and b) work properly with `--parallel` to avoid leaving zombie workers - docstrings in `lib/cli/collect-files.js` - refactors in `lib/cli/run-helpers.js` and `lib/cli/watch-run.js`. We now have four methods: - `watchRun()` - serial + watch - `singleRun()` - serial - `parallelWatchRun()` - parallel + watch - `parallelRun()` - parallel - `lib/cli/run.js` and `lib/cli/run-option-metadata.js`: additions for new options and checks for incompatibility - add `lib/reporters/buffered.js` (`Buffered`); this reporter is _not_ re-exported in `Mocha.reporters`, since it should only be invoked internally. - tweak `landing` reporter to avoid referencing `Runner#total`, which is incompatible with parallel mode. It didn't need to do so in the first place! - the `tap` reporter now outputs the plan at the _end_ instead of at the beginning (avoiding a call to `Runner#grepTotal()`, which is incompatible with parallel mode). This is within spec, so should not be a breaking change. - add `lib/buffered-runner.js` (`BufferedRunner`); subclass of `Runner` which overrides the `run()` method. - There's a little custom finite state machine in here. didn't want to pull in a third-party module, but we should consider doing so if we use FSM's elsewhere. - when `DEBUG=mocha:parallel*` is in the env, this module will output statistics about the worker pool every 5s - the `run()` method looks a little weird because I wanted to use `async/await`, but the method it is overriding (`Runner#run`) is _not_ `async` - traps `SIGINT` to gracefully terminate the pool - pulls in [promise.allsettled](https://npm.im/promise.allsettled) polyfill to handle workers that may have rejected with uncaught exceptions - "bail" support is best-effort. - the `ABORTING` state is only for interruption via `SIGINT` or if `allowUncaught` is true and we get an uncaught exception - `Hook`, `Suite`, `Test`: add a `serialize()` method. This pulls out the most relevant information about the object for transmission over IPC. It's called by worker processes for each event received by its `Runner`; event arguments (e.g., `test` or `suite`) are serialized in this manner. Note that this _limits what reporters have access to_, which may break compatibility with third-party reporters that may use information that is missing from the serialized object. As those cases arise, we can add more information to the serialized objects (in some cases). The `$$` convention tells the _deserializer_ to turn the property into a function which returns the passed value, e.g., `test.fullTitle()`. - `lib/mocha.js`: - refactor `Mocha#reporter` for nicer parameter & variable names - rename `loadAsync` to `lazyLoadFiles`, which is more descriptive, IMO. It's a private property, so should not be a breaking change. - Constructor will dynamically choose the appropriate `Runner` - `lib/runner.js`: `BufferedRunner` needs the options from `Mocha#options`, so I updated the parent method to define the parameter. It is unused here. - add `lib/serializer.js`: on the worker process side, manages event queue serialization; manages deserialization of the event queue in the main process. - I spent a long time trying to get this working. We need to account for things like `Error` instances, with their stack traces, since those can be event arguments (e.g., `EVENT_TEST_FAIL` sends both a `Test` and the `Error`). It's impossible to serialize circular (self-referential) objects, so we need to account for those as well. - Not super happy with the deserialization algorithm, since it's recursive, but it shouldn't be too much of an issue because the serializer won't output circular structures. - Attempted to avoid prototype pollution issues - Much of this works by mutating objects, mostly because it can be more performant. The code can be changed to be "more immutable", as that's likely to be easier to understand, if it doesn't impact performance too much. We're serializing potentially very large arrays of stuff. - The `__type` prop is a hint for the deserializer. This convention allows us to re-expand plain objects back into `Error` instances, for example. You can't send an `Error` instance over IPC! - add `lib/worker.js`: - registers its `run()` function with `workerpool` to be called by main process - if `DEBUG=mocha:parallel*` is set, will output information (on an interval) about long-running test files - afaik the only way `run()` can reject is if `allowUncaught` is true or serialization fails - any user-supplied `reporter` value is replaced with the `Buffered` reporter. thus, reporters are not validated. - the worker uses `Runner`, like usual. - tests: - see `test/integration/options/parallel.spec.js` for the interesting stuff - upgrade `unexpected` for "to have readonly property" assertion - upgrade `unexpected-eventemitter` for support async function support - integration test helpers allow Mocha's developers to use `--bail` and `--parallel`, but will default to `--no-bail` and `--no-parallel`. - etc: - update `.eslintrc.yml` for new Node-only files - increase default timeout to `1000` (also seen in another PR) and use `parallel` mode by default in `.mocharc.yml` - run node unit tests _in serial_ as sort of a smoke test, as otherwise all our tests would be run in parallel - karma, browserify: ignore files for parallel support - force color output in CI. this is nice on travis, but ugly on appveyor. either way, it's easier to read than having no color Ref: #4198

(documentation will be in another PR) Adds "root hook plugins", a system to define root hooks via files loaded with `--require`. This enables root hooks to work in parallel mode. Because parallel mode runs files in a non-deterministic order, and files do not share a `Mocha` instance, it is not possible to share these hooks with other test files. This change also works well with third-party libraries for Mocha which need the behavior; these can now be trivially consumed by adding `--require` or `require: 'some-library'` in Mocha's config file. The way it works is: 1. When a file is loaded via `--require`, we check to see if that file exports a property named `mochaHooks` (can be multiple files). 1. If it does, we save a reference to the property. 1. After Yargs' validation phase, we use async middleware to execute root hook plugin functions--or if they are objects, just collect them--and we flatten all hooks found into four buckets corresponding to the four hook types. 1. Once `Mocha` is instantiated, if it is given a `rootHooks` option, those hooks are applied to the root suite. This works with parallel tests because we can save a reference to the flattened hooks in each worker process, and a new `Mocha` instance is created with them for each test file. * * * Tangential: - Because a root hook plugin can be defined as an `async` function, I noticed that `utils.type()` does not return `function` for async functions; it returns `asyncfunction`. I've added a (Node-specific, for now) test for this. - `handleRequires` is now `async`, since it will need to be anyway to support ESM and calls to `import()`. - fixed incorrect call to `fs.existsSync()` Ref: #4198

> (this PR depends on most other PRs linked to #4198, so they should be merged first; documentation will be in another PR) This PR adds support for running test files in parallel via `--parallel`. For many cases, this should "just work." When the `--parallel` flag is supplied, Mocha will swap out the default `Runner` (`lib/runner.js`) for `BufferedRunner` (`lib/buffered-runner.js`). `BufferedRunner` _extends_ `Runner`. `BufferedRunner#run()` is the main point of extension. Instead of executing the tests in serial, it will create a pool of worker processes (not worker _threads_) based on the maximum job count (`--jobs`; defaults to `<number of CPU cores> - 1`). Both `BufferedRunner` and the `worker` module consume the abstraction layer, [workerpool](https://npm.im/workerpool). `BufferedRunner#run()` does _not_ load the test files, unlike `Runner#run()`. Instead, it has a list of test files, and puts these into an async queue. The `EVENT_RUN_BEGIN` event is then emitted. As files enter the queue, `BufferedRunner#run()` tells `workerpool` to execute the `run()` function of the pool. `workerpool` then launches as many worker processes are needed--up to the maximum--and executes the `run()` function with a single filepath and any options for a `Mocha` instance. The first time `lib/worker.js` is invoked, it will "bootstrap" itself, by handling `--require`'d modules and validating the UI. Note that _reporter validation_ does not occur. Once bootstrapped, it instantiate `Mocha`, add the single file, swap any reporter out for the `Buffered` reporter (`lib/reporters/buffered.js`) then execute `Mocha#run()`, which invokes `Runner#run()`. The `Buffered` reporter listens for events emitting from the `Runner` instance, like a reporter usually does. But instead of outputting to the console, it buffers the events in a queue. Once the file has completed running, the queue is drained: the events collected are (trivially) serialized for transmission back to the main process. `BufferedRunner#run()` receives the list of events, trivially _deserializes_ them, and re-emits the events to whatever the chosen reporter is (e.g., the `spec` reporter). In this way, the reporters don't know that the tests were run in parallel. Practically, the user will see reporter output in "chunks" instead of the "stream" of results they usually expect. This method ensures that while the test files run in a nondeterministic order, the reporter output will be deterministic for any given test file. Once the result (the queue of events) has been returned to the main process, the worker process stays open, but waits for further instruction. If there are more files in `BufferedRunner#run()`'s queue, `workerpool` will instruct the worker to take the next file from the list, and so on, and so forth. When all files have executed, the pool terminates, the `EVENT_RUN_END` event is emitted, and the reporter handles it. > (this section is pasted from the documentation with minimal edits) Due to the nature of the following reporters, they cannot work when running tests in parallel: - `markdown` - `progress` - `json-stream` These reporters expect Mocha to know _how many tests it plans to run_ before execution. This information is unavailable in parallel mode, as test files are loaded only when they are about to be run. In serial mode, tests results will "stream" as they occur. In parallel mode, reporter output is _buffered_; reporting will occur after each file is completed. In practice, the reporter output will appear in "chunks" (but will otherwise be identical). In parallel mode, we have no guarantees about the order in which test files will be run--or what process runs them--as it depends on the execution times of the test files. Because of this, the following options _cannot be used_ in parallel mode: - `--file` - `--sort` - `--delay` Because running tests in parallel mode uses more system resources at once, the OS may take extra time to schedule and complete some operations. For this reason, test timeouts may need to be increased either globally or otherwise. When used with `--bail` (or `this.bail()`) to exit after the first failure, it's likely other tests will be running at the same time. Mocha must shut down its worker processes before exiting. Likewise, subprocesses may throw uncaught exceptions. When used with `--allow-uncaught`, Mocha will "bubble" this exception to the main process, but still must shut down its processes. > _NOTE: This only applies to test files run parallel mode_. A root-level hook is a hook in a test file which is _not defined_ within a suite. An example using the `bdd` interface: ```js // test/setup.js beforeEach(function() { doMySetup(); }); afterEach(function() { doMyTeardown(); }); ``` When run (in the default "serial" mode) via `mocha --file "./test/setup.js" "./test/**/*.spec.js"`, `setup.js` will be executed _first_, and install the two hooks shown above for every test found in `./test/**/*.spec.js`. **When Mocha runs in parallel mode, test files do not share the same process.** Consequently, a root-level hook defined in test file _A_ won't be present in test file _B_. There are a (minimum of) two workarounds for this: 1. `require('./setup.js')` or `import './setup.js'` at the top of every test file. Best avoided for those averse to boilerplate. 1. _Recommended_: Define root-level hooks in a required file, using the new (also as of VERSION) Root Hook Plugin system. Parallel mode is only available in Node.js. If you find your tests don't work properly when run with `--parallel`, either shrug and move on, or use this handy-dandy checklist to get things working: - ✅ Ensure you are using a supported reporter. - ✅ Ensure you are not using other unsupported flags. - ✅ Double-check your config file; options set in config files will be merged with any command-line option. - ✅ Look for root-level hooks in your tests. Move them into a root hook plugin. - ✅ Do any assertion, mock, or other test libraries you're consuming use root hooks? They may need to be migrated for compatibility with parallel mode. - ✅ If tests are unexpectedly timing out, you may need to increase the default test timeout (via `--timeout`) - ✅ Ensure your tests do not depend on being run in a specific order. - ✅ Ensure your tests clean up after themselves; remove temp files, handles, sockets, etc. Don't try to share state or resources between test files. Some types of tests are _not_ so well-suited to run in parallel. For example, extremely timing-sensitive tests, or tests which make I/O requests to a limited pool of resources (such as opening ports, or automating browser windows, hitting a test DB, or remote server, etc.). Free-tier cloud CI services may not provide a suitable multi-core container or VM for their build agents. Regarding expected performance gains in CI: your mileage may vary. It may help to use a conditional in a `.mocharc.js` to check for `process.env.CI`, and adjust the job count as appropriate. It's unlikely (but not impossible) to see a performance gain from a job count _greater than_ the number of available CPU cores. That said, _play around with the job count_--there's no one-size-fits all, and the unique characteristics of your tests will determine the optimal number of jobs; it may even be that fewer is faster! - updated signal handling in `bin/mocha` to a) better work with Windows, and b) work properly with `--parallel` to avoid leaving zombie workers - docstrings in `lib/cli/collect-files.js` - refactors in `lib/cli/run-helpers.js` and `lib/cli/watch-run.js`. We now have four methods: - `watchRun()` - serial + watch - `singleRun()` - serial - `parallelWatchRun()` - parallel + watch - `parallelRun()` - parallel - `lib/cli/run.js` and `lib/cli/run-option-metadata.js`: additions for new options and checks for incompatibility - add `lib/reporters/buffered.js` (`Buffered`); this reporter is _not_ re-exported in `Mocha.reporters`, since it should only be invoked internally. - tweak `landing` reporter to avoid referencing `Runner#total`, which is incompatible with parallel mode. It didn't need to do so in the first place! - the `tap` reporter now outputs the plan at the _end_ instead of at the beginning (avoiding a call to `Runner#grepTotal()`, which is incompatible with parallel mode). This is within spec, so should not be a breaking change. - add `lib/buffered-runner.js` (`BufferedRunner`); subclass of `Runner` which overrides the `run()` method. - There's a little custom finite state machine in here. didn't want to pull in a third-party module, but we should consider doing so if we use FSM's elsewhere. - when `DEBUG=mocha:parallel*` is in the env, this module will output statistics about the worker pool every 5s - the `run()` method looks a little weird because I wanted to use `async/await`, but the method it is overriding (`Runner#run`) is _not_ `async` - traps `SIGINT` to gracefully terminate the pool - pulls in [promise.allsettled](https://npm.im/promise.allsettled) polyfill to handle workers that may have rejected with uncaught exceptions - "bail" support is best-effort. - the `ABORTING` state is only for interruption via `SIGINT` or if `allowUncaught` is true and we get an uncaught exception - `Hook`, `Suite`, `Test`: add a `serialize()` method. This pulls out the most relevant information about the object for transmission over IPC. It's called by worker processes for each event received by its `Runner`; event arguments (e.g., `test` or `suite`) are serialized in this manner. Note that this _limits what reporters have access to_, which may break compatibility with third-party reporters that may use information that is missing from the serialized object. As those cases arise, we can add more information to the serialized objects (in some cases). The `$$` convention tells the _deserializer_ to turn the property into a function which returns the passed value, e.g., `test.fullTitle()`. - `lib/mocha.js`: - refactor `Mocha#reporter` for nicer parameter & variable names - rename `loadAsync` to `lazyLoadFiles`, which is more descriptive, IMO. It's a private property, so should not be a breaking change. - Constructor will dynamically choose the appropriate `Runner` - `lib/runner.js`: `BufferedRunner` needs the options from `Mocha#options`, so I updated the parent method to define the parameter. It is unused here. - add `lib/serializer.js`: on the worker process side, manages event queue serialization; manages deserialization of the event queue in the main process. - I spent a long time trying to get this working. We need to account for things like `Error` instances, with their stack traces, since those can be event arguments (e.g., `EVENT_TEST_FAIL` sends both a `Test` and the `Error`). It's impossible to serialize circular (self-referential) objects, so we need to account for those as well. - Not super happy with the deserialization algorithm, since it's recursive, but it shouldn't be too much of an issue because the serializer won't output circular structures. - Attempted to avoid prototype pollution issues - Much of this works by mutating objects, mostly because it can be more performant. The code can be changed to be "more immutable", as that's likely to be easier to understand, if it doesn't impact performance too much. We're serializing potentially very large arrays of stuff. - The `__type` prop is a hint for the deserializer. This convention allows us to re-expand plain objects back into `Error` instances, for example. You can't send an `Error` instance over IPC! - add `lib/worker.js`: - registers its `run()` function with `workerpool` to be called by main process - if `DEBUG=mocha:parallel*` is set, will output information (on an interval) about long-running test files - afaik the only way `run()` can reject is if `allowUncaught` is true or serialization fails - any user-supplied `reporter` value is replaced with the `Buffered` reporter. thus, reporters are not validated. - the worker uses `Runner`, like usual. - tests: - see `test/integration/options/parallel.spec.js` for the interesting stuff - upgrade `unexpected` for "to have readonly property" assertion - upgrade `unexpected-eventemitter` for support async function support - integration test helpers allow Mocha's developers to use `--bail` and `--parallel`, but will default to `--no-bail` and `--no-parallel`. - etc: - update `.eslintrc.yml` for new Node-only files - increase default timeout to `1000` (also seen in another PR) and use `parallel` mode by default in `.mocharc.yml` - run node unit tests _in serial_ as sort of a smoke test, as otherwise all our tests would be run in parallel - karma, browserify: ignore files for parallel support - force color output in CI. this is nice on travis, but ugly on appveyor. either way, it's easier to read than having no color Ref: #4198

also increase default timeout Ref: #4198

grouped these together: - Site can now use emoji :party: - Using the `.single-column` class will render a `ul` element as a "normal" list - Suppress the warning about `text` being an unknown language coming out of 11ty/Prism/markdown-it Ref: #4198

- `landing` reporter befouls the terminal up if `SIGINT` is issued - `--exit` test used an incorrect description, and also did not reliably handle `SIGINT` when running - `runMocha` helper returns the child process Ref: #4198

- fix webhooks - remove webhook for `mochajs/mocha` - add v14 to Appveyor Ref: #4198

- make sure that template strings aren't used in debug statements, since they are eager Ref: #4198

should keep more garbage out of the coverage Ref: #4198

- add `createInvalidPluginError` for reporters, UIs, and future plugins - ensures the original error is output if the module exists, but it throws (see `test/node-unit/cli/fixtures/bad-module.fixture.js`) - remove unneeded `process.cwd()` from call to `path.resolve()` Ref: #4198

- increase default timeout for wiggle room - specify `watch-ignore` in case we run our own tests in watch mode - reformat `.mocharc.yml` - `integration/fixtures/uncaught/listeners.fixture.js`: reduce number of listeners created to avoid max listener warning - `integration/fixtures/diffs.spec.js`: do not pass `-C`; the helper already does this - `integration/options/watch.spec.js`: do not use context object; be more specific about spawn options. tweak timings - `node-unit/mocha.spec.js`: make more unit-test-like insofar as that's possible when testing w/ `require()` - `node-unit/cli/config.spec.js`: rewiremock fixes; assertion tweaks; add test for `.cjs` extension - `node-unit/cli/options.spec.js`: rewiremock fixes; increase timeout - `unit/hook-timeout.spec.js`: do not override default (configured) timeout - `unit/runner.spec.js`: leverage [unexpected-eventemitter](https://npm.im/unexpected-eventemitter) - `unit/throw.spec.js`: proper teardown: remove uncaught exception listeners - `unit/timeout.spec.js`: increase timeout to _greater than_ default (configured) value - `example/config/.mocharc.yml`: quote yaml strings BONUS - fixes a weird call to `Mocha.unloadFile()` from `Mocha#unloadFiles()` Ref: #4198

* test helper improvements - enables `RawResult` (the result of `invokeMocha()` or `invokeMochaAsync()`) to check the exit code via `to have code` assertion - add passed/failing/pending assertions for `RawRunResult` (the result of `runMocha()` or `runMochaAsync()`) - expose `getSummary()`, which can be used with a `RawResult` (when not failing) - reorganize the module a bit - create `runMochaAsync()` and `runMochaJSONAsync()` which are like `runMocha()` and `runMochaJSON()` except return `Promise`s - better trapping of JSON parse errors - better default handling of `STDERR` output in subprocesses (print it instead of suppress it!) - do not let the `DEBUG` env variable reach subprocesses _unless it was explicitly supplied_ - add an easily copy-paste-able `command` prop to summary - add some missing docstrings Ref: #4198 * increase timeout in watch test for CI the same code should be in PR #4240

(documentation will be in another PR) Adds "root hook plugins", a system to define root hooks via files loaded with `--require`. This enables root hooks to work in parallel mode. Because parallel mode runs files in a non-deterministic order, and files do not share a `Mocha` instance, it is not possible to share these hooks with other test files. This change also works well with third-party libraries for Mocha which need the behavior; these can now be trivially consumed by adding `--require` or `require: 'some-library'` in Mocha's config file. The way it works is: 1. When a file is loaded via `--require`, we check to see if that file exports a property named `mochaHooks` (can be multiple files). 1. If it does, we save a reference to the property. 1. After Yargs' validation phase, we use async middleware to execute root hook plugin functions--or if they are objects, just collect them--and we flatten all hooks found into four buckets corresponding to the four hook types. 1. Once `Mocha` is instantiated, if it is given a `rootHooks` option, those hooks are applied to the root suite. This works with parallel tests because we can save a reference to the flattened hooks in each worker process, and a new `Mocha` instance is created with them for each test file. * * * Tangential: - Because a root hook plugin can be defined as an `async` function, I noticed that `utils.type()` does not return `function` for async functions; it returns `asyncfunction`. I've added a (Node-specific, for now) test for this. - `handleRequires` is now `async`, since it will need to be anyway to support ESM and calls to `import()`. - fixed incorrect call to `fs.existsSync()` Ref: #4198

> (this PR depends on most other PRs linked to #4198, so they should be merged first; documentation will be in another PR) This PR adds support for running test files in parallel via `--parallel`. For many cases, this should "just work." When the `--parallel` flag is supplied, Mocha will swap out the default `Runner` (`lib/runner.js`) for `BufferedRunner` (`lib/buffered-runner.js`). `BufferedRunner` _extends_ `Runner`. `BufferedRunner#run()` is the main point of extension. Instead of executing the tests in serial, it will create a pool of worker processes (not worker _threads_) based on the maximum job count (`--jobs`; defaults to `<number of CPU cores> - 1`). Both `BufferedRunner` and the `worker` module consume the abstraction layer, [workerpool](https://npm.im/workerpool). `BufferedRunner#run()` does _not_ load the test files, unlike `Runner#run()`. Instead, it has a list of test files, and puts these into an async queue. The `EVENT_RUN_BEGIN` event is then emitted. As files enter the queue, `BufferedRunner#run()` tells `workerpool` to execute the `run()` function of the pool. `workerpool` then launches as many worker processes are needed--up to the maximum--and executes the `run()` function with a single filepath and any options for a `Mocha` instance. The first time `lib/worker.js` is invoked, it will "bootstrap" itself, by handling `--require`'d modules and validating the UI. Note that _reporter validation_ does not occur. Once bootstrapped, it instantiate `Mocha`, add the single file, swap any reporter out for the `Buffered` reporter (`lib/reporters/buffered.js`) then execute `Mocha#run()`, which invokes `Runner#run()`. The `Buffered` reporter listens for events emitting from the `Runner` instance, like a reporter usually does. But instead of outputting to the console, it buffers the events in a queue. Once the file has completed running, the queue is drained: the events collected are (trivially) serialized for transmission back to the main process. `BufferedRunner#run()` receives the list of events, trivially _deserializes_ them, and re-emits the events to whatever the chosen reporter is (e.g., the `spec` reporter). In this way, the reporters don't know that the tests were run in parallel. Practically, the user will see reporter output in "chunks" instead of the "stream" of results they usually expect. This method ensures that while the test files run in a nondeterministic order, the reporter output will be deterministic for any given test file. Once the result (the queue of events) has been returned to the main process, the worker process stays open, but waits for further instruction. If there are more files in `BufferedRunner#run()`'s queue, `workerpool` will instruct the worker to take the next file from the list, and so on, and so forth. When all files have executed, the pool terminates, the `EVENT_RUN_END` event is emitted, and the reporter handles it. > (this section is pasted from the documentation with minimal edits) Due to the nature of the following reporters, they cannot work when running tests in parallel: - `markdown` - `progress` - `json-stream` These reporters expect Mocha to know _how many tests it plans to run_ before execution. This information is unavailable in parallel mode, as test files are loaded only when they are about to be run. In serial mode, tests results will "stream" as they occur. In parallel mode, reporter output is _buffered_; reporting will occur after each file is completed. In practice, the reporter output will appear in "chunks" (but will otherwise be identical). In parallel mode, we have no guarantees about the order in which test files will be run--or what process runs them--as it depends on the execution times of the test files. Because of this, the following options _cannot be used_ in parallel mode: - `--file` - `--sort` - `--delay` Because running tests in parallel mode uses more system resources at once, the OS may take extra time to schedule and complete some operations. For this reason, test timeouts may need to be increased either globally or otherwise. When used with `--bail` (or `this.bail()`) to exit after the first failure, it's likely other tests will be running at the same time. Mocha must shut down its worker processes before exiting. Likewise, subprocesses may throw uncaught exceptions. When used with `--allow-uncaught`, Mocha will "bubble" this exception to the main process, but still must shut down its processes. > _NOTE: This only applies to test files run parallel mode_. A root-level hook is a hook in a test file which is _not defined_ within a suite. An example using the `bdd` interface: ```js // test/setup.js beforeEach(function() { doMySetup(); }); afterEach(function() { doMyTeardown(); }); ``` When run (in the default "serial" mode) via `mocha --file "./test/setup.js" "./test/**/*.spec.js"`, `setup.js` will be executed _first_, and install the two hooks shown above for every test found in `./test/**/*.spec.js`. **When Mocha runs in parallel mode, test files do not share the same process.** Consequently, a root-level hook defined in test file _A_ won't be present in test file _B_. There are a (minimum of) two workarounds for this: 1. `require('./setup.js')` or `import './setup.js'` at the top of every test file. Best avoided for those averse to boilerplate. 1. _Recommended_: Define root-level hooks in a required file, using the new (also as of VERSION) Root Hook Plugin system. Parallel mode is only available in Node.js. If you find your tests don't work properly when run with `--parallel`, either shrug and move on, or use this handy-dandy checklist to get things working: - ✅ Ensure you are using a supported reporter. - ✅ Ensure you are not using other unsupported flags. - ✅ Double-check your config file; options set in config files will be merged with any command-line option. - ✅ Look for root-level hooks in your tests. Move them into a root hook plugin. - ✅ Do any assertion, mock, or other test libraries you're consuming use root hooks? They may need to be migrated for compatibility with parallel mode. - ✅ If tests are unexpectedly timing out, you may need to increase the default test timeout (via `--timeout`) - ✅ Ensure your tests do not depend on being run in a specific order. - ✅ Ensure your tests clean up after themselves; remove temp files, handles, sockets, etc. Don't try to share state or resources between test files. Some types of tests are _not_ so well-suited to run in parallel. For example, extremely timing-sensitive tests, or tests which make I/O requests to a limited pool of resources (such as opening ports, or automating browser windows, hitting a test DB, or remote server, etc.). Free-tier cloud CI services may not provide a suitable multi-core container or VM for their build agents. Regarding expected performance gains in CI: your mileage may vary. It may help to use a conditional in a `.mocharc.js` to check for `process.env.CI`, and adjust the job count as appropriate. It's unlikely (but not impossible) to see a performance gain from a job count _greater than_ the number of available CPU cores. That said, _play around with the job count_--there's no one-size-fits all, and the unique characteristics of your tests will determine the optimal number of jobs; it may even be that fewer is faster! - updated signal handling in `bin/mocha` to a) better work with Windows, and b) work properly with `--parallel` to avoid leaving zombie workers - docstrings in `lib/cli/collect-files.js` - refactors in `lib/cli/run-helpers.js` and `lib/cli/watch-run.js`. We now have four methods: - `watchRun()` - serial + watch - `singleRun()` - serial - `parallelWatchRun()` - parallel + watch - `parallelRun()` - parallel - `lib/cli/run.js` and `lib/cli/run-option-metadata.js`: additions for new options and checks for incompatibility - add `lib/reporters/buffered.js` (`Buffered`); this reporter is _not_ re-exported in `Mocha.reporters`, since it should only be invoked internally. - tweak `landing` reporter to avoid referencing `Runner#total`, which is incompatible with parallel mode. It didn't need to do so in the first place! - the `tap` reporter now outputs the plan at the _end_ instead of at the beginning (avoiding a call to `Runner#grepTotal()`, which is incompatible with parallel mode). This is within spec, so should not be a breaking change. - add `lib/buffered-runner.js` (`BufferedRunner`); subclass of `Runner` which overrides the `run()` method. - There's a little custom finite state machine in here. didn't want to pull in a third-party module, but we should consider doing so if we use FSM's elsewhere. - when `DEBUG=mocha:parallel*` is in the env, this module will output statistics about the worker pool every 5s - the `run()` method looks a little weird because I wanted to use `async/await`, but the method it is overriding (`Runner#run`) is _not_ `async` - traps `SIGINT` to gracefully terminate the pool - pulls in [promise.allsettled](https://npm.im/promise.allsettled) polyfill to handle workers that may have rejected with uncaught exceptions - "bail" support is best-effort. - the `ABORTING` state is only for interruption via `SIGINT` or if `allowUncaught` is true and we get an uncaught exception - `Hook`, `Suite`, `Test`: add a `serialize()` method. This pulls out the most relevant information about the object for transmission over IPC. It's called by worker processes for each event received by its `Runner`; event arguments (e.g., `test` or `suite`) are serialized in this manner. Note that this _limits what reporters have access to_, which may break compatibility with third-party reporters that may use information that is missing from the serialized object. As those cases arise, we can add more information to the serialized objects (in some cases). The `$$` convention tells the _deserializer_ to turn the property into a function which returns the passed value, e.g., `test.fullTitle()`. - `lib/mocha.js`: - refactor `Mocha#reporter` for nicer parameter & variable names - rename `loadAsync` to `lazyLoadFiles`, which is more descriptive, IMO. It's a private property, so should not be a breaking change. - Constructor will dynamically choose the appropriate `Runner` - `lib/runner.js`: `BufferedRunner` needs the options from `Mocha#options`, so I updated the parent method to define the parameter. It is unused here. - add `lib/serializer.js`: on the worker process side, manages event queue serialization; manages deserialization of the event queue in the main process. - I spent a long time trying to get this working. We need to account for things like `Error` instances, with their stack traces, since those can be event arguments (e.g., `EVENT_TEST_FAIL` sends both a `Test` and the `Error`). It's impossible to serialize circular (self-referential) objects, so we need to account for those as well. - Not super happy with the deserialization algorithm, since it's recursive, but it shouldn't be too much of an issue because the serializer won't output circular structures. - Attempted to avoid prototype pollution issues - Much of this works by mutating objects, mostly because it can be more performant. The code can be changed to be "more immutable", as that's likely to be easier to understand, if it doesn't impact performance too much. We're serializing potentially very large arrays of stuff. - The `__type` prop is a hint for the deserializer. This convention allows us to re-expand plain objects back into `Error` instances, for example. You can't send an `Error` instance over IPC! - add `lib/worker.js`: - registers its `run()` function with `workerpool` to be called by main process - if `DEBUG=mocha:parallel*` is set, will output information (on an interval) about long-running test files - afaik the only way `run()` can reject is if `allowUncaught` is true or serialization fails - any user-supplied `reporter` value is replaced with the `Buffered` reporter. thus, reporters are not validated. - the worker uses `Runner`, like usual. - tests: - see `test/integration/options/parallel.spec.js` for the interesting stuff - upgrade `unexpected` for "to have readonly property" assertion - upgrade `unexpected-eventemitter` for support async function support - integration test helpers allow Mocha's developers to use `--bail` and `--parallel`, but will default to `--no-bail` and `--no-parallel`. - etc: - update `.eslintrc.yml` for new Node-only files - increase default timeout to `1000` (also seen in another PR) and use `parallel` mode by default in `.mocharc.yml` - run node unit tests _in serial_ as sort of a smoke test, as otherwise all our tests would be run in parallel - karma, browserify: ignore files for parallel support - force color output in CI. this is nice on travis, but ugly on appveyor. either way, it's easier to read than having no color Ref: #4198

> (this PR depends on most other PRs linked to #4198, so they should be merged first; documentation will be in another PR) This PR adds support for running test files in parallel via `--parallel`. For many cases, this should "just work." When the `--parallel` flag is supplied, Mocha will swap out the default `Runner` (`lib/runner.js`) for `BufferedRunner` (`lib/buffered-runner.js`). `BufferedRunner` _extends_ `Runner`. `BufferedRunner#run()` is the main point of extension. Instead of executing the tests in serial, it will create a pool of worker processes (not worker _threads_) based on the maximum job count (`--jobs`; defaults to `<number of CPU cores> - 1`). Both `BufferedRunner` and the `worker` module consume the abstraction layer, [workerpool](https://npm.im/workerpool). `BufferedRunner#run()` does _not_ load the test files, unlike `Runner#run()`. Instead, it has a list of test files, and puts these into an async queue. The `EVENT_RUN_BEGIN` event is then emitted. As files enter the queue, `BufferedRunner#run()` tells `workerpool` to execute the `run()` function of the pool. `workerpool` then launches as many worker processes are needed--up to the maximum--and executes the `run()` function with a single filepath and any options for a `Mocha` instance. The first time `lib/worker.js` is invoked, it will "bootstrap" itself, by handling `--require`'d modules and validating the UI. Note that _reporter validation_ does not occur. Once bootstrapped, it instantiate `Mocha`, add the single file, swap any reporter out for the `Buffered` reporter (`lib/reporters/buffered.js`) then execute `Mocha#run()`, which invokes `Runner#run()`. The `Buffered` reporter listens for events emitting from the `Runner` instance, like a reporter usually does. But instead of outputting to the console, it buffers the events in a queue. Once the file has completed running, the queue is drained: the events collected are (trivially) serialized for transmission back to the main process. `BufferedRunner#run()` receives the list of events, trivially _deserializes_ them, and re-emits the events to whatever the chosen reporter is (e.g., the `spec` reporter). In this way, the reporters don't know that the tests were run in parallel. Practically, the user will see reporter output in "chunks" instead of the "stream" of results they usually expect. This method ensures that while the test files run in a nondeterministic order, the reporter output will be deterministic for any given test file. Once the result (the queue of events) has been returned to the main process, the worker process stays open, but waits for further instruction. If there are more files in `BufferedRunner#run()`'s queue, `workerpool` will instruct the worker to take the next file from the list, and so on, and so forth. When all files have executed, the pool terminates, the `EVENT_RUN_END` event is emitted, and the reporter handles it. Note that exclusive tests ("only") cannot work in parallel mode, because we do not load all files up-front. > (this section is pasted from the documentation with minimal edits) Due to the nature of the following reporters, they cannot work when running tests in parallel: - `markdown` - `progress` - `json-stream` These reporters expect Mocha to know _how many tests it plans to run_ before execution. This information is unavailable in parallel mode, as test files are loaded only when they are about to be run. In serial mode, tests results will "stream" as they occur. In parallel mode, reporter output is _buffered_; reporting will occur after each file is completed. In practice, the reporter output will appear in "chunks" (but will otherwise be identical). In parallel mode, we have no guarantees about the order in which test files will be run--or what process runs them--as it depends on the execution times of the test files. Because of this, the following options _cannot be used_ in parallel mode: - `--file` - `--sort` - `--delay` Because running tests in parallel mode uses more system resources at once, the OS may take extra time to schedule and complete some operations. For this reason, test timeouts may need to be increased either globally or otherwise. When used with `--bail` (or `this.bail()`) to exit after the first failure, it's likely other tests will be running at the same time. Mocha must shut down its worker processes before exiting. Likewise, subprocesses may throw uncaught exceptions. When used with `--allow-uncaught`, Mocha will "bubble" this exception to the main process, but still must shut down its processes. > _NOTE: This only applies to test files run parallel mode_. A root-level hook is a hook in a test file which is _not defined_ within a suite. An example using the `bdd` interface: ```js // test/setup.js beforeEach(function() { doMySetup(); }); afterEach(function() { doMyTeardown(); }); ``` When run (in the default "serial" mode) via `mocha --file "./test/setup.js" "./test/**/*.spec.js"`, `setup.js` will be executed _first_, and install the two hooks shown above for every test found in `./test/**/*.spec.js`. **When Mocha runs in parallel mode, test files do not share the same process.** Consequently, a root-level hook defined in test file _A_ won't be present in test file _B_. There are a (minimum of) two workarounds for this: 1. `require('./setup.js')` or `import './setup.js'` at the top of every test file. Best avoided for those averse to boilerplate. 1. _Recommended_: Define root-level hooks in a required file, using the new (also as of VERSION) Root Hook Plugin system. Parallel mode is only available in Node.js. If you find your tests don't work properly when run with `--parallel`, either shrug and move on, or use this handy-dandy checklist to get things working: - ✅ Ensure you are using a supported reporter. - ✅ Ensure you are not using other unsupported flags. - ✅ Double-check your config file; options set in config files will be merged with any command-line option. - ✅ Look for root-level hooks in your tests. Move them into a root hook plugin. - ✅ Do any assertion, mock, or other test libraries you're consuming use root hooks? They may need to be migrated for compatibility with parallel mode. - ✅ If tests are unexpectedly timing out, you may need to increase the default test timeout (via `--timeout`) - ✅ Ensure your tests do not depend on being run in a specific order. - ✅ Ensure your tests clean up after themselves; remove temp files, handles, sockets, etc. Don't try to share state or resources between test files. Some types of tests are _not_ so well-suited to run in parallel. For example, extremely timing-sensitive tests, or tests which make I/O requests to a limited pool of resources (such as opening ports, or automating browser windows, hitting a test DB, or remote server, etc.). Free-tier cloud CI services may not provide a suitable multi-core container or VM for their build agents. Regarding expected performance gains in CI: your mileage may vary. It may help to use a conditional in a `.mocharc.js` to check for `process.env.CI`, and adjust the job count as appropriate. It's unlikely (but not impossible) to see a performance gain from a job count _greater than_ the number of available CPU cores. That said, _play around with the job count_--there's no one-size-fits all, and the unique characteristics of your tests will determine the optimal number of jobs; it may even be that fewer is faster! - updated signal handling in `bin/mocha` to a) better work with Windows, and b) work properly with `--parallel` to avoid leaving zombie workers - docstrings in `lib/cli/collect-files.js` - refactors in `lib/cli/run-helpers.js` and `lib/cli/watch-run.js`. We now have four methods: - `watchRun()` - serial + watch - `singleRun()` - serial - `parallelWatchRun()` - parallel + watch - `parallelRun()` - parallel - `lib/cli/run.js` and `lib/cli/run-option-metadata.js`: additions for new options and checks for incompatibility - add `lib/reporters/buffered.js` (`Buffered`); this reporter is _not_ re-exported in `Mocha.reporters`, since it should only be invoked internally. - tweak `landing` reporter to avoid referencing `Runner#total`, which is incompatible with parallel mode. It didn't need to do so in the first place! - the `tap` reporter now outputs the plan at the _end_ instead of at the beginning (avoiding a call to `Runner#grepTotal()`, which is incompatible with parallel mode). This is within spec, so should not be a breaking change. - add `lib/buffered-runner.js` (`BufferedRunner`); subclass of `Runner` which overrides the `run()` method. - There's a little custom finite state machine in here. didn't want to pull in a third-party module, but we should consider doing so if we use FSM's elsewhere. - when `DEBUG=mocha:parallel*` is in the env, this module will output statistics about the worker pool every 5s - the `run()` method looks a little weird because I wanted to use `async/await`, but the method it is overriding (`Runner#run`) is _not_ `async` - traps `SIGINT` to gracefully terminate the pool - pulls in [promise.allsettled](https://npm.im/promise.allsettled) polyfill to handle workers that may have rejected with uncaught exceptions - "bail" support is best-effort. - the `ABORTING` state is only for interruption via `SIGINT` or if `allowUncaught` is true and we get an uncaught exception - `Hook`, `Suite`, `Test`: add a `serialize()` method. This pulls out the most relevant information about the object for transmission over IPC. It's called by worker processes for each event received by its `Runner`; event arguments (e.g., `test` or `suite`) are serialized in this manner. Note that this _limits what reporters have access to_, which may break compatibility with third-party reporters that may use information that is missing from the serialized object. As those cases arise, we can add more information to the serialized objects (in some cases). The `$$` convention tells the _deserializer_ to turn the property into a function which returns the passed value, e.g., `test.fullTitle()`. - `lib/mocha.js`: - refactor `Mocha#reporter` for nicer parameter & variable names - rename `loadAsync` to `lazyLoadFiles`, which is more descriptive, IMO. It's a private property, so should not be a breaking change. - Constructor will dynamically choose the appropriate `Runner` - `lib/runner.js`: `BufferedRunner` needs the options from `Mocha#options`, so I updated the parent method to define the parameter. It is unused here. - add `lib/serializer.js`: on the worker process side, manages event queue serialization; manages deserialization of the event queue in the main process. - I spent a long time trying to get this working. We need to account for things like `Error` instances, with their stack traces, since those can be event arguments (e.g., `EVENT_TEST_FAIL` sends both a `Test` and the `Error`). It's impossible to serialize circular (self-referential) objects, so we need to account for those as well. - Not super happy with the deserialization algorithm, since it's recursive, but it shouldn't be too much of an issue because the serializer won't output circular structures. - Attempted to avoid prototype pollution issues - Much of this works by mutating objects, mostly because it can be more performant. The code can be changed to be "more immutable", as that's likely to be easier to understand, if it doesn't impact performance too much. We're serializing potentially very large arrays of stuff. - The `__type` prop is a hint for the deserializer. This convention allows us to re-expand plain objects back into `Error` instances, for example. You can't send an `Error` instance over IPC! - add `lib/worker.js`: - registers its `run()` function with `workerpool` to be called by main process - if `DEBUG=mocha:parallel*` is set, will output information (on an interval) about long-running test files - afaik the only way `run()` can reject is if `allowUncaught` is true or serialization fails - any user-supplied `reporter` value is replaced with the `Buffered` reporter. thus, reporters are not validated. - the worker uses `Runner`, like usual. - tests: - see `test/integration/options/parallel.spec.js` for the interesting stuff - upgrade `unexpected` for "to have readonly property" assertion - upgrade `unexpected-eventemitter` for support async function support - integration test helpers allow Mocha's developers to use `--bail` and `--parallel`, but will default to `--no-bail` and `--no-parallel`. - split some node-specific tests out of `test/unit/mocha.spec.js` into `test/node-unit/mocha.spec.js` - etc: - update `.eslintrc.yml` for new Node-only files - increase default timeout to `1000` (also seen in another PR) and use `parallel` mode by default in `.mocharc.yml` - run node unit tests _in serial_ as sort of a smoke test, as otherwise all our tests would be run in parallel - karma, browserify: ignore files for parallel support - force color output in CI. this is nice on travis, but ugly on appveyor. either way, it's easier to read than having no color - move non-CLI-related node-specific files into `lib/nodejs/` - correct some issues with APIs not marked `@private` - add some istanbul directives to ignore some debug statements - add `utils.isBrowser()` for easier mocking of a `process.browser === true` situation - add `createForbiddenExclusivityError()` Ref: #4198

* add support for running tests in parallel mode > (this PR depends on most other PRs linked to #4198, so they should be merged first; documentation will be in another PR) This PR adds support for running test files in parallel via `--parallel`. For many cases, this should "just work." When the `--parallel` flag is supplied, Mocha will swap out the default `Runner` (`lib/nodejs/runner.js`) for `ParallelBufferedRunner` (`lib/nodejs/parallel-buffered-runner.js`). `ParallelBufferedRunner` _extends_ `Runner`. `ParallelBufferedRunner#run()` is the main point of extension. Instead of executing the tests in serial, it will create a pool of worker processes (not worker _threads_) based on the maximum job count (`--jobs`; defaults to `<number of CPU cores> - 1`). Both `ParallelBufferedRunner` and the `worker` module consume the abstraction layer, [workerpool](https://npm.im/workerpool). `ParallelBufferedRunner#run()` does _not_ load the test files, unlike `Runner#run()`. Instead, it has a list of test files, and puts these into an async queue. The `EVENT_RUN_BEGIN` event is then emitted. As files enter the queue, `ParallelBufferedRunner#run()` tells `workerpool` to execute the `run()` function of the pool. `workerpool` then launches as many worker processes are needed--up to the maximum--and executes the `run()` function with a single filepath and any options for a `Mocha` instance. The first time `lib/nodejs/worker.js` is invoked, it will "bootstrap" itself, by handling `--require`'d modules and validating the UI. Note that _reporter validation_ does not occur. Once bootstrapped, it instantiate `Mocha`, add the single file, swap any reporter out for the `ParallelBuffered` reporter (`lib/nodejs/reporters/parallel-buffered.js`) then execute `Mocha#run()`, which invokes `Runner#run()`. The `ParallelBuffered` reporter listens for events emitting from the `Runner` instance, like a reporter usually does. But instead of outputting to the console, it buffers the events in a queue. Once the file has completed running, the queue is drained: the events collected are (trivially) serialized for transmission back to the main process. `ParallelBufferedRunner#run()` receives the list of events, trivially _deserializes_ them, and re-emits the events to whatever the chosen reporter is (e.g., the `spec` reporter). In this way, the reporters don't know that the tests were run in parallel. Practically, the user will see reporter output in "chunks" instead of the "stream" of results they usually expect. This method ensures that while the test files run in a nondeterministic order, the reporter output will be deterministic for any given test file. Once the result (the queue of events) has been returned to the main process, the worker process stays open, but waits for further instruction. If there are more files in `ParallelBufferedRunner#run()`'s queue, `workerpool` will instruct the worker to take the next file from the list, and so on, and so forth. When all files have executed, the pool terminates, the `EVENT_RUN_END` event is emitted, and the reporter handles it. Note that exclusive tests ("only") cannot work in parallel mode, because we do not load all files up-front. > (this section is pasted from the documentation with minimal edits) ### Caveats: Reporters Due to the nature of the following reporters, they cannot work when running tests in parallel: - `markdown` - `progress` - `json-stream` These reporters expect Mocha to know _how many tests it plans to run_ before execution. This information is unavailable in parallel mode, as test files are loaded only when they are about to be run. ### Caveats: Buffered Output In serial mode, tests results will "stream" as they occur. In parallel mode, reporter output is _buffered_; reporting will occur after each file is completed. In practice, the reporter output will appear in "chunks" (but will otherwise be identical). ### Caveats: Nondeterminism In parallel mode, we have no guarantees about the order in which test files will be run--or what process runs them--as it depends on the execution times of the test files. Because of this, the following options _cannot be used_ in parallel mode: - `--file` - `--sort` - `--delay` Because running tests in parallel mode uses more system resources at once, the OS may take extra time to schedule and complete some operations. For this reason, test timeouts may need to be increased either globally or otherwise. ### Caveats: Other Impacted Options When used with `--bail` (or `this.bail()`) to exit after the first failure, it's likely other tests will be running at the same time. Mocha must shut down its worker processes before exiting. Likewise, subprocesses may throw uncaught exceptions. When used with `--allow-uncaught`, Mocha will "bubble" this exception to the main process, but still must shut down its processes. `--forbid-only` does not work in parallel mode, for a similar reason to the unsupported reporters. > _NOTE: This only applies to test files run parallel mode_. ### Caveats: Root Hooks A _root hook_ is a hook in a test file which is _not defined_ within a suite. An example using the `bdd` interface: ```js // test/setup.js beforeEach(function() { doMySetup(); }); afterEach(function() { doMyTeardown(); }); ``` When run (in the default "serial" mode) via `mocha --file "./test/setup.js" "./test/**/*.spec.js"`, `setup.js` will be executed _first_, and install the two hooks shown above for every test found in `./test/**/*.spec.js`. **When Mocha runs in parallel mode, test files do not share the same process.** Consequently, a root hook defined in test file _A_ won't be present in test file _B_. There are a (minimum of) two workarounds for this: 1. `require('./setup.js')` or `import './setup.js'` at the top of every test file. Best avoided for those averse to boilerplate. 1. _Recommended_: Define root-level hooks in a required file, using the new (also as of VERSION) Root Hook Plugin system. ### Caveats: Node.js Only, For Now Parallel mode is only available in Node.js. ### Migration Checklist If you find your tests don't work properly when run with `--parallel`, either shrug and move on, or use this handy-dandy checklist to get things working: - ✅ Ensure you are using a supported reporter. - ✅ Ensure you are not using other unsupported flags. - ✅ Double-check your config file; options set in config files will be merged with any command-line option. - ✅ Look for root-level hooks in your tests. Move them into a root hook plugin. - ✅ Do any assertion, mock, or other test libraries you're consuming use root hooks? They may need to be migrated for compatibility with parallel mode. - ✅ If tests are unexpectedly timing out, you may need to increase the default test timeout (via `--timeout`) - ✅ Ensure your tests do not depend on being run in a specific order. - ✅ Ensure your tests clean up after themselves; remove temp files, handles, sockets, etc. Don't try to share state or resources between test files. ### About Parallelism Some types of tests are _not_ so well-suited to run in parallel. For example, extremely timing-sensitive tests, or tests which make I/O requests to a limited pool of resources (such as opening ports, or automating browser windows, hitting a test DB, or remote server, etc.). Free-tier cloud CI services may not provide a suitable multi-core container or VM for their build agents. Regarding expected performance gains in CI: your mileage may vary. It may help to use a conditional in a `.mocharc.js` to check for `process.env.CI`, and adjust the job count as appropriate. It's unlikely (but not impossible) to see a performance gain from a job count _greater than_ the number of available CPU cores. That said, _play around with the job count_--there's no one-size-fits all, and the unique characteristics of your tests will determine the optimal number of jobs; it may even be that fewer is faster! ### Change Details - updated signal handling in `bin/mocha` to a) better work with Windows, and b) work properly with `--parallel` to avoid leaving zombie workers - docstrings in `lib/nodejs/cli/collect-files.js` - refactors in `lib/nodejs/cli/run-helpers.js` and `lib/nodejs/cli/watch-run.js`. We now have four methods: - `watchRun()` - serial + watch - `singleRun()` - serial - `parallelWatchRun()` - parallel + watch - `parallelRun()` - parallel - `lib/nodejs/cli/run.js` and `lib/nodejs/cli/run-option-metadata.js`: additions for new options and checks for incompatibility - add `lib/nodejs/reporters/buffered.js` (`ParallelBuffered`); this reporter is _not_ re-exported in `Mocha.reporters`, since it should only be invoked internally. - tweak `landing` reporter to avoid referencing `Runner#total`, which is incompatible with parallel mode. It didn't need to do so in the first place! - the `tap` reporter now outputs the plan at the _end_ instead of at the beginning (avoiding a call to `Runner#grepTotal()`, which is incompatible with parallel mode). This is within spec, so should not be a breaking change. - add `lib/nodejs/parallel-buffered-runner.js` (`ParallelBufferedRunner`); subclass of `Runner` which overrides the `run()` method. - There's a little custom finite state machine in here. didn't want to pull in a third-party module, but we should consider doing so if we use FSM's elsewhere. - when `DEBUG=mocha:parallel*` is in the env, this module will output statistics about the worker pool every 5s - the `run()` method looks a little weird because I wanted to use `async/await`, but the method it is overriding (`Runner#run`) is _not_ `async` - traps `SIGINT` to gracefully terminate the pool - pulls in [promise.allsettled](https://npm.im/promise.allsettled) polyfill to handle workers that may have rejected with uncaught exceptions - "bail" support is best-effort. - the `ABORTING` state is only for interruption via `SIGINT` or if `allowUncaught` is true and we get an uncaught exception - `Hook`, `Suite`, `Test`: add a `serialize()` method. This pulls out the most relevant information about the object for transmission over IPC. It's called by worker processes for each event received by its `Runner`; event arguments (e.g., `test` or `suite`) are serialized in this manner. Note that this _limits what reporters have access to_, which may break compatibility with third-party reporters that may use information that is missing from the serialized object. As those cases arise, we can add more information to the serialized objects (in some cases). The `$$` convention tells the _deserializer_ to turn the property into a function which returns the passed value, e.g., `test.fullTitle()`. - `lib/nodejs/mocha.js`: - refactor `Mocha#reporter` for nicer parameter & variable names - rename `loadAsync` to `lazyLoadFiles`, which is more descriptive, IMO. It's a private property, so should not be a breaking change. - Constructor will dynamically choose the appropriate `Runner` - `lib/nodejs/runner.js`: `ParallelBufferedRunner` needs the options from `Mocha#options`, so I updated the parent method to define the parameter. It is unused here. - add `lib/nodejs/serializer.js`: on the worker process side, manages event queue serialization; manages deserialization of the event queue in the main process. - I spent a long time trying to get this working. We need to account for things like `Error` instances, with their stack traces, since those can be event arguments (e.g., `EVENT_TEST_FAIL` sends both a `Test` and the `Error`). It's impossible to serialize circular (self-referential) objects, so we need to account for those as well. - Not super happy with the deserialization algorithm, since it's recursive, but it shouldn't be too much of an issue because the serializer won't output circular structures. - Attempted to avoid prototype pollution issues - Much of this works by mutating objects, mostly because it can be more performant. The code can be changed to be "more immutable", as that's likely to be easier to understand, if it doesn't impact performance too much. We're serializing potentially very large arrays of stuff. - The `__type` prop is a hint for the deserializer. This convention allows us to re-expand plain objects back into `Error` instances, for example. You can't send an `Error` instance over IPC! - add `lib/nodejs/worker.js`: - registers its `run()` function with `workerpool` to be called by main process - if `DEBUG=mocha:parallel*` is set, will output information (on an interval) about long-running test files - afaik the only way `run()` can reject is if `allowUncaught` is true or serialization fails - any user-supplied `reporter` value is replaced with the `ParallelBuffered` reporter. thus, reporters are not validated. - the worker uses `Runner`, like usual. - tests: - see `test/integration/options/parallel.spec.js` for the interesting stuff - upgrade `unexpected` for "to have readonly property" assertion - upgrade `unexpected-eventemitter` for support async function support - integration test helpers allow Mocha's developers to use `--bail` and `--parallel`, but will default to `--no-bail` and `--no-parallel`. - split some node-specific tests out of `test/unit/mocha.spec.js` into `test/node-unit/mocha.spec.js` - add some missing coverage in `test/node-unit/worker.spec.js` - etc: - update `.eslintrc.yml` for new Node-only files - increase default timeout to `1000` (also seen in another PR) and use `parallel` mode by default in `.mocharc.yml` - run node unit tests _in serial_ as sort of a smoke test, as otherwise all our tests would be run in parallel - karma, browserify: ignore files for parallel support - force color output in CI. this is nice on travis, but ugly on appveyor. either way, it's easier to read than having no color - move non-CLI-related node-specific files into `lib/nodejs/nodejs/` - correct some issues with APIs not marked `@private` - add some istanbul directives to ignore some debug statements - add `utils.isBrowser()` for easier mocking of a `process.browser === true` situation - add `createForbiddenExclusivityError()` Ref: #4198 Signed-off-by: Christopher Hiller <boneskull@boneskull.com>

boneskull added type: feature enhancement proposal area: usability concerning user experience or interface semver-minor implementation requires increase of "minor" version number; "features" area: node.js command-line-or-Node.js-specific type: performance Performance improvements labels Mar 3, 2020

boneskull self-assigned this Mar 3, 2020

boneskull changed the title ~~[DO NOT MERGE] WIP: concurrency based on worker threads; see #2839~~ [DO NOT MERGE] WIP: run test concurrently; see #2839 Mar 12, 2020

boneskull changed the title ~~[DO NOT MERGE] WIP: run test concurrently; see #2839~~ [DO NOT MERGE] WIP: run tests concurrently; see #2839 Mar 12, 2020

boneskull force-pushed the boneskull/issue/2839-concurrent branch 2 times, most recently from fa5c779 to 6d321d2 Compare March 18, 2020 22:06

boneskull force-pushed the boneskull/issue/2839-concurrent branch 2 times, most recently from 09cb328 to 314ef78 Compare March 26, 2020 21:45

boneskull mentioned this pull request Apr 1, 2020

Spike: migrate tests to Jest (ts-jest) loopbackio/loopback-next#3159

Closed

5 tasks

boneskull force-pushed the boneskull/issue/2839-concurrent branch from 5f7da79 to 4f0b384 Compare April 9, 2020 00:36

craigtaub pushed a commit that referenced this pull request May 21, 2020

remove cruft from wallaby config

2afb9f6

also increase default timeout Ref: #4198

craigtaub pushed a commit that referenced this pull request May 21, 2020

travis: add job names, add Node.js v14 to matrix

ddaecbc

- fix webhooks - remove webhook for `mochajs/mocha` - add v14 to Appveyor Ref: #4198

craigtaub pushed a commit that referenced this pull request May 21, 2020

better debug output

d51a728

- make sure that template strings aren't used in debug statements, since they are eager Ref: #4198

craigtaub pushed a commit that referenced this pull request May 21, 2020

fix nyc ignorelist

6db2282

should keep more garbage out of the coverage Ref: #4198

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DO NOT MERGE] WIP: run tests concurrently; see #2839 #4198

[DO NOT MERGE] WIP: run tests concurrently; see #2839 #4198

boneskull commented Mar 3, 2020 •

edited

boneskull commented Mar 5, 2020

boneskull commented Mar 5, 2020

boneskull commented Mar 12, 2020

boneskull commented Mar 12, 2020

boneskull commented Mar 12, 2020

boneskull commented Mar 12, 2020

coveralls commented Mar 12, 2020 •

edited

boneskull commented Mar 27, 2020

boneskull commented Mar 27, 2020

boneskull commented Mar 27, 2020

boneskull commented Mar 27, 2020

giltayar commented Mar 28, 2020

juergba commented Mar 28, 2020

boneskull commented Mar 29, 2020

boneskull commented Mar 31, 2020 •

edited

boneskull commented Apr 17, 2020

[DO NOT MERGE] WIP: run tests concurrently; see #2839 #4198

[DO NOT MERGE] WIP: run tests concurrently; see #2839 #4198

Conversation

boneskull commented Mar 3, 2020 • edited

boneskull commented Mar 5, 2020

boneskull commented Mar 5, 2020

boneskull commented Mar 12, 2020

boneskull commented Mar 12, 2020

boneskull commented Mar 12, 2020

boneskull commented Mar 12, 2020

coveralls commented Mar 12, 2020 • edited

boneskull commented Mar 27, 2020

boneskull commented Mar 27, 2020

boneskull commented Mar 27, 2020

boneskull commented Mar 27, 2020

giltayar commented Mar 28, 2020

juergba commented Mar 28, 2020

boneskull commented Mar 29, 2020

boneskull commented Mar 31, 2020 • edited

boneskull commented Apr 17, 2020

boneskull commented Mar 3, 2020 •

edited

coveralls commented Mar 12, 2020 •

edited

boneskull commented Mar 31, 2020 •

edited