[Feature]: A middleground between full isolation and --runInBand

[glenjamin]

🚀 Feature Proposal

Provide some way to mark test suites as safe to run without isolation, and then allow those suites to share a process - cutting down on the overhead from the cost of spinning up new workers and re-initialising the modules.

Motivation

In the days before jest, NodeJS test runners tended to load all of the tests into a single process, and then execute them (eg. mocha).

If you had a lot of fast tests, this worked pretty well, but if you had a lot of tests which did a lot of setup (like jsdom), this ended up being quite slow - as there was no way to parallelise them.

Then along came jest with a new execution model, each file would be isolated and run in a subprocess, allowing them to all run in parallel.

Effectivetly, jest made your slow tests faster - but the tradeoff is that it made fast tests slower (mostly due to the extra subprocess and isolation overheads).

The --runInBand option is often suggested in blogs about speeding up jest testsuites - but that loses out on the parallelism entirely.

It would be nice if across a large project's testsuite we could make use of both execution models, so that test suites for simple, pure functions could all re-use the same worker (and module cache?), but the more complex tests could continue to take advantage of jest's more powerful features.

Example

I'm not entirely sure how a test would be marked as "simple" or "unisolated". I suspect that because it would need to change the worker behaviour, that this would need to be done with a configuration option that gave a filename pattern?

Pitch

When jest first came onto the scene it was rapidly adopted in part for it's speed advantages, but nowawdays there are a bunch of reports and blog posts where people are complaining that it's not fast enough.

I believe that these complaints have the same root - that jest is slower than a basic test runner for tests which would still work fine in a basic test runner.

Having a first party story for addressing this would be excellent.

I'm interested in experimenting in this space, so if there are ways to extend jest to try out models like this from outside core then I'd be happy to receive some pointers and I'll share any progress here.

[glenjamin]

While digging through #7963 for ideas, I came across this comment, which is a pretty clever way of achieving something close to what I've described above - albeit somewhat cryptically and in a way that'd break the CLI arguments that do test filtering.

#7963 (comment)

[JoostK]

I would support such a mode of operation. We're seeing tremendous overhead of having to reload all modules for each test suite (800ms for a test suite that takes ~50ms) and as a little experiment I bundled all code and specs into a single bundle (using esbuild this took 5s) to see our test execution times drop tenfold, from 70s to 6s (so 11s in total including the bundling step). The primary issue with this approach is that it affects the location of snapshot files and there is no more progress information, as everything would then execute as a single test suite.

As an alternative approach I played around with caching the full Script instances in-memory, instead of requesting the file again from the transformer and then having to recreate the Script instance. This approach reduced module loading overhead by ~4x, to ~200ms, but I have not measured memory usage at all (whether this approach is acceptable depends a lot on usage patterns, e.g. in our case most of the modules have to be loaded for each suite regardless so just caching them in their entirety is beneficial in our case, but may not be for others).

[Rugvip]

Also interested in this idea as we're seeing some very large overhead to requiring the same modules over and over. I think in particular a possible approach could be to share a common cache for package imports marked with "sideEffects": false

[jokeyrhyme]

For our current project, we have some tests that are resource-intensive / latency-sensitive and should not be run concurrently with other tests, but the majority of our tests are fine with concurrency

To accommodate this:

• we rename these special tests to have .runinband. in the filename
• we run jest with --runInBand --testPathPattern=runinband
• then, we run jest again but with --testPathIgnorePatterns=runinband to run the rest of our tests with default concurrency

While this works, it's quite awkward, and it means that --changedSince and other jest features don't quite work as well as they could

It would be better for our use case if we could do something like jest.runInBand(true) inside the special test files, so that the scheduler could alternate between worker-pools and in-band modes as required within the one invocation of jest (instead of having to run it twice with different flags)

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. Please open a new issue if the issue is still relevant, linking to this one.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.