jest 25 performance #9457
Comments
|
Sorry about the regression but
means we can do absolutely nothing. This is the first I've heard of performance regressing, everywhere else I've heard from 10-40% improvement in performance going from 24 to 25. You need to provide some sort of reproduction, otherwise we'll have to close this issue as it's not actionable at all as it stands. If you want to see this adressed, you'll need to spend some time putting together a reproduction case, or hope somebody else does so. |
|
ok i will see if i can pull our 10 slowest tests maybe, then try to run them in 24 vs 25. in the meantime, what do you recommend with regards to clearing cache before running tests in CI? do it? don't do it? |
|
Your configuration, especially
I personally think it's a good idea just to be sure there's nothing stale laying around giving false negative or positives. Does it make a huge difference to the runtime of your tests to not clear the cache? |
|
it appears to be quite a bit slower when run after clearing cache. thanks for the tips i'll look into it and see if i can attempt a repro |
|
FWIW, I've also noticed that v25 is either slightly slower or right on par with v24. Have not seen anywhere near 10-40% improvement. |
|
I saw a significant speedup over jest 24 as noted here: #7811 (comment) The above was tested on osx. However, the exact same setup runs much, much slower on our CI which runs CentOS. Linux specific issue? I/O related issues when writing cache files? Is it possible to turn off cache generation altogether to rule this out? |
|
I think I found the culprit in our case, it's the This is reproducible both on OSX and CentOS, so contrary to my previous comment the issue does not appear Linux-specific. |
|
Interesting! We've updated Istanbul to v3, maybe something in there has regressed. We've added support for v8 code coverage, so I might also have messed up the refactoring when doing so |
|
Yes! That's consistent with what I'm seeing as well. We are running with coverage in CI where it's 2x slower. And when I run locally without covg is quite fast. @SimenB is there any config option to use the older Istanbul? :) |
|
Echoing what @csvan said it would be nice to turn off cache generation in CI if that is in fact a culprit since we delete it prior to building anyway |
|
I'm unable to reproduce this - the repos I test have about the same performance with |
|
just ran our CI build w/ coverage disabled, I think @csvan is on to something. The tests run in 4:00 w/ coverage turned off vs 11 min w/ coverage turned on. I will try to see if i can create repro this weekend at some point. our exinfo from the build agent: 00:03:31.992 System:
00:03:31.992 OS: Linux 3.10 CentOS Linux 7 (Core)
00:03:31.992 CPU: (8) x64 Intel Core Processor (Skylake, IBRS)
00:03:31.992 Binaries:
00:03:31.992 Node: 10.16.0 - ~/workspace/grocery-electrode/tools/nix_64/nodejs-10.16.0/bin/node
00:03:31.992 npm: 6.9.0 - ~/workspace/grocery-electrode/tools/nix_64/npm-6.9.0/node_modules/.bin/npm
00:03:31.993 npmPackages:
00:03:31.993 jest: 25.1.0 => 25.1.0 |
|
We're seeing a similar issue. Upgrading Jest 25 made our tests slower when using coverage (166s with Jest 24 vs. 381s with Jest 25). With Jest 25 displaying many of these errors while running the checks: Downgrading to Jest 24 makes those errors go away. I also noticed Jest 25 handles the |
|
Any JS traces at all? Would be interesting to see where it died.
We upgraded to Micromatch 4, it might've changed something. No changes to it on purpose, though. Could you open up a separate issue with a reproduction? |
This was printed: EDIT: Actually, I'm seeing heap errors even with coverage disabled.
Will do. |
|
Chiming in again. Coverage is definitely slower, and seems to be spurious. Here's the timings for OSX. Timings for CI (travis). |
|
@milesj what is v25 circus? |
|
It's jests new runner, which is supposed to be faster, but it never is from what I've seen. https://www.npmjs.com/package/jest-circus |
|
@EvHaus Traces from JSDOM is interesting (might also be completely coincidental, of course). Could you try installing |
|
@SimenB Rolling back just |
|
Interesting! If you have time, it'd be lovely if you could test
Or just link in |
|
For the following tests I don't have coverage enabled.
Stack tracesThese are some of the stack traces from the Hope this helps |
|
I don't know if this will help, but my organization had a massive slow down from Jest 24 to Jest 25 (18 minutes to 28 minutes) that went away after turning off coverage collection (down to 10 minutes). |
|
@rosasynstylae out of curiosity, does your code have a lot of snapshot tests? |
|
@benmonro It does, yes. |
|
So does ours! @SimenB do you think lots of snapshots in a repo could cause this? |
* Initial playwright * Updated e2e to use playwright and typescript * Update set up environment and jest package * Add changelog * Add await to uncheck * Fix formatting * Revert jset back to ~24, as >25 runs slower, see jestjs/jest#9457 * Removed some unnecessary uses of waitForSelector * Fix eslint issue * Fix the e2e tests with latest updates * Running most tests, with typescript now * Fix any outstanding queries for the tests to work * Update changelog * Remove unnecessary jest version and unnecessary transform setting * Fix test case broken after rebase * Add fix to make e2e tests more robust * Making sure dropdown value is correct * Reove the wcpay condition for features number
|
Hi, @billyvg ! What is a soft that you used to monitor tests? #9457 (comment) |
|
@billyvg already found it, but thanks for an answer. Cannot understand how to monitor tests there, any guides/videos/topics? |
|
@Lertis It's a bit experimental, but here's what we did: getsentry/sentry#22237 |
|
@SimenB Regression in micromatch was fixed in 4.0.4 (see micromatch/micromatch#179), would it be possible to bump the dependency and see if it improves the performance in jest cases that regressed before? |
|
It's in semver range, so you can update locally |
|
@SimenB Yes, but considering that it's an important fix, wouldn't it make sense to force update globally? |
|
I checked on my codebase and cannot see any performance difference with micromatch@4.0.4 |
|
@atsikov Thanks for checking! Guess it only makes difference when there is a significant amount of files excluded from collecting coverage. |
|
@kibertoad you are right, I should have checked changes in micromatch before testing. |
|
I'm not seeing an improvement with 4.0.4, but as has been discussed here and elsewhere that was probably not the cause of the regression in the first place. |
|
My team upgraded Jest from 24 to 26 and ran into similar performance woes, regardless of whether we're collecting coverage (our tests were taking ~30-40% longer). I've tried a few things mentioned in this thread:
Baseline performance with Jest 24: Performance after upgrading to Jest 26 💥: Performance with Jest 26 and Performance with Jest 26 and I'm not exactly sure what it is about JSDOM 15/16 that is causing this for us, but it seems we don't have an issue with JSDOM 14. |
|
I was able to fix and resolve the OOM errors we were having by doing the following:
// We have hundreds of tests that use this
beforeEach(async () => {
// Inject a giant HTML string into the DOM to simulate a real website
document.documentElement.innerHTML = TEST_SUITE.dom;
// THIS WAS THE MOST IMPORTANT FIX
// Without this, memory would never get released between test suites and would reach the Node memory ceiling
await new Promise((resolve) => {
setTimeout(() => resolve(), 1);
})
});
After this, all the 2 big lesson for us were to:
Hope this can help others who are having issues. |
A few weeks ago I played around with replacing babel with |
|
I tried https://github.com/capricorn86/happy-dom as JSDOM alternative, that was a few month ago. I did not support everything when need and I had no time to take a deeper look into it. However the tests which ran were not significantly faster. (Our setup is very complex.) |
|
This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days. |
|
As an addendum, I suppose we can say that Jest performance is pretty great as of Jest 29 (at least). |
|
This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days. |
|
Hello 'github-actions[bot]',
Thank you for contacting us! The Notable office is currently closed in observance of the Presidents' Day holiday.
We will resume normal business hours on Tuesday, February 20th, from 9am to 9pm EST and look forward to assisting you at that time.
Best Regards,
The Notable Team
|
|
Who knew that the achilles heel of ActionsBot was vacation auto-replies. |
|
Hello 'Christopher,
Thank you for contacting us! The Notable office is currently closed in observance of the Presidents' Day holiday.
We will resume normal business hours on Tuesday, February 20th, from 9am to 9pm EST and look forward to assisting you at that time.
Best Regards,
The Notable Team
|
and others
💥 Regression Report
we upgraded jest from 24 to 25 and saw our unit tests go from taking 5min 23sec in jenkins to now over 11 minutes. only a few snapshot tests broke in the upgrade, we
-u'd them, but this is a severe regression imo. please help me understand how we can fix this. We clear cache in CI to ensure we always run the latest.A clear and concise description of what the regression is.
run time went from 5:23, to 11:00
Last working version
24.8.0
Worked up to version:
24.8.0
Stopped working in version:
25.1.0
To Reproduce
sorry can't share our codebase
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Link to repl or repo (highly encouraged)
Please provide either a repl.it demo or a minimal repository on GitHub.
Issues without a reproduction link are likely to stall.
Run
npx envinfo --preset jestPaste the results here:
System: OS: macOS Mojave 10.14.6 CPU: (12) x64 Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz Binaries: Node: 10.16.0 - ~/.nvm/versions/node/v10.16.0/bin/node Yarn: 1.19.0 - ~/.nvm/versions/node/v10.16.0/bin/yarn npm: 6.13.6 - ~/.nvm/versions/node/v10.16.0/bin/npmThe text was updated successfully, but these errors were encountered: