Improve Jest startup time and test runtime, particularly when running with coverage, by caching micromatch and avoiding recreating RegExp instances #10131

lencioni · 2020-06-05T13:48:01Z

I was profiling some Jest runs at Airbnb and noticed that on my
MacBook Pro, we can spend over 2 seconds at Jest startup time in
SearchSource getTestPaths. I believe that this will grow as the size
of the codebase increases.

Looking at the call stacks, it appears to be calling micromatch
repeatedly, which calls picomatch, which builds a regex out of the
globs. It seems that the parsing and regex building also triggers the
garbage collector frequently.

Upon testing, I noticed that the globs don't actually change between
these calls, so we can save a bunch of work by making a micromatch
matcher and reusing that function for all of the paths.

micromatch has some logic internally to handle lists of globs that
may include negated globs. A naive approach of just checking if it
matched any of the globs won't capture that, so I copied and
simplified the logic from within micromatch.

https://github.com/micromatch/micromatch/blob/fe4858b0/index.js#L27-L77

In my profiling of this change locally, this brings down the time of
startRun from about 2000ms to about 200ms.

Before:

Before, zoomed in:

After:

Avoid recreating RegExp instances in regexToMatcher

After optimizing globsToMatcher, I noticed that there was still a
lot of unnecessary overhead at Jest startup time spent recreating
the same RegExp instances repeatedly. Thankfully, we can be a little
smarter about this and create them all ahead of time and just reuse
them.

On top of my globsToMatcher optimization, this brings the speed of
the ArrayMap in startRun down from about 160ms to about 7ms.

Before:

After:

Optimize micromatch and RegExps in shouldInstrument

I've been profiling running Jest with code coverage at Airbnb, and
noticed that shouldInstrument is called often and is fairly
expensive. It also seems to call micromatch and new RegExp
repeatedly, both of which can be optimized by caching the work to
convert globs and strings into matchers and regexes.

I profiled this change by running a set of 27 fairly simple tests.
Before this change, about 6-7 seconds was spent in shouldInstrument.
After this change, only 400-500 ms is spent there. I would expect
this delta to increase along with the number of tests and size of
their dependency graphs.

A typical shouldInstrument call before this change:

A typical shouldInstrument call after this change (most are actually too fast to even show up in the profiler now):

I was profiling some Jest runs at Airbnb and noticed that on my
MacBook Pro, we can spend over 30 seconds after running Jest with code
coverage as the coverage reporter adds all of the untested files. I
believe that this will grow as the size of the codebase increases.

Looking at the call stacks, it appears to be calling micromatch
repeatedly, which calls picomatch, which builds a regex out of the
globs. It seems that the parsing and regex building also triggers the
garbage collector frequently.

Since this is in a tight loop and the globs won't change between
checks, we can greatly improve the performance here by using
micromatch.matcher.

This optimization reduces the block of time here from about 30s to
about 10s. The aggregated total time of coverage reporter's
onRunComplete goes from 23s to 600ms.

Before:

Before, zoomed in:

After:

After, zoomed in:

Summary

Motivation: Improve slow Jest startup and runtime speed, particularly when running with coverage

Test plan

I ran jest in the Airbnb frontend monorepo with and without coverage options, with a path argument.

thymikee

Nice!

packages/jest-core/src/SearchSource.ts

CHANGELOG.md

packages/jest-core/src/SearchSource.ts

SimenB · 2020-06-05T16:56:25Z

packages/jest-core/src/SearchSource.ts

@@ -37,8 +37,55 @@ export type TestSelectionConfig = {
  watch?: boolean;
 };

-const globsToMatcher = (globs: Array<Config.Glob>) => (path: Config.Path) =>
-  micromatch([replacePathSepForGlob(path)], globs, {dot: true}).length > 0;
+const globsMatchers = new Map<


how does this work with watch mode, and changing the globs? I guess since the key is the glob itself and we don't collect all entries in the Map we won't pull stale data out?

Yep, that's my understanding!

SimenB · 2020-06-05T18:49:00Z

@jonschlinkert hiya, thoughts on this?

I was profiling some Jest runs at Airbnb and noticed that on my MacBook Pro, we can spend over 2 seconds at Jest startup time in SearchSource getTestPaths. I believe that this will grow as the size of the codebase increases. Looking at the call stacks, it appears to be calling micromatch repeatedly, which calls picomatch, which builds a regex out of the globs. It seems that the parsing and regex building also triggers the garbage collector frequently. Upon testing, I noticed that the globs don't actually change between these calls, so we can save a bunch of work by making a micromatch matcher and reusing that function for all of the paths. micromatch has some logic internally to handle lists of globs that may include negated globs. A naive approach of just checking if it matched any of the globs won't capture that, so I copied and simplified the logic from within micromatch. https://github.com/micromatch/micromatch/blob/fe4858b0/index.js#L27-L77 In my profiling of this change locally, this brings down the time of startRun from about 2000ms to about 200ms.

lencioni · 2020-06-08T14:01:18Z

To avoid having to stack a bunch of separate PRs, I've added another commit to this one to optimize the startup even more.

packages/jest-core/src/SearchSource.ts

After optimizing globsToMatcher, I noticed that there was still a lot of unnecessary overhead at Jest startup time spent recreating the same RegExp instances repeatedly. Thankfully, we can be a little smarter about this and create them all ahead of time and just reuse them. On top of my globsToMatcher optimization, this brings the speed of the ArrayMap in startRun down from about 160ms to about 7ms.

lencioni · 2020-06-08T16:12:20Z

Alright friends, I think I'm done hacking on this now. Please let me know if you'd like me to make any changes. I'm really looking forward to seeing how this performs for us in CI!

packages/jest-core/src/__tests__/SearchSource.test.ts

packages/jest-transform/src/shouldInstrument.ts

packages/jest-util/package.json

SimenB · 2020-06-08T16:32:01Z

packages/jest-util/src/globsToMatcher.ts

+  globs: Array<Config.Glob>,
+): (path: Config.Path) => boolean {
+  if (globs.length === 0) {
+    return (_: Config.Path): boolean => false;


Are these type annotations needed? It's not inferred from the outer function

I didn't expect them to be needed, but vscode showed me an eslint error when I didn't have them:

Missing return type on function. eslint@typescript-eslint/explicit-module-boundary-types

Hmm, wonder if that's fixed by the fresh 3.2 release? Either typescript-eslint/typescript-eslint#2169 or typescript-eslint/typescript-eslint#2176

Yeah, seems promising. It looks like Jest is still on v2.30 though, so I think I'd like to say that updating the eslint plugin is out of scope for this PR if that's alright with you.

FWIW the breaking changes seem to be pretty okay for this repo: https://github.com/typescript-eslint/typescript-eslint/releases/tag/v3.0.0

yeah, that's out of scope for this 😅

@SimenB I'm happy to tackle that upgrade, if you want to create an issue and assign it to me :)

Thanks @G-Rath! #10177

I'd like to start using this in more places to improve performance. Moving it to jest-util seems like a better spot. Now that it is a standalone module, I decided to write some unit tests for this function. In doing so, I uncovered a small difference between the behavior of this function and micromatch when overlapping glob patterns are used which I also fixed.

While incorporating this function into more places, I discovered a discrepancy here with how micromatch works. We can fix this by creating a fast path for when there are no globs at all.

I've been profiling running Jest with code coverage at Airbnb, and noticed that shouldInstrument is called often and is fairly expensive. It also seems to call micromatch and `new RegExp` repeatedly, both of which can be optimized by caching the work to convert globs and strings into matchers and regexes. I profiled this change by running a set of 27 fairly simple tests. Before this change, about 6-7 seconds was spent in shouldInstrument. After this change, only 400-500 ms is spent there. I would expect this delta to increase along with the number of tests and size of their dependency graphs.

I was profiling some Jest runs at Airbnb and noticed that on my MacBook Pro, we can spend over 30 seconds after running Jest with code coverage as the coverage reporter adds all of the untested files. I believe that this will grow as the size of the codebase increases. Looking at the call stacks, it appears to be calling micromatch repeatedly, which calls picomatch, which builds a regex out of the globs. It seems that the parsing and regex building also triggers the garbage collector frequently. Since this is in a tight loop and the globs won't change between checks, we can greatly improve the performance here by using our new and optimized globsToMatcher function, which avoids re-parsing globs unnecessarily. This optimization reduces the block of time here from about 30s to about 10s. The aggregated total time of coverage reporter's onRunComplete goes from 23s to 600ms.

lencioni · 2020-06-08T17:31:09Z

packages/jest-transform/src/shouldInstrument.ts

-      [replacePathSepForGlob(path.relative(config.rootDir, filename))],
-      options.collectCoverageFrom,
-    ).length === 0
+    !globsToMatcher(options.collectCoverageFrom)(


I just noticed that this will end up using the dot: true option where before that wasn't used here (and above in this file). This seems like it should be fine, but I don't really have enough context here to know for sure.

yeah, seems fine. HasteMap should filter out any dotfiles we don't want already (I think)

lencioni · 2020-06-08T18:32:31Z

packages/jest-util/src/globsToMatcher.ts

+  }
+>();
+
+// Every time micromatch is called, it will parse the glob strings and turn them


One of the windows builds was flaking and there's no way to re-run, so I decided to add some code comments to this file.

AdaskoTheBeAsT · 2020-06-08T21:24:14Z

amazing job @lencioni :)
I am just curious how this will speed up tests in our project - on my win machine it takes 45s just to run first test from pack of few hundreds - looking forward for new version to test :)

SimenB · 2020-06-08T21:34:36Z

This LGTM, I just want some more people to review it before merging

The logic here might be a little confusing, so I am adding some comments that I hope will help make it easier for future explorers to understand. While I was doing this, I noticed a small way to simplify this function even more.

AdaskoTheBeAsT · 2020-06-15T20:32:38Z

@cpojer @thymikee @jeysal can you please review this pull request? :) whole community keep fingers crossed for increased performance :)

7rulnik · 2020-06-22T08:45:05Z

Tested this branch on my company project: duration decreased from 11 minutes to 4 minutes. Seems that it is one of the main reasons for the #9457 (comment) regression. @lencioni awesome job, thanks!

csvan · 2020-06-22T15:20:00Z

@cpojer @thymikee @jeysal the sanity of a million JS developers is in your hands!

cpojer

Thanks for working on this!

SimenB · 2020-06-23T15:17:47Z

Thanks @lencioni! Published as 26.1.0

lencioni · 2020-06-23T16:22:38Z

In CI we currently spread our tests across 10 separate machines.

When running all tests, the 10 machines on 26.0.1 had runtimes of:

10m 28s
9m 47s
10m 11s
9m 59s
10m 9s
11m 8s
11m 26s
9m 53s
10m 33s
13m 39s

On 26.1.0:

8m 8s
10m 0s
9m 28s
9m 36s
9m 8s
9m 50s
10m 41s
9m 21s
10m 8s
9m 54s

n=1

Not as much as I had wished for, but this is definitely a nice win for us! Thank you for the reviews and the release!

… with coverage, by caching micromatch and avoiding recreating RegExp instances (jestjs#10131) Co-authored-by: Christoph Nakazawa <cpojer@fb.com>

github-actions · 2021-05-11T06:06:31Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

facebook-github-bot added the cla signed label Jun 5, 2020

lencioni force-pushed the searchsource-micromatch branch from e13b694 to 29eb389 Compare June 5, 2020 14:02

thymikee reviewed Jun 5, 2020

View reviewed changes

packages/jest-core/src/SearchSource.ts Outdated Show resolved Hide resolved

lencioni mentioned this pull request Jun 5, 2020

Reduce micromatch overhead in jest-haste-map HasteFS #10132

Closed

lencioni force-pushed the searchsource-micromatch branch from 29eb389 to f6ad57c Compare June 5, 2020 15:22

lencioni commented Jun 5, 2020

View reviewed changes

packages/jest-core/src/SearchSource.ts Outdated Show resolved Hide resolved

lencioni force-pushed the searchsource-micromatch branch from f6ad57c to dbb20fe Compare June 5, 2020 16:16

lencioni requested a review from thymikee June 5, 2020 16:27

SimenB reviewed Jun 5, 2020

View reviewed changes

lencioni force-pushed the searchsource-micromatch branch from dbb20fe to a40693c Compare June 5, 2020 17:01

lencioni requested a review from SimenB June 5, 2020 17:10

SimenB approved these changes Jun 5, 2020

View reviewed changes

SimenB requested review from jeysal and cpojer June 5, 2020 18:49

lencioni force-pushed the searchsource-micromatch branch from a40693c to 1b67559 Compare June 6, 2020 14:25

lencioni force-pushed the searchsource-micromatch branch from 1b67559 to 65a0405 Compare June 7, 2020 00:39

lencioni changed the title ~~Cache micromatch in SearchSource globsToMatcher~~ Cache micromatch in SearchSource globsToMatcher and avoid recreating RegExp instances in regexToMatcher Jun 8, 2020

SimenB reviewed Jun 8, 2020

View reviewed changes

packages/jest-core/src/SearchSource.ts Outdated Show resolved Hide resolved

lencioni force-pushed the searchsource-micromatch branch 2 times, most recently from 5669574 to 069905a Compare June 8, 2020 15:51

lencioni changed the title ~~Cache micromatch in SearchSource globsToMatcher and avoid recreating RegExp instances in regexToMatcher~~ Improve Jest startup time and test runtime, particularly when running with coverage, by caching micromatch and avoiding recreating RegExp instances Jun 8, 2020

SimenB reviewed Jun 8, 2020

View reviewed changes

lencioni added 2 commits June 8, 2020 11:37

Teach globsToMatcher to work with empty globs

9c01c20

While incorporating this function into more places, I discovered a discrepancy here with how micromatch works. We can fix this by creating a fast path for when there are no globs at all.

lencioni force-pushed the searchsource-micromatch branch from 692fce3 to f3e1f02 Compare June 8, 2020 16:39

lencioni added 2 commits June 8, 2020 11:53

lencioni force-pushed the searchsource-micromatch branch from f3e1f02 to f51fd34 Compare June 8, 2020 16:53

lencioni commented Jun 8, 2020

View reviewed changes

Add code comments to globsToMatcher

42ab2e9

The logic here might be a little confusing, so I am adding some comments that I hope will help make it easier for future explorers to understand. While I was doing this, I noticed a small way to simplify this function even more.

lencioni force-pushed the searchsource-micromatch branch from f588ada to 42ab2e9 Compare June 9, 2020 13:06

7rulnik mentioned this pull request Jun 20, 2020

jest 25 performance #9457

Open

cpojer approved these changes Jun 23, 2020

View reviewed changes

Update globsToMatcher.ts

03c8004

cpojer merged commit 504cace into jestjs:master Jun 23, 2020

lencioni deleted the searchsource-micromatch branch June 23, 2020 17:03

ahnpnl mentioned this pull request Jun 23, 2020

perf(compiler): use globsToMatcher from jest-util kulshekhar/ts-jest#1754

Merged

2 tasks

SimenB mentioned this pull request Apr 12, 2021

fix(util): Replace micromatch with picomatch to fix issues with negated globs #11287

Merged

1 task

github-actions bot locked as resolved and limited conversation to collaborators May 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve Jest startup time and test runtime, particularly when running with coverage, by caching micromatch and avoiding recreating RegExp instances #10131

Improve Jest startup time and test runtime, particularly when running with coverage, by caching micromatch and avoiding recreating RegExp instances #10131

lencioni commented Jun 5, 2020 •

edited

thymikee left a comment

SimenB Jun 5, 2020

lencioni Jun 5, 2020

SimenB commented Jun 5, 2020

lencioni commented Jun 8, 2020

lencioni commented Jun 8, 2020

SimenB Jun 8, 2020

lencioni Jun 8, 2020

SimenB Jun 8, 2020 •

edited

lencioni Jun 8, 2020

lencioni Jun 8, 2020

SimenB Jun 8, 2020

G-Rath Jun 19, 2020

SimenB Jun 19, 2020

lencioni Jun 8, 2020

SimenB Jun 8, 2020

lencioni Jun 8, 2020

AdaskoTheBeAsT commented Jun 8, 2020

SimenB commented Jun 8, 2020 •

edited

AdaskoTheBeAsT commented Jun 15, 2020

7rulnik commented Jun 22, 2020

csvan commented Jun 22, 2020

cpojer left a comment

SimenB commented Jun 23, 2020

lencioni commented Jun 23, 2020

github-actions bot commented May 11, 2021

Improve Jest startup time and test runtime, particularly when running with coverage, by caching micromatch and avoiding recreating RegExp instances #10131

Improve Jest startup time and test runtime, particularly when running with coverage, by caching micromatch and avoiding recreating RegExp instances #10131

Conversation

lencioni commented Jun 5, 2020 • edited

Summary

Test plan

thymikee left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SimenB commented Jun 5, 2020

lencioni commented Jun 8, 2020

lencioni commented Jun 8, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SimenB Jun 8, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AdaskoTheBeAsT commented Jun 8, 2020

SimenB commented Jun 8, 2020 • edited

AdaskoTheBeAsT commented Jun 15, 2020

7rulnik commented Jun 22, 2020

csvan commented Jun 22, 2020

cpojer left a comment

Choose a reason for hiding this comment

SimenB commented Jun 23, 2020

lencioni commented Jun 23, 2020

github-actions bot commented May 11, 2021

lencioni commented Jun 5, 2020 •

edited

SimenB Jun 8, 2020 •

edited

SimenB commented Jun 8, 2020 •

edited