Skip to content

Commit

Permalink
feat(typescript-estree): switch to globby (#2418)
Browse files Browse the repository at this point in the history 
Fixes #2398

If the user has a particularly large node_modules folder and uses globs for `parserOption.project`, then the `glob` library can spend a decent chunk of time searching the `node_modules` folder.
In some cases, this can be on the order of hundreds to thousands of milliseconds.
This wouldn't be a problem, but for safety and correctness during a persistent parse, we have to do this check for every call to the parser (i.e. every file that's being linted).
Over a whole project, this can easily add up to many, many seconds wasted.

Previously, we:
- applied the project globs, one by one
- then manually excluded `tsconfig`s from the list.

This meant that we are always slow. I remember I did this because I had issues getting `glob`'s `ignore` option to work at all.

## The solution

`globby` is a better glob library:
- it accepts an array of globs, which saves us doing manual looping
- it supports exclusion globs (globs prefixed with `!`), which  are evaluated as part of the glob process
- it has caching built in by default

This allows us to evaluate all of the `project` globs at once, as opposed to one at a time (so should be less duplicated work).
This also allows us to evaluate the `projectFolderIgnoreList` at the same time as the `project` globs (so should be no useless work done).

All of these together should cut the glob evaluation time down to ~50ms for the first parse, and ~2ms for each parse after that (due to caching).
For comparison, previously, in bad cases we would spend ~3-500ms, per project, per parsed file.

Example to illustrate how much faster this can potentially be:
For a project that provides 2 globs and has 100 files.
Before: 300ms * 2 * 100 = 60,000ms (60s)
After: 50ms + 2 * 100 = 250ms

This should also save a non-trival amount of time in other, more optimal setups.

BREAKING CHANGE:
- removes the ability to supply a `RegExp` to `projectFolderIgnoreList`, and changes the meaning of the string value from a regex to a glob.
  • Loading branch information
@bradzacher
bradzacher committed Aug 29, 2020
1 parent 9273441 commit 3a7ec9b
Show file tree
Hide file tree
Showing 7 changed files with 128 additions and 68 deletions.
7 changes: 4 additions & 3 deletions packages/parser/README.md
View file
Expand Up @@ -57,7 +57,7 @@ interface ParserOptions {
lib?: string[];

project?: string | string[];
projectFolderIgnoreList?: (string | RegExp)[];
projectFolderIgnoreList?: string[];
tsconfigRootDir?: string;
extraFileExtensions?: string[];
warnOnUnsupportedTypeScriptVersion?: boolean;
Expand Down Expand Up @@ -156,12 +156,13 @@ This option allows you to provide the root directory for relative tsconfig paths

### `parserOptions.projectFolderIgnoreList`

Default `["/node_modules/"]`.
Default `["**/node_modules/**"]`.

This option allows you to ignore folders from being included in your provided list of `project`s.
Any resolved project path that matches one or more of the provided regular expressions will be removed from the list.
This is useful if you have configured glob patterns, but want to make sure you ignore certain folders.

It accepts an array of globs to exclude from the `project` globs.

For example, by default it will ensure that a glob like `./**/tsconfig.json` will not match any `tsconfig`s within your `node_modules` folder (some npm packages do not exclude their source files from their published packages).

### `parserOptions.extraFileExtensions`
Expand Down
9 changes: 4 additions & 5 deletions packages/typescript-estree/README.md
View file
Expand Up @@ -183,14 +183,13 @@ interface ParseAndGenerateServicesOptions extends ParseOptions {
project?: string | string[];

/**
* If you provide a glob (or globs) to the project option, you can use this option to blacklist
* certain folders from being matched by the globs.
* Any project path that matches one or more of the provided regular expressions will be removed from the list.
* If you provide a glob (or globs) to the project option, you can use this option to ignore certain folders from
* being matched by the globs.
* This accepts an array of globs to ignore.
*
* Accepts an array of strings that are passed to new RegExp(), or an array of regular expressions.
* By default, this is set to ["/node_modules/"]
*/
projectFolderIgnoreList?: (string | RegExp)[];
projectFolderIgnoreList?: string[];

/**
* The absolute path to the root directory for all provided `project`s.
Expand Down
2 changes: 1 addition & 1 deletion packages/typescript-estree/package.json
View file
Expand Up @@ -43,7 +43,7 @@
"@typescript-eslint/types": "3.10.1",
"@typescript-eslint/visitor-keys": "3.10.1",
"debug": "^4.1.1",
"glob": "^7.1.6",
"globby": "^11.0.1",
"is-glob": "^4.0.1",
"lodash": "^4.17.15",
"semver": "^7.3.2",
Expand Down
11 changes: 5 additions & 6 deletions packages/typescript-estree/src/parser-options.ts
View file
Expand Up @@ -142,14 +142,13 @@ interface ParseAndGenerateServicesOptions extends ParseOptions {
project?: string | string[];

/**
* If you provide a glob (or globs) to the project option, you can use this option to blacklist
* certain folders from being matched by the globs.
* Any project path that matches one or more of the provided regular expressions will be removed from the list.
* If you provide a glob (or globs) to the project option, you can use this option to ignore certain folders from
* being matched by the globs.
* This accepts an array of globs to ignore.
*
* Accepts an array of strings that are passed to new RegExp(), or an array of regular expressions.
* By default, this is set to ["/node_modules/"]
* By default, this is set to ["**\/node_modules/**"]
*/
projectFolderIgnoreList?: (string | RegExp)[];
projectFolderIgnoreList?: string[];

/**
* The absolute path to the root directory for all provided `project`s.
Expand Down
67 changes: 25 additions & 42 deletions packages/typescript-estree/src/parser.ts
View file
@@ -1,5 +1,5 @@
import debug from 'debug';
import { sync as globSync } from 'glob';
import { sync as globSync } from 'globby';
import isGlob from 'is-glob';
import semver from 'semver';
import * as ts from 'typescript';
Expand Down Expand Up @@ -113,7 +113,7 @@ function resetExtra(): void {
*/
function prepareAndTransformProjects(
projectsInput: string | string[] | undefined,
ignoreListInput: (string | RegExp)[] | undefined,
ignoreListInput: string[],
): string[] {
let projects: string[] = [];

Expand All @@ -133,47 +133,21 @@ function prepareAndTransformProjects(
}

// Transform glob patterns into paths
projects = projects.reduce<string[]>(
(projects, project) =>
projects.concat(
isGlob(project)
? globSync(project, {
cwd: extra.tsconfigRootDir,
})
: project,
),
[],
);

// Normalize and sanitize the ignore regex list
const ignoreRegexes: RegExp[] = [];
if (Array.isArray(ignoreListInput)) {
for (const ignore of ignoreListInput) {
if (ignore instanceof RegExp) {
ignoreRegexes.push(ignore);
} else if (typeof ignore === 'string') {
ignoreRegexes.push(new RegExp(ignore));
}
}
} else {
ignoreRegexes.push(/\/node_modules\//);
}

// Remove any paths that match the ignore list
const filtered = projects.filter(project => {
for (const ignore of ignoreRegexes) {
if (ignore.test(project)) {
return false;
}
}

return true;
});
const globbedProjects = projects.filter(project => isGlob(project));
projects = projects
.filter(project => !isGlob(project))
.concat(
globSync([...globbedProjects, ...ignoreListInput], {
cwd: extra.tsconfigRootDir,
}),
);

log('parserOptions.project matched projects: %s', projects);
log('ignore list applied to parserOptions.project: %s', filtered);
log(
'parserOptions.project (excluding ignored) matched projects: %s',
projects,
);

return filtered;
return projects;
}

function applyParserOptionsToExtra(options: TSESTreeOptions): void {
Expand Down Expand Up @@ -278,9 +252,18 @@ function applyParserOptionsToExtra(options: TSESTreeOptions): void {
extra.filePath = ensureAbsolutePath(extra.filePath, extra);

// NOTE - prepareAndTransformProjects relies upon having the correct tsconfigRootDir in extra
const projectFolderIgnoreList = (options.projectFolderIgnoreList ?? [])
.reduce<string[]>((acc, folder) => {
if (typeof folder === 'string') {
acc.push(folder);
}
return acc;
}, [])
// prefix with a ! for not match glob
.map(folder => (folder.startsWith('!') ? folder : `!${folder}`));
extra.projects = prepareAndTransformProjects(
options.project,
options.projectFolderIgnoreList,
projectFolderIgnoreList,
);

if (
Expand Down
12 changes: 3 additions & 9 deletions packages/typescript-estree/tests/lib/parse.test.ts
View file
Expand Up @@ -592,7 +592,7 @@ describe('parse()', () => {

const testParse = (
filePath: 'ignoreme' | 'includeme',
projectFolderIgnoreList: TSESTreeOptions['projectFolderIgnoreList'] = [],
projectFolderIgnoreList?: TSESTreeOptions['projectFolderIgnoreList'],
) => (): void => {
parser.parseAndGenerateServices(code, {
...config,
Expand All @@ -606,14 +606,8 @@ describe('parse()', () => {
expect(testParse('includeme')).not.toThrow();
});

it('ignores a folder when given a string regexp', () => {
const ignore = ['/ignoreme/'];
expect(testParse('ignoreme', ignore)).toThrow();
expect(testParse('includeme', ignore)).not.toThrow();
});

it('ignores a folder when given a RegExp', () => {
const ignore = [/\/ignoreme\//];
it('ignores a folder when given a string glob', () => {
const ignore = ['**/ignoreme/**'];
expect(testParse('ignoreme', ignore)).toThrow();
expect(testParse('includeme', ignore)).not.toThrow();
});
Expand Down
88 changes: 86 additions & 2 deletions yarn.lock
View file
Expand Up @@ -1405,11 +1405,32 @@
call-me-maybe "^1.0.1"
glob-to-regexp "^0.3.0"

"@nodelib/fs.scandir@2.1.3":
version "2.1.3"
resolved "https://registry.yarnpkg.com/@nodelib/fs.scandir/-/fs.scandir-2.1.3.tgz#3a582bdb53804c6ba6d146579c46e52130cf4a3b"
integrity sha512-eGmwYQn3gxo4r7jdQnkrrN6bY478C3P+a/y72IJukF8LjB6ZHeB3c+Ehacj3sYeSmUXGlnA67/PmbM9CVwL7Dw==
dependencies:
"@nodelib/fs.stat" "2.0.3"
run-parallel "^1.1.9"

"@nodelib/fs.stat@2.0.3", "@nodelib/fs.stat@^2.0.2":
version "2.0.3"
resolved "https://registry.yarnpkg.com/@nodelib/fs.stat/-/fs.stat-2.0.3.tgz#34dc5f4cabbc720f4e60f75a747e7ecd6c175bd3"
integrity sha512-bQBFruR2TAwoevBEd/NWMoAAtNGzTRgdrqnYCc7dhzfoNvqPzLyqlEQnzZ3kVnNrSp25iyxE00/3h2fqGAGArA==

"@nodelib/fs.stat@^1.1.2":
version "1.1.3"
resolved "https://registry.yarnpkg.com/@nodelib/fs.stat/-/fs.stat-1.1.3.tgz#2b5a3ab3f918cca48a8c754c08168e3f03eba61b"
integrity sha512-shAmDyaQC4H92APFoIaVDHCx5bStIocgvbwQyxPRrbUY20V1EYTbSDchWbuwlMG3V17cprZhA6+78JfB+3DTPw==

"@nodelib/fs.walk@^1.2.3":
version "1.2.4"
resolved "https://registry.yarnpkg.com/@nodelib/fs.walk/-/fs.walk-1.2.4.tgz#011b9202a70a6366e436ca5c065844528ab04976"
integrity sha512-1V9XOY4rDW0rehzbrcqAmHnz8e7SKvX27gh8Gt2WgB0+pdzdiLV83p72kZPU+jvMbS1qU5mauP2iOvO8rhmurQ==
dependencies:
"@nodelib/fs.scandir" "2.1.3"
fastq "^1.6.0"

"@octokit/auth-token@^2.4.0":
version "2.4.0"
resolved "https://registry.yarnpkg.com/@octokit/auth-token/-/auth-token-2.4.0.tgz#b64178975218b99e4dfe948253f0673cbbb59d9f"
Expand Down Expand Up @@ -2042,6 +2063,11 @@ array-union@^1.0.2:
dependencies:
array-uniq "^1.0.1"

array-union@^2.1.0:
version "2.1.0"
resolved "https://registry.yarnpkg.com/array-union/-/array-union-2.1.0.tgz#b798420adbeb1de828d84acd8a2e23d3efe85e8d"
integrity sha512-HGyxoOTYUyCM6stUe6EJgnd4EoewAI7zMdfqO+kGjnlZmBDz/cR5pf8r/cR4Wq60sL/p0IkcjUEEPwS3GFrIyw==

array-uniq@^1.0.1:
version "1.0.3"
resolved "https://registry.yarnpkg.com/array-uniq/-/array-uniq-1.0.3.tgz#af6ac877a25cc7f74e058894753858dfdb24fdb6"
Expand Down Expand Up @@ -3433,6 +3459,13 @@ dir-glob@^2.2.2:
dependencies:
path-type "^3.0.0"

dir-glob@^3.0.1:
version "3.0.1"
resolved "https://registry.yarnpkg.com/dir-glob/-/dir-glob-3.0.1.tgz#56dbf73d992a4a93ba1584f4534063fd2e41717f"
integrity sha512-WkrWp9GR4KXfKGYzOLmTuGVi1UWFfws377n9cc55/tb6DuqyF6pcQ5AbiHEshaDpY9v6oaSr2XCDidGmMwdzIA==
dependencies:
path-type "^4.0.0"

doctrine@1.5.0:
version "1.5.0"
resolved "https://registry.yarnpkg.com/doctrine/-/doctrine-1.5.0.tgz#379dce730f6166f76cefa4e6707a159b02c5a6fa"
Expand Down Expand Up @@ -3977,6 +4010,18 @@ fast-glob@^2.2.6:
merge2 "^1.2.3"
micromatch "^3.1.10"

fast-glob@^3.1.1:
version "3.2.4"
resolved "https://registry.yarnpkg.com/fast-glob/-/fast-glob-3.2.4.tgz#d20aefbf99579383e7f3cc66529158c9b98554d3"
integrity sha512-kr/Oo6PX51265qeuCYsyGypiO5uJFgBS0jksyG7FUeCyQzNwYnzrNIMR1NXfkZXsMYXYLRAHgISHBz8gQcxKHQ==
dependencies:
"@nodelib/fs.stat" "^2.0.2"
"@nodelib/fs.walk" "^1.2.3"
glob-parent "^5.1.0"
merge2 "^1.3.0"
micromatch "^4.0.2"
picomatch "^2.2.1"

fast-json-stable-stringify@2.x, fast-json-stable-stringify@^2.0.0:
version "2.1.0"
resolved "https://registry.yarnpkg.com/fast-json-stable-stringify/-/fast-json-stable-stringify-2.1.0.tgz#874bf69c6f404c2b5d99c481341399fd55892633"
Expand All @@ -3987,6 +4032,13 @@ fast-levenshtein@^2.0.6, fast-levenshtein@~2.0.6:
resolved "https://registry.yarnpkg.com/fast-levenshtein/-/fast-levenshtein-2.0.6.tgz#3d8a5c66883a16a30ca8643e851f19baa7797917"
integrity sha1-PYpcZog6FqMMqGQ+hR8Zuqd5eRc=

fastq@^1.6.0:
version "1.8.0"
resolved "https://registry.yarnpkg.com/fastq/-/fastq-1.8.0.tgz#550e1f9f59bbc65fe185cb6a9b4d95357107f481"
integrity sha512-SMIZoZdLh/fgofivvIkmknUXyPnvxRE3DhtZ5Me3Mrsk5gyPL42F0xr51TdRXskBxHfMp+07bcYzfsYEsSQA9Q==
dependencies:
reusify "^1.0.4"

fb-watchman@^2.0.0:
version "2.0.1"
resolved "https://registry.yarnpkg.com/fb-watchman/-/fb-watchman-2.0.1.tgz#fc84fb39d2709cf3ff6d743706157bb5708a8a85"
Expand Down Expand Up @@ -4363,7 +4415,7 @@ glob-parent@^3.1.0:
is-glob "^3.1.0"
path-dirname "^1.0.0"

glob-parent@^5.0.0:
glob-parent@^5.0.0, glob-parent@^5.1.0:
version "5.1.1"
resolved "https://registry.yarnpkg.com/glob-parent/-/glob-parent-5.1.1.tgz#b6c1ef417c4e5663ea498f1c45afac6916bbc229"
integrity sha512-FnI+VGOpnlGHWZxthPGR+QhR78fuiK0sNLkHQv+bL9fQi57lNNdquIbna/WrfROrolq8GK5Ek6BiMwqL/voRYQ==
Expand Down Expand Up @@ -4438,6 +4490,18 @@ globals@^12.1.0:
dependencies:
type-fest "^0.8.1"

globby@^11.0.1:
version "11.0.1"
resolved "https://registry.yarnpkg.com/globby/-/globby-11.0.1.tgz#9a2bf107a068f3ffeabc49ad702c79ede8cfd357"
integrity sha512-iH9RmgwCmUJHi2z5o2l3eTtGBtXek1OYlHrbcxOYugyHLmAsZrPj43OtHThd62Buh/Vv6VyCBD2bdyWcGNQqoQ==
dependencies:
array-union "^2.1.0"
dir-glob "^3.0.1"
fast-glob "^3.1.1"
ignore "^5.1.4"
merge2 "^1.3.0"
slash "^3.0.0"

globby@^9.2.0:
version "9.2.0"
resolved "https://registry.yarnpkg.com/globby/-/globby-9.2.0.tgz#fd029a706c703d29bdd170f4b6db3a3f7a7cb63d"
Expand Down Expand Up @@ -4666,6 +4730,11 @@ ignore@^5.0.5, ignore@~5.1.4:
resolved "https://registry.yarnpkg.com/ignore/-/ignore-5.1.4.tgz#84b7b3dbe64552b6ef0eca99f6743dbec6d97adf"
integrity sha512-MzbUSahkTW1u7JpKKjY7LCARd1fU5W2rLdxlM4kdkayuCwZImjkpluF9CM1aLewYJguPDqewLam18Y6AU69A8A==

ignore@^5.1.4:
version "5.1.8"
resolved "https://registry.yarnpkg.com/ignore/-/ignore-5.1.8.tgz#f150a8b50a34289b33e22f5889abd4d8016f0e57"
integrity sha512-BMpfD7PpiETpBl/A6S498BaIJ6Y/ABT93ETbby2fP00v4EbvPBXWEoaR1UBPKs3iR53pJY7EtZk5KACI57i1Uw==

import-fresh@^2.0.0:
version "2.0.0"
resolved "https://registry.yarnpkg.com/import-fresh/-/import-fresh-2.0.0.tgz#d81355c15612d386c61f9ddd3922d4304822a546"
Expand Down Expand Up @@ -6199,6 +6268,11 @@ merge2@^1.2.3:
resolved "https://registry.yarnpkg.com/merge2/-/merge2-1.3.0.tgz#5b366ee83b2f1582c48f87e47cf1a9352103ca81"
integrity sha512-2j4DAdlBOkiSZIsaXk4mTE3sRS02yBHAtfy127xRV3bQUFqXkjHCHLW6Scv7DwNRbIWNHH8zpnz9zMaKXIdvYw==

merge2@^1.3.0:
version "1.4.1"
resolved "https://registry.yarnpkg.com/merge2/-/merge2-1.4.1.tgz#4368892f885e907455a6fd7dc55c0c9d404990ae"
integrity sha512-8q7VEgMJW4J8tcfVPy8g09NcQwZdbwFEqhe/WZkoIzjn/3TGDwtOCYtXGxA3O8tPzpczCCDgv+P2P5y00ZJOOg==

merge@^1.2.1:
version "1.2.1"
resolved "https://registry.yarnpkg.com/merge/-/merge-1.2.1.tgz#38bebf80c3220a8a487b6fcfb3941bb11720c145"
Expand Down Expand Up @@ -7036,7 +7110,7 @@ performance-now@^2.1.0:
resolved "https://registry.yarnpkg.com/performance-now/-/performance-now-2.1.0.tgz#6309f4e0e5fa913ec1c69307ae364b4b377c9e7b"
integrity sha1-Ywn04OX6kT7BxpMHrjZLSzd8nns=

picomatch@^2.0.4, picomatch@^2.0.5:
picomatch@^2.0.4, picomatch@^2.0.5, picomatch@^2.2.1:
version "2.2.2"
resolved "https://registry.yarnpkg.com/picomatch/-/picomatch-2.2.2.tgz#21f333e9b6b8eaff02468f5146ea406d345f4dad"
integrity sha512-q0M/9eZHzmr0AulXyPwNfZjtwZ/RBZlbN3K3CErVrk50T2ASYI7Bye0EvekFY3IP1Nt2DHu0re+V2ZHIpMkuWg==
Expand Down Expand Up @@ -7637,6 +7711,11 @@ retry@^0.10.0:
resolved "https://registry.yarnpkg.com/retry/-/retry-0.10.1.tgz#e76388d217992c252750241d3d3956fed98d8ff4"
integrity sha1-52OI0heZLCUnUCQdPTlW/tmNj/Q=

reusify@^1.0.4:
version "1.0.4"
resolved "https://registry.yarnpkg.com/reusify/-/reusify-1.0.4.tgz#90da382b1e126efc02146e90845a88db12925d76"
integrity sha512-U9nH88a3fc/ekCF1l0/UP1IosiuIjyTh7hBvXVMHYgVcfGvt897Xguj2UOLDeI5BG2m7/uwyaLVT6fbtCwTyzw==

rimraf@*, rimraf@^3.0.0, rimraf@^3.0.2:
version "3.0.2"
resolved "https://registry.yarnpkg.com/rimraf/-/rimraf-3.0.2.tgz#f1a5402ba6220ad52cc1282bac1ae3aa49fd061a"
Expand Down Expand Up @@ -7668,6 +7747,11 @@ run-async@^2.2.0, run-async@^2.4.0:
resolved "https://registry.yarnpkg.com/run-async/-/run-async-2.4.1.tgz#8440eccf99ea3e70bd409d49aab88e10c189a455"
integrity sha512-tvVnVv01b8c1RrA6Ep7JkStj85Guv/YrMcwqYQnwjsAS2cTmmPGBBjAjpCW7RrSodNSoE2/qg9O4bceNvUuDgQ==

run-parallel@^1.1.9:
version "1.1.9"
resolved "https://registry.yarnpkg.com/run-parallel/-/run-parallel-1.1.9.tgz#c9dd3a7cf9f4b2c4b6244e173a6ed866e61dd679"
integrity sha512-DEqnSRTDw/Tc3FXf49zedI638Z9onwUotBMiUFKmrO2sdFKIbXamXGQ3Axd4qgphxKB4kw/qP1w5kTxnfU1B9Q==

run-queue@^1.0.0, run-queue@^1.0.3:
version "1.0.3"
resolved "https://registry.yarnpkg.com/run-queue/-/run-queue-1.0.3.tgz#e848396f057d223f24386924618e25694161ec47"
Expand Down

0 comments on commit 3a7ec9b

Please sign in to comment.