ESLint checking from the 1st compilation blocks 2nd webpack compilation (tapAfterCompileToAddDependencies)

[grumd]

Current behavior

1. During the initial compilation, fork-ts-checker starts doing its thing, and one of these things is getDependencies.
2. Dependencies resolution will be written to dependenciesPromise
3. Next webpack compilation will stumble upon tapAfterCompileToAddDependencies hook.
4. If dependencies weren't resolved yet, this compilation will be blocked until fork-ts-checker finishes its job.

• My first initial compilation takes ~22 seconds.
• Initial TS+ESLint checks take something like 2 minutes.
• Without this plugin an HMR recompilation takes 1-2 seconds.
• Issue: With the plugin, an HMR recompilation will have to wait for dependenciesPromise to resolve which means I can't use the app for the next 2 minutes while plugin is doing all the checks.

Illustration of the issue at hand, hope it's not too messy (I added a couple of logs to show how much time tapAfterCompileToAddDependencies takes).

Note: I can disable ESLint and this doesn't happen. getDependencies takes so long for ESLint specifically.

Expected behavior

Not sure how this should be fixed, but maybe this can work?

• Don't tap into afterCompile hook.
• Wait until getDependencies resolves and only after that tap some hook and add resolved dependencies into next webpack compilation?

I couldn't really make this work locally for me, but if you can point me to the right solution, I can make a PR.

Steps to reproduce the issue

• Have a project big enough for ESLint's getDependencies to take considerable time.
• Compile with webpack and immediately do an HMR recompilation - easiest use-case is when lazyCompilation is enabled.
• Notice how 2nd webpack compilation will only finish when fork-ts-checker is finished.

Issue reproduction repository

I can try to create one if description above isn't enough.

Environment

• fork-ts-checker-webpack-plugin: 6.2.6
• typescript: 4.1.2 (irrelevant I think)
• eslint: 7.8.1
• webpack: 5.35.0
• os: win10

[grumd]

I really love this extension, works pretty much flawlessly at 6.2.6 except for this one thing. Thank you for your work

[piotr-oles]

Thanks for reporting that, maybe #613 will solve it 🤔 Maybe getDependencies is traversing node_modules so that PR should improve time in that case

[piotr-oles]

Could you check if this problem still exists with 6.2.7?

[grumd]

@piotr-oles Unfortunately still happens in 6.2.7 :(

Seems like it shaved 20s off the 120s total.
Disabling a few heavy ESLint rules could take it down to 55s, but it's still a lot.

[grumd]

I'll be debugging for a bit right now to find what takes the most time, because atm ESLint's getDependencies seems to take <1s

[grumd]

Intermediate findings: await previousReport.close(); seems to take ~1 minute for me.
(L90 in /src/hooks/tapStartToConnectAndRunReporter.ts)

With logging setup like this:

I get this in terminal:

[grumd]

Removing await from await previousReport.close(); just causes it to get stuck at the next line, const report = await reporter.getReport(change);. Wild guess: while the other thread is busy with eslint/ts checks, we can't communicate with it until it finishes the job?

[grumd]

Doing this "fixed" my issue, 2nd compilation is 3-4 seconds now, but I don't know the implications of doing this. It seems to work, but maybe there's some bugs I just didn't find yet.

Edit: seems like eventually plugin crashes with an unhandled promise rejection because it can't find a report...

[piotr-oles]

You're right - the eslint code is synchronous so it's blocking the thread... I think we could move dependency computation to the main process or a separate process, but it's non-trivial change :/

[rkriauciukas-freesat]

I came here to just confirm I am having the same issue - I think it was due to recent version upgrade to one of eslint-related packages.

Basically it always hangs for me on 94% after seal in webpack watch mode and on subsequent recompiles after the first compilation.
I've just done the tweak @grumd suggested - and it is great, webpack does not get blocked anymore.

Any robust fixes for this one planned and on the roadmap?

[piotr-oles]

I think I will try to move dependencies resolution to the main thread :) I'm not sure when I find time for this. Feel free to open PR ;)

[caiquelsousa]

Same problem here, and I just realized that this problem was introduced in the @6.0.4 version because the @6.0.3 version has the asynchronous process working just perfectly...

v6.0.3...v6.0.4

Also want to thank you guys for this lib it's really awesome!

[caiquelsousa]

More specifically the problem is with this line #68
/src/watch/InclusiveNodeWatchFileSystem.ts

  watch(
    files: Iterable<string>,
    dirs: Iterable<string>,
    missing: Iterable<string>,
    startTime?: number,
    options?: Partial<WatchFileSystemOptions>,
    callback?: Function,
    callbackUndelayed?: Function
  ): Watcher {
    clearFilesChange(this.compiler);
    const isIgnored = createIsIgnored(options?.ignored);

    // use standard watch file system for files and missing
    const standardWatcher = this.watchFileSystem.watch(
      files,
      dirs, // <-- this line is the problem
      missing,
      startTime,
      options,
      callback,
      callbackUndelayed
    );

It's watching to many files and directory, and I believe that the filesystem starts to watch all files under the dir when added

this also happens with files

I believe that we shouldn't be watching files or dir of node_modules.
In fact it becomes blazing fast when you do something like this:

  watch(
    files: Iterable<string>,
    dirs: Iterable<string>,
    missing: Iterable<string>,
    startTime?: number,
    options?: Partial<WatchFileSystemOptions>,
    callback?: Function,
    callbackUndelayed?: Function
  ): Watcher {
    clearFilesChange(this.compiler);
    const isIgnored = createIsIgnored(options?.ignored);

    const isProjectPath = (path: string) => {
      return !path.includes('node_modules');
    };
    
    // use standard watch file system for files and missing
    const standardWatcher = this.watchFileSystem.watch(
      [...files].filter(isProjectPath),
      [...dirs].filter(isProjectPath),
      [...missing].filter(isProjectPath),
      startTime,
      options,
      callback,
      callbackUndelayed
    );

status	files	dirs	missing
before	6099	4	28963
after	672	0	4004

it's a compilation difference between 20 segs and 2 mins

[piotr-oles]

@caiquelsousa
This is how webpack watch system works - I don't want to change it because it will probably break something in someone's project. Have you tried ignored option in webpack configuration?https://webpack.js.org/configuration/watch/#watchoptionsignored

[caiquelsousa]

yeah, that's not the problem I just checked again and it seems that when I use fork-ts-checker-webpack-plugin with react-dev-utils/WatchMissingNodeModulesPlugin it adds the path '/Users/${USER}/Projects/${PROJECT}/node_modules' to my dirs list and then I have a problem

so when I remove the WatchMissingNodeModulesPlugin it makes fork-ts-checker-webpack-plugin unlocks the main thread

[rkriauciukas-freesat]

In my setup there is no WatchMissingNodeModulesPlugin plugin used - but I've added the watchOptions.ignored option to webpack.config.js and will see if it has any positive effect.

[piotr-oles]

@rkriauciukas-freesat I think it's not related to this issue - @caiquelsousa could you open a new issue regarding WatchMissingNodeModulesPlugin?

[grumd]

@caiquelsousa I'll chime in to say that using 6.0.3 still results in the same problem I have.

[amertak]

We are dealing with the same issue, we have a large repository and '94% after seal' in subsequent builds takes sometimes more than 5 minutes. Tried 7.0.0-alpha.3 with no luck. Not sure if it's connected to using workspaces or large or wrongly configure file dependencies.

[rkriauciukas-freesat]

@amertak I am too still having the same issue - hoped that v6.2.12 of the package would fix the issue, but no luck.
I am now resorting to using just ts-loader + babel-loader and disabling fork-ts-checker-webpack-plugin: without fail it always recompiles just fine, but is slower to do that... My work involves loading up transpiled JS onto hardware, which in itself is a longish process, so if Webpack is taking longer or gets stuck, that just adds to the frustration.

I am on Windows10 + WSL1, so given WSL1 issues with file locking mechanism etc, maybe Webstorm JS sub-processes lock some files and obstruct something or maybe there is another reason for this.

Would love to see this fixed.

Maybe we can add separate config flag for those who want to use it?

[piotr-oles]

@rkriauciukas-freesat , @amertak do you use eslint feature? Or only type-checking?

[amertak]

@piotr-oles I do not have eslint enabled in this plugin. Using webstorm plugin.

[rkriauciukas-freesat]

@piotr-oles Yes, I am relying heavily on ESLint, but not calling it from Webpack (it is disabled by default in Webpack).
My Webstorm is setup to use Typescript Checker + ESLint Checker and the former is pointed to the below .eslintrc.js.

I've tried disabling those in Webstorm, but do not seem to make any difference.

.eslintrc.js: is setup to use parser: '@typescript-eslint/parser' with below parser options:

parserOptions: {
        // Load ESLint-specific TS config which covers more files and extends the main one
        project: path.join(__dirname, './tsconfig.eslint.json'),
        ecmaVersion: 2018, // Allows for the parsing of modern ECMAScript features
        sourceType: 'module', // Allows for the use of imports
        ecmaFeatures: {
            jsx: true, // Allows for the parsing of JSX
        },
    },

Below is ./tsconfig.eslint.json - which extends the main ./tsconfig.json:

{
    "extends": "./tsconfig.json",
    "include": [
        "**.js*",
        "src/**/*.ts*",
        "src/**/*.js*",
        "tests/**/*.ts*",
        "tests/**/*.js*",

        "src/**/*.test.ts",
        "src/**/*.test.js",
        "tests/**/*.test.ts",
        "tests/**/*.test.js",

        "G3Emu/**/*.ts*",
        "G3Emu/**/*.js*",
    ],
    "exclude": [
        "node_modules",
        "build",
        "build/**/*"
    ]
}

The main `./tsconfig.json':

{
    "compilerOptions": {
        "target": "ES6",
        "lib": [
            "ES6",
            "ES2015",
            "ES2020",
            "dom",
            "dom.iterable",
            "esnext"
        ],
        "allowJs": true,
        "skipLibCheck": false,
        "esModuleInterop": true,
        "allowSyntheticDefaultImports": true,
        "strict": false,
        "forceConsistentCasingInFileNames": true,
        "module": "esnext",
        "moduleResolution": "node",
        "resolveJsonModule": true,
        "isolatedModules": false,
        "noEmit": false,
        "jsx": "preserve",
        "removeComments": false,
        "sourceMap": true,
        "inlineSources": true,
        "sourceRoot": "/",
        "incremental": true
    },
    "include": [
        "src/**/*.ts*",
        "src/**/*.js*",
        "G3Emu/**/*.ts*",
        "G3Emu/**/*.js*"
    ],
    "exclude": [
        "./node_modules/*",
        "node_modules",
        "typings",
        "src/**/*.spec.*",
        "src/**/*.test.*",
        "test/**/*.spec.*",
        "test/**/*.test.*",
        "G3Emu/**/*.spec.*",
        "G3Emu/**/*.test.*",
        "build"
    ]
}

[piotr-oles]

That's good because the next version of the plugin should fix this issue but we are also dropping support for eslint :)

[piotr-oles]

🎉 This issue has been resolved in version 6.3.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[piotr-oles]

🎉 This issue has been resolved in version 7.0.0-alpha.15 🎉

The release is available on:

• npm package (@alpha dist-tag)
• GitHub release

Your semantic-release bot 📦🚀