Race condition sometimes causes pnpm install to fail when projects define executable bins

[stefan-toubia]

pnpm version: 7.3.0

Code to reproduce the issue:

https://github.com/stefan-toubia/pnpm-multi-lockfile/tree/install-race

Suppose you have a repo that defines a couple projects, as well as shared tools that provide some executables meant to be shared by all projects. In this example, I've created 2 projects and 2 shared tools that provide eslint and stylelint by re-exporting their own binstubs.

/project-a
  "dependencies": {
    "project-b": "workspace:^1.0.0",
    "tool-a": "workspace:^1.0.0",
    "tool-b": "workspace:^1.0.0"
  }
/project-b
  "dependencies": {
    "gifsicle": "^7.0.1",
    "tool-a": "workspace:^1.0.0"
  }
/common/tool-a
  "dependencies": {
    "eslint": "^8.26.0"
  },
  "bin": {
    "eslint": "./node_modules/.bin/eslint"
  },
/common/tool-b
  "dependencies": {
    "stylelint": "^14.14.0",
    "tool-a": "workspace:^1.0.0"
  },
  "bin": {
    "stylelint": "./node_modules/.bin/stylelint"
  },

It seems like install is not installing projects and creating binstubs in a proper topological ordering. Right now this type of "warning" is very common in our repo, though most of the time said warning seems to fix itself at some point, maybe the operation gets retried after the warning is emitted?

 WARN  Failed to create bin at /Users/stoubia/repos/pnpm-multi-lockfile/project-a/node_modules/.bin/eslint. The source file at /Users/stoubia/repos/pnpm-multi-lockfile/common/tool-a/node_modules/.bin/eslint does not exist.

$ pnpm i
Scope: all 4 workspace projects
Lockfile is up-to-date, resolution step is skipped
Packages: +380
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Packages are hard linked from the content-addressable store to the virtual store.
  Content-addressable store is at: /Users/stoubia/Library/Caches/pnpm/store/v3
  Virtual store is at:             node_modules/.pnpm
 WARN  Failed to create bin at /Users/stoubia/repos/pnpm-multi-lockfile/project-a/node_modules/.bin/eslint. The source file at /Users/stoubia/repos/pnpm-multi-lockfile/common/tool-a/node_modules/.bin/eslint does not exist.
 WARN  Failed to create bin at /Users/stoubia/repos/pnpm-multi-lockfile/project-a/node_modules/.bin/stylelint. The source file at /Users/stoubia/repos/pnpm-multi-lockfile/common/tool-b/node_modules/.bin/stylelint does not exist.
Progress: resolved 380, reused 380, downloaded 0, added 380, done

I've attempted to reproduce it but it's I have not figured out how to repeat the error condition consistently.

Sometimes pnpm install fails on the first install, but then works on the second install.

$ pnpm i
Scope: all 4 workspace projects
Lockfile is up-to-date, resolution step is skipped
Already up-to-date
 ENOENT  ENOENT: no such file or directory, chmod '/Users/stoubia/repos/pnpm-multi-lockfile/common/tool-a/node_modules/.bin/eslint'

$ pnpm i
Scope: all 4 workspace projects
Lockfile is up-to-date, resolution step is skipped
Already up-to-date

Expected behavior:

Is it safe to assume that binstubs should be initialized in topological order within the worksapce?
Unless there's something inherently wrong with the way the example project has been setup in terms of defining binstubs that point to node_modules/.bin, it seems like pnpm needs to ensure proper topological ordering so that each project is first installed, then binstubs are setup, then other repo projects binstubs are not created until all of its workspace dependencies have been fully initialized with its own binstubs.

Actual behavior:

Some race condition causes pnpm install to fail periodically on the first install. I could be wrong in the diagnosis, or it could be multiple problems. Also note that this behavior seems to be independent of shared-workspace-lockfile or recursive-install, I've tried all combinations and those settings seem to have not effect.

Additional information:

• node -v prints: v16.15.1
• Windows, macOS, or Linux?: macos

[stefan-toubia]

I didn't see this before, but this seems like a duplicate of this ancient issue #1801

This comment was useful and will likely solve my issues, though it still seems like ensuring binstubs are created in topological order would be the right fix.

[zkochan]

I don't get any errors with pnpm v8.6.6