Dependency resolution not deterministic

[unional]

pnpm version:

7.28.0 with lockfile v6

Code to reproduce the issue:

I was work on this: #5585 (comment)

I delete the lockfile and node_modules folder and start again,
this time it resolves the dependency issue.

I then remove the overrides field and test how it works.

When I look at the lock file, I saw many changes.
I was expecting it just remove the overrides and maybe adjust (mess up) dependencies related to webpack.

But I found changes like these:

Actual behavior:

eslint-plugin-harmony:
  specifier: ^7.1.1
- version: 7.1.1(@typescript-eslint/eslint-plugin@5.54.0)(eslint-config-prettier@8.6.0)(eslint@8.35.0)(typescript@4.9.5)  
+ version: 7.1.1(@typescript-eslint/eslint-plugin@5.54.0)(eslint@8.35.0)(typescript@4.9.5)

'@storybook/addon-docs':
  specifier: ^7.0.0-beta.47
- version: 7.0.0-beta.56(@storybook/mdx1-csf@1.0.0-next.1)(react-dom@16.14.0)(react@16.14.0)
+ version: 7.0.0-beta.56(react-dom@16.14.0)(react@16.14.0)

... and many more

Since the actual version installed is the same (they are the latest),
the resolution really should stay the same.

Additional information:

• node -v prints: v16.19.0
• Windows, macOS, or Linux?: macOS

[unional]

The lockfile got messed up every time I install or update some dependency.

I have to delete the lock file and node_modules every time to fix this.
Basically throwing the benefit of lockfile out of the window.

I guess pretty soon even that will stop working and I'll be completely stucked.

[unional]

This is happening very often to me.

[zkochan]

I got a similar issue too. It is important to fix.

In my case it happened when I added a new dependency to a project in a workspace.

[zkochan]

try v8.0.0-rc.1

or v7.30.4

[zkochan]

@unional can you confirm that your issue has been fixed?

[unional]

It happens to my work repo, not in OSS, so will check it tomorrow.

[zkochan]

@unional were you able to check it?

[unional]

I'm validating this, using 7.30.5.

I'm running pnpm up -r "@myscope/*".
I saw changes like these:

      '@typescript-eslint/eslint-plugin':
        specifier: ^5.54.1
-        version: 5.56.0(@typescript-eslint/parser@5.56.0)(eslint@8.36.0)(typescript@4.9.5)
+        version: 5.56.0(@typescript-eslint/parser@5.57.0)(eslint@8.36.0)(typescript@4.9.5)
      eslint-plugin-harmony:
        specifier: ^7.1.1
-        version: 7.1.2(@typescript-eslint/eslint-plugin@5.56.0)(eslint-config-prettier@8.8.0)(eslint@8.36.0)(typescript@4.9.5)
+        version: 7.1.2(@typescript-eslint/eslint-plugin@5.57.0)(eslint-config-prettier@8.8.0)(eslint@8.36.0)(typescript@4.9.5)

  /@storybook/client-logger@7.0.0-rc.7:
-    resolution: {integrity: sha1-ivPqyXGd4YdP8GAQXcKKW9cGjKE=, tarball: https://art.code.pan.run:443/artifactory/api/npm/npm-panvirt/@storybook/client-logger/-/client-logger-7.0.0-rc.7.tgz}
+    resolution: {integrity: sha1-UisEGG5CDa7eK8sjD9NrHEYTJQE=, tarball: https://art.code.pan.run:443/artifactory/api/npm/npm-panvirt/@storybook/client-logger/-/client-logger-7.0.0-rc.8.tgz}

and a whole bunch more:

I don't expect other dependencies to be changed.

If I update the package.json manually and run pnpm i, I still got similar behavior, but the changes are fewer, relatively:

The dependency I'm updating has nothing to do with eslint and storybook.

[unional]

Some of the changes are add and remove of tarball:

  /@formatjs/ecma402-abstract@1.7.0:
-    resolution: {integrity: sha512-0IQF4oDZdO8ruyrNJZuRle3F/YiGgRwTNrZyMI1N1X8GERZusOrXU9Stw+j/lyyfDWaJK44b+Qnri/qfLPCtGQ==, tarball: <some-url>}
+    resolution: {integrity: sha512-0IQF4oDZdO8ruyrNJZuRle3F/YiGgRwTNrZyMI1N1X8GERZusOrXU9Stw+j/lyyfDWaJK44b+Qnri/qfLPCtGQ==}
    dependencies:
      tslib: 2.5.0

  /react-docgen-typescript@2.2.2(typescript@4.9.5):
-    resolution: {integrity: sha1-RhEFXlae3AcSBKrbIOHJPhqxZZw=}
+    resolution: {integrity: sha1-RhEFXlae3AcSBKrbIOHJPhqxZZw=, tarball: <some-url>}
    peerDependencies:
      typescript: '>= 4.3.x'
    dependencies:
      typescript: 4.9.5
    dev: true

[unional]

  /react-intl@5.25.1(react@16.14.0)(typescript@4.9.5):
-    resolution: {integrity: sha1-aKc678SFyb9wBiOBrn9vR5FoCHk=, tarball: <some-url>}
+    resolution: {integrity: sha1-aKc678SFyb9wBiOBrn9vR5FoCHk=}
    peerDependencies:
      react: ^16.3.0 || 17 || 18
      typescript: ^4.5
+    peerDependenciesMeta:
+      typescript:
+        optional: true

  /uni-require@0.0.1(@types/node@18.15.7):
    resolution: {integrity: sha1-C8EXUUVgWNhmuRMZWyFxLn0nObw=}
    peerDependencies:
      '@types/node': '*'
    dependencies:
      '@types/node': 18.15.7
+    dev: false

[unional]

One good thing is this time it doesn't cause webpack to fail (due to getting two versions of webpack), that's the main issue I was having.

Will continue to monitor to see if that happens again or not.

[unional]

FYI it still happens with 7.30.5 and 8.1.0

[unional]

@zkochan can you reopen this? It is still happening in 8.2.0

you can see that pnpm why webpack says there is only one version of webpack installed (5.79.0):

pnpm why webpack

devDependencies:
@pats/just-sparky 6.1.1
└─┬ @sparky/framework 0.0.29 peer
  └─┬ @sparky/webpack-config 0.3.39
    └─┬ webpack 5.79.0 peer
      └─┬ terser-webpack-plugin 5.3.7
        └── webpack 5.79.0 peer

But the build error shows there is webpack@5.78.0 there:

TypeError: The 'compilation' argument must be an instance of Compilation
    at Function.getCompilationHooks (/project/node_modules/.pnpm/webpack@5.78.0_esbuild@0.16.17/node_modules/webpack/lib/NormalModule.js:228:10)
    at /project/node_modules/.pnpm/webpack-manifest-plugin@4.1.1_webpack@5.78.0/node_modules/webpack-manifest-plugin/dist/index.js:58:42
    at _next43 (eval at create (/project/node_modules/.pnpm/tapable@2.2.1/node_modules/tapable/lib/HookCodeFactory.js:19:10), <anonymous>:50:1)
    at _next21 (eval at create (/project/node_modules/.pnpm/tapable@2.2.1/node_modules/tapable/lib/HookCodeFactory.js:19:10), <anonymous>:97:1)
    at Hook.eval [as call] (eval at create (/project/node_modules/.pnpm/tapable@2.2.1/node_modules/tapable/lib/HookCodeFactory.js:19:10), <anonymous>:143:1)
    at Hook.CALL_DELEGATE [as _call] (/project/node_modules/.pnpm/tapable@2.2.1/node_modules/tapable/lib/Hook.js:14:14)
    at Compiler.newCompilation (/project/node_modules/.pnpm/webpack@5.79.0_esbuild@0.16.17/node_modules/webpack/lib/Compiler.js:1125:26)
    at /project/node_modules/.pnpm/webpack@5.79.0_esbuild@0.16.17/node_modules/webpack/lib/Compiler.js:1169:29
    at Hook.eval [as callAsync] (eval at create (/project/node_modules/.pnpm/tapable@2.2.1/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (/project/node_modules/.pnpm/tapable@2.2.1/node_modules/tapable/lib/Hook.js:18:14)
 ELIFECYCLE  Command failed with exit code 1.

When running pnpm why webpack-manifest-plugin, nothing shows up.

[unional]

Under node_modules/.pnpm, there are "3" versions of webpack there:

[unional]

Or reopen this one. That's the one I originally mentioned about the webpack compilation error:

#5585

[codepunkt]

Same problems here. Doing updates of a single dependency in a workspaces setting causes unrelated updates all over the place, including running into the same webpack problem with multiple versions. @unional how did you solve this?

[unional]

I'm doing workarounds like this:

pnpm up -r abc   // webpack is out of sync


pnpm up -r webpack // realign webpack

It is not optimal as there can be other packages that are affected by this.
But that's what's been working for me (and probably other packages doesn't care that much).

[codepunkt]

@unional We found out that manually changing a dependency version in a package.json of one of our workspaces and running pnpm i afterwards does what we want and only applies minimal updates to our lockfile while using pnpm up to do that screws us over and changes thousands of lines in the lockfile. I'm wondering what setting this is related to...

/cc @zkochan

[unional]

Thanks!

Sometimes I need to update multiple projects using wildcards. Maybe in that case, I can do revert and install. Will need to try that to see if it gets the same result. i.e.:

pnpm up -r "@just-web/*"

git reset pnpm-lock.yaml

pnpm i

[unional]

Unfortunately, I've confirmed that neither workaround works 100% of the time. I have this case where I upgraded some dependencies, undo the changes in the lock file, even nuke the node_modules folder and redo pnpm i. Still get the same webpack error.

[tgdn]

This still happening on 8.15.3 and is a nightmare to fix