Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PNPM mysteriously installs unrelated packages #5132

Closed
octogonz opened this issue Aug 1, 2022 · 6 comments · Fixed by #5140
Closed

PNPM mysteriously installs unrelated packages #5132

octogonz opened this issue Aug 1, 2022 · 6 comments · Fixed by #5140

Comments

@octogonz
Copy link
Member

octogonz commented Aug 1, 2022

pnpm version: 6.33.0

Code to reproduce the issue:

  1. Create an empty project:
    C:\>md test
    C:\>cd test
    C:\test>pnpm init
    
  2. Install the @babel/parser package:
    C:\test>pnpm install @babel/parser@7.17.8
    
    

Expected behavior:

Only @babel/parser should get installed, because its package.json file specifies no dependencies:

@babel/parser/package.json

{
  "name": "@babel/parser",
  "version": "7.17.8",
  "description": "A JavaScript parser",
  "author": "The Babel Team (https://babel.dev/team)",
  "homepage": "https://babel.dev/docs/en/next/babel-parser",
  "bugs": "https://github.com/babel/babel/issues?utf8=%E2%9C%93&q=is%3Aissue+label%3A%22pkg%3A+parser+%28babylon%29%22+is%3Aopen",
  "license": "MIT",
  "publishConfig": {
    "access": "public"
  },
  "keywords": [
    "babel",
    "javascript",
    "parser",
    "tc39",
    "ecmascript",
    "@babel/parser"
  ],
  "repository": {
    "type": "git",
    "url": "https://github.com/babel/babel.git",
    "directory": "packages/babel-parser"
  },
  "main": "./lib/index.js",
  "types": "./typings/babel-parser.d.ts",
  "files": [
    "bin",
    "lib",
    "typings"
  ],
  "engines": {
    "node": ">=6.0.0"
  },
  "devDependencies": {
    "@babel/code-frame": "^7.16.7",
    "@babel/helper-fixtures": "^7.17.0",
    "@babel/helper-validator-identifier": "^7.16.7",
    "charcodes": "^0.2.0"
  },
  "bin": "./bin/babel-parser.js"
}

Actual behavior:

For some reason @babel/types and (its dependencies) to-fast-properties and @babel/helper-validator-identifier are also getting installed:

pnpm-lock.yaml

lockfileVersion: 5.3

specifiers:
  '@babel/parser': 7.17.8

dependencies:
  '@babel/parser': 7.17.8

packages:

  /@babel/helper-validator-identifier/7.18.6:
    resolution: {integrity: sha512-MmetCkz9ej86nJQV+sFCxoGGrUbU3q02kgLciwkrt9QqEB7cP39oKEY0PakknEO0Gu20SskMRi+AYZ3b1TpN9g==}
    engines: {node: '>=6.9.0'}
    dev: false

  /@babel/parser/7.17.8:
    resolution: {integrity: sha512-BoHhDJrJXqcg+ZL16Xv39H9n+AqJ4pcDrQBGZN+wHxIysrLZ3/ECwCBUch/1zUNhnsXULcONU3Ei5Hmkfk6kiQ==}
    engines: {node: '>=6.0.0'}
    hasBin: true
    dependencies:
      '@babel/types': 7.18.9
    dev: false

  /@babel/types/7.18.9:
    resolution: {integrity: sha512-WwMLAg2MvJmt/rKEVQBBhIVffMmnilX4oe0sRe7iPOHIGsqpruFHHdrfj4O1CMMtgMtCU4oPafZjDPCRgO57Wg==}
    engines: {node: '>=6.9.0'}
    dependencies:
      '@babel/helper-validator-identifier': 7.18.6
      to-fast-properties: 2.0.0
    dev: false

  /to-fast-properties/2.0.0:
    resolution: {integrity: sha512-/OaKK0xYrs3DmxRYqL/yDc+FxFUVYhDlXMhRmv3z915w2HF1tnN1omB354j8VUGO/hbRzyD6Y3sA7v7GS/ceog==}
    engines: {node: '>=4'}
    dev: false

This doesn't repro with NPM 8.11.0 or Yarn 1.22.19 -- those package managers install @babel/parser only.

Additional information:

  • node -v prints: 14.19.3
  • Windows, macOS, or Linux?: Windows and Mac
@octogonz octogonz changed the title PNPM mysteriously installs @types packages PNPM mysteriously installs unrelated packages Aug 1, 2022
@octogonz
Copy link
Member Author

octogonz commented Aug 1, 2022

To clarify, these extraneous packages are not linked into the project's node_modules folder. They are installed as "hoisted" dependencies under node_modules/.pnpm/:

node_modules/.modules.yaml

hoistPattern:
  - '*'
hoistedDependencies:
  /@babel/helper-validator-identifier/7.18.6:
    '@babel/helper-validator-identifier': private
  /@babel/types/7.18.9:
    '@babel/types': public
  /to-fast-properties/2.0.0:
    to-fast-properties: private
included:
  dependencies: true
  devDependencies: true
  optionalDependencies: true
injectedDeps: {}
layoutVersion: 5
nodeLinker: isolated
packageManager: pnpm@6.33.0
pendingBuilds: []
prunedAt: Mon, 01 Aug 2022 01:06:56 GMT
publicHoistPattern:
  - '*types*'
  - '*eslint*'
  - '@prettier/plugin-*'
  - '*prettier-plugin-*'
registries:
  default: https://registry.npmjs.org/
skipped: []
storeDir: C:\Users\Owner\.pnpm-store\v3
virtualStoreDir: C:\test\node_modules\.pnpm

I encountered this while investigating an unwanted upgrade. The appearance of these packages in pnpm-lock.yaml seems to be new. When installing a lockfile that does not have these dependencies (probably from an older PNPM?) that difference seems to be triggering an unwanted upgrade. (?)

@octogonz
Copy link
Member Author

octogonz commented Aug 1, 2022

image

@zkochan is it true that...

  1. Our Git-tracked files (package.json, pnpmfile.cjs, etc) are not the complete specification of dependencies for a package? In fact PNPM secretly applies its own fixups of the package.json dependencies; and
  2. pnpmfile.cjs does not have the final word about dependencies, because PNPM's fixups are applied after the readPackage() hook; and
  3. The table of fixup rules varies with each PNPM release (because @yarnpkg/extensions is bundled into the PNPM package); and
  4. Differences across versions can cause pnpm install to interpret the lockfile as being inconsistent, which can trigger an unwanted upgrade

If so, this is counterintuitive. Can we disable this magic somehow? Or even better, is there a way for us to store the fixup rules explicitly in a Git-tracked file that our developers can easily be aware of and inspect?

@octogonz
Copy link
Member Author

octogonz commented Aug 1, 2022

Looks like the implementation is here:

import { packageExtensions as compatPackageExtensions } from '@yarnpkg/extensions'

const hooks: ReadPackageHook[] = []
if (!isEmpty(overrides ?? {})) {
hooks.push(createVersionsOverrider(overrides!, lockfileDir))
}
hooks.push(createPackageExtender(fromPairs(compatPackageExtensions)))
if (!isEmpty(packageExtensions ?? {})) {
hooks.push(createPackageExtender(packageExtensions!))
}

@octogonz
Copy link
Member Author

octogonz commented Aug 1, 2022

After a bunch of experiments, I determined that this behavior was introduced in PNPM version 6.32.12.

Empirical results:

  1. Start with an empty package.json and do pnpm install @babel/parser@7.17.8. The extra entries are present.
  2. Downgrade to 6.32.11 and run pnpm install @babel/parser@7.17.8 (or pnpm install). The extra entries are still present.
  3. Downgrade to 6.32.11, empty package.json, and delete pnpm-lock.yaml, and delete node_modules/.modules.yaml. Then run pnpm install @babel/parser@7.17.8. The extra entries are STILL present. Somehow the bad state is remembered. ❓
  4. Downgrade to 6.32.11, empty package.json, and delete pnpm-lock.yaml, and delete node_modules/.pnpm/lock.yaml. Then run pnpm install @babel/parser@7.17.8. Okay the extra entries are removed. ✔
  5. Upgrade to 6.32.12 (without emptying package.json). Run pnpm install. The extra entries stay removed.
  6. Upgrade to 6.32.12 (without emptying package.json). Run pnpm install @babel/parser@7.17.8. The extra entries are re-added.

In the past, deleting pnpm-lock.yaml was sufficient to fully reset PNPM's version selections. It seems that now we must delete both pnpm-lock.yaml and node_modules/.pnpm/lock.yaml, otherwise some state is persisted.

For the 7.x series, the problem is introduced in version 7.0.1.

Summary

👉 The most recent releases unaffected by this problem are 6.32.11 and 7.0.0.

In order to eliminate the extra dependencies, you must do all of the following:

  • Downgrade PNPM
  • Delete pnpm-lock.yaml
  • Delete node_modules/.pnpm/lock.yaml

@octogonz
Copy link
Member Author

octogonz commented Aug 1, 2022

I discussed this with Zoltan, who responded quickly and was very helpful. 🙏 The plan is to implement a setting to optionally disable the @yarnpkg/extensions fixups.

It's also worth considering whether these fixups should be applied earlier, since pnpmfile.cjs is less useful if it is not the final authority about version selections.

zkochan added a commit that referenced this issue Aug 1, 2022
@zkochan
Copy link
Member

zkochan commented Aug 1, 2022

#5140

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants