npm v7 does not install linked packages dependencies

[garciaalvaro]

Current Behavior:

npm v7 does not install linked local packages dependencies.

Expected Behavior:

In npm v6 the dependencies of a local linked package are installed. This means that if app has my-local-pkg as a dependency, running npm install generates app/node_modules folder and also my-local-pkg/node_modules (with its dependencies).
In npm v7, however, only app/node_modules is generated.

Steps To Reproduce:

I created the following repository to illustrate it. The README.md has the instructions on how to reproduce it.
Simplified, the steps are:

1. Add the following folder structure: ./app and ./my-local-pkg, each with their own package.json
2. Include this in the ./app/package.json:

{
  "name": "app",
  "dependencies": {
    "my-local-pkg": "file:../my-local-pkg",
    "prettier": "^2.2.1"
  }
}

3. Include this in the ./my-local-pkg/package.json:

{
  "name": "my-local-pkg",
  "dependencies": {
    "typescript": "^4.1.3"
  }
}

4. Go to ./app and run npm install.
   npm version 6 produces:

node_modules
	.bin
	my-local-pkg
		node_modules
			typescript
	prettier

while npm version 7 produces:

node_modules
	.bin
	my-local-pkg
	prettier

Compare the two outputs, npm v7 does not install my-local-pkg dependencies, there is no ./my-local-pkg/node_modules folder generated.

Environment:

npm v7.1.2

[raphinesse]

Same behavior with npm@7.2.0

[rafgraph]

I'd also add that for me the expected behavior for local installs npm install ../some-package is to result in the same package-lock.json as installs from the registry with the only difference being the "resolved" field of local install pointing to ../some-package instead of to the registry (and other related metadata like "link": true).

This means that the dependencies of the locally installed package should end up in the top level of node_modules unless there is a version conflict. I believe that v6 puts all of the local install package's dependencies in its node_modules and not in the top level node_modules (hopefully v7 will change this).

Having the install behavior be consistent will make it easier to develop npm packages by being able to install the package locally in development to test it out and know that it will act the same as installing it from the registry when it's published.

For example, with Rollpkg:

git clone git@github.com:rafgraph/rollpkg.git
mkdir test && cd test
npm init --yes
# compare package-lock.json after running the two commands
npm i --save-dev ../rollpkg
# vs
npm i --save-dev rollpkg

Note that Rollpkg, a zero config build tool to create packages with Rollup and TypeScript, has Jest and ESLint as dependencies (and provides default configs for these), but let's the user call jest and eslint instead of wrapping those commands, which relies on those bins being in the top level of node_modules. The advantage of not wrapping the jest and eslint commands is that if the user needs to use a specific version of Jest or ESlint all they have to do is install that version and it will override the version provided by Rollpkg.

[ljharb]

@rafgraph for that, jest and eslint should be peer deps - and linking with peer deps means you have to link the peer deps also.

[rafgraph]

Not exactly peer deps, it's a default with optional override. I know this isn't a standard use case, but regardless a consistent package-lock.json would be nice to have.

[darcyclarke]

@garciaalvaro if the path is set within the root directory we will resolve the dependencies in v7 but, if as you've shown, the path is outside the fs root then we treat it like a linked dep and won't resolve those deps. We understand that this is a change from v6.

[garciaalvaro]

@darcyclarke Thank you for the information.
If I understood correctly, the behaviour is intended. Is there some way to keep/emulate the previous behaviour and install those dependencies too?

[darcyclarke]

@garciaalvaro I'm going to bring this up at our next Open RFC call & see what the consensus is in terms of supporting the old behaivour; I forgot the exact reasoning behind the change but I can imagine that it would have reduced some complexity & potentially cleaned up a number of issues/edgecases. I'll circle back here after the call on Wednesday.

[alexander-schranz]

We do have the same problem im @sulu where the package.json can not be the in the parent directory as it would block the package.json which developers could maybe create to build there own application/website css/js.

@darcyclarke I did read in the meeting doc that:

we have options here in terms of making this more configurable

Is this a suggestion to add options or are here already some options to get the old behaviour back?

[jsejcksn]

Not sure if this is the same issue or if I should open a new one—please let me know.
I cannot npm link typescript in v7, but can in 6:

v7 (fail):

example % nvm ls --no-alias
       v14.15.5
->      v15.8.0

example % node --version
v15.8.0

example % npm --version
7.5.2

example % npm -g ls
/Users/jesse/.nvm/versions/node/v15.8.0/lib
├── @typescript-eslint/eslint-plugin@4.14.2
├── @typescript-eslint/parser@4.14.2
├── eslint@7.19.0
├── npm@7.5.2
└── typescript@4.1.3


example % ls -a
.	..

example % npm link typescript
npm ERR! code 127
npm ERR! path /Users/jesse/.nvm/versions/node/v15.8.0/lib/node_modules/typescript
npm ERR! command failed
npm ERR! command sh -c gulp build-eslint-rules
npm ERR! sh: gulp: command not found

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/jesse/.npm/_logs/2021-02-10T01_33_13_597Z-debug.log

debug.log

example % cat /Users/jesse/.npm/_logs/2021-02-10T01_33_13_597Z-debug.log
0 verbose cli [
0 verbose cli   '/Users/jesse/.nvm/versions/node/v15.8.0/bin/node',
0 verbose cli   '/Users/jesse/.nvm/versions/node/v15.8.0/bin/npm',
0 verbose cli   'link',
0 verbose cli   'typescript'
0 verbose cli ]
1 info using npm@7.5.2
2 info using node@v15.8.0
3 timing config:load:defaults Completed in 1ms
4 timing config:load:file:/Users/jesse/.nvm/versions/node/v15.8.0/lib/node_modules/npm/npmrc Completed in 0ms
5 timing config:load:builtin Completed in 1ms
6 timing config:load:cli Completed in 1ms
7 timing config:load:env Completed in 0ms
8 timing config:load:file:/Users/jesse/example/.npmrc Completed in 0ms
9 timing config:load:project Completed in 1ms
10 timing config:load:file:/Users/jesse/.npmrc Completed in 2ms
11 timing config:load:user Completed in 2ms
12 timing config:load:file:/Users/jesse/.nvm/versions/node/v15.8.0/etc/npmrc Completed in 0ms
13 timing config:load:global Completed in 0ms
14 timing config:load:cafile Completed in 0ms
15 timing config:load:validate Completed in 1ms
16 timing config:load:setUserAgent Completed in 1ms
17 timing config:load:setEnvs Completed in 0ms
18 timing config:load Completed in 8ms
19 verbose npm-session d489e3dfc3b55581
20 timing npm:load Completed in 15ms
21 timing arborist:ctor Completed in 1ms
22 timing arborist:ctor Completed in 0ms
23 timing arborist:ctor Completed in 0ms
24 timing idealTree:init Completed in 5ms
25 timing idealTree:userRequests Completed in 2ms
26 silly idealTree buildDeps
27 silly placeDep ROOT typescript@4.1.3 OK for:  want: file:../.nvm/versions/node/v15.8.0/lib/node_modules/typescript
28 timing idealTree:#root Completed in 4ms
29 timing idealTree:node_modules/typescript Completed in 0ms
30 timing idealTree:buildDeps Completed in 5ms
31 timing idealTree:fixDepFlags Completed in 0ms
32 timing idealTree Completed in 13ms
33 timing reify:loadTrees Completed in 13ms
34 timing reify:diffTrees Completed in 1ms
35 silly reify moves {}
36 timing reify:retireShallow Completed in 1ms
37 timing reify:createSparse Completed in 1ms
38 timing reify:loadBundles Completed in 0ms
39 timing reifyNode:node_modules/typescript Completed in 34ms
40 timing reify:unpack Completed in 34ms
41 timing reify:unretire Completed in 0ms
42 timing build:queue Completed in 0ms
43 timing build:deps Completed in 0ms
44 timing build:queue Completed in 0ms
45 info run typescript@4.1.3 prepare ../.nvm/versions/node/v15.8.0/lib/node_modules/typescript gulp build-eslint-rules
46 info run typescript@4.1.3 prepare { code: 127, signal: null }
47 timing reify:rollback:createSparse Completed in 1ms
48 timing reify:rollback:retireShallow Completed in 0ms
49 timing command:link Completed in 89ms
50 verbose stack Error: command failed
50 verbose stack     at ChildProcess.<anonymous> (/Users/jesse/.nvm/versions/node/v15.8.0/lib/node_modules/npm/node_modules/@npmcli/promise-spawn/index.js:64:27)
50 verbose stack     at ChildProcess.emit (node:events:378:20)
50 verbose stack     at maybeClose (node:internal/child_process:1067:16)
50 verbose stack     at Socket.<anonymous> (node:internal/child_process:453:11)
50 verbose stack     at Socket.emit (node:events:378:20)
50 verbose stack     at Pipe.<anonymous> (node:net:665:12)
51 verbose pkgid typescript@4.1.3
52 verbose cwd /Users/jesse/example
53 verbose Darwin 20.3.0
54 verbose argv "/Users/jesse/.nvm/versions/node/v15.8.0/bin/node" "/Users/jesse/.nvm/versions/node/v15.8.0/bin/npm" "link" "typescript"
55 verbose node v15.8.0
56 verbose npm  v7.5.2
57 error code 127
58 error path /Users/jesse/.nvm/versions/node/v15.8.0/lib/node_modules/typescript
59 error command failed
60 error command sh -c gulp build-eslint-rules
61 error sh: gulp: command not found
62 verbose exit 127

v6 (success):

example % nvm use default
Now using node v14.15.5 (npm v6.14.11)

example % node --version
v14.15.5

example % npm --version
6.14.11

example % npm link typescript
/Users/jesse/example/node_modules/typescript -> /Users/jesse/.nvm/versions/node/v14.15.5/lib/node_modules/typescript

[m90]

I just tried updating to 8.8.0 in a setup that was locked to npm 6 until now and I think I am running into a similar issue as @alexander-schranz where npm 6 was exposing a behavior different from what npm 8.8.0 is doing, even when using install-links=true

Considering a setup as described in this issue where i have packages app1 and app2 that install a local package package via file:./../package, with npm 6 I could:

• edit contents of package
• see changes reflected in app1 and app2 on rebuild

With npm 8.8.0 I need to:

• edit contents of package
• nuke node_modules of app1 and app2 (not sure why it's needed but I couldn't get it to work without)
• run npm i for both app1 and app2
• trigger a rebuild to see changes reflected

[wyckster]

Hello again, @nlf, I wasn't able to get correct behaviour with npm 8.8.0. (Refer to my comment in which I outlined a repro)

Perhaps I don't understand what I need to do differently, but this does not work:

/# sudo docker run --rm -it node:18 /bin/bash
/# git clone https://github.com/wyckster/npm-deps.git
/# cd npm-deps/main
/# npm --version
8.8.0
/# npm install
/# node app.js
node:internal/modules/cjs/loader:942
  throw err;
  ^

Error: Cannot find module '@modules/something'
Require stack:
- /npm-deps/modules/problem/index.js
- /npm-deps/main/app.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:939:15)
    at Module._load (node:internal/modules/cjs/loader:780:27)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/npm-deps/modules/problem/index.js:1:25)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Module._load (node:internal/modules/cjs/loader:827:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/npm-deps/modules/problem/index.js', '/npm-deps/main/app.js' ]
}

Node.js v18.1.0

I also tried this, which also does not work

/# sudo docker run --rm -it node:18 /bin/bash
/# git clone https://github.com/wyckster/npm-deps.git
/# cd npm-deps/main
/# npm --version
8.8.0
/# npm install --install-links
npm ERR! code ENOENT
npm ERR! syscall open
npm ERR! path /npm-deps/main/node_modules/@modules/something/package.json
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, open '/npm-deps/main/node_modules/@modules/something/package.json'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2022-05-04T16_01_23_231Z-debug-0.log

So, can you please explain how I can take advantage of the fix you mentioned? What do I have to do differently?

Currently my workaround is still:

npm install -g npm@6

[nmccready]

At this point I agree with others npm 6 should be forked and start over and or abandon npm and go to pnpm.

[siyb]

Does apache/cordova-electron#229 fix this issue as well and we can we expect a fix to hit the shelves?

[wyckster]

@nlf Can you help me figure out how to take advantage of this fix, please? See my failed attempt to demonstrate that the issue is fixed from May 4, 2022 and my repro of the problem from Aug 24, 2021

[fabb]

All that said, we will consider making --install-links the default behavior in the next major version (v9) which you should start to see release candidates for in the coming weeks/months.

@darcyclarke --install-links sounds cool, if it wasn't for this bug 🙈: #4863

[wyckster]

@nlf To me it seems like this bug is not fixed. I've demonstrated my failed attempt to take advantage of this fix. Can you please tell me what I did wrong or why I cannot install? It's a nice self-contained repro, so hopefully someone can show me how to make this --install-links work correctly.

[jasminen]

@nlf The most upsetting feature here is actually that npm link now REMOVES linked pkg deps that are already installed in the parent's node_modules, forcing me to explicitly add those peerDeps to the parent's package.json if i need to use them in my parent project.
Can this be somehow fixed/reverted/added as a feature?

[drik98]

Facing the same issues as @jasminen

[mryellow]

I'm still finding npm link to be an absolute pain to work with. Hours spent chasing non-issues.

[shamik-fw]

Still facing issues with --install-link it seems to make my current locally linked package work but it then messes up with my other locally referenced ( not linked ) components/packages which are present in my parent's project directory. And also npm link is really a pain to work with as mentioned by @jasminen we need to install all local's peerDeps in parent.

Maybe a solution to it could be somehow making my linked package refer to its own node_modules for its deps rather than searching for those in parent's.

[wyckster]

npm is still broken as far as I'm concerned.

For what it's worth, I had to switch to using yarn to achieve what I want. The package.json remains the same, but I use yarn install in place of npm install because it handles the linked dependencies of local dependencies according to my expectations.

[alexander-schranz]

@wyckster interesting I failed with yarn also what version are you using of yarn?

[wyckster]

@alexander-schranz

@wyckster interesting I failed with yarn also what version are you using of yarn?

Yarn will correctly install the example in my repro. I just confirmed this to be true with the node:19 docker image: node v19.9.0 and yarn 1.22.19.

Whereas npm v9.6.3 (which comes with that same docker image) still fails to install local dependencies of local dependencies correctly.

[alexander-schranz]

I still run into issues with npm >= 7, pnpm and yarn.

But today I tried bun from @oven-sh where example @Jarred-Sumner works on. They released 1.0 and seems correclty handle also dependencies of local dependencies and install them. In my case i was able to build our complex setup with it, which was currently only possible with npm@6. While we still use some kind of symlinks which simulates child paths so not sure if it will work in all cases. In my case bun run preinstall && bun install && bun run build did work.

[caramboleyo]

yarn 1.22.21 makes a full copy (not a symlink) of the local package into the node_modules folder and installs all deps properly into the projects node_modules folder.

This was the last straw that made me switch to Deno two years ago, still not fixed.
Well done.

A locally linked package should NOT be treated any different than a package installed from a remote source. That means its deps need to be installed into the projects node_modules folder and nowhere else.

[brianbraunstein]

I thought this issue was causing problems in my scenario, but then I realized the npm behavior makes sense, given the nodejs module resolution behavior.

I think my scenario is pretty common so I'll elaborate in case it helps others.

Scenario

my_repo
├── a
├── b
├── c

• a depends on b
• b depends on c
• a and b are not published externally, they're only used within the context of my_repo.
• c could be published externally (or it could also be a remote package, I just made it local here because it was easy for testing, this is irrelevant though).
• Key requirements: fast refresh/hot reload/etc features (like nextjs fast refresh), must work for both packages (even if running within a local docker container, which requires some volume mounts/etc).

Relevant Nodejs module resolution behavior.

The key is the phrase "real path" in the following: https://nodejs.org/api/esm.html#resolution-algorithm-specification section ESM_RESOLVE.7.4 which says:

• "Set resolved to the real path of resolved, maintaining the same URL querystring and fragment components".

So this means that symlinks are resolved before looking for node_modules directories.

What this means is the following works:

my_repo
├── a
│   └── node_modules
│       └── b -> ../../b
├── b
│   └── node_modules
│       └── c -> ../../c/
└── c

but the following does NOT work:

my_repo
├── a
│   ├── a.mjs
│   └── node_modules
│       ├── b -> ../../b
│       └── c -> ../../c/
├── b
│   └── b.mjs
└── c
    └── c.mjs

Note that this has nothing to do with npm, just nodejs.

That means if you want to symlink from a to b to support "fast refresh", you're also stuck making sure that b has its node_modules setup correctly (i.e. npm install has been run within b).

Thoughts

One could argue that cd a; npm install should setup a/node_modules and a/package-lock.json to look the same as if b was a remote package, I thought this at first too. However, if you say --install-links=false, you're specifically requesting a symlink, which means npm knows nodejs will use b/node_modules when doing lookups for b dependencies.

Setting up a/package-lock.json to include b dependencies would then be odd and problematic. Imagine a2 is just like a, it also depends on b.

• You then have 3 different package-lock.json files that all cover b dependencies, and they could potentially conflict, meanwhile you only have 1 node_modules folder containing the dependencies that will actually get used (the one inside of b).
• You're working on a, you tweak b, all is good, "fast refresh" is working. Then 4 days later you go to work on a2 and you forgot you tweaked b and changed its dependencies. The symlink a2/node_modules/b -> ../../b is still there, so the code works fine, utilizing the forgotten tweak and dep change, but your a2/package-lock.json is now incorrect. Because a2 is working via the symlink, you may never remember to update the package-lock.json, you might even check it into your repo.

Given this, I can see why npm behaves like it does when requesting symlinking.