[BUG] regression - npm_config_prefix no longer works with global npx

[rwoll]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

Using the npm_config_prefix results in a no such file or directory error when attempting to run a command via npx globally on npm@8.16.0. The same command used to work on npm@8.15.1.

Expected Behavior

Global npx command works in conjunction with npm_config_prefix outside of a project/workspace.

Steps To Reproduce

$ mkdir repro
$ cd repro
$ npm i -g npm@8.15.1
$ npm_config_prefix=/tmp/.foo npx serve@latest --help
# observe `serve` CLI is executed successfully and the help message is printed

Now, upgrade npm and repeat:

$ npm i -g npm@8.16.0
$ npm_config_prefix=/tmp/.foo npx serve@latest --help
npm ERR! code ENOENT
npm ERR! syscall lstat
npm ERR! path /private/tmp/.foo/lib
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, lstat '/private/tmp/.foo/lib'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! A complete log of this run can be found in:
npm ERR!     /…/.npm/_logs/2022-08-05T21_22_57_389Z-debug-0.log

💥 Observe npx errors and complains This is related to npm not being able to find a file..

Environment

• npm:
  • GOOD: 8.15.1
  • BAD: 8.16.0
• Node.js: v16.14.0
• OS Name: macOS, Ubuntu 22
• System Model Name: M1 Macbook Pro, GitHub Action Runner
• npm config:

$ npm config ls
; node bin location = /Users/…/.nvm/versions/node/v16.14.0/bin/node
; node version = v16.14.0
; npm local prefix = /Users/…/Desktop/repro
; npm version = 8.16.0
; cwd = /Users/…/Desktop/repro
; HOME = /Users/…
; Run `npm config ls -l` to show all defaults.

Additionally, npm_config_prefix is set when running npx command.

[rwoll]

Relates microsoft/playwright#16281. It looks like this is not particular to serve package, as it was originally discovered in test suite of playwright package.

[wraithgar]

$npm_config_prefix is very buggy and sets both the global and local prefix when used. There's another issue that already digs into it, and it's not something that is going to be easy to fix.

The reason this is happening now is because npx/npm exec were not even looking in the global space before for installed packages, but they are now, meaning they are also susceptible to this overarching bug.

I would recommend NOT using $npm_config_prefix, even if this error specifically is fixed. It does highly unexpected things under the hood. Please use --prefix so that your expected behavior is executed.

[rwoll]

Thanks! For some additional context, Playwright discourage's using itself as a globally installed package. Instead, we ask users to explicitly depend on the package and invoke it via npx within their project. This ensures there's no confusion about which version of Playwright they are getting and which binary deps (browsers) they are getting.

Playwright prints a warning if it thinks its being used in the global context. We have tests that check this message is present under some cases and absent in others. (i.e. In CI, we are testing the Playwright CLI itself.) The tests all set npm_config_prefix, npm_config_cache, npm_config_registry so they are (ideally) hermetic.

In this case, I don't think --prefix will be appropriate. Do you have any other suggestions based on the additional context?

My alternative ideas are to:

1. Skip setting the above variables, run tests sequentially, and nuke the global install directory (and cache) between tests
2. Run each test in a Docker container (but this means we lose testing the behavior cross-platform)
3. Look to see if nvm can provide an isolated npm env per test

Thanks for your time!

[ljharb]

You can make an npmrc file that sets prefix?

[rwoll]

@ljharb It looks like that gives the same error as it's equivalent to the env var:

$ npm config --global set prefix /tmp/bar
$ npx serve
npm ERR! code ENOENT
npm ERR! syscall lstat
npm ERR! path /private/tmp/bar
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, lstat '/private/tmp/bar'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/…/.npm/_logs/2022-08-09T04_50_53_983Z-debug-0.log

Even doing:

$ npx --prefix=/tmp/bar serve

produces the same error (with fresh install/config). Given the above, it looks like with 8.16.0 you can no longer use a non-default global install directory.

[wraithgar]

There are two things happening here. First, the global prefix has a completely different layout than the local one, the second is that setting --prefix changes both the local and global prefix. The reasons for this are largely lost to time, and untangling that is already on our radar.

What this functionally means is that when you set --prefix, any global scope is ignored. npx/npm exec need to follow suit.

#5291 fixes this specific case, leaving the --prefix untangling for a future date.

[rwoll]

Thanks so much @wraithgar! Appreciate the clarifications and the workaround/fix.