[BUG] EPERM issues since v6.11.0 #476

rowanoulton · 2019-11-13T00:38:50Z

What / Why

We recently upgraded from npm 6.0.1 to 6.12.0 and several of our engineers are encountering persistent EPERM issues when they run npm install.

From all angles the folders in question have the right permissions set, and we've ruled out nvm as a possible reason for this failure too. No combination of uninstalling and reinstalling or cache clearing has resolved it.

Whenever they run npm install, they get output like this:

code EPERM
npm ERR! syscall spawn
npm ERR! errno EPERM
npm ERR! Error: spawn EPERM
npm ERR!     at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!     at Object.spawn (child_process.js:540:9)
npm ERR!     at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:216:24
npm ERR!     at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24
npm ERR!  OperationalError: spawn EPERM
npm ERR!     at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!     at Object.spawn (child_process.js:540:9)
npm ERR!     at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:216:24
npm ERR!     at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24 {
npm ERR!   cause: Error: spawn EPERM
npm ERR!       at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!       at Object.spawn (child_process.js:540:9)
npm ERR!       at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:216:24
npm ERR!       at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24 {
npm ERR!     errno: 'EPERM',
npm ERR!     code: 'EPERM',
npm ERR!     syscall: 'spawn'
npm ERR!   },
npm ERR!   stack: 'Error: spawn EPERM\n' +
npm ERR!     '    at ChildProcess.spawn (internal/child_process.js:394:11)\n' +
npm ERR!     '    at Object.spawn (child_process.js:540:9)\n' +
npm ERR!     '    at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:216:24\n' +
npm ERR!     '    at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24',
npm ERR!   errno: 'EPERM',
npm ERR!   code: 'EPERM',
npm ERR!   syscall: 'spawn',
npm ERR!   parent: 'webapp'
npm ERR! }
npm ERR! 
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR! 
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator.
npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/REDACTED/.npm/_logs/2019-11-12T21_45_00_580Z-debug.log

I walked back the version of npm until 6.10.3, where installs work again. The mention of pacote in the above logs seems suspicious when you compare it to the release notes for 6.11.0:

dc8f9e5 pacote@9.5.7: Infer the ownership of all unpacked files in node_modules, so that we never have user-owned files in root-owned folders, or root-owned files in user-owned folders. (@isaacs)

Note that upgrading to 6.13.0 does not fix the problem.

Also note that we have some private git+ssh urls in our package.json

Where

npm public registry

How

Current Behavior

Install fails with above error output, EPERM

Steps to Reproduce

npm i -g npm@6.11.0 && npm install

Expected Behavior

Install works as before

The text was updated successfully, but these errors were encountered:

isaacs · 2019-11-13T20:13:13Z

It looks like the issue is involving a git repo dependency that has a prepare script of some kind.

Can you share some details about your dependencies? I know that they're private git deps, but how are you referencing them? What are they doing?

rowanoulton · 2019-11-13T23:36:13Z

Hi @isaacs, thanks for the quick response.

In addition to public npm registry dependencies, we have two types of private modules:

Internal modules we publish to our Artifactory registry. These are namespaced and configured with .npmrc. They take the format of @company-name/module
Dependencies on repositories from our Github Enterprise instance, of the format git+ssh://git@company-url.com:company/repo-name.git#<full sha1 commit hash>

Does that answer your question?

doochik · 2019-11-14T14:42:13Z

I think this issue is related to #286

Since v6.11 npm calls git with the directory owner user
https://github.com/npm/pacote/blob/v9.5.8/lib/util/git.js#L216

Examples:
I'm user1 and group1. Run npm install in directory /workdir which is owned by user2 and group2. npm runs git with uid=user2 and gid=group2 and fails to start with Error: spawn EPERM. Because user1 hasn't rights to start process for user2.

Another example is for docker. I have docker container for build and mount /workdir inside. Inside container /workdir is owned by unknown user. And we have error

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://git@<private server>/admin-template.git
npm ERR! 
npm ERR! No user exists for uid 1000
npm ERR! fatal: Could not read from remote repository.

isaacs · 2019-11-14T17:11:55Z

Ok, thanks. So this isn't about prepare scripts, it's just failing to run git because the uid/gid is getting set to the owner of the folder that it's unpacking to. The private-ness of the git repo is a red herring in this case, I think, because what you're describing seems to be about spawning git in the first place, regardless of the repo you're fetching from.

I think the problem here is that it's mixing "folder owner uid" with "euid of child process" under the same uid/gid config param.

We'll need to dig into it a bit more, because if we just roll that back, we'll be back to the state of checking out root-owned files in user-owned locations. Setting an effective uid on the spawned process does make the files default to being owned by that user, which is nice, but we should be skipping that unless the current user is root.

So, it might still fail if the user1 doesn't have permission to write to the folders owned by user2, but I think that would've been a problem before 6.11 as well. I'm assuming that /workdir has a mode of 775 rather than 755?

isaacs · 2019-11-14T17:12:57Z

Note (for myself, mostly) that means you'll end up with files being owned by user2 in a folder owned by user1, but since the files checked out are owned by user2, then user2 should have permission to chown them over to user1.

CNSKnight · 2019-11-14T22:51:50Z

+1 npm 10.-12.

$ npm install --save https://github.com/CNSKnight/mifterialize-tree.git
npm ERR! code EPERM
npm ERR! syscall spawn
npm ERR! errno EPERM
npm ERR! Error: spawn EPERM
npm ERR!     at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!     at spawn (child_process.js:540:9)
npm ERR!     at execFile (child_process.js:224:17)
npm ERR!     at tryCatcher (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
npm ERR!     at ret (eval at makeNodePromisifiedEval (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/promisify.js:184:12), <anonymous>:16:23)
npm ERR!     at opts.retry.retries (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:192:14)
npm ERR!     at /home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24
npm ERR!  OperationalError: spawn EPERM
npm ERR!     at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!     at spawn (child_process.js:540:9)
npm ERR!     at execFile (child_process.js:224:17)
npm ERR!     at tryCatcher (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
npm ERR!     at ret (eval at makeNodePromisifiedEval (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/promisify.js:184:12), <anonymous>:16:23)
npm ERR!     at opts.retry.retries (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:192:14)
npm ERR!     at /home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24 {
npm ERR!   cause: Error: spawn EPERM
npm ERR!       at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!       at spawn (child_process.js:540:9)
npm ERR!       at execFile (child_process.js:224:17)
npm ERR!       at tryCatcher (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
npm ERR!       at ret (eval at makeNodePromisifiedEval (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/promisify.js:184:12), <anonymous>:16:23)
npm ERR!       at opts.retry.retries (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:192:14)
npm ERR!       at /home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24 {
npm ERR!     errno: 'EPERM',
npm ERR!     code: 'EPERM',
npm ERR!     syscall: 'spawn'
npm ERR!   },
npm ERR!   stack: 'Error: spawn EPERM\n' +
npm ERR!     '    at ChildProcess.spawn (internal/child_process.js:394:11)\n' +
npm ERR!     '    at spawn (child_process.js:540:9)\n' +
npm ERR!     '    at execFile (child_process.js:224:17)\n' +
npm ERR!     '    at tryCatcher (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)\n' +
npm ERR!     '    at ret (eval at makeNodePromisifiedEval (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/promisify.js:184:12), <anonymous>:16:23)\n' +
npm ERR!     '    at opts.retry.retries (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:192:14)\n' +
npm ERR!     '    at /home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24',
npm ERR!   errno: 'EPERM',
npm ERR!   code: 'EPERM',
npm ERR!   syscall: 'spawn',
npm ERR!   parent: 'ANCILLARIES'
npm ERR! }

doochik · 2019-11-15T08:28:49Z

I'm assuming that /workdir has a mode of 775 rather than 755?

Yes, 775. I have write access to workdir.
In docker I’m root and have access to whole fs

bertrama · 2019-11-18T12:25:36Z

I believe this is also related to issue #296 .

sppatel · 2019-11-21T12:15:29Z

I am seeing similar issues too. What is strange is that if I run npm install w/ the package locks removed, npm install succeeds with 6.12.0. At that point I will have fresh package-lock files. I then wipe node_modules and re-run npm install and it fails with the EPERM error.

If I roll back npm versions v10.16.2 (the version I was on previously) - no issues.

sppatel · 2019-11-21T12:52:07Z

Two questions on the above comment that I don't quite understand.

(1) Why is this not seen on an npm install w/o a package lock file.
(2) Why is this not more widespread and seen consistently between users? I could not get others to repro.

SebCorbin · 2019-11-28T11:17:49Z

I'm encountering this issue too in a directory created by docker (I think, it's been a while).

I've fixed that by changing the group of the directory to my default user group, all runs fine now

bertrama · 2019-12-02T19:09:59Z

One example way to reproduce:

Installing a dependency from a git repository
Running in linux or osx.
The package directory’s group is not the user’s default group.
The package directory has the setgid bit turned on (i.e. chmod g+s or 2775) such that the not-default-group is preserved when npm install creates new directories.

In my ubuntu environment, my default username/group might be bertrama/bertrama and I’m a member of a group cdrom. Steps for me to reproduce are:

mkdir -m 2775 eperm-test
chgrp cdrom eperm-test
cd eperm-test
npm init -y
npm add left-pad@git://github.com/left-pad/left-pad

I haven't tested, but you may be able to substitute chgrp cdrom eperm-test with chgrp "$(groups | awk '{print $2}')" eperm-test.

Fix npm/cli#476

@isaacs

Fix: npm/cli#476 PR-URL: #23 Credit: @isaacs Close: #23 Reviewed-by: @darcyclarke

isaacs · 2019-12-04T20:14:37Z

Fixed in pacote 9.5.10, will be included in the next npm v6 release.

sppatel · 2019-12-05T23:05:22Z

Really appreciate the quick turn-around on this. Thank you. Is there any rough (and I mean rough 😄 ) estimate as to when the next release is targeted for?

isaacs · 2019-12-10T00:09:24Z

in about an hour

sppatel · 2019-12-10T11:41:05Z

Confirmed fixed! TY!

dmikalova · 2020-05-06T22:36:40Z

I'm still getting this issue on NPM 6.14.4 with a container volume mount in Jenkins. Jenkins docker runs with a user that doesn't exist on the container, but the volume mounts are owned by that user, and I end up using root for unrelated reasons. NPM then fails because the owner of the dir doesn't exist. In brief, where user 11028 doesn't exist:

docker run -t -d -u 11028:100 --user root:root -w /tmp/jenkins-ea2356b0/workspace/x -v /tmp/jenkins-ea2356b0/workspace/x:/tmp/jenkins-ea2356b0/workspace/x:rw,z x
...
npm install
...
npm ERR! No user exists for uid 11028

NPM is the only thing that ends up having this behavior. Copying the dir to a temp dir and running let's me continue with the build.

darcyclarke added Bug thing that needs fixing Epic a larger scope of work that will be tracked in a Project Board labels Nov 19, 2019

darcyclarke added this to To do in Community & Open Source via automation Nov 19, 2019

mikemimik added the Release 6.x work is associated with a specific npm 6 release label Nov 26, 2019

mikemimik added this to the Release 6.14.0 milestone Nov 26, 2019

mikemimik added the Priority Backlog a "backlogged" item that will be tracked in a Project Board label Dec 3, 2019

mikemimik removed this from the Release 6.13.2 milestone Dec 3, 2019

darcyclarke assigned isaacs and darcyclarke Dec 4, 2019

darcyclarke removed Priority Backlog a "backlogged" item that will be tracked in a Project Board Epic a larger scope of work that will be tracked in a Project Board labels Dec 4, 2019

isaacs added a commit to npm/pacote that referenced this issue Dec 4, 2019

fix: Do not drop perms in git when not root

5d8196b

Fix npm/cli#476

isaacs mentioned this issue Dec 4, 2019

fix: Do not drop perms in git when not root npm/pacote#23

Closed

isaacs added a commit to npm/pacote that referenced this issue Dec 4, 2019

fix: Do not drop perms in git when not root

5f33040

Fix: npm/cli#476 PR-URL: #23 Credit: @isaacs Close: #23 Reviewed-by: @darcyclarke

isaacs closed this as completed in npm/pacote@bad55cd Dec 4, 2019

isaacs mentioned this issue Dec 10, 2019

deps: upgrade npm to 6.13.3 nodejs/node#30873

Closed

jgehrcke mentioned this issue Dec 10, 2019

[BUG] EPERM during npm install (breakage between npm 6.9 and 6.13) #578

Closed

This was referenced Dec 18, 2019

[Snyk] Upgrade npm from 6.2.0 to 6.13.3 restinpeaceinout/atom#84

Open

[Snyk] Upgrade npm from 6.2.0 to 6.13.4 restinpeaceinout/atom#87

Open

doochik mentioned this issue Dec 23, 2019

[BUG] Failed to install npm package from git in docker since v6.11.0 #624

Closed

jmyersmsft mentioned this issue Jan 6, 2020

Npm Task fail to install packages microsoft/azure-pipelines-tasks#11811

Closed

snyk-bot mentioned this issue Jan 7, 2020

[Snyk] Upgrade npm from 6.10.3 to 6.13.4 srdjan-19/rent-a-car-service#4

Open

This was referenced Jan 17, 2020

[Snyk] Upgrade npm from 6.8.0 to 6.13.4 silver-xu/yame-ui#8

Open

[Snyk] Upgrade npm from 6.2.0 to 6.13.6 restinpeaceinout/atom#102

Open

snyk-bot mentioned this issue Feb 11, 2020

[Snyk] Upgrade npm from 6.13.1 to 6.13.6 codeztech-atique/Meanstack-Crud-Frontend-Application#4

Open

This was referenced Feb 25, 2020

[Snyk] Upgrade npm from 6.4.1 to 6.13.7 ali8889/etherwallet#10

Open

[Snyk] Upgrade npm from 6.2.0 to 6.13.7 restinpeaceinout/atom#124

Open

This was referenced Mar 10, 2020

[Snyk] Upgrade npm from 6.11.2 to 6.14.2 Hypnos3/RedMatic#4

Open

[Snyk] Upgrade npm from 6.2.0 to 6.13.7 xalpinistx/ExpressProject#6

Merged

snyk-bot mentioned this issue Apr 22, 2020

[Snyk] Upgrade npm from 6.10.1 to 6.14.4 aminatakonate000/Graviton-App#3

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] EPERM issues since v6.11.0 #476

[BUG] EPERM issues since v6.11.0 #476

rowanoulton commented Nov 13, 2019 •

edited

isaacs commented Nov 13, 2019

rowanoulton commented Nov 13, 2019

doochik commented Nov 14, 2019

isaacs commented Nov 14, 2019

isaacs commented Nov 14, 2019

CNSKnight commented Nov 14, 2019

doochik commented Nov 15, 2019 •

edited

bertrama commented Nov 18, 2019

sppatel commented Nov 21, 2019

sppatel commented Nov 21, 2019 •

edited

SebCorbin commented Nov 28, 2019 •

edited

bertrama commented Dec 2, 2019

isaacs commented Dec 4, 2019

sppatel commented Dec 5, 2019

isaacs commented Dec 10, 2019

sppatel commented Dec 10, 2019

dmikalova commented May 6, 2020 •

edited

[BUG] EPERM issues since v6.11.0 #476

[BUG] EPERM issues since v6.11.0 #476

Comments

rowanoulton commented Nov 13, 2019 • edited

What / Why

Where

How

Current Behavior

Steps to Reproduce

Expected Behavior

isaacs commented Nov 13, 2019

rowanoulton commented Nov 13, 2019

doochik commented Nov 14, 2019

isaacs commented Nov 14, 2019

isaacs commented Nov 14, 2019

CNSKnight commented Nov 14, 2019

doochik commented Nov 15, 2019 • edited

bertrama commented Nov 18, 2019

sppatel commented Nov 21, 2019

sppatel commented Nov 21, 2019 • edited

SebCorbin commented Nov 28, 2019 • edited

bertrama commented Dec 2, 2019

isaacs commented Dec 4, 2019

sppatel commented Dec 5, 2019

isaacs commented Dec 10, 2019

sppatel commented Dec 10, 2019

dmikalova commented May 6, 2020 • edited

rowanoulton commented Nov 13, 2019 •

edited

doochik commented Nov 15, 2019 •

edited

sppatel commented Nov 21, 2019 •

edited

SebCorbin commented Nov 28, 2019 •

edited

dmikalova commented May 6, 2020 •

edited