[BUG] Failed to install npm package from git in docker since v6.11.0

[doochik]

What / Why

This bug is related to #476 (comment) and #514

When

npm install packages from git in docker

Where

• packages from public git repositories

How

Current Behavior

# npm i
npm ERR! code 128
npm ERR! Command failed: git clone --depth=1 -q -b v2.0.5 git://github.com/boblauer/MockDate.git /root/.npm/_cacache/tmp/git-clone-c9699b7f
npm ERR! fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-c9699b7f': Permission denied
npm ERR! 

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-12-23T09_49_05_990Z-debug.log

Steps to Reproduce

• Create simple package.json with git dependency

{
  "name": "test-npm",
  "dependencies": {
    "mockdate": "git://github.com/boblauer/MockDate.git#v2.0.5"
  }
}

• run docker run --network=host -it -v $(pwd):/var/work/test -w /var/work/test node:12.14.0 /bin/bash
• ls -l shows that files are owned by unknown user and unknown group (this is typical behaviour on build agents like teamcity, teamcity clones git repo and mounts it inside docker container)

# ls -l
-rw-r--r-- 1 20857 115755  114 Dec 23 09:46 package.json

• run npm install

# Depends on environment you can get

npm ERR! code 128
npm ERR! Command failed: git clone --depth=1 -q -b v2.0.5 git://github.com/boblauer/MockDate.git /root/.npm/_cacache/tmp/git-clone-c9699b7f
npm ERR! fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-c9699b7f': Permission denied
npm ERR! 

# or

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://git@<private server>/admin-template.git
npm ERR! 
npm ERR! No user exists for uid 1000
npm ERR! fatal: Could not read from remote repository.

• run npm i -g npm@6.10 (downgrade to npm 6.10.X)
• run npm install

# all works well!

updated 1 package and audited 1 package in 1.355s
found 0 vulnerabilities

Expected Behavior

• npm install / npm ci works well like npm@6.10

[SabatierBoris]

I have the same issue with npm 6.13.4.
It's working with npm 6.4.1

[doochik]

@isaacs please, take a look on this

[isaacs]

Is the npm command being run as root, or some other user?

I'd like to try to zoom in on the problem and leave the docker aspect out of it.

If I understand this correctly:

• The user running npm install is root
• The $HOME directory is /root or some other root owned directory
• Thus the cache folder in $HOME/.npm is root-owned.
• But pacote is executing a git command based on the user of the ultimate target of the extraction, not the cwd where the git action is taking place, and it fails.

Simpler reproduction, sans docker:

$ sudo su
# echo $HOME
/var/root
# npm i git://github.com/isaacs/abbrev-js
npm ERR! code 128
npm ERR! Command failed: git clone --mirror -q git://github.com/isaacs/abbrev-js.git /var/root/.npm/_cacache/tmp/git-clone-35b93572/.git
npm ERR! fatal: could not create leading directories of '/var/root/.npm/_cacache/tmp/git-clone-35b93572/.git'
npm ERR!

npm ERR! A complete log of this run can be found in:
npm ERR!     /var/root/.npm/_logs/2019-12-30T20_15_05_627Z-debug.log

I will take a look today and see how hard this'll be to fix.

[isaacs]

Note for myself and @npm/cli-team: this problem does not occur with pacote v10, so it's strictly a pacote v9 issue.

To verify this, I applied this diff to pacote:

diff --git a/lib/git.js b/lib/git.js
index ade670b..a79a0e1 100644
--- a/lib/git.js
+++ b/lib/git.js
@@ -36,6 +36,7 @@ class GitFetcher extends Fetcher {
   constructor (spec, opts) {
     super(spec, opts)
     this.resolvedRef = null
+    this.spec.hosted = null
     if (this.spec.hosted)
       this.from = this.spec.hosted.shortcut({ noCommittish: false })
 

(This is required because otherwise pacote v10 will try to fetch github repos from their static tarball endpoint, as that's usually much faster.)

Then ran:

# pacote extract git+ssh://git@github.com/isaacs/abbrev-js abbrev-from-git
{
  resolved: 'git+ssh://git@github.com/isaacs/abbrev-js#b8f3a2fc0c3bb8ffd8b0d0072cc6b5a3667e963c',
  integrity: 'sha512-NFzZKD4tPyRbi5lxIbIhczbuAaIlTCMCwR9GGG9dqptIOM5G7e9w80NOInG5Ny6yUS/QcCp0XHxVPOKrItUUIg==',
  from: 'git+ssh://git@github.com/isaacs/abbrev-js.git'
}
# ls -laF abbrev-from-git/
total 32
drwxr-xr-x    6 isaacs  staff   192 Dec 30 12:22 ./
drwxr-xr-x  133 isaacs  staff  4256 Dec 30 12:21 ../
-rw-r--r--    1 isaacs  staff  2011 Oct 26  1985 LICENSE
-rw-r--r--    1 isaacs  staff   499 Oct 26  1985 README.md
-rw-r--r--    1 isaacs  staff  1763 Oct 26  1985 abbrev.js
-rw-r--r--    1 isaacs  staff   529 Oct 26  1985 package.json

[isaacs]

Looks like npm/pacote#26 will fix this problem, assuming that I am understanding the root cause. (At the very least, it fixes a problem, which is that running npm install of a git dep as a root user with a root-owned npm cache will not fail, and it seems like that's the thing happening here.)

[doochik]

Yes, npm install is being run as root.

Thanks for the fix!

[simonkotwicz]

@isaacs Installing npm packages from git private repositories in docker still fails (since v6.11.0 and still fails with 6.13.6) when using a ~/.netrc file or ~/.git-credentials file

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t https://github.com/some-private-repo
npm ERR!
npm ERR! remote: Password authentication is not available for Git operations.
npm ERR! remote: You must use a personal access token or SSH key.
npm ERR! remote: See https://github.com/settings/tokens or https://github.com/settings/ssh
npm ERR! fatal: unable to access 'https://github.com/some-private-repo/': The requested URL returned error: 403
npm ERR!
npm ERR! exited with error code: 128

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-02-19T02_07_01_401Z-debug.log

[isaacs]

@simonkotwicz That's a different issue. Can you please post a new issue describing the situation, ideally with a more fleshed-out reproduction case?

[simonkotwicz]

opened #950

[augnustin]

Using npm in a docker, the issue still occurs to me.

With:

npm install
npm ERR! code 128
npm ERR! command failed
npm ERR! command git clone https://repo:token@my-own-gitlab.com/ui/myy-repo.git /root/.npm/_cacache/tmp/git-clone-4af321f1 --recurse-submodules
npm ERR! fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-4af321f1': Permission denied

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-03-23T10_34_17_446Z-debug.log

npm --version
7.6.3

node --version
v15.12.0

Any idea?

[kowalk]

same here ;/

[kowalk]

@augnustin after downgrade to version 6.10 all is working fine...

[yuiseki]

I'm facing same problem at npm@v7.7.0 on docker

[Coding-Kiwi]

I also face this issue with node 7 (every subversion)
Works for me on 6.14.13

npm i -g npm@6.14.13

[dmercertaylor]

I ran into this issue on several subversions, including 7.13.0, on WSL Debian. Interestingly, the project was installed in /home/<non-root user> a user, and running npm i as that user works fine, only root seems to have this problem. Install as root also works fine if the project is installed outside of /home. Honestly, I assumed this was a user error with permissions in my case, I'm surprised to see other people get this.

[MrJuliuss]

Hi, same issue with Npm 7. It works fine with 6.14.*

[ljharb]

A fresh issue describing the failure on npm 7 would be helpful.

[codegod100]

broken for me on 7

[MrJuliuss]

I tried something : use node user instead of root user, and it works.

[molinto]

Broken for me too on:
node: v14.17.3
npm: v6.14.13

[mrzealot]

Broken for me as well (with a package having github dependencies).
node: v16.5.0
npm: v7.20.0

Any info on a fix?

[nettybun]

None of these install methods work:

❯ npm i -D "github:antfu/esbuild-node-loader#0.2.0"
❯ npm i -D "git+https://github.com/antfu/esbuild-node-loader#0.2.0" 
❯ npm i -D "git+https://github.com/antfu/esbuild-node-loader.git#0.2.0"
❯ npm i -D "git://github.com/antfu/esbuild-node-loader#0.2.0"
❯ npm i -D "https://github.com/antfu/esbuild-node-loader.git#0.2.0"

They all throw:

npm ERR! code 128
npm ERR! command failed
npm ERR! command git --no-replace-objects clone --mirror -q ssh://git@github.com/antfu/esbuild-node-loader.git /home/today/.npm/_cacache/tmp/git-clone-e55e766c/.git
npm ERR! fatal: destination path '/home/today/.npm/_cacache/tmp/git-clone-e55e766c/.git' already exists and is not an empty directory.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/today/.npm/_logs/2021-08-08T01_12_10_223Z-debug.log

I have rm -rf the _cacache/tmp/ directory but it doesn't help.

[androidfans]

same here

[eisenyang]

I run into this error, Then I tried yarn, it worked.

[goncalves-diogo]

Hi, I've also come across the same error, If you look deeper into the logs you can see that the issue lies on the _cacache directory. NPM doesn't have permission to write into it. In my case, It was trying to write to the folder on /root/.npm/_cacache which isn't really a good idea because it's root.
That being I think there are two possible temporary solutions to this problem:

1. Give write permission give something like sudo chown -R 1000:1000 /current/cache/folder (If this folder is root, don't do it)
2. Change the npm cache directory npm config set cache ~/random/place --global. (NOTE: It might be required to do step 1 with the new repository).

[eMPee584]

2. Change the npm cache directory `npm config set cache ~/random/place --global`. (NOTE: It might be required to do step 1 with the new repository).

Good idea, but with /tmp/npm.cache at 777 permissions, it will still throw the same problem..

npm ERR! Cloning into '/tmp/npm.cache/_cacache/tmp/git-clone-596d9ec4'...
npm ERR! /tmp/npm.cache/_cacache/tmp/git-clone-596d9ec4/.git: Permission denied

Also, because I was running under sudo, I tried prepending env -i to not forward SUDO_UID variable, but that doesn't fix it either.

[likemusic]

0 verbose cli [ '/usr/local/bin/node', '/usr/local/bin/npm', 'install' ]
1 info using npm@8.1.2
2 info using node@v17.0.1
3 timing npm:load:whichnode Completed in 1ms
4 timing config:load:defaults Completed in 5ms
5 timing config:load:file:/usr/local/lib/node_modules/npm/npmrc Completed in 4ms
6 timing config:load:builtin Completed in 5ms
7 timing config:load:cli Completed in 3ms
8 timing config:load:env Completed in 0ms
9 timing config:load:file:/vue2-datepicker/.npmrc Completed in 1ms
10 timing config:load:project Completed in 3ms
11 timing config:load:file:/root/.npmrc Completed in 1ms
12 timing config:load:user Completed in 1ms
13 timing config:load:file:/usr/local/etc/npmrc Completed in 2ms
14 timing config:load:global Completed in 7ms
15 timing config:load:validate Completed in 1ms
16 timing config:load:credentials Completed in 3ms
17 timing config:load:setEnvs Completed in 3ms
18 timing config:load Completed in 32ms
19 timing npm:load:configload Completed in 32ms
20 timing npm:load:setTitle Completed in 0ms
21 timing npm:load:setupLog Completed in 2ms
22 timing config:load:flatten Completed in 10ms
23 timing npm:load:cleanupLog Completed in 6ms
24 timing npm:load:configScope Completed in 0ms
25 timing npm:load:projectScope Completed in 1ms
26 timing npm:load Completed in 56ms
27 timing arborist:ctor Completed in 2ms
28 timing idealTree:init Completed in 3386ms
29 timing idealTree:userRequests Completed in 0ms
30 silly idealTree buildDeps
31 silly fetch manifest date-parse-multiformats@git+https://git@github.com/likemusic/date-parse-multiformats.git
32 timing arborist:ctor Completed in 1ms
33 silly placeDep ROOT date-parse-multiformats@ OK for: vue2-datepicker@3.10.2 want: git+https://git@github.com/likemusic/date-parse-multiformats.git
34 timing idealTree:#root Completed in 3131ms
35 timing idealTree:node_modules/date-parse-multiformats Completed in 0ms
36 timing idealTree:buildDeps Completed in 3135ms
37 timing idealTree:fixDepFlags Completed in 39ms
38 timing idealTree Completed in 6565ms
39 timing command:install Completed in 6611ms
40 verbose stack Error: An unknown git error occurred
40 verbose stack     at makeError (/usr/local/lib/node_modules/npm/node_modules/@npmcli/git/lib/make-error.js:28:13)
40 verbose stack     at /usr/local/lib/node_modules/npm/node_modules/@npmcli/git/lib/spawn.js:36:26
40 verbose stack     at async Object.withTempDir (/usr/local/lib/node_modules/npm/node_modules/@npmcli/fs/lib/with-temp-dir.js:23:14)
40 verbose stack     at async Arborist.[nodeFromEdge] (/usr/local/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:1061:19)
40 verbose stack     at async Arborist.[buildDepStep] (/usr/local/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:930:11)
40 verbose stack     at async Arborist.buildIdealTree (/usr/local/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:216:7)
40 verbose stack     at async Promise.all (index 1)
40 verbose stack     at async Arborist.reify (/usr/local/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:149:5)
40 verbose stack     at async Install.install (/usr/local/lib/node_modules/npm/lib/install.js:170:5)
41 verbose cwd /vue2-datepicker
42 verbose Linux 5.10.16.3-microsoft-standard-WSL2
43 verbose argv "/usr/local/bin/node" "/usr/local/bin/npm" "install"
44 verbose node v17.0.1
45 verbose npm  v8.1.2
46 error code 128
47 error An unknown git error occurred
48 error command git --no-replace-objects clone https://git@github.com/likemusic/date-parse-multiformats.git /root/.npm/_cacache/tmp/git-cloneMplQnl --recurse-submodules --depth=1
49 error fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-cloneMplQnl': Permission denied
50 verbose exit 128

If run manually run git --no-replace-objects clone https://git@github.com/likemusic/date-parse-multiformats.git /root/.npm/_cacache/tmp/git-cloneMplQnl --recurse-submodules --depth=1 it works as expected without any errors.

[wongjiahau]

I'm facing this issue when trying to install from a Gitlab private repo using HTTPS with node@16.13.1 and npm@8.3.0. Things worked when I downgraded npm as per #624 (comment).

[likemusic]

When I previously post log for issue I investigated the problem.

This happens bacuase npm tries to clone (git clone subprocess) in temp directory for current user but it runs as current directory owner user where npm runs.

In my case the owner of directory was node but I run npm as root user. So npm start process as root-user (because I run it as root user), but when git clone supbrocess runs it tries clone in temp directory for root user as node user whose haven't enough rigth to create subdirectories during git clone.

To prevent this issue (untill it would be fixed) you must run npm command as user who is current directory owner, for example:

runuser -u node -- npm install

Or change owner for current directory:

chown $(whoami) .

[wongjiahau]

@likemusic For my case the directory that contains package.json is in fact not owned by root, and after doing chown root:root . the problem is fixed indeed. Thanks!

[murtycodes]

This worked for me: RUN apk add --no-cache git

https://stackoverflow.com/a/63033749/6884069

[georgisoft2020]

When there is a reference to a git library in package json, like : git:// it really shows the error : npm ERR! No user exists for uid 1000
That's why for me a workaround was to use the command npm install with other user inside docker container.
First check what users do you have on that docker with the help of cat /etc/passwd , then use an user different from root.
runuser -u otheruser -- npm install