Extracting the tar content of undefined failed

[opiation]

Do you want to request a feature or report a bug?
Reporting a bug when running yarn install to install node dependencies. For severity, this bug seems critical considering it essentially prevents me from acquiring node dependencies.

What is the current behavior?
Fails sometimes with errors like the following:

yarn install v1.9.4
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
error https://registry.yarnpkg.com/lodash/-/lodash-4.17.10.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, chmod '/usr/local/share/.cache/yarn/v2/npm-lodash-4.17.10-1b7793cf7259ea38fb3661d4d38b3260af8ae4e7/_cacheHas.js'"
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

yarn install v1.9.4
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
error https://registry.yarnpkg.com/lodash/-/lodash-4.17.10.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "EEXIST: file already exists, mkdir '/usr/local/share/.cache/yarn/v2/npm-lodash-4.17.10-1b7793cf7259ea38fb3661d4d38b3260af8ae4e7'"
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

yarn install v1.9.4
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
error https://registry.yarnpkg.com/fbjs/-/fbjs-0.8.17.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, chmod '/usr/local/share/.cache/yarn/v2/npm-fbjs-0.8.17-c4d598ead6949112653d6588b01a5cdcd9f90fdd/lib/resolveImmediate.js'"
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command

The occurrence of this error is the challenging part. It does not always fail and does not always fail with the same dependency. The installation is sometimes successful after 3-5 tries.

If the current behavior is a bug, please provide the steps to reproduce.
I've attempted to install dependencies on bare-metal and in a node:8-alpine docker container. Both can sometimes encounter the error. I've tested this on my personal device in Montreal, Canada (Mac OS X10.13), on a AWS EC2 instance (Ubuntu 18.04), on a GCE instance (Ubuntu 16.04) and on a production server in France (Debian 8). Each of them can sometimes encounter this error. I've also tried installing with and without yarn.lock to no avail. Find a package.json that appears to sometimes reproduce the issue in this gist. The issue does not seem to happen with projects having fewer dependencies.

What is the expected behavior?
Successful installation of all packages like npm install or npm ci which deterministically succeeds without any tar or caching errors.

Please mention your node.js, yarn and operating system version.
Tested with the following version:
Node: 8 LTS, 10
Yarn: 1.9.2, 1.9.4
OS: Ubuntu 18.04 LTS, Ubuntu 16.04 LTS, Debian 8, Mac OSX 10.13
Registrie: registry.yarnpkg.com, registry.npmjs.org, private registry

If you require any additional information, don't hesitate to request it. FWIW, reducing the network-concurrency seems to produce a slightly higher success ratio but not consistently enough to conclude the errors are related. It may be an area to investigate however. I've unfortunately exhausted all the time I could afford to spend on this after a few days of troubleshooting. I've reluctantly had to migrate all our CI builds back to using npm install/npm ci :(

[Titozzz]

Same issue, it's blocking my CI too, we updated recently to yarn 1.9.2

[adrienharnay]

@opiation The error is indeed random but I may have found the cause: do you have distant git URLs in your package.json without .git at the end? We had two of those and adding .git fixed the issue. Not sure why the error message doesn't directly state this is is the problem though.

[adrienharnay]

Also, maybe related: #5536

[opiation]

@adrienharnay, can you define what you mean by distant? For the record, here's the package.json I used. There's only one github dependency and I still get errors without it. I'm not sure how I would append .git to non-git dependencies, unless I've misunderstood your suggestion.

[adrienharnay]

Distant wasn't the right word, I just meant packages installed from Git 🙂

Could you try this?

"storybook-addon-markdown": "https://github.com/mihalik/storybook-addon-markdown.git"

[opiation]

As per my previous comment, I still encounter the issue without storybook-addon-markdown dependency. Thus, I don't believe this issue stems from yarn improperly handling git URLs.

[adrienharnay]

Indeed, I read too quickly. Well, that fixed our bug, but I have no idea about yours 😕 Sorry

[Titozzz]

@opiation Did you update you yarn.lock file too? Cause I had to do that

[opiation]

@Titozzz, I encounter this error with and without the yarn.lock file. I have deleted and recreated the lock file several times to no avail.

[ekarlso]

I get this too and I dont have any packages from git.

[luwes]

I wanted to get around this issue (#6256) by using the tarball versions of the packages but indeed the error above is thrown for tarball urls on self-hosted Github enterprise.

github.com hosted tarballs somehow did work. e.g.
https://github.com/luwes/chameleon/archive/grasshopper-v0.0.1-beta.4.tar.gz

[kvnify]

I'm seeing the same issue with a project we have. However when I remove the deps that run a prepare script as part of the install (due to being git urls) then it works. These happen to be pointing git urls, but I think it's actually the prepare that seem to kick off more yarn install processes that seem to subvert the mutex flag for some reason. I wonder if that's because the other processes are kicked off by the root process rather than by different root processes. I don't know if this information helps or if its actually way off base. But I figured I would share what I've found regardless.

[adrienharnay]

@khendry I got the issue again, and you are right, it comes from git dependencies which have a prepare script in their package.json! 👍

[opnet]

I've been tracking this down with a project we have and so far narrowed it down to the concurrent install the git-fetcher starts here. If the package that is being installed by the git-fetcher has any of the same dependencies of the currently installing package a race condition is created where the duplicated packages will be untarred to offline cache at the same time.

I haven't seen enough of the code base to understand where/what the correct fix is, but that's the start of the issue.

[bjentsch]

Any news on this? We're facing this issue too.

[idexter]

Same problem. It's impossible to use yarn with CI. Every 2nd build failed with this error 😞

[holyxiaoxin]

try delete node_modules,

yarn cache clean
yarn install --network-concurrency 1

[muuki88]

Thanks for sharing this. It's at least a workaround 🤗 , but no real solution if you want your build time reasonable fast 😅

[kvnify]

We did try using the --network-concurrency flag with no success. So that does not actually resolve this particular issue. The flag addresses concurrency at a higher level than where the issue occurs.

[idexter]

For me --network-concurrency 1 solves the problem. I know it temporary fix, but it works. But value must be exactly 1.

[kvnify]

I spoke too soon. I had asked my team mate if we'd tried this, rather than actually trying myself and he was very confident that we had...he was mistaken and miss-read the previous post thinking it was related to the mutex flag, not network concurrency. We've since re-tried and can confirm this does seem to also address the issue for us.

[hulkish]

setting --network-concurrency 1 does not actually work for me.

right now, the only workaround I've come across involves completely regenning yarn.lock. The error I get is:

2.054 Performing "GET" request to "https://<private-artifactory-npm-registry>/@myorg/eslint-plugin-import/-/@myorg/eslint-plugin-import-3.0.0.tgz".
verbose 2.519 Error: https://<private-artifactory-npm-registry>/@myorg/eslint-plugin-import/-/@myorg/eslint-plugin-import-3.0.0.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "Unexpected end of data"
    at MessageError.ExtendableBuiltin (/Users/me/.nvm/versions/node/v8.12.0/lib/node_modules/yarn/lib/cli.js:237:66)
    at new MessageError (/Users/me/.nvm/versions/node/v8.12.0/lib/node_modules/yarn/lib/cli.js:266:123)
    at Extract.<anonymous> (/Users/me/.nvm/versions/node/v8.12.0/lib/node_modules/yarn/lib/cli.js:59446:14)
    at emitOne (events.js:121:20)
    at Extract.emit (events.js:211:7)
    at Extract.module.exports.Extract.destroy (/Users/me/.nvm/versions/node/v8.12.0/lib/node_modules/yarn/lib/cli.js:135306:17)
    at Extract.module.exports.Extract._final (/Users/me/.nvm/versions/node/v8.12.0/lib/node_modules/yarn/lib/cli.js:135364:34)
    at callFinal (/Users/me/.nvm/versions/node/v8.12.0/lib/node_modules/yarn/lib/cli.js:70270:10)
    at _combinedTickCallback (internal/process/next_tick.js:139:11)
    at process._tickCallback (internal/process/next_tick.js:181:9)

Update: I've just found that using --skip-integrity-check allows me to bypass this error. While obviously that's really a solution. This looks like kind of an important bug in the integrity check logic.

Im using node@8.12.0, yarn@1.12.0

[hulkish]

@arcanis @rally25rs more details on this error:

So, this seems pretty odd to me that it's failing integrity checksum, considering the sha1's are the same:

Error: sha1-Sl7Hxk364iw6FBJNus3uhG2Ay8Q= integrity checksum failed when using sha1: wanted sha1-Sl7Hxk364iw6FBJNus3uhG2Ay8Q= but got sha1-AHoWKXweP+Pg9aZkGBsAjFruGaM=. (77 bytes)
    at Transform.on (/Users/shargrove/.nvm/versions/node/v8.12.0/lib/node_modules/yarn/lib/cli.js:32831:19)
    at emitNone (events.js:111:20)
    at Transform.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:139:11)
    at process._tickCallback (internal/process/next_tick.js:181:9)

Update: After seeing above, I confirmed that --skip-integrity-check bypasses this error. Looks like a more serious bug in integrity check logic.

[arcanis]

Given that this problem suddenly appears much more than usual and that it's during the npm registry being especially flaky, it's quite likely my hypothesis isn't far off

[mjniday]

@arcanis just started having this problem today. I can confirm that by removing that npmrc auth token line the issue was resolved.

[achillesrasquinha]

There's no ~/.npmrc created in my case. But regenerating yarn.lock worked for me.

Simply,

$ rm yarn.lock && yarn

EDIT: Faced this issue twice only to end up landing here. 😄

[jeongukjae]

In my case, I use CircleCI, circleci/node:10.11.0 docker image, and yarn@1.12.3, and there is no ~/.npmrc. Thank you @achillesrasquinha. It works for me.

[ghost]

I have been facing this issue over a week. yarn install --network-concurrency 1 solved issue but it is very very slow.

In btw, this info could be helpful to anyone.
I was using a custom npm package (in house) in my project. Always I am getting same issue like .cache/v4 but showing different package names every fail. After spending much time, I find one random observation.
My project and custom npm package is using same yarn build for build the bundle. I have updated my custom package build script name to another name as yarn build:p. Then it is start working. I ran build many times. It was not failed. Not sure How these 2 are dependent but worked for me.

[0x80]

Removing .npmrc only didn't do it for me. I also had to remove my yarn.lock file like @davidalee mentioned. I don't know why he's getting thumbs down for it 🤷‍♂️

Not sure if removing .npmrc had any effect for me.

[zedtux]

I'm not really fan of deleting the yarn.lock file so what I did is just remove the har-validator package from the yarn.lock and then re-run yarn which fixed the issue for me.

[phattruong-agilityio]

Yes rm yarn.lock work for me. Facing issue with package har-validator-5.1.2.

error https://registry.yarnpkg.com/har-validator/-/har-validator-5.1.2.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "Unexpected end of data"

[ondratra]

Hi, har-validator-5.1.2 was unpublished from npm as stated here ahmadnassri/node-har-validator#112 (comment) , thus you need to upgrade your dependencies via yarn upgrade(this has probably the same effect as removing yarn.lock that was recommended by the others).

I suppose this issue can be closed.

[opiation]

Removing yarn.lock did not work for me, as mentioned in my initial issue report. Nor did removing .npmrc. Besides, as far as I am aware, the node:10-alpine docker image does not have or create a .npmrc file.

Lastly, the error is not limited to the har-validator package. In fact, I've never encountered it with that package. I did encounter it with packages lodash, fbjs, react, and a host of others.

I summarized my attempts that still reliably reproduce this issue in a previous comment. For the record, when testing with docker, I can reproduce the problem including only the package.json thus, no yarn.lock, no .npmrc, no node_modules. I can still reproduce this issue on my local machine, on a GCE instance and with Gitlab.com's CI. Neither --network-concurrency=1 nor --skip-integrity-check seem to resolve the issue for me. Thus, I would hesitate to recommend closing this issue, especially since all the above-mentioned tests work using npm install, assuming that yarn install should be a drop-in replacement for npm install given the same package.json.

[arcanis]

The problem is that the npm registry is generally unstable and returns errors (at a higher rate when multiple requests are firing apparently - maybe some kind of per-ip throttling?). For some reason they are not properly caught by Yarn, which blindly tries to hash them and compare them to the expected hash - which fails.

So there is a bug in Yarn (we should print a more helpful error), but given that the real problem is how flaky the npm registry is, it's not my priority at the moment (I'd definitely review a PR, though!).

As for why it doesn't happen with npm: they retry their requests until they work. Yarn has a mechanism to do that, but not on the part that specifically compute the hash.

I'd suggest to use an offline mirror to stop relying on the npm registry for your installs.

[arcanis]

#6817 will "fix" that by showing the actual status code returned by the registry. I'd prefer it to be stable rather than blindly retry until it works so I haven't added the retry code, but if there's no improvements to the horizon we might have to do that.

In the meantime I'll close this discussion, as the error messages will change and this thread grown quite large (we can open new ones to discuss each status code individually).

[jaychoumylove]

There's no ~/.npmrc created in my case. But regenerating yarn.lock worked for me.

Simply,

$ rm yarn.lock && yarn

thank you,
rm -rf ./yarn.lock && yarn
it's work for me!

[mrseanryan]

in case it helps anyone:

• this same error occurred for me, when I had forgotten to log in to npm (doh!)

[middlehut]

For me the issue was resolved with service docker restart (Ubuntu 18.04).

[jleppert]

I've been experiencing intermittent and non-deterministic errors like this one. I restart my build, nothing else having changed and it works. Does anyone have any alternatives to yarn?

[quarklemotion]

I starting getting this same error on every build (errors on a different npm module each time) after making a PR to update our base docker image from node:8.12.0 to node:8.13.0. I inspected these node docker images and discovered that the pre-installed yarn version was changed from v1.9.4 to v1.12.3. See: associated git commit. I tried some of the suggested fixes in this thread with no luck in resolving the error. I was able to fix the issue by simply downgrading the yarn version in my Dockerfile to v1.9.4. I know this version of yarn has been problematic for others, but for me the more recent yarn version is triggering the issue. I'll note that I am using an .npmrc file that provides credentials to access private modules via jfrog artifactory and we have artifactory set up to mirror/proxy all npm modules.

[kunokdev]

Why is this closed? Still breaking CI

[arcanis]

In the meantime I'll close this discussion, as the error messages will change and this thread grown quite large (we can open new ones to discuss each status code individually).

I'll go ahead and lock this thread since it seems to me it reached past its usefulness. As a reminder:

• If you have this error message, you are very likely using an old release. Upgrade to 1.13+ to get the true error message. Chances are the registry is returning an HTTP 500 for some reason.

• If you still get errors that seem to come from Yarn itself open a new thread and detail how to reproduce the problem. If you don't provide a reproduction we won't be able to provide a fix, and will likely have to ask you to investigate yourself.