New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
yarn install sporadically fails with integrity check failure #6407
Comments
Are you sure this doesn't happen with the 1.9.4? Afaik this is caused by the npm registry sometimes returning 500 at random (the npm client retries until it works, but we don't do this yet), so this should affect all Yarn releases. |
I think integrity checking was added with 1.10.0. See https://github.com/yarnpkg/yarn/pull/5042/files?utf8=%E2%9C%93&diff=unified and https://twitter.com/yarnpkg/status/1037631294170189824 |
I know :) But the integrity errors are caused by the registry returning 500 errors that are not properly reported as such. |
@arcanis We have been testing for flakiness recently, just reverting yarn back to 1.9.4 has taken our builds from a 36% success rate to 88% |
@Dahaden Can you share details regarding the errors? In particular:
The 1.10 ships with the new integrity feature, which requires us to migrate your lockfiles by querying the npm registry for each dependency in order to add the field. If done during CI, it could increase the opportunities for the registry to return an 500 somewhere. If the integrity fields were added before being checked-in, that would alleviate the issue (since the CI wouldn't have to query the network for this anymore) cc @imsnif If you have the opportunity to retry the 1.10, can you add the following line into a
|
Yeah, as @arcanis said, the migration to 1.10 is more network intensive. If on a flaky network (or if the npm registry is having networking issues at the time) it might explain it. Following. |
This should hopefully be mitigated with #6413 |
We did populate the integrity field in the lockfile prior to pushing, so it should be all good. |
In my case, this .yarnrc workaround did the job. |
add a .yarnrc file also solved my problem, thanks. |
is working |
|
One package that keeps failing the integrity check is this one I hope this helps |
anyway to fix this? or to disable integrity check? |
That seems strange, we have a CI test that uses it. Can you share the output of |
Minus your possible npm auth tokens, of course (we maybe should remove it from the output ...)! |
another strange thing is that if I do the following things:
config: yarn config v1.12.3
verbose 0.282 Checking for configuration file "/Users/sibelius/Dev/e/f/f-server/.npmrc".
verbose 0.283 Checking for configuration file "/Users/sibelius/.npmrc".
verbose 0.283 Found configuration file "/Users/sibelius/.npmrc".
verbose 0.284 Checking for configuration file "/usr/local/etc/npmrc".
verbose 0.284 Checking for configuration file "/Users/sibelius/Dev/e/f/f-server/.npmrc".
verbose 0.285 Checking for configuration file "/Users/sibelius/Dev/e/f/.npmrc".
verbose 0.285 Checking for configuration file "/Users/sibelius/Dev/e/.npmrc".
verbose 0.285 Checking for configuration file "/Users/sibelius/Dev/.npmrc".
verbose 0.285 Checking for configuration file "/Users/sibelius/.npmrc".
verbose 0.285 Found configuration file "/Users/sibelius/.npmrc".
verbose 0.286 Checking for configuration file "/Users/.npmrc".
verbose 0.29 Checking for configuration file "/Users/sibelius/Dev/e/f/f-server/.yarnrc".
verbose 0.29 Found configuration file "/Users/sibelius/Dev/e/f/f-server/.yarnrc".
verbose 0.29 Checking for configuration file "/Users/sibelius/.yarnrc".
verbose 0.29 Found configuration file "/Users/sibelius/.yarnrc".
verbose 0.291 Checking for configuration file "/usr/local/etc/yarnrc".
verbose 0.291 Checking for configuration file "/Users/sibelius/Dev/e/f/f-server/.yarnrc".
verbose 0.291 Found configuration file "/Users/sibelius/Dev/e/f/f-server/.yarnrc".
verbose 0.291 Checking for configuration file "/Users/sibelius/Dev/e/f/.yarnrc".
verbose 0.291 Found configuration file "/Users/sibelius/Dev/e/f/.yarnrc".
verbose 0.292 Checking for configuration file "/Users/sibelius/Dev/e/.yarnrc".
verbose 0.292 Checking for configuration file "/Users/sibelius/Dev/.yarnrc".
verbose 0.292 Checking for configuration file "/Users/sibelius/.yarnrc".
verbose 0.292 Found configuration file "/Users/sibelius/.yarnrc".
verbose 0.292 Checking for configuration file "/Users/.yarnrc".
verbose 0.301 current time: 2018-11-08T11:36:25.250Z
info yarn config
{ 'version-tag-prefix':
'v',
'version-git-tag':
true,
'version-commit-hooks':
true,
'version-git-sign':
false,
'version-git-message':
'v%s',
'init-version':
'1.0.0',
'init-license':
'MIT',
'save-prefix':
'^',
'bin-links':
true,
'ignore-scripts':
false,
'ignore-optional':
false,
registry:
'https://registry.yarnpkg.com',
'strict-ssl':
true,
'user-agent':
'yarn/1.12.3 npm/? node/v10.13.0 darwin x64',
lastUpdateCheck:
1541611519487,
'unsafe-disable-integrity-migration':
true,
'yarn-offline-mirror':
'/Users/sibelius/Dev/e/f/f-server/yarn-offline-cache',
'yarn-offline-mirror-pruning':
true,
'experimental-pack-script-packages-in-mirror':
true }
info npm config
{ 'init.author.name':
'Sibelius Seraphini',
'init.author.email':
'sibeliusseraphini@gmai.com',
'init.author.url':
'https://github.com/sibelius',
'//registry.npmjs.org/:_authToken':
'blah',
progress:
true,
python:
'/usr/bin/python' }
|
|
here are two prints of what keep happening with mongodb-memory-server@2.7.0:
version "2.7.0"
resolved "https://registry.yarnpkg.com/mongodb-memory-server/-/mongodb-memory-server-2.7.0.tgz#663345e8fe38e3b76c703fcc94f691c192fcbd66"
- integrity sha512-T9zBEN3/y7/s4F83B2jAlLHtjjSEp50GQ2J0b7QMbAwM/G7Rkxzdf3cCfzOChDBfI0lQto09EOTdDam6mm0REQ==
+ integrity sha1-M2tbFYi0Q8ExxGJmGySYCdh3/qY=
dependencies:
"@babel/runtime" "^7.1.2"
debug "^4.1.0" then mongodb-memory-server@2.7.0:
version "2.7.0"
resolved "https://registry.yarnpkg.com/mongodb-memory-server/-/mongodb-memory-server-2.7.0.tgz#663345e8fe38e3b76c703fcc94f691c192fcbd66"
-integrity sha512-T9zBEN3/y7/s4F83B2jAlLHtjjSEp50GQ2J0b7QMbAwM/G7Rkxzdf3cCfzOChDBfI0lQto09EOTdDam6mm0REQ==
+integrity sha1-M2tbFYi0Q8ExxGJmGySYCdh3/qY=
dependencies:
"@babel/runtime" "^7.1.2"
debug "^4.1.0" same machine, same yarn (I'm using 1.12.3 now) |
Hmm - ping @imsnif? |
Thanks for the ping @arcanis! @sibelius - I cannot reproduce this issue with that package. What I do:
I repeated this process a few times and still go the same results. A few options:
If this still happens to you consistently, I'd be very happy to be able to reproduce and debug this. For this I need your help: could you trim an example repo down to the minimum needed to reproduce this consistently (eg. EDIT: (just to add an explanation) Thanks for reporting this and helping out. I really want to get to the bottom of these because I've seen them pop up here and there and could never reproduce consistently. |
this is my .yarnrc
this is happening in a yarn workspaces, I'm gonna try to create a repro tks for the help |
here is a repro using workspaces https://github.com/sibelius/yarn-workspaces-mms check the commits where I change yarn.lock doing the below commands:
|
The only diff I'm getting is this:
which is not ideal, but out of scope for now. :) How often does this happen to? I tried 3-4 times and everything stays a-ok. |
sha1 -> sha512 sibelius/yarn-workspaces-mms@d41d463 |
For sure. I just need to be able to reproduce it, otherwise I'm just guessing. Does it happen to you every time with this repo? |
not every yarn install, try to delete node_modules and yarn.lock and try to do yarn again this is my mac os x version 10.14.1, not sure if this helps or not node v10.12.0, using nvm 0.31.1 |
I've now run it 10 times and no reproduction. I'd really like to check and see if you're getting a different response from the registry for some reason. If you have the jq json parser, would you mind running this command a few times and seeing if you sometimes get
BTW - I assume you don't start out with the local cache folder, right? |
it keeps returning sha512 I think the problem is with the offline cache, I've disabled this for now and it keeps on sha512 I've used the cache because native packages always recompiled, but this is another issue |
Hey @sibelius, that's quite interesting. If you delete the offline cache and then re-enable it, does the issue reoccur? |
I have many .yarnrc in a lot of different folders, because I work in a lot of different projects I've deleted all of them, and for now, it is working fine not sure if this was a problem with some old yarn-offline-cache folder |
Alright, so the problematic path seems to be here: https://github.com/yarnpkg/yarn/blob/master/src/resolvers/registries/npm-resolver.js#L170 I'm guessing (all I can do right now unfortunately*) that we go to the offline mirror to get the integrity from there, we get the old sha1 integrity and are satisfied with it and thus yarn.lock is updated with the old sha1. What I'm not fully understanding is why this happens sporadically. If this is the case, why is the offline mirror bypassed every now and then. @arcanis - do you have any ideas? I wouldn't want to solve this by always going to the registry to get the integrity for obvious reasons - but I would love to solve the flakiness here as soon as we understand it better. *This might theoretically be reproducible by starting an offline mirror, artificially creating a manifest that has only sha1 integrity and running this a few times. If there aren't any ideas, I'll try to get to that soon. |
can we do not update yarn.lock when resolving from offline cache? |
@sibelius - perhaps, but I'd first like to make sure this is indeed the issue before we implement the solution. |
Encounter this problems with yarn I had to use yvm and manually install previous versions of yarn and found that the version
|
It could happen when you are using yarn local cache with the yarn registry set to a 3rd party mirror which does not yet provide the |
I'm having similar sporadical fails, when using offline and verdaccio registry proxy. in fact, if i repeat the cycle, i still end up with error on
and btw, if i have the |
more fun, is that if i retry the same command, it passes, and the calculated integrity also changes
|
#7499 did not work for me, I manually updated to 1.18.0 (should include this pr) and I am still seeing this when hitting the github package registry (GPR).
I am unsure if my problem is actually specific to GPR or the same as this issue. EDIT: |
^^^ Joining the chorus: We've had huge problems with integrity check failures, and socket connection failures, on 1.19.0. Dropping our pipelines back to 1.18.0 resolves the issues. |
Having this issue with yarn 1.19.0 as well. The sequence is like this:
The same error is happening on my Linux box, my buddies WSL 1 environment, and on our Linux CI servers, so its not something specific to my machine. Artifacts are resolved from our private npm repository using Nexus. I've even erased and updated our These problems all seemed to start around the time we added a yarn-offline-mirror, but we also updated yarn around the same time. Next time it happens I'll try disabling the offline mirror and see if that fixes the integrity error. |
Happened again, and commenting the offline mirror lines from So at least in my case, the yarn offline cache is the culprit here. Deleting it and recreating it only solves the problem temporarily. |
Also, |
yarn v1.19.1 pre_build:
commands:
- echo ">> installing dependencies"
- yarn --update-checksums # Update checksums in the yarn.lock lockfile if there’s a mismatch between them and their package’s checksum.
- yarn install
- yarn lint
- yarn tsc
- yarn test |
|
This is super frustrating. I do lots of things, and randomly it will work. |
Why not use the SHA512 and the old sha1 both together to avoid this? |
It happend for there are multiple registry domains in my lock file. |
Closing as fixed in v2 where the integrity check has been improved |
Do you want to request a feature or report a bug?
bug
What is the current behavior?
When running yarn install (on our CI) we are finding that random dependencies fail to download due to integrity check failures:
e.g.
If the current behavior is a bug, please provide the steps to reproduce.
Have a somewhat yarn project setup with a somewhat large number of dependencies - keep yarn installing until integrity check fails.
What is the expected behavior?
No integrity checks occur
Please mention your node.js, yarn and operating system version.
node: v10.9.0
yarn: v1.10.0
OS: NixOS (Linux)
This could possibly be occurring due to network failures on our end, but I don't consider this likely considering our build machines are EC2 instances with solid internet connections.
As a workaround for now, we are rolling back to v1.9.4
The text was updated successfully, but these errors were encountered: