Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix cache integrity check false-positives across multiple registries #7595

Merged
merged 3 commits into from Oct 3, 2019

Conversation

Blasz
Copy link
Contributor

@Blasz Blasz commented Oct 3, 2019

Summary

Improve cache integrity error messages and fixes the cache integrity check issue described here: #7584 (comment)

I determined it is related to an internal npm registry that we use.

The registry that we use does not return an integrity field and so the cached versions of packages resolved from the registry uses sha1 hashes. When the same package is then resolved from npm instead of our own registry, it has a sha512 integrity field and causes a clash.

I'm still not so sure about erroring out in these scenarios rather than just invalidating the cache for the problem packages since users will manually need to do that after the error was thrown.

Also now that the cacheIntegrity field from cache is being used, I wonder whether the separate -integrity and no-integrity suffixed caches are required anymore?

Test plan

Added tests for the previous cache integrity changes. I saw test cases for sha384 integrity hashes so I wonder if those need to be accounted for, as the same false positive error will occur there as well. If npm ever decides to change their default integrity hash to something other than sha512 this check would also fail.

A better way of handling this could be to compute the remote integrity hash against the cache if it did not match the hash algorithms already stored in cache and then append that value to the cached integrity hash if it matched and throw an error otherwise.

This prevents broken installs as a result of a package being resolved
from a registry that does not return an sha512 integrity field, such as
an internal npm registry, where the package was previously cached from
a registry that did return a sha512 integrity hash.
@Blasz
Copy link
Contributor Author

Blasz commented Oct 3, 2019

cc @arcanis

@arcanis
Copy link
Member

arcanis commented Oct 3, 2019

Hey @Blasz! Thanks for working on it! It looks good to me, especially with the tests you've added ⭐

Also now that the cacheIntegrity field from cache is being used, I wonder whether the separate -integrity and no-integrity suffixed caches are required anymore?

Good question 🤔 Probably not if we always compute both the sha1 and the sha512 regardless of the registry, but I'm not 100% sure there isn't some edge case somewhere in the logic so I prefer to keep it for now.

A better way of handling this could be to compute the remote integrity hash against the cache if it did not match the hash algorithms already stored in cache and then append that value to the cached integrity hash if it matched and throw an error otherwise.

Yeah, but it would make the code even quite spaghetti 🤔

@arcanis arcanis merged commit c040e79 into yarnpkg:master Oct 3, 2019
@Blasz Blasz deleted the fix-cache-integrity branch October 3, 2019 23:33
arcanis pushed a commit that referenced this pull request Oct 8, 2019
…7595)

* Add tests for cache integrity check

* Always store sha1 hash as part of cacheIntegrity

This prevents broken installs as a result of a package being resolved
from a registry that does not return an sha512 integrity field, such as
an internal npm registry, where the package was previously cached from
a registry that did return a sha512 integrity hash.

* Improve cache integrity check error messages
@Daniel15
Copy link
Member

Daniel15 commented Oct 11, 2019

This is probably an edge case, but the build server that publishes the Chocolatey package is hitting this error when trying to check out the Yarn Git repo now:

stderr: error: unable to create file __tests__/fixtures/install/install-update-auth-invalid-cache-integrity/.yarn-cache/v6/npm-safe-buffer-5.1.1-893312af69b2123def71f57889001671eeb2c853-integrity/node_modules/safe-buffer/.yarn-metadata.json: Filename too long

It's currently trying to check out the code to C:\Program Files (x86)\Jenkins\workspace\yarn-chocolatey. I can probably move the directory around to avoid this (and will probably move this to GitHub Actions in the future), but I wonder if other people working on Yarn could hit the same issue.

VincentBailly pushed a commit to VincentBailly/yarn that referenced this pull request Jun 10, 2020
…arnpkg#7595)

* Add tests for cache integrity check

* Always store sha1 hash as part of cacheIntegrity

This prevents broken installs as a result of a package being resolved
from a registry that does not return an sha512 integrity field, such as
an internal npm registry, where the package was previously cached from
a registry that did return a sha512 integrity hash.

* Improve cache integrity check error messages
VincentBailly pushed a commit to VincentBailly/yarn that referenced this pull request Jun 10, 2020
…arnpkg#7595)

* Add tests for cache integrity check

* Always store sha1 hash as part of cacheIntegrity

This prevents broken installs as a result of a package being resolved
from a registry that does not return an sha512 integrity field, such as
an internal npm registry, where the package was previously cached from
a registry that did return a sha512 integrity hash.

* Improve cache integrity check error messages
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants