Disables by default the migrations #6465

arcanis · 2018-10-02T10:50:41Z

Summary

While not breaking technically speaking, the automatic migration is causing unexpected behaviors to our users. Such a behavior should only happen when bumping the major version.

To prevent that, the unsafe-disable-integrity-migration settings will default to true for the duration of the 1.X branch. It will default to false again starting with the 2.0.

In order to migrate your project now, just add the following line into your project-local yarnrc:

unsafe-disable-integrity-migration "false"

Note that this only affect the lockfile entries already present in your lockfile. Yarn will continue to create the new entries using the integrity field. My guess is that by the time the 2.0 lands, enough projects will have been migrated to use the integrity field that this will be a painless transition.

Test plan

CI

imsnif

What do you think about adding a warning message that would explain this to the user (or link to a help page with more details)? unsafe-disable-integrity-migration which defaults to true is a bit of a handful (double negative + unsafe). (EDIT: double negative if it's explicitly disabled, that is :) )

Also about migrating: I realize this is inevitable, but we'll still have mutations between versions (if I add a package with integrity, it will be stripped when someone runs yarn with a previous version).

Gudahtt · 2018-10-02T11:19:58Z

src/config.js

@@ -408,6 +408,8 @@ export default class Config {
      }
    }

+    const toBool = value => value !== 'false' && value !== 'no' && value !== '0';


The boolifyWithDefault function in src/util.js might be appropriate to use here instead of introducing a new function.

arcanis · 2018-10-02T12:13:36Z

What do you think about adding a warning message that would explain this to the user (or link to a help page with more details)?

Not sure - should we print the warning every time? It would be very verbose (especially if it includes a link). It sounds like the kind of thing that users will just ignore since "it works".

Gudahtt · 2018-10-02T12:30:51Z

I would suggest that the option could be renamed to avoid the double-negative, but then the unsafe- prefix ceases to make sense. We don't want to imply that migrations are unsafe.

I don't think printing a warning each time is warranted - there is nothing actionable here for the user to do.

arcanis · 2018-10-02T12:35:52Z

Yep, the double negation was because I preferred to make it explicit that we recommend this option not to be turned on (a bit like DO_NOT_USE_OR_YOU_WILL_BE_FIRED).

buildsize · 2018-10-02T12:38:28Z

File name	Previous Size	New Size	Change
yarn-[version].noarch.rpm	1.09 MB	1.09 MB	40 bytes (0%)
yarn-[version].js	4.26 MB	4.26 MB	247 bytes (0%)
yarn-legacy-[version].js	4.44 MB	4.44 MB	243 bytes (0%)
yarn-v[version].tar.gz	1.1 MB	1.1 MB	1.69 KB (0%)
yarn_[version]all.deb	803.62 KB	803.55 KB	-76 bytes (0%)

imsnif · 2018-10-02T12:56:42Z

A message each time might indeed be excessive. Maybe some documentation somewhere like here then? https://yarnpkg.com/lang/en/docs/yarnrc/

arcanis · 2018-10-02T13:43:05Z

@imsnif Do you have an idea why the bailout tests are failing?

imsnif · 2018-10-02T13:54:28Z

@arcanis - my guess would be something to do with https://github.com/yarnpkg/yarn/blob/master/src/cli/commands/install.js#L465
But I'll have to take a closer look. Can do later tonight if it can wait.

arcanis · 2018-10-02T14:22:06Z

Thanks, that was it, problem fixed!

rjmunro · 2018-10-08T10:27:02Z

There are 2 kinds of project out there. Ones with 1.9 type lock files (no integrity), and ones with 1.10 lock files (with integrity). Yarn should work with both, and preserve them it that state until the project decides to upgrade them to the new format. A warning should be given to remind people to do that.

Making a third, hybrid format lockfile that has only some integrity entries is no use to 1.10 users because it doesn't really check the integrity, and even worse for 1.9 users because they will strip the integrity lines and have to commit the yarn.lock file each time a 1.10 user adds something.

arcanis · 2018-10-08T17:33:53Z

Making a third, hybrid format lockfile that has only some integrity entries is no use to 1.10 users because it doesn't really check the integrity

Yarn will check the integrity. It just will keep using the sha1 for old ones. The main reason we've disabled the migration by default is that the lockfile should not change when you run yarn install on an up-to-date project, except between major releases.

Yarn v1.10 adds `integrity`s but v1.9 removes them. Add this option to prevent them from adding unnecessary diffs all the time. This is the default value for Yarn v1.12+ (ref: yarnpkg/yarn#6465).

borekb · 2018-11-02T07:51:52Z

A bit OT but since we're fighting this integrity field in our team where not everyone is on the same Yarn version (I know...), is there a way to pass unsafe-disable-integrity-migration from the command-line?

rjmunro · 2018-11-02T10:17:25Z

@arcanis I'm seeing random integrity entires being added, depending on who / where yarn is being run, then being removed again when someone on an old version runs it. These all go into version control. It's a total mess. There should only ever be all or no entries. Which state a particular lock file is in should be left alone unless the user changes it (e.g. by running something like a special command, maybe yarn upgrade-lockfile).

At the very least, weather or not to include the entries should be something that happens to a project, not something done by a user.

arcanis · 2018-11-02T14:31:34Z

@borekb @rjmunro this doesn't entirely answer your question, but are you aware that you can very easily force all of your team to use the exact version of Yarn you want them to use?

Just create a .yarnrc in your repositories, add yarn-path "./scripts/yarn-1.12.1.js" inside, add the single-file release to the right place, commit all this, and voilà. As long as your developers have Yarn 1.0 globally installed (which is more than one year old), Yarn will transparently use the version you've selected over the global one 🙂

We use this pattern at work and it's extremely handy to force everyone to upgrade at once, or to rollback everyone if we find a regression.

borekb · 2018-11-05T08:58:47Z

@arcanis That is a clever trick, thank you!

Changelog tracks back up to 1.12.0 only. ## 1.12.3 **Important:** This release contains a cache bump. It will cause the very first install following the upgrade to take slightly more time, especially if you don't use the [Offline Mirror](https://yarnpkg.com/blog/2016/11/24/offline-mirror/) feature. After that everything will be back to normal. - Fixes an issue with `yarn audit` when using workspaces [6625](yarnpkg/yarn#6639) - [**Jeff Valore**](https://twitter.com/codingwithspike) - Uses `NODE_OPTIONS` to instruct Node to load the PnP hook, instead of raw CLI arguments **Caveat:** This change might cause issues for PnP users having a space inside their cwd (cf [nodejs/node24065](nodejs/node#24065)) [6479](yarnpkg/yarn#6629) - [**Maël Nison**](https://twitter.com/arcanis) - Fixes Gulp when used with Plug'n'Play [6623](yarnpkg/yarn#6623) - [**Maël Nison**](https://twitter.com/arcanis) - Fixes an issue with `yarn audit` when the root package was missing a name [6611](yarnpkg/yarn#6611) - [**Jack Zhao**](https://github.com/bugzpodder) - Fixes an issue with `yarn audit` when a package was depending on an empty range [6611](yarnpkg/yarn#6611) - [**Jack Zhao**](https://github.com/bugzpodder) - Fixes an issue with how symlinks are setup into the cache on Windows [6621](yarnpkg/yarn#6621) - [**Yoad Snapir**](https://github.com/yoadsn) - Upgrades `inquirer`, fixing `upgrade-interactive` for users using both Node 10 and Windows [6635](yarnpkg/yarn#6635) - [**Philipp Feigl**](https://github.com/pfeigl) - Exposes the path to the PnP file using `require.resolve('pnpapi')` [6643](yarnpkg/yarn#6643) - [**Maël Nison**](https://twitter.com/arcanis) ## 1.12.2 This release doesn't actually exists and was caused by a quirk in our systems. ## 1.12.1 - Ensures the engine check is ran before showing the UI for `upgrade-interactive` [6536](yarnpkg/yarn#6536) - [**Orta Therox**](https://github.com/orta) - Restores Node v4 support by downgrading `cli-table3` [6535](yarnpkg/yarn#6535) - [**Mark Stacey**](https://github.com/Gudahtt) - Prevents infinite loop when parsing corrupted lockfiles with unterminated strings [4965](yarnpkg/yarn#4965) - [**Ryan Hendrickson**](https://github.com/rhendric) - Environment variables now have to **start** with `YARN_` (instead of just contain it) to be considered [6518](yarnpkg/yarn#6518) - [**Michael Gmelin**](https://blog.grem.de) - Fixes the `extensions` option when used by `resolveRequest` [6479](yarnpkg/yarn#6479) - [**Maël Nison**](https://twitter.com/arcanis) - Fixes handling of empty string entries for `bin` in package.json [6515](yarnpkg/yarn#6515) - [**Ryan Burrows**](https://github.com/rhburrows) - Adds support for basic auth for registries with paths, such as artifactory [5322](yarnpkg/yarn#5322) - [**Karolis Narkevicius**](https://twitter.com/KidkArolis) - Adds 2FA (Two Factor Authentication) support to publish & alike [6555](yarnpkg/yarn#6555) - [**Krzysztof Zbudniewek**](https://github.com/neonowy) - Fixes how the `files` property is interpreted to bring it in line with npm [6562](yarnpkg/yarn#6562) - [**Bertrand Marron**](https://github.com/tusbar) - Fixes Yarn invocations on Darwin when the `yarn` binary was symlinked [6568](yarnpkg/yarn#6568) - [**Hidde Boomsma**](https://github.com/hboomsma) - Fixes `require.resolve` when used together with the `paths` option [6565](yarnpkg/yarn#6565) - [**Maël Nison**](https://twitter.com/arcanis) ## 1.12.0 - Adds initial support for PnP on Windows [6447](yarnpkg/yarn#6447) - [**John-David Dalton**](https://twitter.com/jdalton) - Adds `yarn audit` (and the `--audit` flag for all installs) [6409](yarnpkg/yarn#6409) - [**Jeff Valore**](https://github.com/rally25rs) - Adds a special logic to PnP for ESLint compatibility (temporary, until [eslint/eslint10125](eslint/eslint#10125) is fixed) [6449](yarnpkg/yarn#6449) - [**Maël Nison**](https://twitter.com/arcanis) - Makes the PnP hook inject a `process.versions.pnp` variable when setup (equals to `VERSIONS.std`) [6464](yarnpkg/yarn#6464) - [**Maël Nison**](https://twitter.com/arcanis) - Disables by default (configurable) the automatic migration of the `integrity` field. **It will be re-enabled in 2.0.** [6465](yarnpkg/yarn#6465) - [**Maël Nison**](https://twitter.com/arcanis) - Fixes the display name of the faulty package when the NPM registry returns corrupted data [6455](yarnpkg/yarn#6455) - [**Grey Baker**](https://github.com/greysteil) - Prevents crashes when running `yarn outdated` and the NPM registry forgets to return the `latest` tag [6454](yarnpkg/yarn#6454) - [**mad-mike**](https://github.com/mad-mike) - Fixes `yarn run` when used together with workspaces and PnP [6444](yarnpkg/yarn#6444) - [**Maël Nison**](https://twitter.com/arcanis) - Fixes an edge case when peer dependencies were resolved multiple levels deep (`webpack-dev-server`) [6443](yarnpkg/yarn#6443) - [**Maël Nison**](https://twitter.com/arcanis)

arcanis requested a review from imsnif October 2, 2018 10:50

imsnif requested changes Oct 2, 2018

View reviewed changes

Gudahtt reviewed Oct 2, 2018

View reviewed changes

edmorley mentioned this pull request Oct 2, 2018

Unpin the default Yarn version from 1.9.x heroku/heroku-buildpack-nodejs#569

Merged

Maël Nison added 2 commits October 2, 2018 13:17

Disables by default the migrations

5dd3e6c

Uses boolifyWithDefaults

026ff3e

arcanis force-pushed the default-disable-migrations branch from b156a23 to 026ff3e Compare October 2, 2018 12:17

Gudahtt approved these changes Oct 2, 2018

View reviewed changes

Makes flow happy(ier)

aefdeb1

imsnif approved these changes Oct 2, 2018

View reviewed changes

Fixes tests

7bec845

arcanis force-pushed the default-disable-migrations branch from e5f6db5 to 7bec845 Compare October 2, 2018 13:59

Merge branch 'master' into default-disable-migrations

20bc65d

arcanis merged commit 6ea3ff5 into yarnpkg:master Oct 2, 2018

imsnif mentioned this pull request Oct 2, 2018

docs(integrity): document unsafe-disable-integrity-migration yarnpkg/website#865

Merged

zephraph mentioned this pull request Oct 8, 2018

Preempt issues w/ yarn 1.11+ from removing integrity checks artsy/palette#111

Merged

ylemkimon mentioned this pull request Oct 30, 2018

Do not remove integrity property in Yarn lockfiles greenkeeperio/greenkeeper#955

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disables by default the migrations #6465

Disables by default the migrations #6465

arcanis commented Oct 2, 2018 •

edited

imsnif left a comment •

edited

Gudahtt Oct 2, 2018

arcanis commented Oct 2, 2018

Gudahtt commented Oct 2, 2018

arcanis commented Oct 2, 2018

buildsize bot commented Oct 2, 2018 •

edited

imsnif commented Oct 2, 2018

arcanis commented Oct 2, 2018

imsnif commented Oct 2, 2018

arcanis commented Oct 2, 2018

rjmunro commented Oct 8, 2018

arcanis commented Oct 8, 2018

borekb commented Nov 2, 2018

rjmunro commented Nov 2, 2018 •

edited

arcanis commented Nov 2, 2018

borekb commented Nov 5, 2018

Disables by default the migrations #6465

Disables by default the migrations #6465

Conversation

arcanis commented Oct 2, 2018 • edited

imsnif left a comment • edited

Choose a reason for hiding this comment

Gudahtt Oct 2, 2018

Choose a reason for hiding this comment

arcanis commented Oct 2, 2018

Gudahtt commented Oct 2, 2018

arcanis commented Oct 2, 2018

buildsize bot commented Oct 2, 2018 • edited

imsnif commented Oct 2, 2018

arcanis commented Oct 2, 2018

imsnif commented Oct 2, 2018

arcanis commented Oct 2, 2018

rjmunro commented Oct 8, 2018

arcanis commented Oct 8, 2018

borekb commented Nov 2, 2018

rjmunro commented Nov 2, 2018 • edited

arcanis commented Nov 2, 2018

borekb commented Nov 5, 2018

arcanis commented Oct 2, 2018 •

edited

imsnif left a comment •

edited

buildsize bot commented Oct 2, 2018 •

edited

rjmunro commented Nov 2, 2018 •

edited