Perf ? compressionLevel without affecting lock file

[belgattitude]

I've been interested about carbon emissions recently and started tuning yarn install on multiple repos (ie: gha yarn install with action/cache gist).

Intro

Right now I'm looking into compressionLevel and here's some results:

ie: projection 100 install/week with compressionLevel=0 saves ♻️ ~53 minutes of CI/month from compressionLevel=mixed (>25% speedup).

CI Scenario	Install	@action/cache restore	setup	Total	Projection
yarn 4 mixed	±42s	2s (190MB)	0s	±44s	293mins/mo
yarn 4 no-comp	±32s	4s (160MB)	0s	±36s	240mins:/mo
pnpm 7	±17s	8s (245MB)	2s	±27s	180mins/mo

Or locally (using taskset to emulate single core)

Command	Mean [s]	Min [s]	Max [s]	Relative
YARN_COMPRESSION_LEVEL=0 taskset -c 0 yarn install --immutable --mode=skip-build	8.430 ± 0.067	8.361	8.531	1.00
YARN_COMPRESSION_LEVEL=mixed taskset -c 0 yarn install --immutable --mode=skip-build	11.331 ± 0.086	11.249	11.458	1.34 ± 0.01

Hyperfine/Taskset cmd

hyperfine --runs=5 --export-markdown ".yarn/bench-compression-mixed-vs-zero-single-cores.md" \
          --prepare "YARN_COMPRESSION_LEVEL=0 yarn install; npx --yes rimraf '**/node_modules'" \
          "YARN_COMPRESSION_LEVEL=0 taskset -c 0 yarn install --immutable --mode=skip-build" \
          --prepare "YARN_COMPRESSION_LEVEL=mixed yarn install; npx --yes rimraf '**/node_modules'" \
          "YARN_COMPRESSION_LEVEL=mixed taskset -c 0 yarn install --immutable --mode=skip-build" 

The benchmarks runs with node_module linker (pnp has issues with vercel/nft...) and are based on https://github.com/belgattitude/nextjs-monorepo-example (monorepo with nextjs, storybook, linters...)

For more info, please look at this PR belgattitude/nextjs-monorepo-example#2799. Very easy to reproduce.

To sum up the compressionLevel:0 is very helpful option 🙏 👍

Drawbacks

Currently the compressionLevel:0 has some drawbacks. We can't set it differently locally (saves 100's of MB of disk storage) and on the CI. Cause it affect the yarn.lock file (metadata and cheksums) and in the big picture (hardlinks-global, renovate bot, changesets ....) it will cause issues.

Possible improvements

I'm wondering if there's a strong reason to store the cheksums in yarn.lock ? What if

• checksums were stored in a different file(s) ? ie: ./yarn/cache/tbd, install-state...

That way:

• There's a clear separation of concerns between install and cache management. So we can run no-compress on ci, docker... and keep compression locally.
• Side-effect: The lock file will be smaller (might bring a performance gains as it's read for each yarn invocation).

Implementation ideas

Before going further, I'd like to gather some feedbacks from @arcanis or any yarn team member. I'm not sure of the feasibility.

[arcanis]

The checksums are intended to make sure that, when running yarn install on a fresh clone, the dependencies you get are the same one that were originally installed. We can't put them elsewhere than in a file checked-in inside the repo, ie the lockfile.

[belgattitude]

Thanks that makes sense to me.

Is it only computed from the cache zipfile content (or in combination with something else)?

yarn install on a fresh clone the dependencies you get are the same one that were originally installed

When I see the lock I'm still having the impression that the resolution field would be sufficient. On fresh clone the npm archives would be downloaded with the correct version. (I assume npm tarballs are immutable and safe, but not seeing the big picture yet - offline mode, pnp, extra security...).

__metadata:
  version: 7
  cacheKey: 9c0 # will change when using different compressionLevels

"@adobe/css-tools@npm:^4.0.1":
  version: 4.0.1
  resolution: "@adobe/css-tools@npm:4.0.1"
  # Will change when using different compressionLevels
  checksum: dd065e95339bd9b3846f3b27488c7141b8e8a1ca176eb2807fb4d18bbe817a39dac89925cf5a6de882dbc823c0ad4af44c63ff01be671e011482d768e7fb83cf
  languageName: node
  linkType: hard

[arcanis]

I assume npm tarballs are immutable and safe

Various security policies refuse to assume this (what if a malicious actor infiltrated the registry?) - additionally, it's handy when checking for corruption (I sometime edit my zip when doing some debug, the checksums prevent me from committing my changes), and it's important for non-npm registries (git repos, url tarballs, etc).

[belgattitude]

Makes sense, the non npm registries is the example that made me click the most.

Compression level to zero for both local and ci is totally an option actually.

I really appreciated your lights on this. Thanks a lot