FeatureRequest: Allow install packages only from lock file

[thynson]

Original comment
Users of docker can benifit from it, by changing their Dockerfile to

ADD pnpm-lock.yaml ./
RUN pnpm install --ci --only-from-lockfile # Proposed option for this feature
ADD package.json ./
RUN pnpm test-lockfile # Need to verify the lockfile is correct
ADD . ./

instead of the current popular pattern:

ADD package.json pnpm-lock.yaml ./
RUN pnpm install --frozen-lockfile
ADD . ./

The new Dockerfile pattern will allow pnpm install --ci --only-from-lockfile layer be reused even when package.json is modified, as long as lockfile is not modified, e.g. version bump.

[thynson]

I think there are many things need to considered, e.g. lifecycle script, and maybe monorepo will also be affected.

[zkochan]

Well, w/o a package.json you cannot have scripts. But installation of dependencies should be possible w/o the manifest. In monorepo as well, all the necessary information is present in the lockfile

[thynson]

It may be possible to create a new command pnpm postinstall that verify the lockfile and run scripts. The Dockerfile should like:

ADD pnpm-lock.yaml ./
RUN pnpm install --ci --only-from-lockfile # Proposed option for this feature
ADD . ./ # package.json will be included
RUN pnpm postinstall # Verify the lockfile and run scripts

[zkochan]

I am not sure what advantage this gives you then.

[thynson]

The new Dockerfile pattern will allow pnpm install --ci --only-from-lockfile layer be reused even when package.json is modified, as long as lockfile is not modified, e.g. version bump.

To boost the Docker build process.
Docker build images by layering content generated each step in the Dockerfile. That is why common pattern of Dockerfile looks like this:

ADD package.json package-lock.json ./  
# npm ci will also be cached if package.json package-lock.json is not modified
RUN npm ci 
ADD . ./

instead of the naive way

# Add all the things of the project. If any files of your project, the digest of above 
# layer is changed and all following steps can not be cached any more
ADD . ./  

 # You will reinstall all packages even if you just changed a bit of your code
RUN npm ci

While even using the suggested pattern, anyone modify the package.json for just adding a script or just bump the version will also make cache invalided. Since installing pacakges is the step consumes most of the time, it's worth to seperate installing packages as a single step to be cache friendly.

[thynson]

Actually this feature will made monorepo be easily build by docker. Currently, it's very hard to write and maintains a Dockerfile for monorepo using the pattern described before.

[kaidjohnson]

Currently, our first docker layer for all builds within our monorepo is the pnpm install layer, which requires copying both the lockfile and all package.json files in the repo. If a script on one of those packages changes, or even some other arbitrary metadata, but dependencies are left untouched, the entire build cache is invalidated.

I've toyed with switching from a shared lockfile to individual package locks for better control over cache-invalidation, but the initial development install performance is 5-10x slower, which could result in moving the problem from our CI pipelines to local developer machines.

I've also looked into running a filtered install based on the specific application that is being built, but pnpm throws a slew of warnings when packages declared in the workspace lockfile are not included, even if those packages are not part of the required dependency tree, resulting in reduced confidence in our production builds.

For each approach I've looked into, I keep coming back to this topic. If we were able to install dependencies from the lockfile only, the problem is largely mitigated, resulting in hours of weekly CI congestion being removed.

Running postinstall after copying the package.json files for the entire monorepo only allows the dependency download layer not to be cache-invalidated, but still has the problem of all downstream-builds being invalidated should arbitrary package metadata have been changed within the workspace. I suppose at that point, postinstall could be run using filters to target just the build slice that is needed for the specific image, allowing some improvement in docker layering strategies that could further improve cache-invalidation problems. Now that I think about it, postinstall would only need to run postinstall scripts for workspace/local packages, as installing external dependencies from the npm repository would include their package.json files already, so all third-party dependency install would be completely cached in the initial pnpm install --from-lockfile layer. Given that line of thinking, we wouldn't even need a special postinstall command from pnpm, as we can already manually run postinstall scripts on workspace-local packages.

[zkochan]

I think there are too many edge cases. Like the bin field for instance. I don't think duplicating a lot of fields in the lockfile is a good idea. Maybe we could create a new manifest file. Something like dev-package.json that would contain some additional fields that are not needed in the docker image. Then package.json would be modified less frequently.

[thynson]

#1670 could be another way to boost the docker build process.

If we were able to save all required package to local store at the first time, the install can be done offline.

COPY pnpm-lock.yaml ./
RUN pnpm store add-from-lock-file pnpm-lock.yaml # Proposed command for #1670
COPY . .
RUN pnpm install --prefer-offline

As long as the lockfile is not changed, the cache will be still valid and the actual install process can be done without network I/O.

[kaidjohnson]

That solution gets to the heart of it. It's less about the install and more about the dependency download required for install. 👍

[thynson]

Just tried with following Dockerfile.

FROM node:14-buster

# yq is not availabe from apt yet, install it by copying from the docker image published by its author
COPY --from=mikefarah/yq /usr/bin/yq /usr/bin/yq

RUN npm install -g pnpm

WORKDIR /opt/app

COPY pnpm-lock.yaml ./

# Parse the lockfile add them to store
RUN pnpm store add $(yq -M  eval '.packages | keys' pnpm-lock.yaml | sed 's/- \///g' | sed -r 's/\/([^_/]+)(_[^_]+)?$/@\1/g')

COPY package.json pnpmfile.js ./

RUN pnpm i --offline

COPY . ./

RUN pnpm run build

Now the building process is much improved, note that the step RUN pnpm m i --offline could still take about one minutes (around 30s for pnpm to construct virtual store with 1300+ packages and 30s for packages that are building from source like sqlite3 and etc).

For monospace project, since there is no easy way to copy only package.json while keep the directory structure, the following Dockerfile works at the best effort.

FROM node:14-buster

# yq is not availabe from apt yet, install it by copying from the docker image published by its author
COPY --from=mikefarah/yq /usr/bin/yq /usr/bin/yq

RUN npm install -g pnpm

WORKDIR /opt/app

COPY pnpm-lock.yaml ./

# Parse the lockfile add them to store
RUN pnpm store add $(yq -M  eval '.packages | keys' pnpm-lock.yaml | sed 's/- \///g' | sed -r 's/\/([^_/]+)(_[^_]+)?$/@\1/g')

COPY . ./

RUN pnpm m i --offline

RUN pnpm run build

@kaidjohnson Hope this could help you.

[jetzhliu]

There is an easy way to find dependencies in lock file:

sed -n 's/^  \/\(.*\)\/\([^\/]*\):$/\1@\2/p' pnpm-lock.yaml

[kaidjohnson]

Thanks, @thynson! This approach looked promising, but I see the install from a frozen-lockfile taking about half the time as installing to the store and then installing offline. Even if we do get the store hydration layer cached by docker, the offline install is within a margin of error of the frozen lockfile install, so the benefit is undermined by sometimes needing to re-hydrate the store, so it's more expensive to do it this way, unfortunately.

Measures:

Before
pnpm i -r --frozen-lockfile : ~2:00

After
pnpm store add $(...from lockfile...): ~1:55
pnpm i -r --offline : ~2:05

[thynson]

Yeah, it seems that your CI environment have much better network bandwidth than ours.
On my environment:

Before:
pnpm i -r --frozen-lockfile : ~3:30
After:
pnpm store add $(...from lockfile...): ~3:00
pnpm i -r --offline : ~1:15

While offline installation on host cost around 15s, it seems that the virtual store consturction is relative slow in docker, possibily due to high hard-link creation cost on layered filesystem. See spotify/docker-client#1158.

[thynson]

Just done an experiment:

FROM node:14-buster 
RUN npm install -g pnpm
WORKDIR /opt/app

COPY . ./

# Install all packages and then remove all contents except the virtual store
RUN pnpm m i \
  && mv node_modules/.pnpm /virtual-store \
  && rm -rf * \
  && mkdir node_modules && mv /virtual-store node_modules/.pnpm

COPY . ./

# Here the virtual store is already presented
RUN pnpm m i --offline

Output (You need to scroll to the most right side to see time costed by each step):

[+] Building 38.8s (12/12) FINISHED                                                                                                                                                                                       
 => [internal] load build definition from Dockerfile                                                                                                                                                                 0.0s
 => => transferring dockerfile: 292B                                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                                    0.0s
 => => transferring context: 35B                                                                                                                                                                                     0.0s
 => [internal] load metadata for docker.io/library/node:14-buster                                                                                                                                                    1.6s
 => [internal] load build context                                                                                                                                                                                    0.1s
 => => transferring context: 12.26kB                                                                                                                                                                                 0.1s
 => CACHED [1/7] FROM docker.io/library/node:14-buster@sha256:810fc73fffa02751facec95f355202b2976441ac772bdb4bd650bd1b87e88776                                                                                       0.0s
 => [2/7] RUN npm install -g pnpm                                                                                                                                                                                    2.3s
 => [3/7] WORKDIR /opt/app                                                                                                                                                                                           0.0s
 => [4/7] COPY . ./                                                                                                                                                                                                  0.1s
 => [5/7] RUN pnpm m i   && mv node_modules/.pnpm /virtual-store   && rm -rf *   && mkdir node_modules && mv /virtual-store node_modules/.pnpm                                                                      22.1s
 => [6/7] COPY . ./                                                                                                                                                                                                  0.1s 
 => [7/7] RUN pnpm m i --offline                                                                                                                                                                                     2.3s 
 => exporting to image                                                                                                                                                                                              10.3s 
 => => exporting layers                                                                                                                                                                                             10.3s 
 => => writing image sha256:bca482f2a5f236d3827ad7e3add9db308c6647753223b9725edb27be7245db49                                                                                                                         0.0s 

It concludes that if the virtual store is present, and then the installation process can be done very quickly.
@zkochan Can a virtual store be constructed with only docker file?

[thynson]

I've created PR #3294 for this feature.