Issue with pnpmfileChecksum and frozen-lockfile

[zkochan]

          Hey guys, I want to be clear here. We've identified that we're using 9.0.2 to both generate the lock file and we're also using 9.0.2 in the Docker image. This is still an ongoing issue.

In our local machines:

❯ sudo corepack prepare pnpm@latest --activate
Password:
Preparing pnpm@latest for immediate activation...
❯ pnpm -v
9.0.2
❯ rm -rf pnpm-lock.yaml
❯ pnpm i
Scope: all 8 workspace projects
packages/server                          |  WARN  deprecated json2csv@5.0.7
 WARN  3 deprecated subdependencies found: dommatrix@1.0.3, eivindfjeldstad-dot@0.0.1, uuid@3.3.2
Packages: +154 -150
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------------------------------------
Progress: resolved 1655, reused 1539, downloaded 0, added 151, done

In our Dockerfile:

FROM node:20.12.0-slim as corepack

ARG PACKAGE
RUN corepack enable && corepack prepare pnpm@9.0.2 --activate 

The result:

❯ docker compose build server
[+] Building 22.6s (16/27)                                                                                             docker:orbstack
 => [server internal] load build definition from Dockerfile                                                                       0.0s
 => => transferring dockerfile: 1.75kB                                                                                            0.0s
 => [server internal] load metadata for docker.io/library/node:20.12.0-slim                                                       0.5s
 => [server internal] load .dockerignore                                                                                          0.0s
 => => transferring context: 405B                                                                                                 0.0s
 => [server internal] load build context                                                                                          0.4s
 => => transferring context: 28.43MB                                                                                              0.4s
 => CACHED [server corepack 1/4] FROM docker.io/library/node:20.12.0-slim@sha256:dcb9e35d8afca163a231cdfad9657d2360947f212faf0fa  0.0s
 => [server corepack 2/4] RUN corepack enable && corepack prepare pnpm@9.0.2 --activate                                           1.2s
 => [server corepack 3/4] RUN corepack prepare npm@10.3.0 --activate                                                              1.2s
 => [server corepack 4/4] WORKDIR /app                                                                                            0.0s
 => [server ssl 1/2] RUN apt-get update && apt-get install -y openssl apt-transport-https ca-certificates curl gnupg graphicsma  15.7s
 => [server ssl 2/2] RUN apt-get update && apt-get install -y python3.11-venv                                                     3.1s
 => [server dependencies 1/8] COPY [package.json, pnpm-lock.yaml, pnpm-workspace.yaml, tsconfig.json, ./]                         0.0s
 => CANCELED [server stage-4 1/6] RUN curl -sLf --retry 3 --tlsv1.2 --proto "=https" 'https://packages.doppler.com/public/cli/gp  0.7s
 => [server dependencies 2/8] COPY packages/server/package.json ./packages/server/package.json                                    0.0s
 => [server dependencies 3/8] COPY packages/prisma ./packages/prisma                                                              0.1s
 => [server dependencies 4/8] COPY packages/server/package.json ./packages/server/package.json                                    0.0s
 => ERROR [server dependencies 5/8] RUN pnpm i --frozen-lockfile --prod                                                           0.5s
------
 > [server dependencies 5/8] RUN pnpm i --frozen-lockfile --prod:
0.381 Scope: all 3 workspace projects
0.483  ERR_PNPM_LOCKFILE_CONFIG_MISMATCH  Cannot proceed with the frozen installation. The current "pnpmfileChecksum" configuration doesn't match the value found in the lockfile
0.483
0.483 Update your lockfile using "pnpm install --no-frozen-lockfile"
------
failed to solve: process "/bin/sh -c pnpm i --frozen-lockfile --prod" did not complete successfully: exit code: 1

This is also causing build failures in Render.com, which is using the latest version of pnpm. According to your release notes it should work with a v6 lockfile, but it's not working with a v9 lockfile. We've never had these issues with pnpm@8 and we can't upgrade to v9 because we can't build our local Docker images.

Unsure if it matters, but our machines are Darwin based while our Docker images are Linux. Unsure how you're generating the pnpmfileChecksum but if that is dependent on the architecture then it could cause this problem.

Originally posted by @divmgl in #7938 (comment)

[zkochan]

@KSXGitHub could the checksum of the .pnpmfile.cjs differ in different envs?

[zkochan]

@divmgl do you have the same .pnpmfile.cjs file both locally (when you create the lockfile) and in the docker, when you run pnpm install --frozen-lockfile?

[divmgl]

@zkochan checking now

[zkochan]

I guess if git changes the line endings, the checksum could differ

[divmgl]

!!!!! @zkochan we did not! that fixed it! thanks a million

[KSXGitHub]

could the checksum of the .pnpmfile.cjs differ in different envs?

I don't think it could. And CRLFs are normalized into LFs by the hash function so it should be a problem either. But I don't remember the hash function trimming new lines, so if pnpmfile final newlines were trimmed, it could cause hash difference.

[KSXGitHub]

we did not! that fixed it! thanks a million

Oh, it was because the pnpmfiles were different?

[divmgl]

Quick observation though: while this does fix things locally, this will break Render.com builds because the .pnpmfile.cjs file is not included.

@KSXGitHub yep, we had a custom .pnpmfile.cjs file that was not included in our Docker images.

[zkochan]

Maybe we could ignore the pnpmfileChecksum check if --frozen-lockfile is used and .pnpmfile.cjs is not present.

(or add a new flag, I don't know which is better)

[divmgl]

(or add a new flag, I don't know which is better)

I think adding a flag makes more sense here since we're having these issues in cases where the pnpm options are not under our control.

Regardless, I believe I found a way to get our .pnpmfile.cjs file into the Render build. We have a monorepo and Render only uses a subset of our monorepo, so we're thinking about fixing this by moving our .pnpmfile.cjs to our package (which is called packages/app) and then doing a symlink from the root of our repo to the file in our package:

❯ ln -s packages/app/.pnpmfile.cjs .pnpmfile.cjs

That will allow us to both use pnpm i locally and pnpm i in Render without having to customize the pnpm options. So hopefully it will work. We're testing it now.

The next issue will come when we have more than one frontend package deployed on Render, as we'll then need to duplicate .pnpmfile.cjs in every package.

[zkochan]

(or add a new flag, I don't know which is better)

I think adding a flag makes more sense here since we're having these issues in cases where the pnpm options are not under our control.

If the options are not under your control then how do you expect to use the said flag?

[divmgl]

If the options are not under your control then how do you expect to use the said flag?

Ah, that's true :(. We're having a different issue now we're the .pnpmfile.cjs is not being picked up by GHA, so we're looking into that now lol

[divmgl]

We're thinking about just removing our .pnpmfile.cjs file at this point. It exists to avoid installing Webpack (as we have dependencies that are using Webpack for some reason) but it's causing more headaches for us.

[KSXGitHub]

It exists to avoid installing Webpack (as we have dependencies that are using Webpack for some reason) but it's causing more headaches for us.

Can you use overrides? Many of pnpmfile use cases can be replaced with pnpm.overrides.

[zkochan]

overrides cannot remove dependencies.

I think in this scenario it would be OK to ignore the missing pnpmfile.

[KSXGitHub]

If webpack was an optional dependencies then there's ignoredOptionalDependencies.

[zkochan]

We could add a way to remove other types of dependencies using settings. It was already requested once.

[KSXGitHub]

We could add a way to remove other types of dependencies using settings. It was already requested once.

Can and should it be integrated to overrides? For example: "webpack": "-" means removing webpack.

[zkochan]

I think that should be fine