v7.20.0 fetch not populating store-dir in Docker

[72636c]

pnpm version: 7.20.0–7.26.3

Code to reproduce the issue:

We have a pattern of building a base Docker image using pnpm fetch then running a containerised pnpm install --offline --recursive in subsequent steps. This allows us to amortise dependency retrieval across build agents/steps and avoid awkwardly mounting nested node_modules in monorepos.

The following pseudocode works in v7.19.0 and below:

# docker build
RUN pnpm config set store-dir /workdir/.pnpm-store
RUN pnpm fetch

# docker run --volume /workdir/.pnpm-store
pnpm install --offline --recursive

I've created a minimal repro in GitHub Actions:

• Repository: https://github.com/72636c/pnpm-no-offline-tarball
• Workflow: https://github.com/72636c/pnpm-no-offline-tarball/actions/runs/3819799104

Expected behavior:

+ Falling back to copying packages from store
+ Packages are copied from the content-addressable store to the virtual store.
+   Content-addressable store is at: /workdir/.pnpm-store/v3
+   Virtual store is at:             node_modules/.pnpm
+ Done in 721ms

Actual behavior:

-  ERR_PNPM_NO_OFFLINE_TARBALL  A package is missing from the store but cannot download it in offline mode. The missing package may be downloaded from https://registry.npmjs.org/@types/node/-/node-18.11.18.tgz.

Additional information:

• node -v prints: v18.12.1
• Windows, macOS, or Linux?: Linux

[garth]

I have the same issue. It seems that the setting for the store location is not respected by pnpm install any more.

First set the store location:

pnpm config set store-dir /usr/.pnpm-store

The run pnpm fetch and you see:

Packages are hard linked from the content-addressable store to the virtual store.
  Content-addressable store is at: /usr/.pnpm-store/v3
  Virtual store is at:             node_modules/.pnpm

But then run pnpm install and you see:

Packages are hard linked from the content-addressable store to the virtual store.
  Content-addressable store is at: /root/.local/share/pnpm/store/v3
  Virtual store is at:             node_modules/.pnpm

The content addressable store for pnpm install is (I'm guessing) the default value and not the value set in the config.

[dkozickis]

Reproducible on 7.24.3

[garth]

Just tested in v7.26... still broken

[zkochan]

Does it work if you replace pnpm config set with npm config set?

[garth]

Yes, using npm config set fixes the issue. After making this change, both pnpm fetch and pnpm install see the same value.

Should we no longer be using pnpm config set?

[zkochan]

pnpm config set should be fixed. For now you can use npm config set instead.

[garth]

But it seems that some functions (eg pnpm fetch) are able to read the config set by pnpm config set. So maybe the issue is elsewhere?

[zkochan]

In 7.20 we added our own implementation of pnpm config. Before that, pnpm config was just passing through to npm config.

[72636c]

72636c/pnpm-no-offline-tarball#1 looks happy, thanks for the workaround!

E: worth noting that this will only work with npm <9 as per #5621

[72636c]

This is a bit more worrying now that Node.js 18.14.0 (Active LTS) bundles npm 9.3.1.

[garth]

E: worth noting that this will only work with npm <9 as per #5621

Just got blocked by this one since we updated our builds to use node v18.14.0 today.

npm ERR! `store-dir` is not a valid npm option

[tadhglewis]

I'm running into this issue when migrating to pnpm as well :/

[zkochan]

Fixed in v7.28.0 via #6132

[deadcoder0904]

i still get this error:

FROM node:20-alpine AS base

# mostly inspired from https://github.com/BretFisher/node-docker-good-defaults/blob/main/Dockerfile & https://github.com/remix-run/example-trellix/blob/main/Dockerfile

# Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.
RUN apk add --no-cache libc6-compat
RUN corepack enable && corepack prepare pnpm@8.15.3 --activate 
# set the store dir to a folder that is not in the project
RUN pnpm config set store-dir ~/.pnpm-store
RUN pnpm fetch

# 1. Install all dependencies including dev dependencies
FROM base AS deps

# Root user is implicit so you don't have to actually specify it. From https://stackoverflow.com/a/45553149/6141587
# USER root
# RUN mkdir -p /app
# RUN chown -R node:node /app
USER node
# WORKDIR now sets correct permissions if you set USER first
WORKDIR /app

# Install dependencies based on the preferred package manager
COPY --chown=node:node package.json pnpm-lock.yaml* ./
COPY --chown=node:node /src/app/db/migrations ./migrations

USER root
RUN pnpm install --recursive --offline
USER node

# 2. Setup production node_modules
FROM base as production-deps
WORKDIR /app

COPY --from=deps /app/node_modules ./node_modules
COPY --chown=node:node package.json pnpm-lock.yaml* ./
RUN pnpm prune --prod

# 3. Rebuild the source code only when needed
FROM base AS builder
WORKDIR /app
COPY --from=deps --chown=node:node /app/node_modules ./node_modules

COPY --chown=node:node . .

# This will do the trick, use the corresponding env file for each environment.
COPY --chown=node:node .env.production .env.production
RUN mkdir -p /data

ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1

# Create /data/users.prod.sqlite using Volume Mount
# RUN npm run db:migrate:prod
RUN pnpm build

# 3. Production image, copy all the files and run next
FROM base AS runner
WORKDIR /app

ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1

# RUN addgroup -g 1001 -S nodejs
# RUN adduser -S nextjs -u 1001

COPY --from=builder --chown=node:node /app/public ./public
COPY --from=production-deps --chown=node:node /app/node_modules ./node_modules
# COPY --from=builder --chown=node:node /data /data

# Automatically leverage output traces to reduce image size
# https://nextjs.org/docs/advanced-features/output-file-tracing
COPY --from=builder --chown=node:node /app/.next/standalone ./
COPY --from=builder --chown=node:node /app/.next/static ./.next/static

# Move the drizzle directory to the runtime image
COPY --from=builder --chown=node:node /app/src/app/db/migrations ./migrations

# Move the run script and litestream config to the runtime image
COPY --from=builder --chown=node:node /app/scripts/drizzle-migrate.mjs ./scripts/drizzle-migrate.mjs
# COPY --from=builder --chown=node:node /app/scripts/init.sh ./init.sh
COPY --from=builder --chown=node:node /app/scripts/run.sh ./run.sh
# RUN chmod +x init.sh
RUN chmod +x run.sh

# RUN chown -R node:node /app/node_modules
# Create data directory or else `npm run db:migrate:prod` will fail with TypeError: Cannot open database because the directory does not exist
# RUN mkdir -p /data

# USER nextjs

EXPOSE 3000

# ENV PORT 3000
# ENV HOSTNAME localhost

# CMD ["npm", "run", "start"]
# ENTRYPOINT ["init.sh"]

CMD ["sh", "run.sh"]

The error:

#14 [web deps 4/4] RUN pnpm install --recursive --offline
#14 0.846 Lockfile is up to date, resolution step is skipped
#14 0.886 Progress: resolved 1, reused 0, downloaded 0, added 0
#14 1.087 .                                        | +334 ++++++++++++++++++++++++++++++++
#14 1.291 undefined
#14 1.291  ERR_PNPM_NO_OFFLINE_TARBALL  A package is missing from the store but cannot download it in offline mode. The missing package may be downloaded from https://registry.npmjs.org/@t3-oss/env-nextjs/-/env-nextjs-0.9.2.tgz.
#14 ERROR: process "/bin/sh -c pnpm install --recursive --offline" did not complete successfully: exit code: 1
------
 > [web deps 4/4] RUN pnpm install --recursive --offline:
0.846 Lockfile is up to date, resolution step is skipped
0.886 Progress: resolved 1, reused 0, downloaded 0, added 0
1.087 .                                        | +334 ++++++++++++++++++++++++++++++++
1.291 undefined
1.291  ERR_PNPM_NO_OFFLINE_TARBALL  A package is missing from the store but cannot download it in offline mode. The missing package may be downloaded from https://registry.npmjs.org/@t3-oss/env-nextjs/-/env-nextjs-0.9.2.tgz.
------
failed to solve: process "/bin/sh -c pnpm install --recursive --offline" did not complete successfully: exit code: 1
make: *** [Makefile:3: build-production] Error 17

How do I fix it?

npm config set doesn't work. I get invalid option. I tried npm config set registry but then it wants https url.

[deadcoder0904]

I think my issue is a separate one so I'll open a new one.

For now, I removed --recursive --offline flag as it was the one causing errors.

These 2 commands are working fine after removing them:

RUN pnpm config set store-dir ~/.pnpm-store
RUN pnpm fetch