make-dedicated-lockfile doesn't recognize workspaces packages, tries to download from npm registry

[rdsedmundo]

pnpm version: 6.3.0

Code to reproduce the issue:

1- Create a workspace with packages A and B.
2- Import A in B (add it as a dependency).
2- Run make-dedicated-lockfile.

Expected behavior:

Workspaces should be recognized and linked internally.

Actual behavior:

pnpm doesn't recognize the workspaces packages and tries to download them from npm.

 ERROR  GET https://registry.npmjs.org/@acme%2Flib-client: Not Found - 404
@acme/lib-client is not in the npm registry, or you have no permission to fetch it.

No authorization header was set for the request.

Additional information:

• node -v prints: 14.16.1
• Windows, macOS, or Linux?: macOS

Related #2198

[owl-from-hogvarts]

Linked via global store packages have same behavior: fail even if they are in global node_modules and versions are satisfied.

[TeChn4K]

Has anyone found a workaround?

[zkochan]

Maybe we can do or implement a workaround using injected dependencies: https://pnpm.io/package_json#dependenciesmetainjected

[xpluscal]

@zkochan is there an update on this issue?

[xpluscal]

or @TeChn4K @rdsedmundo did you guys find a successful workaround?

[commoddity]

Having this same issue. Wondering if anybody has found a solution?

[zkochan]

Try to use the pnpm deploy command instead.

[TeChn4K]

This new deploy command is a great news, especially for building docker images of single packages.
Thanks @zkochan for taking care of this.

[rhyek]

I don't understand how pnpm deploy is meant to replace make-dedicated-lockfile. I am building several docker images in a matrix in github actions. for each app I am only copying into the docker image the folder belonging to that app as well as folders belonging to its workspace production dependencies.

I need to run pnpm i --prod inside the docker image during build, but since some apps have workspace Dev dependencies (which I am not copying), pnpm fails since it thinks it needs to get them from the npm registry (issue reported here).

It doesn't fail, however, if I also copy into the image the lockfile. npm i --prod then works. However that lockfile is for the whole monorepo. I need a dedicated one so as to not interfere with docker layer caches.

How does pnpm deploy help for my use case?

[rhyek]

I think the key here is I am doing application builds inside the docker image builds which are in turn done in parallel using a github actions matrix, as mentioned. I believe this is not compatible with the idea of pnpm deploy which I think wants you to first build your apps on your host, run pnpm deploy, then copy the result to a docker image.

But this does not take into consideration that the host might have a different OS or cpu architecture. On the other hand, in the case of multi stage docker builds where you build the app and pnpm deploy in an initial stage, docker layer caches are affected.

[matzeeable]

@rhyek I think pnpm deploy is the perfect command for your use case. In general, it tries to isolate a single package with all its dependencies (including workspace dependencies) into a folder which is completely unrelated to the monorepo. This can become very helpful with Docker Multi Stage Builds, as also described here.

Unfortunately, I also run into the issue with npm registry when using the --prod flag as described in the doc:

In case you build your project before deployment, also use the --prod option to skip devDependencies installation.

In our case, we build the typescript project before with tsc and need to run the deploy command with --prod. It runs into the following error:

When using without --prod (installing dev dependencies) it works. Unfortunately, I notice here that workspace dependencies are not copied, but are still symlinked. This means: In a Docker Multi Stage Build, the command COPY --from=pruned /app/pruned . will not be sufficient because workspace dependencies are missing.

EDIT: I found #5078 which describes that workspace dependencies should be injected. However, this only works with the workspace: protocol (107d011#diff-ae95f3a0ecedfd96f49ca26a073cc0d91e67c12ae49b97497ecad63134d877a8R7-R11).

If you want to make pnpm deploy make work without workspace: protocol and make the result COPYable by Dockerfile, check this:

Expand me

/**
 * @see https://github.com/pnpm/pnpm/blob/8e5b77ef61d7cab6ed914cbf1d4ef69acd4c8ad0/packages/types/src/misc.ts#L4
 */
const DEPENDENCIES_FIELDS = ["optionalDependencies", "dependencies", "devDependencies"];
const MAIN_PKG_SCOPE = "@your-scope";

module.exports = {
    hooks: {
        readPackage: (pkg) => {
            /**
             * Inject when using Deploy as it currently does only support the `workspace:` protocol:
             *
             * @see https://github.com/pnpm/pnpm/issues/3442#issuecomment-1228105319
             * @see https://github.com/pnpm/pnpm/issues/3114#issuecomment-1192654437
             * @see https://github.com/pnpm/pnpm/commit/107d01109adeee50a40b749d5bbe4bc3eb3bc068#diff-ae95f3a0ecedfd96f49ca26a073cc0d91e67c12ae49b97497ecad63134d877a8R7-R11
             */
            if (process.env.PNPM_INJECT_WITHOUT_WORKSPACE_PROTOCOL) {
                pkg.dependenciesMeta = pkg.dependenciesMeta || {};
                for (const depField of DEPENDENCIES_FIELDS) {
                    for (const [depName] of Object.entries(pkg[depField] ?? {})) {
                        if (depName.startsWith(MAIN_PKG_SCOPE)) {
                            pkg[depField][depName] = "workspace:*";

                            // This does not work as expected, always receiving an "is not in the npm registry" error
                            /*pkg.dependenciesMeta[depName] = {
                                injected: true
                            };*/
                        }
                    }
                }
            }
            return pkg;
        }
    }
};

Afterwards, run PNPM_INJECT_WITHOUT_WORKSPACE_PROTOCOL=1 pnpm --filter <your-package> deploy pruned.

Unfortunately, pnpm --filter <your-package> --prod deploy pruned still does not work and results in "is not in the npm registry" errors. @zkochan any ideas? 😊

EDIT 2: Figured out that only peerDependencies are affected by the NPM registry issue. Fixed it by removing the peerDependencies from package.json before pnpm deploy.

[codepunkt]

@matzeeable I don't see how this is helpful with multi-stage docker builds. As far as I've understood, we need to at least install all pnpm dependencies based on the toplevel lockfile first, and then continue on using pnpm deploy. This would re-install the dependencies for the whole monorepo on every build of an individual app or service in the first docker build stage, which is exactly what we tried to avoid using make-dedicated-lockfile.

[0x80]

I have created isolate-package which I think might be useful to you.

It was quite a trip trying to figure this out isolating the PNPM lockfile. First I started with make-dedicated-lockfile but that was not the right path, and then tried multiple other approaches but eventually the solution was not that complicated.

I will write about it in the docs at some point, but I have other priorities now, and I doubt anyone will be interested in the details :)

NPM and Yarn v1 are also supported. For modern versions of Yarn I plan to make a fallback to generating NPM lockfiles. NPM has this built-in tool called Arborist and it does almost exactly what I wanted, but still required a little hack moving the node_modules from the root and back again.

Anyway, I hope you find it useful. I needed it for Firebase deployments. I have created a monorepo boilerplate that showcases how it can be used in that context.