Current understanding of how pnpm works internally

When performing an install, pnpm kicks off a bunch of asynchronous "fetchers". This can perform various IO like downloading tarballs. It also can load packages internally (directory fetcher).

This work can be incredibly IO heavy, and is not immediately needed, but it kicked off early to optimize performance.
This is the line where the "forking" happens: https://github.com/pnpm/pnpm/blob/main/pkg-manager/package-requester/src/packageRequester.ts#L348
doFetchToStore is itself asynchronous, but does not block execution. Instead it reports success via deferred promises.

NOTE this means that the fetchers can fail early in install, but not have the failure reported until later (once actually used)

Later on in the install, the "fetch" result is actually referenced https://github.com/pnpm/pnpm/blob/main/pkg-manager/core/src/install/link.ts#L428

At an even later step, the local packages are sym-linked together: https://github.com/pnpm/pnpm/blob/main/pkg-manager/core/src/install/link.ts#L273

The regression occurs because the directory fetcher expects the dependencies to be sym-linked, which does not happen until after the fetch is complete. Which is a circular dependency, and fails.

Solution

There are two parts of this.

First, we need to defer the execution of the directory fetcher. We don't want to delay all fetchers (e.g. tarballs) because that provides significant performance boost and do not have a similar dependency issue.

Second, we need to perform the sym-linking before the fetching is called, that will ensure the local packages are "installed" before being referenced.

Opened up PR adding a test to demonstrate this regression: https://github.com/pnpm/pnpm/pull/7048/files

Note question regarding expected format of package.json of injected dependencies.

If we expect this package to be "exportable" (e.g. publishConfig is applied and workspace:* is resolved) then something like the solution above is required.

If we do not expect the fetched directory to be "exportable", then the solution can perhaps be simpler and simply move the logic to a deploy-specific method

Opened a separate PR as a possible implementation of the solution.

There are couple parts that feel a bit hacky... trying to delay invocations of promises and immediately invoking in others...

Definitely open to alternate solutions or implementationos

A reported failure in keycloak is further impacted by this issue.

Even once the dependencies are resolved, it reports not-installed.
This is caused by the shared directory not having a version field and failing this check

That fix is trivial enough on their side, but probably could benefit from better messaging. Can consider as improvement to DX once solution is implemented

This issue was fixed in this PR #7058

nacho@Ignacios-MacBook-Pro pnpm-install-error-example % pnpm i
Scope: all 4 workspace projects
Packages: +3
+++
Progress: resolved 3, reused 2, downloaded 1, added 3, done
package-b prepare$ pnpm run build
│ > @namespace/package-b@1.0.0 build /private/tmp/pnpm-install-error-example/package-b
│ > tsc ./src/index.ts --outDir ./dist
└─ Done in 1s
Done in 1.5s
nacho@Ignacios-MacBook-Pro pnpm-install-error-example % pnpm -v
8.7.4