Not working with PNPM

[theprobugmaker]

Bug report

Describe the bug

I tried using Next.js with Rush.js which is a tool that helps developers create and maintain monorepo projects. It has support for NPM, Yarn and PNPM.
I tried using with PNPM cause it's really good compared to Yarn and NPM but I was surprised that it wasn't working with Next.js.
Next.js is working only with Yarn and NPM and it appears that Next.js's server-renderer bundler is not correctly externalizing modules with PNPM.

To Reproduce

https://github.com/zefexdeveloper/nextjs-pnpm

You can clone the repository and follow the readme. There's also a yarn branch with a working example.

Expected behavior

It should work just like with Yarn/NPM.

Additional context

I would like to invite @octogonz (Rush.js), @timneutkens @Timer (Next.js) and @zkochan (PNPM). Great people that are related to those projects and it would be amazing to hear from them to see what might be the problem.

[octogonz]

From a preliminary investigation, we found that Next.js's server-renderer bundler is not correctly externalizing modules installed by PNPM. For example, if we compare the file .next\server\pages\index.js between the Yarn vs PNPM branches from @zefexdeveloper's repro:

For Yarn we will see this:

.next\server\pages\index.js

/***/ "react":
/*!************************!*\
  !*** external "react" ***!
  \************************/

...whereas for PNPM we see this:

.next\server\pages\index.js

/***/ "./node_modules/.pnpm/react@16.13.1/node_modules/react/cjs/react.development.js":
/*!**************************************************************************************!*\
  !*** ./node_modules/.pnpm/react@16.13.1/node_modules/react/cjs/react.development.js ***!
  \**************************************************************************************/
/*! no static exports found */
/***/ (function(module, exports, __webpack_require__) {

"use strict";
/** @license React v16.13.1
 * react.development.js

...along with full copies of every NPM dependency, copied inline into every .js file. For example the same is seen in other files .next\server\pages\_document.js, .next\server\pages\_app.js, etc.

As a result, React reports a warning about hooks. The warning is complaining that multiple instances of React are getting loaded (into the Node.js server-side renderer, not the web browser runtime). Next.js seems to be using Webpack to bundle everything, so maybe it is just a configuration problem. Webpack itself certainly is compatible with PNPM.

[octogonz]

The webpack configuration is here: packages/next/build/webpack-config.ts

It references packages/next/lib/resolve-request.ts, which looks to be doing custom module resolution. Perhaps it does not handle symlinks correctly according to the Node.js standard.

[octogonz]

I also found an older PR #8668 where @arcanis fixed Next.js to work with Yarn Plug'n'Play. It is fortunately a pretty small diff. @zkochan I bet these are the exact same places that need to be fixed up to eliminate the incompatibility with PNPM. Maybe this is easy to solve!

[theprobugmaker]

Thank you @bitfrost for the PR, I hope that @Timer or @timneutkens can merge.

[octogonz]

I see that PR #16614 was closed recently.

Is PR #16369 expected to fix this issue?

[elliottsj]

Since #17279 was merged, Next.js works with pnpm.

I proposed some integration tests to help ensure pnpm continues to work in the future: #17882

However, there are some outstanding issues using Next.js on Vercel with pnpm, likely due to how Vercel's file system works:

• Support for pnpm as an alternative to Yarn vercel#3896 (reply in thread)
• https://github.com/elliottsj/nextjs-vercel-pnpm-issue

[Timer]

Closing as fixed by #17279

[vasiliicuhar]

@Timer I think that has been partially fixed. It still does not work when deploying to Vercel.
At first I thought it was because of Rushjs, but it does not work with Pnpm without Rush as well

Error: ENOENT: no such file or directory, lstat '/vercel/workpath0/app/url'
--
15:28:10.448 | at realpathSync (fs.js:1643:7)
15:28:10.448 | at handleExternals (/vercel/workpath0/app/node_modules/.pnpm/next@10.0.5_react-dom@17.0.1+react@17.0.1/node_modules/next/dist/build/webpack-config.js:66:21)

I spent a lot of time yesterday trying to reproduce it locally but I have no idea what is different between my environment and Vercel build pipeline.

What I tried so far:

• adding url and querystring dependencies to next using readPackage hook
• changing rush / Pnpm default settings
• in case it's a difference between Mac and Linux, I tried to install packages and build it in a docker container image node:12

Workaround:
Install url and querystring packages as direct dependencies of the project seems to help, but I'd really like to understand why Vercel environment is so specific, since everywhere else #17279 seems to work

Versions:
next@10.0.5, react-dom@17.0.1, react@17.0.1

package.json

{
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start"
  },
  "dependencies": {
    "next": "~10.0.5",
    "react": "~17.0.1",
    "react-dom": "~17.0.1",
    "postcss": "~8.2.4",
    "tailwindcss": "~2.0.2",
    "autoprefixer": "~10.2.1"
  }
}

[elliottsj]

@vasiliicuhar try this: #20949

[danvoyce]

This is still very broken... Here's a minimal reproducible repo https://github.com/looopdotdev/looop-2.1

I override the install command in Vercel like so npm i pnpm -g && pnpm i. (I've also simply tried pnpm i, but the build errors)

All works fine with my app, until I add getServerSideProps() {... in one of the page components. As soon as I add that I get a big 500 - internal server error on the deployed page.

https://looop-2-1.vercel.app/

Please help. I'm about to go live with my site and need to simply add some meta data. I really don't want to have to move away from pnpm or Vercel.

Thanks 🙏

[PabloSzx]

This is still very broken... Here's a minimal reproducible repo https://github.com/looopdotdev/looop-2.1

I override the install command in Vercel like so npm i pnpm -g && pnpm i. (I've also simply tried pnpm i, but the build errors)

All works fine with my app, until I add getServerSideProps() {... in one of the page components. As soon as I add that I get a big 500 - internal server error on the deployed page.

https://looop-2-1.vercel.app/

Please help. I'm about to go live with my site and need to simply add some meta data. I really don't want to have to move away from pnpm or Vercel.

Thanks 🙏

It seems like adding "--shamefully-hoist" in the install command fixes it

EDIT: DOESN'T WORK FOR Next.js 12 Serverless Functions in Vercel

EDIT2: IT WORKS AGAIN FOR Next.js 12.0.3 🎉

[danvoyce]

It seems like adding "--shamefully-hoist" in the install command fixes it

Dude, thanks so much. Literally save my bacon!!!

[elliottsj]

These are my Vercel command overrides which seem to work (for https://github.com/elliottsj/elliott.dev):

@danvoyce I'm curious about the underlying error for your 500 page. Anything in your app's logs?

[PabloSzx]

@danvoyce I'm curious about the underlying error for your 500 page. Anything in your app's logs?

I could reproduce this issue and I got the most useless error possible

These are my Vercel command overrides which seem to work (for https://github.com/elliottsj/elliott.dev):

I feel like that recommendation is not the best one, this uses all the pnpm cache and everything seems to be working fine, very fast builds (faster than yarn & npm)

npx pnpm install --store=node_modules/.pnpm-store --shamefully-hoist

[elliottsj]

@PabloSzx I suppose it's a tradeoff. Unfortunately --shamefully-hoist provides fewer correctness guarantees than pnpm's default strict behaviour.

For example, let's say your project declares a dependency on foo@1, and foo@1 has a dependency on bar@2.

With --shamefully-hoist, your project can require('bar') without error since it's hoisted to node_modules/bar/.

But if foo@1.1 is released (a minor bump) with a new dependency on bar@3 (a major bump), then your project's require('bar') can break the app in unexpected ways. With pnpm's default strict behaviour, this issue is caught at build time.

If forced to choose between --shamefully-hoist and slower builds due to being unable to use cached node_modules, I would probably choose the latter.

Of course, ideally Vercel should cache node_modules correctly with pnpm without having to resort to --shamefully-hoist.

[SirensOfTitan]

Just to add to data here: We're running Next on a non-Vercel host, and our app also doesn't work without use of the --shamefully-hoist.

This issue should probably be reopened, or alternatively another issue cut. It feels like a fairly high-quality bug with easy reproduction and a couple canned projects that show the issue.

[dotsinspace]

This is still very broken... Here's a minimal reproducible repo https://github.com/looopdotdev/looop-2.1
I override the install command in Vercel like so npm i pnpm -g && pnpm i. (I've also simply tried pnpm i, but the build errors)
All works fine with my app, until I add getServerSideProps() {... in one of the page components. As soon as I add that I get a big 500 - internal server error on the deployed page.
https://looop-2-1.vercel.app/
Please help. I'm about to go live with my site and need to simply add some meta data. I really don't want to have to move away from pnpm or Vercel.
Thanks pray

It seems like adding "--shamefully-hoist" in the install command fixes it

vercel CI goes into infinte loop. even this is not working

[bitfrost]

Hmm, we have been using pnpm with no issues for a long time with no issues in many projects with next.js since the above issue was fixed (no use of --shamefully-hoist latest versions of pnpm) next. 10.1.3 (I will update / try the above repo later today).

[elliottsj]

For what it's worth, I opened a PR a while ago to fix a specific issue which surfaces when using pnpm: #21048

May or may not be related to the issues people are experiencing here, but the PR includes updates to the integration tests which should help catch many potential problems. If a maintainer has time to comment on it, it would be appreciated.

[PabloSzx]

The --shamefully-hoist workaround seems to be broken for Next.js 12 in Serverless Functions in Vercel, RIP

Can this start having a bigger priority? @Timer @styfle

EDIT: It seems like in 12.0.3 it was fixed, I think the issue was related to #30786 🎉

[balazsorban44]

This issue has been automatically locked due to no recent activity. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.