9.5.0 introduced compile error / breaks when using the pnpm package manager #15616

bitfrost · 2020-07-29T00:58:48Z

Bug report

Upgrading a hello world project from 9.4.4 to 9.5.0 when using pnpm introduces a compile error for next build (see below)

To Reproduce

install pnpm on your system.

clone example repo https://github.com/bitfrost/example_pnpm_next_issue
cd into repo

run:

pnpm install
pnpm next build

Get

next "build"

Creating an optimized production build

Compiled successfully.

Automatically optimizing pages ..
Error occurred prerendering page "/404". Read more: https://err.sh/next.js/prerender-error
Error: Minified React error #321; visit https://reactjs.org/docs/error-decoder.html?invariant=321 for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
at Z (/Users/kobrien/dev/nx-5/.next/server/pages/_document.js:1028:404)
at module.exports.mbhs.exports.useContext (/Users/kobrien/dev/nx-5/.next/server/pages/_document.js:1033:261)
at Html (/Users/kobrien/dev/nx-5/.next/server/pages/_document.js:257:29)
at d (/Users/kobrien/dev/nx-5/node_modules/.pnpm/react-dom@16.13.1_react@16.13.1/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:36:498)
at $a (/Users/kobrien/dev/nx-5/node_modules/.pnpm/react-dom@16.13.1_react@16.13.1/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:39:16)
at a.b.render (/Users/kobrien/dev/nx-5/node_modules/.pnpm/react-dom@16.13.1_react@16.13.1/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:44:476)
at a.b.read (/Users/kobrien/dev/nx-5/node_modules/.pnpm/react-dom@16.13.1_react@16.13.1/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:44:18)
at renderToStaticMarkup (/Users/kobrien/dev/nx-5/node_modules/.pnpm/react-dom@16.13.1_react@16.13.1/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:54:462)
at renderDocument (/Users/kobrien/dev/nx-5/node_modules/.pnpm/next@9.5.0_react-dom@16.13.1+react@16.13.1/node_modules/next/dist/next-server/server/render.js:3:594)
at renderToHTML (/Users/kobrien/dev/nx-5/node_modules/.pnpm/next@9.5.0_react-dom@16.13.1+react@16.13.1/node_modules/next/dist/next-server/server/render.js:47:72)

Error occurred prerendering page "/". Read more: https://err.sh/next.js/prerender-error
Error: Minified React error #321; visit https://reactjs.org/docs/error-decoder.html?invariant=321 for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
at Z (/Users/kobrien/dev/nx-5/.next/server/pages/_document.js:1028:404)
at Object.module.exports.mbhs.exports.useState (/Users/kobrien/dev/nx-5/.next/server/pages/_document.js:1034:277)
at Link (/Users/kobrien/dev/nx-5/.next/server/pages/index.js:1975:50)
at d (/Users/kobrien/dev/nx-5/node_modules/.pnpm/react-dom@16.13.1_react@16.13.1/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:36:498)
at $a (/Users/kobrien/dev/nx-5/node_modules/.pnpm/react-dom@16.13.1_react@16.13.1/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:39:16)
at a.b.render (/Users/kobrien/dev/nx-5/node_modules/.pnpm/react-dom@16.13.1_react@16.13.1/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:44:476)
at a.b.read (/Users/kobrien/dev/nx-5/node_modules/.pnpm/react-dom@16.13.1_react@16.13.1/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:44:18)
at renderToString (/Users/kobrien/dev/nx-5/node_modules/.pnpm/react-dom@16.13.1_react@16.13.1/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:54:364)
at Object.renderPage (/Users/kobrien/dev/nx-5/node_modules/.pnpm/next@9.5.0_react-dom@16.13.1+react@16.13.1/node_modules/next/dist/next-server/server/render.js:45:686)
at Function.getInitialProps (/Users/kobrien/dev/nx-5/.next/server/pages/_document.js:222:19)

Build error occurred
Error: Export encountered errors on following paths:
/
/404
at exportApp (/Users/kobrien/dev/nx-5/node_modules/.pnpm/next@9.5.0_react-dom@16.13.1+react@16.13.1/node_modules/next/dist/export/index.js:24:1103)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async build (/Users/kobrien/dev/nx-5/node_modules/.pnpm/next@9.5.0_react-dom@16.13.1+react@16.13.1/node_modules/next/dist/build/index.js:37:218)
Automatically optimizing pages . ERROR Command failed with exit code 1.

Move dependency to "next": "9.4.4" in package.json

repeat:

pnpm install
pnpm next build

project works and builds

Expected behavior

pnpm and be used with a minimal next.js app

System information

OS: macos
Version of Next.js: 9.5.0
Version of Node.js: 12.6.3

IanMitchell · 2020-07-29T01:21:25Z

~~Seeing the same thing, but on regular npm~~ - nvm

bitfrost · 2020-07-29T01:56:48Z

Interesting @IanMitchell . With the above example repo, I don't see this using npm (6.14.4).

Can you confirm my example repo doesn't work with you for npm (and if so what version of npm / node?) (Look out for stray node_modules/ in parent directories, i have seen those cause similar webpack errors around duplicate 'react/package versions'). Locally I can use the example repo and npm to build and run via next without any issues. Tested on 2 machines.

IanMitchell · 2020-07-29T02:00:01Z

Whoops, you're right - I wasn't seeing the same thing, I was getting a different bug covered by an existing report / PR. My bad for not reading the stack trace carefully enough, sorry!

bitfrost · 2020-07-29T02:48:27Z

I can repo with 9.4.5-canary.8 (9.4.5-canary.7 issue is not present)
I will dig through these v9.4.5-canary.7...v9.4.5-canary.8

bitfrost · 2020-07-29T03:41:32Z

Via some bi-secting This appears to be introduced by
e125d90
@timneutkens . I am not grokking why yet, but will dig a bit more tomorrow.

bitfrost · 2020-07-29T05:12:19Z

removing the use of this hook gets rid of the 'you may be violating the rules of hooks' error. I tweaked (_document.tsx)
// const { inAmpMode } = useContext(DocumentComponentContext)._documentProps
const inAmpMode = true; <-- not a real fix, e125d90 did introduce using the hook versus a class component

The above hack lead me to this apparently bad import

https://github.com/vercel/next.js/blob/canary/packages/next/next-server/server/lib/path-match.ts#L1
notice the direct import from compiled/dist/

import * as pathToRegexp from 'next/dist/compiled/path-to-regexp'

This package is missing here:

https://github.com/vercel/next.js/tree/canary/packages/next/compiled

changing path-match.ts and removing the use of a hook in _document.tsx
fixes things when using pnpm

Change:
import * as pathToRegexp from 'path-to-regexp' + package.json update adding 'path-to-regexp'

Not sure yet why I am getting the hook error only from pnpm.

I will dig more tomorrow potentially raise a PR

Timer · 2020-07-29T23:13:17Z

I've unmarked this as p0 because this appears to be a bug with pnpm. pnpm is duplicating copies of react and react-dom which causes three instances to be loaded at build-time:

/example_pnpm_next_issue/node_modules/react/index.js
/example_pnpm_next_issue/node_modules/.pnpm/next@9.5.0_react-dom@16.13.1+react@16.13.1/node_modules/react/index.js
/.pnpm/styled-jsx@3.3.0_react@16.13.1/node_modules/react/index.js

Timer · 2020-07-29T23:16:06Z

Marking this as a duplicate of #9022

bitfrost · 2020-07-30T13:55:52Z

@Timer I followed the following procedure and found the same instance of react. pnpm points things via symlinks links to single shared versions of these packages. Can you share how you produced the above snippet?
Using the facebook suggested method, I got a 'true' using the below:

https://reactjs.org/warnings/invalid-hook-call-warning.html
// Add this in node_modules/react-dom/index.js
window.React1 = require('react');

// Add this in your component file
require('react-dom');
window.React2 = require('react');
console.log(window.React1 === window.React2);

Same version of pnpm, this works fine previous to this change.

bitfrost · 2020-07-30T16:02:08Z

This does feel like a pnpm bug , the work around here fixes my example repo
pnpm/pnpm#2695

zkochan · 2020-08-01T20:15:08Z

pnpm is duplicating copies of react and react-dom which causes three instances to be loaded at build-time

I don't see duplicated.

This
/.pnpm/next@9.5.0_react-dom@16.13.1+react@16.13.1/node_modules/react/

and this
/.pnpm/styled-jsx@3.3.0_react@16.13.1/node_modules/react/

are just symlinks to the same directory:
/.pnpm/react@16.13.1/node_modules/react

Seems like a bug in React. React should resolve these symlinks to their real locations. That's how node's resolution algorithm works.

bitfrost · 2020-08-04T17:49:19Z

I agree @zkochan not pnpm bug, all same copies of React. While the workaround 'worked', it should not be needed at all.

elliottsj · 2020-08-16T17:56:15Z

tl;dr resolveRequest should be updated to follow symlinks, which I believe should fix this issue.

Did some debugging; here's what I found.

The issue is in this handleExternals function where Next.js determines which modules webpack should include in the server bundle (written to .next/server/pages/_document.js) and which should be treated as "external", i.e. require()'d by Node.js at run time. We end up with two instances of the 'react' module because one is included in the bundle and one is require()'d by Node.js.

I added some log statements to this function and ran next dev:

with npm

> my-app@0.1.0 dev /Users/spencerelliott/Dev/elliottsj/my-app
> next dev

ready - started server on http://localhost:3000
context /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/next/dist/pages
res     /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
externalizing module
context /Users/spencerelliott/Dev/elliottsj/my-app/pages
res     /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
externalizing module
context /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/next/dist/pages
res     /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
externalizing module

with pnpm

> my-app@0.1.0 dev /Users/spencerelliott/Dev/elliottsj/my-app
> next dev

ready - started server on http://localhost:3000
context /Users/spencerelliott/Dev/elliottsj/my-app/pages
res     /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
externalizing module
context /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/next@9.5.2_ac1925d3b18bab26b8f85315f9db8478/node_modules/next/dist/pages
res     /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/next@9.5.2_ac1925d3b18bab26b8f85315f9db8478/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
baseRes !== res. bundling module.
context /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/next@9.5.2_ac1925d3b18bab26b8f85315f9db8478/node_modules/next/dist/pages
res     /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/next@9.5.2_ac1925d3b18bab26b8f85315f9db8478/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
baseRes !== res. bundling module.
context /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/styled-jsx@3.3.0_react@16.13.1/node_modules/styled-jsx/dist
res     /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/styled-jsx@3.3.0_react@16.13.1/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
baseRes !== res. bundling module.
context /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/styled-jsx@3.3.0_react@16.13.1/node_modules/styled-jsx/dist
res     /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/styled-jsx@3.3.0_react@16.13.1/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
baseRes !== res. bundling module.

Using npm, we get our desired result: the 'react' module is always externalized.

Using pnpm, 'react' is only externalized when the importer is in the /Users/spencerelliott/Dev/elliottsj/my-app/pages directory (aka the context). From these other contexts, React is bundled instead:

node_modules/.pnpm/next@9.5.2_ac1925d3b18bab26b8f85315f9db8478/node_modules/next/dist/pages
node_modules/.pnpm/styled-jsx@3.3.0_react@16.13.1/node_modules/styled-jsx/dist

This happens because handleExternals performs a check to make sure a module is resolved to the same location from both the context and the project directory dir before externalizing it:

next.js/packages/next/build/webpack-config.ts

Line 599 in 07df897

res = resolveRequest(request, `${context}/`)

next.js/packages/next/build/webpack-config.ts

Line 637 in 07df897

baseRes = resolveRequest(request, `${dir}/`)

If baseRes !== res, then the module is forced to be bundled.

If we take this example:

res     /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/next@9.5.2_ac1925d3b18bab26b8f85315f9db8478/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js

Both of these paths actually resolve to the same location on disk:

$ cd node_modules/.pnpm/next@9.5.2_ac1925d3b18bab26b8f85315f9db8478/node_modules/react
$ pwd -P
/Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/react@16.13.1/node_modules/react
$ cd -
$ cd node_modules/react
$ pwd -P
/Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/react@16.13.1/node_modules/react

So, I think the root cause of this issue is that the resolveRequest function does not follow symlinks. If it followed symlinks, then baseRes would always equal res, so React would be externalized as we expect.

elliottsj · 2020-08-24T21:34:12Z

I made a pull request which updates the behaviour of resolveRequest so it follows symlinks: #16369

This fixes the problem described in this issue.

However it does not resolve the issue which this is marked as a duplicate of: #9022. Importing React components from outside of the project folder is a slightly different issue due to the fact that you can have different copies of the React package on disk.

Appreciate if a maintainer can re-open this issue.

keyworks · 2020-08-29T01:27:27Z

Would love to see #16369 merged as well. We are currently on a pnpm monorepo and this issue is preventing us from upgrading to 9.5 from 9.4. #9022 seems to be a slew of issues where duplicate react is a symptom and not necessarily related to this specific issue that @elliottsj has root caused so I too would appreciate if the maintainers can perhaps re-open this and consider #16369.

bitfrost · 2020-08-30T00:41:22Z

@elliottsj I missed this PR and created a similar one (but made it pnpm specific) and attached it to
#16471 as well. I would be thrilled if one of them is merged. In the mean time I postinstall the patched files into next.js (ick)

theprobugmaker · 2020-08-30T02:19:44Z

@bitfrost What you mean with postinstall?

bitfrost · 2020-08-30T04:33:19Z

In my own projects, I setup something like:
"scripts": {
"postinstall": "a script to copy files into node_modues/next/dist with my patched PR files"
}
so post pnpm install , next.js gets 'fixed' and I can continue to work using pnpm + next.

balazsorban44 · 2022-01-29T15:05:40Z

This issue has been automatically locked due to no recent activity. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.

bitfrost changed the title ~~9.5.0 introduce compile error when using the pnpm package manager~~ 9.5.0 introduced compile error / breaks when using the pnpm package manager Jul 29, 2020

Timer added kind: bug labels Jul 29, 2020

Timer added this to the iteration 6 milestone Jul 29, 2020

ijjk self-assigned this Jul 29, 2020

Timer removed the priority: p0 label Jul 29, 2020

Timer closed this as completed Jul 29, 2020

Timer assigned Timer and unassigned ijjk Jul 29, 2020

rfestag mentioned this issue Aug 1, 2020

pnpm appears to be pulling in multiple versions of React when used with NextJS 9.5.0+ pnpm/pnpm#2743

Closed

This was referenced Aug 16, 2020

Update nextjs monorepo to v9.5.2 (minor) elliottsj/elliott.dev#221

Closed

Follow symlinks in resolveRequest #16369

Closed

sergioutama mentioned this issue Aug 29, 2020

Fix missing nodemon dependency kyle-mccarthy/nest-next#56

Merged

illarionvk mentioned this issue Sep 10, 2020

9.5.x build fails if node_modules folder is a symlink and imported packages use hooks #16988

Closed

remorses referenced this issue Sep 12, 2020

Prevents externalizing non-hoisted packages (#8739)

9770d19

vercel locked as resolved and limited conversation to collaborators Jan 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

9.5.0 introduced compile error / breaks when using the pnpm package manager #15616

9.5.0 introduced compile error / breaks when using the pnpm package manager #15616

bitfrost commented Jul 29, 2020 •

edited

IanMitchell commented Jul 29, 2020 •

edited

bitfrost commented Jul 29, 2020

IanMitchell commented Jul 29, 2020

bitfrost commented Jul 29, 2020

bitfrost commented Jul 29, 2020

bitfrost commented Jul 29, 2020

Timer commented Jul 29, 2020

Timer commented Jul 29, 2020

bitfrost commented Jul 30, 2020

bitfrost commented Jul 30, 2020

zkochan commented Aug 1, 2020 •

edited

bitfrost commented Aug 4, 2020

elliottsj commented Aug 16, 2020

elliottsj commented Aug 24, 2020 •

edited

keyworks commented Aug 29, 2020

bitfrost commented Aug 30, 2020

theprobugmaker commented Aug 30, 2020

bitfrost commented Aug 30, 2020 •

edited

balazsorban44 commented Jan 29, 2022

9.5.0 introduced compile error / breaks when using the pnpm package manager #15616

9.5.0 introduced compile error / breaks when using the pnpm package manager #15616

Comments

bitfrost commented Jul 29, 2020 • edited

Bug report

To Reproduce

Expected behavior

System information

IanMitchell commented Jul 29, 2020 • edited

bitfrost commented Jul 29, 2020

IanMitchell commented Jul 29, 2020

bitfrost commented Jul 29, 2020

bitfrost commented Jul 29, 2020

bitfrost commented Jul 29, 2020

Timer commented Jul 29, 2020

Timer commented Jul 29, 2020

bitfrost commented Jul 30, 2020

bitfrost commented Jul 30, 2020

zkochan commented Aug 1, 2020 • edited

bitfrost commented Aug 4, 2020

elliottsj commented Aug 16, 2020

with npm

with pnpm

elliottsj commented Aug 24, 2020 • edited

keyworks commented Aug 29, 2020

bitfrost commented Aug 30, 2020

theprobugmaker commented Aug 30, 2020

bitfrost commented Aug 30, 2020 • edited

balazsorban44 commented Jan 29, 2022

bitfrost commented Jul 29, 2020 •

edited

IanMitchell commented Jul 29, 2020 •

edited

zkochan commented Aug 1, 2020 •

edited

elliottsj commented Aug 24, 2020 •

edited

bitfrost commented Aug 30, 2020 •

edited