Storybook v6.5.9 dev server crashes when used with pnpm on a big monorepo

[LucaColonnello]

Describe the bug
When using pnpm with Storybook v6.5.9, in a lerna monorepo with 80+ packages, running pnpm start-storybook results in a long compilation time (minutes) and then the dev server eventually crashes when the browser loads the storybook's localhost url.

This is the error, it seems a memory leak occurs.

<--- Last few GCs --->

[54907:0x7fa458008000]   342201 ms: Scavenge 4000.6 (4124.9) -> 3989.9 (4127.9) MB, 6.9 / 0.0 ms  (average mu = 0.166, current mu = 0.015) allocation failure 
[54907:0x7fa458008000]   342267 ms: Scavenge 4003.4 (4127.9) -> 3992.7 (4129.9) MB, 11.9 / 0.0 ms  (average mu = 0.166, current mu = 0.015) allocation failure 
[54907:0x7fa458008000]   346845 ms: Mark-sweep 4006.2 (4130.4) -> 3966.0 (4133.7) MB, 4544.9 / 1.9 ms  (average mu = 0.175, current mu = 0.185) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

This does not happen in the same codebase using yarn v1.22.
It also works perfectly fine with 20 something stories, still slow, but it works.

It never works if I add any mdx stories, regardless of the total of stories.

I tried to get down to the issue debugging, and I'm wondering if this could be an issue with the watcher, as it doesn't happen in the prod build, which still takes minutes to compile, but it loads and works perfectly fine when opening the static output.

To Reproduce
So sorry I don't have a repo as this is internal and this is hard to reproduce; it only happens with this many components.

Totally appreciate if a repo with this issue is required, but I thought to ask before making that effort as I've seen this has been raised in the past #12348 , but none of the fixes suggested there worked.

I've been on this for weeks now, so I thought to reach out and see if there was any suggestion based on past experiences, hoping this might just be a config issue.

System

Environment Info:

  System:
    OS: macOS 12.3.1
    CPU: (16) x64 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
  Binaries:
    Node: 16.13.1 - ~/.nvm/versions/node/v16.13.1/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 8.1.2 - ~/.nvm/versions/node/v16.13.1/bin/npm
  Browsers:
    Chrome: 104.0.5112.79
    Firefox: 103.0.1
    Safari: 15.4
  npmPackages:
    @storybook/addon-essentials: ^6.3.12 => 6.5.9 
    @storybook/addon-knobs: ^6.3.1 => 6.4.0 
    @storybook/addons: ^6.3.11 => 6.5.9 
    @storybook/api: ^6.4.0 => 6.5.9 
    @storybook/components: ^6.4.0 => 6.5.9 
    @storybook/core-events: ^6.4.0 => 6.5.9 
    @storybook/react: ^6.3.12 => 6.5.9 
    @storybook/storybook-deployer: ^2.8.10 => 2.8.12 
    @storybook/theming: ^6.3.10 => 6.5.9 

Additional context

pnpm-workspace.yaml

packages:
  # all packages in direct subdirs of packages/
  - 'packages/*'

.npmrc

scripts-prepend-node-path=true
auto-install-peers=true
link-workspace-packages=false
public-hoist-pattern[]=*eslint*
public-hoist-pattern[]=*prettier*
public-hoist-pattern[]=*babel*
public-hoist-pattern[]=*storybook*

.storybook/main.js

module.exports = {
  features: {
    postcss: false,
    babelModeV7: true,
  },
  stories: [
    '../packages/**/*stories.mdx',
    '../packages/**/stories.jsx',
  ],
  addons: ['@storybook/addon-essentials', '@storybook/addon-knobs'],
};

[smalljiaot]

I have exactly the same issue when creating a react project in NX (nx.dev) and giving it a storybook project.

[LucaColonnello]

@smalljiaot so are you able to reproduce this on a brand new repo? It would be awesome if we could have a test repo, cause I believe it would speed up assessment for the Storybook team! 🙏

[LucaColonnello]

Updating to webpack 5 solves this issue with pnpm, but Storybook is now taking minutes to load, it's stuck regardless of how many stories I add. It eventually works, but it can take up to 5 mins, even just for 1 story.

[LucaColonnello]

After debugging storybook locally, I found out the issue here.
Essentially, the problem is linked to the fact that storybook is picking up stories from node_modules.

In fact, when listing all stories manually, no glob pattern, it works.
I noticed this by switching to storyStoreV7 mode, as a bug printed out the full path of a story with an error, and it was coming from a sub-package symlinked by pnpm in node_modules.

I believe the fact that pnpm creates a nested node_modules tree is the problem here, as stories files will end up in the node_modules of a package importing another one from the same monorepo.

In my case it was like this:

packages/
   my-pkg-1/
      stories.jsx
   my-pkg-2/      <- depends on my-pkg-1
      node_modules/
             my-pkg-1/
                   stories.jsx
      stories.jsx

So with a simple ./packages/**/stories.jsx pattern, all of these stories files are loaded.
If you think about a monorepo, this can happen a lot if node_modules are not hoisted.

Even when using yarn or npm, node_modules are not necessarily always hoisted, as if there are conflicts in resolution, specific versions would be installed locally for a package rather than hoisted.

I'll open a new issue with these details...

[akanshgulati]

Thanks @LucaColonnello, you saved a lot of our time.

[LucaColonnello]

Solved by #22873