File watcher limit exceeded when using histoire build

[anoack93]

When using histoire@0.10.7 (vite 3.1.0) and executing histoire build in our GitLab CI/CD pipeline the job fails due to a file system limit which is being reached. I use histoire with nuxt and also run nuxt build in advance. I don't get why histoire needs file system watchers, when running the production build. Is that a known problem?

histoire build
node:internal/errors:465
    ErrorCaptureStackTrace(err);
    ^
Error: ENOSPC: System limit for number of file watchers reached, watch '/builds/foo-bar/starter/.npm/_cacache/content-v2/sha512/64/36/...'
    at FSWatcher.<computed> (node:internal/fs/watchers:244:19)
    at Object.watch (node:fs:2264:34)
    at createFsWatchInstance (file:///builds/foo-bar/starter/node_modules/vite/dist/node/chunks/dep-665b0112.js:49738:17)
    at setFsWatchListener (file:///builds/foo-bar/starter/node_modules/vite/dist/node/chunks/dep-665b0112.js:49785:15)
    at NodeFsHandler$1._watchWithNodeFs (file:///builds/foo-bar/starter/node_modules/vite/dist/node/chunks/dep-665b0112.js:49940:14)
    at NodeFsHandler$1._handleFile (file:///builds/foo-bar/starter/node_modules/vite/dist/node/chunks/dep-665b0112.js:50004:23)
    at NodeFsHandler$1._addToNodeFs (file:///builds/foo-bar/starter/node_modules/vite/dist/node/chunks/dep-665b0112.js:50246:21)
Emitted 'error' event on FSWatcher instance at:
    at FSWatcher._handleError (file:///builds/foo-bar/starter/node_modules/vite/dist/node/chunks/dep-665b0112.js:51434:10)
    at NodeFsHandler$1._addToNodeFs (file:///builds/foo-bar/starter/node_modules/vite/dist/node/chunks/dep-665b0112.js:50254:18) {
  errno: -28,
  syscall: 'watch',
  code: 'ENOSPC',
  path: '/builds/foo-bar/starter/.npm/_cacache/content-v2/sha512/64/36/...',
  filename: '/builds/foo-bar/starter/.npm/_cacache/content-v2/sha512/64/36/...'
}

[Akryum]

I guess this is caused by the watcher that is used to scan for markdown files:

histoire/packages/histoire/src/node/build.ts

Line 45 in b57aad8

const { stop } = await createMarkdownFilesWatcher(ctx)

[anoack93]

I tried to find the root cause of this problem, but I was not very successful yet.

I found out the problem occurs only in the CI/CD pipeline because of the way we cache node modules.

We are using npm i --cache .npm --prefer-offline to create the npm cache directly inside the working directory, where also the nuxt 3 application and histoire lives. This is needed for GitLab CI caching. This causes the file watcher problem. I could verify that the problem does not occur if the caching directory is not changed.
So it seems like some scanning mechanism in histoire build causes too many files in the .npm directory being watched.
Thats why I added the .npm directory to storyIgnored

  storyIgnored: [
    '**/node_modules/**',
    '**/dist/**',
    '**/.npm/**',
  ],

but this does not fix the problem either, so I don't know whether createMarkdownFilesWatcher causes the problem, as it should use the storyIgnored config too.

But in general it would probably be more reliable to not use file watchers when indexing files in build mode.

[Akryum]

But in general it would probably be more reliable to not use file watchers when indexing files in build mode.

Agreed

[Qualterink]

Hi,

I have the same problem after upgrade my Library (not histoire). I removed from build 'umd' format and I set only 'es' format.

[freakzlike]

The issue still seems to exist and I am unable to build within the CI/CD Pipeline. Even if I don't have any *.story.md in my repository. I just wonder which files are being watch then?

histoire/packages/histoire/src/node/markdown.ts

Lines 138 to 141 in b2bc009

	const watcher = chokidar.watch(['**/*.story.md'], {
	cwd: ctx.root,
	ignored: ctx.config.storyIgnored,
	})

[anoack93]

@freakzlike I was able to resolve this issue by increasing the file watcher limit right before running the build in the pipeline

echo fs.inotify.max_user_watches=524288 | tee -a /etc/sysctl.conf && sysctl -p

the actual way to change the limit depends on the base image your are using to run your CI/CD jobs