Too many open files / UnknownContentCollectionError: site does not build #7073

fviolette · 2023-05-12T09:38:49Z

What version of `astro` are you using?

2.4.5

Are you using an SSR adapter? If so, which one?

None

What package manager are you using?

pnpm

What operating system are you using?

Mac

What browser are you using?

Chrome

Describe the Bug

Site does not build using Astro 2.4.5 (with the latest MDX integration).
The issue seems to start with Astro 2.4.4.

I've used the docs site as an example.
My private site has the same issue.

EMFILE: too many open files, open '/home/projects/github-xwpzeh/src/pages/[lang]/[...slug].astro'

UnknownContentCollectionError: [object Object]
    at Module.getEntrySlug (file://file:///home/projects/github-xwpzeh/node_modules/.pnpm/astro@2.4.5_@types+node@18.6.4_sass@1.54.3/node_modules/astro/dist/content/utils.js:298:11)
    at async eval (file://file:///home/projects/github-xwpzeh/node_modules/.pnpm/astro@2.4.5_@types+node@18.6.4_sass@1.54.3/node_modules/astro/dist/content/vite-plugin-content-virtual-mod.js:99:20)
    at async Promise.all (index 103)
    at async getStringifiedLookupMap (file://file:///home/projects/github-xwpzeh/node_modules/.pnpm/astro@2.4.5_@types+node@18.6.4_sass@1.54.3/node_modules/astro/dist/content/vite-plugin-content-virtual-mod.js:90:3)
    at async Context.load (file://file:///home/projects/github-xwpzeh/node_modules/.pnpm/astro@2.4.5_@types+node@18.6.4_sass@1.54.3/node_modules/astro/dist/content/vite-plugin-content-virtual-mod.js:52:36)
    at async Object.load (file://file:///home/projects/github-xwpzeh/node_modules/.pnpm/vite@4.3.2_@types+node@18.6.4_sass@1.54.3/node_modules/vite/dist/node/chunks/dep-7efa13d7.js:42887:32)
    at async loadAndTransform (file://file:///home/projects/github-xwpzeh/node_modules/.pnpm/vite@4.3.2_@types+node@18.6.4_sass@1.54.3/node_modules/vite/dist/node/chunks/dep-7efa13d7.js:53326:24)

The troubleshooting link provided is https://docs.astro.build/en/reference/errors/unknown-content-collection-error/

Link to Minimal Reproducible Example

https://stackblitz.com/edit/github-xwpzeh?file=package.json

Participation

I am willing to submit a pull request for this issue.

The text was updated successfully, but these errors were encountered:

ChrisChinchilla · 2023-05-15T13:53:44Z

Also same for me, and the site used to build/serve. Same problem locally on macOS and using Netlify. I have a reasonable amount of content, but it used to work.

ezzle · 2023-05-16T07:49:16Z

More detailed error message :
`node:fs:601
handleErrorFromBinding(ctx);
^

Error: EMFILE: too many open files, open 'D:\Workspace\astro-esse\node_modules\vscode-oniguruma\release\onig.wasm'
at Object.openSync (node:fs:601:3)
at Object.readFileSync (node:fs:469:35)
at getOniguruma (D:\Workspace\astro-esse\node_modules\astro\node_modules\shiki\dist\index.js:1926:26)
at getHighlighter (D:\Workspace\astro-esse\node_modules\astro\node_modules\shiki\dist\index.js:2774:34)
at getViteErrorPayload (file:///D:/Workspace/astro-esse/node_modules/astro/dist/core/errors/dev/vite.js:92:29)
at Timeout._onTimeout (file:///D:/Workspace/astro-esse/node_modules/astro/dist/vite-plugin-astro-server/response.js:18:67)
at listOnTimeout (node:internal/timers:569:17)
at process.processTimers (node:internal/timers:512:7) {
}

Node.js v18.15.0`

infovore · 2023-05-16T09:51:27Z

Yep, same here. This bug comes in around 2.4.4 - 2.4.3 is entirely fine.

Simple reproduction: make a new Astro project with npm create, use the basic blog template, throw a few hundred posts into src/content/blog, and watch something have a meltdown. It looks like it might be related to Shiki.

ChrisChinchilla · 2023-05-16T10:02:47Z

I'm slowly digging into this… I seem to have things sort of working for serve now, but not yet for build. Will summarise my findings when done. Even working, the processes are much slower than before.

infovore · 2023-05-16T10:44:51Z

@ChrisChinchilla looking at the error messages and changes between 2.4.3 and 2.4.4, my gut says that #6916 might be the source.

bluwy · 2023-05-16T11:01:42Z

Looks like my suggestion here did cause issues 😬 Perhaps making this part serial or using a promise queue would fix it.

astro/packages/astro/src/content/vite-plugin-content-virtual-mod.ts

Lines 106 to 132 in d7007a1

    
           await Promise.all( 
        
           	contentGlob 
        
           		// Ignore underscore files in lookup map 
        
           		.filter((e) => !hasUnderscoreBelowContentDirectoryPath(pathToFileURL(e), contentDir)) 
        
           		.map(async (filePath) => { 
        
           			const info = getEntryInfo({ contentDir, entry: filePath }); 
        
           			// Globbed entry outside a collection directory 
        
           			// Log warning during type generation, safe to ignore in lookup map 
        
           			if (info instanceof NoCollectionError) return; 
        
           			const contentEntryType = contentEntryConfigByExt.get(extname(filePath)); 
        
           			if (!contentEntryType) return; 
        
           			const { id, collection, slug: generatedSlug } = info; 
        
           			const slug = await getEntrySlug({ 
        
           				id, 
        
           				collection, 
        
           				generatedSlug, 
        
           				fs, 
        
           				fileUrl: pathToFileURL(filePath), 
        
           				contentEntryType, 
        
           			}); 
        
           			filePathByLookupId[collection] = { 
        
           				...filePathByLookupId[collection], 
        
           				[slug]: rootRelativePath(root, filePath), 
        
           			}; 
        
           		}) 
        
           );

ChrisChinchilla · 2023-05-16T12:33:21Z

Here's a brain dump of my notes and things I found that may or may not help others 🤷

Rogue .vscode folder

too many open files errors

https://facebook.github.io/watchman/docs/install#macos-file-descriptor-limits

Remove minor React component

Seemed to be posts folder, tried to debug which one

Renamed hash from imported medium posts

Change file casing to all lower case, remove extra special symbols

Too many updates to types at once? So try sere adding chunks of content at a time

That gets serve to run

Build still stuck and somewhat quiet about what's happening

Debug doesn't do much
Downgrade to Astro 2.4.4
icon and svgo issues
natemoo-re/astro-icon#93
natemoo-re/astro-icon#65

Then build works!

02:02:59 PM [build] Waiting for the @astrojs/image integration...
02:03:01 PM [@astrojs/image] "~/assets/images/default.png" image could not be fetched
Completed in 12.81s.

Completed in 61.75s.

@astrojs/sitemap: `sitemap-index.xml` is created.

02:03:12 PM [build] Waiting for the astro-compress integration...
Successfully compressed a total of 4 CSS files for 1010 Bytes.
Error: Cannot compress file /Users/chrisward/Library/Mobile Documents/com~apple~CloudDocs/OSS projects/chris-chinchilla/new-site/dist/blog/2015/2015-01-26-review-of-triumph-of-the-nerds-and-kobold-guide-to-board-game-design/index.html!
Error: Cannot compress file /Users/chrisward/Library/Mobile Documents/com~apple~CloudDocs/OSS projects/chris-chinchilla/new-site/dist/blog/2015/2015-03-22-add-your-suggestions-to-what-came-next/index.html!
Successfully compressed a total of 976 HTML files for 4.62 MB.
02:03:47 PM [build] 978 page(s) built in 1641.25s
02:03:47 PM [build] Complete!

ChrisChinchilla · 2023-05-16T12:34:31Z

Going to now see if that gets a Netlify build to work, no…

It gets very close and then times out :(

fviolette · 2023-05-18T17:25:09Z

Works with 2.5.0 on my end.

ezzle · 2023-05-19T06:36:46Z

Still not working for me

New error after npm install astro@latest

Logs

[vite] Error when evaluating SSR module D:\Workspace\astro-esse\src\content\config.ts: failed to import "astro:content"
|- UnknownContentCollectionError: [object Object]
    at getEntrySlug (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/utils.js:325:11) 
    at async file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/vite-plugin-content-virtual-mod.js:110:22
    at async Promise.all (index 8189)
    at async getStringifiedLookupMap (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/vite-plugin-content-virtual-mod.js:86:3)
    at async Context.load (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/vite-plugin-content-virtual-mod.js:47:38)
    at async Object.load (file:///D:/Workspace/astro-esse/node_modules/vite/dist/node/chunks/dep-934dbc7c.js:42856:32)
    at async loadAndTransform (file:///D:/Workspace/astro-esse/node_modules/vite/dist/node/chunks/dep-934dbc7c.js:53282:24)

 error   [object Object]
  File:
    D:\Workspace\astro-esse\node_modules\astro\dist\content\utils.js:325:11
  Stacktrace:
UnknownContentCollectionError: [object Object]
    at getEntrySlug (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/utils.js:325:11) 
    at async handleEvent (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/types-generator.js:194:27)
    at async runEvents (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/types-generator.js:259:24)
    at async Object.init (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/types-generator.js:61:5)
    at async attachListeners (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/server-listeners.js:46:5)
    at async attachContentServerListeners (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/server-listeners.js:27:5)
    at async dev (file:///D:/Workspace/astro-esse/node_modules/astro/dist/core/dev/dev.js:63:3)       
    at async runCommand (file:///D:/Workspace/astro-esse/node_modules/astro/dist/cli/index.js:151:7)  
    at async cli (file:///D:/Workspace/astro-esse/node_modules/astro/dist/cli/index.js:209:5)

ChrisChinchilla · 2023-05-19T06:38:22Z

Same, I also forgot to note that I could only get anything to work after changing the open file limit on macOS, as I can’t do that on Netlify, there I had to temporarily remove a load of content to get it to work. Not ideal…

bluwy · 2023-05-19T10:15:12Z

I've made an attempt to fix this at #7125. Appreciate if you can try it with npm install astro@next--content-serial to see if it works.

infovore · 2023-05-19T10:47:39Z

Still not working for me

Well, as far as I can see, there's nothing in 2.5.0 that alters what I feel the cause of the issue was.

ezzle · 2023-05-19T18:36:15Z

Build OK, but very slow

Logs

This is a  content-serial-20230519093503  prerelease build
    Feedback? https://astro.build/issues

20:01:13 [content] Watching src/content/ for changes`

But access to pages too too too much slow.

And finally an error when trying to get a page

error [object Object]
File:
D:\Workspace\astro-esse\node_modules\astro\dist\content\utils.js:325:11
Code:
324 | } catch (e) {
> 325 | throw new AstroError(AstroErrorData.UnknownContentCollectionError, { cause: e });
| ^
326 | }
327 | const { slug: frontmatterSlug } = await contentEntryType.getEntryInfo({
328 | fileUrl,
Stacktrace:
UnknownContentCollectionError: [object Object]
at getEntrySlug (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/utils.js:325:11)
at async handleEvent (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/types-generator.js:194:27)
at async runEvents (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/types-generator.js:259:24)
at async Object.init (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/types-generator.js:61:5)
at async attachListeners (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/server-listeners.js:46:5)
at async attachContentServerListeners (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/server-listeners.js:27:5)
at async dev (file:///D:/Workspace/astro-esse/node_modules/astro/dist/core/dev/dev.js:63:3)
at async runCommand (file:///D:/Workspace/astro-esse/node_modules/astro/dist/cli/index.js:151:7)
at async cli (file:///D:/Workspace/astro-esse/node_modules/astro/dist/cli/index.js:209:5)

</details>

ezzle · 2023-05-20T04:11:26Z

Build OK, but very slow

How Big is my project :

*.xml           238 MB            2/2 files     100%
*.mdx           129 MB  101539/101539 files     100%
*.png           831 KB          10/10 files     100%
*.md            448 KB          75/75 files     100%
*.ts            448 KB        105/105 files     100%

ChrisChinchilla · 2023-05-20T10:53:48Z

Will try later myself when back at a computer. Initially my project is about 8000 items of content, but even cutting it down to a fraction of that I an’t get it to build locally without upping the open file limit (on macOS anyway) and as stated, it all used to work totally fine and fairly speedily. Right now, building and serving is slow Lee than some old school SSGs like Jekyll.

bluwy · 2023-05-22T10:07:59Z

Thanks for the feedback. I've cut another release for astro@next--content-serial using p-limit to run 10 at a time instead of serially, maybe that could fix it. If not, maybe we should revert the original PR.

Also, if you're using MDX, you can keep an eye on #7151 which should also improve things.

ezzle · 2023-05-22T10:56:10Z

Thanks for the new release. Unfortunately still not the green light

Logs

 astro  v0.0.0-content-serial-20230522100058 started in 613ms

  ┃ Local    http://localhost:3003/
  ┃ Network  use --host to expose

  ▶ This is a  content-serial-20230522100058  prerelease build
    Feedback? https://astro.build/issues

12:52:45 [content] Watching src/content/ for changes
 error   EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx'
Error: EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx'
 error   EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx'
Error: EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx' (x2)
 error   EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx'
Error: EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx' (x3)
 error   EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx'
Error: EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx' (x4)
 error   EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx'
Error: EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx' (x5)
 error   EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx'
Error: EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx' (x6)
 error   EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx'
Error: EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx' (x7)
 error   EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx'
Error: EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx' (x8)
 error   EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx'
Error: EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx' (x9)
 error   EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx'
Error: EMFILE: too many open files, open 'D:\Workspace\astro-esse\src\content\essential\en\slot\realised_by_data_objects.mdx' (x10)
Sourcemap for "D:/Workspace/astro-esse/src/pages/essential/[...slug].astro" points to missing source files
 error   [object Object]
  File:
    D:\Workspace\astro-esse\node_modules\astro\dist\content\utils.js:325:11
  Stacktrace:
UnknownContentCollectionError: [object Object]
    at getEntrySlug (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/utils.js:325:11)        
    at async handleEvent (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/types-generator.js:202:27)
    at async runEvents (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/types-generator.js:267:24)
    at async Object.init (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/types-generator.js:69:5)
    at async attachListeners (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/server-listeners.js:46:5)
    at async attachContentServerListeners (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/server-listeners.js:27:5)
    at async dev (file:///D:/Workspace/astro-esse/node_modules/astro/dist/core/dev/dev.js:63:3)
    at async runCommand (file:///D:/Workspace/astro-esse/node_modules/astro/dist/cli/index.js:151:7)
    at async cli (file:///D:/Workspace/astro-esse/node_modules/astro/dist/cli/index.js:209:5)

Best regards

Tshitshi

bluwy · 2023-05-22T11:51:41Z

@ezzle which release had started causing issues for you? The original issue mentioned 2.4.4 which I'm pretty sure I've fixed the cause.

Given you have more than 100k files in src/content, I think you'll always be hitting the limit because Vite is also watching the files (see Vite's troubleshooting guide). However that link doesn't apply to Windows (and I'm not sure how to do the same on Windows), you can perhaps also try configuring vite.server.ignored in astro.config.js.

ematipico · 2023-05-22T12:36:30Z

Given you have more than 100k files in src/content, I think you'll always be hitting the limit because Vite is also watching the files

When generating the content collections, should we turn off the watcher?

ezzle · 2023-05-22T12:56:09Z

@bluwy

Thank you for the support.

The issue in my case is related with the number of files in 'src/content'. It appears when >= +-4000K with any Astro version. I am very new in the Astro community, coming fron ApostropheCMS eco-system. I am hoping to achieve a more reactive & lean alternative with astro & alpine as base. Unfortunately, the capacity to serve efficiently >= 1000K files is mandatory. Do you think Astro can help meet that content volume requirement ?

The current astro version is 2.5.1. Previously I have been using the default version of the astro-doc project : ["astro": "^2.3.1"].

By the way uninstalling the patch is worst. I got the following error

npm run dev

> docs@0.0.7 dev
> astro dev

   astro  v2.5.1 started in 609ms

  ┃ Local    http://localhost:3003/
  ┃ Network  use --host to expose

14:44:37 [content] Watching src/content/ for changes
[object Object]
[object Object] (x2)
14:44:42 [vite] Error when evaluating SSR module D:\Workspace\astro-esse\src\content\config.ts: failed to import "astro:content"
|- UnknownContentCollectionError: [object Object]
    at getEntrySlug (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/utils.js:325:11)        
    at async file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/vite-plugin-content-virtual-mod.js:110:22
    at async Promise.all (index 8189)
    at async getStringifiedLookupMap (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/vite-plugin-content-virtual-mod.js:86:3)
    at async Context.load (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/vite-plugin-content-virtual-mod.js:47:38)
    at async Object.load (file:///D:/Workspace/astro-esse/node_modules/vite/dist/node/chunks/dep-4d3eff22.js:42892:32)
    at async loadAndTransform (file:///D:/Workspace/astro-esse/node_modules/vite/dist/node/chunks/dep-4d3eff22.js:53318:24)

 error   [object Object]
  File:
    D:\Workspace\astro-esse\node_modules\astro\dist\content\utils.js:325:11
  Stacktrace:
UnknownContentCollectionError: [object Object]
    at getEntrySlug (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/utils.js:325:11)        
    at async handleEvent (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/types-generator.js:202:27)
    at async runEvents (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/types-generator.js:267:24)
    at async Object.init (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/types-generator.js:69:5)
    at async attachListeners (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/server-listeners.js:46:5)
    at async attachContentServerListeners (file:///D:/Workspace/astro-esse/node_modules/astro/dist/content/server-listeners.js:27:5)
    at async dev (file:///D:/Workspace/astro-esse/node_modules/astro/dist/core/dev/dev.js:63:3)
    at async runCommand (file:///D:/Workspace/astro-esse/node_modules/astro/dist/cli/index.js:151:7)
    at async cli (file:///D:/Workspace/astro-esse/node_modules/astro/dist/cli/index.js:209:5)`

With the patch installed, the build terminates normally

` astro  v0.0.0-content-serial-20230522100058 started in 740ms

  ┃ Local    http://localhost:3003/
  ┃ Network  use --host to expose

  ▶ This is a  content-serial-20230522100058  prerelease build
    Feedback? https://astro.build/issues

14:46:13 [content] Watching src/content/ for changes
14:48:21 [content] Unsupported file types found. Prefix with an underscore (`_`) to ignore:
- essential-schema.ts
14:48:21 [content] Types generated
14:48:21 [astro] update D:/Workspace/astro-esse/.astro/types.d.ts

But installing the patch produce warning for conflicting dependencies version

npm install astro@next--content-serial
npm WARN ERESOLVE overriding peer dependency
npm WARN While resolving: docs@0.0.7
npm WARN Found: astro@2.5.1
npm WARN node_modules/astro
npm WARN   peer astro@"^2.5.0" from @astrojs/markdown-remark@2.2.1
npm WARN   node_modules/@astrojs/mdx/node_modules/@astrojs/markdown-remark
npm WARN     @astrojs/markdown-remark@"^2.1.3" from @astrojs/mdx@0.18.4
npm WARN     node_modules/@astrojs/mdx
npm WARN   4 more (astro-auto-import, astro-og-canvas, ...)
npm WARN
npm WARN Could not resolve dependency:
npm WARN peer astro@"^1.0.0 || ^2.0.0-beta" from astro-auto-import@0.2.1
npm WARN node_modules/astro-auto-import
npm WARN   dev astro-auto-import@"^0.2.1" from the root project
npm WARN ERESOLVE overriding peer dependency

Best regards

Tshitshi

@ezzle which release had started causing issues for you? The original issue mentioned 2.4.4 which I'm pretty sure I've fixed the cause.

Given you have more than 100k files in src/content, I think you'll always be hitting the limit because Vite is also watching the files (see Vite's troubleshooting guide). However that link doesn't apply to Windows (and I'm not sure how to do the same on Windows), you can perhaps also try configuring vite.server.ignored in astro.config.js.

ChrisChinchilla · 2023-05-22T19:47:09Z

@bluwy et all, the #7125 change fixed my issues. I get some dependency conflicts, but I guess that's expected and using --force gets things to work fine. Excellent, I look forward to a proper release!

bluwy · 2023-05-23T08:51:26Z

When generating the content collections, should we turn off the watcher?

That makes sense. We're spinning up one-off Vite servers in some places and turning the watcher off could fix it. You can't shut it down entirely though since Vite always spins it up, but we can configure to ignore watching all files.

Do you think Astro can help meet that content volume requirement ?

We should be able to but seeing that many files is a first for me so maybe we're not doing something right at that scale. However, I think it's diverging from the original issue so I'd suggest opening another issue if you can find out which Astro version starts causing issues. The errors you get are because some other dependencies rely on the latest version of Astro, so you might need to downgrade the others too.

@bluwy et all, the #7125 change fixed my issues

Thanks for the update! I'll go ahead and get the fix out then.

This comment was marked as outdated.

Sign in to view

bluwy added - P4: important Violate documented behavior or significantly impacts performance (priority) content-collections labels May 16, 2023

ematipico assigned ematipico and unassigned ematipico May 16, 2023

bluwy mentioned this issue May 19, 2023

Make vite-plugin-content-virtual-mod run getEntrySlug 10 at a time #7125

Merged

bluwy self-assigned this May 19, 2023

This comment was marked as outdated.

Sign in to view

bluwy mentioned this issue May 23, 2023

Prevent Vite watching on Astro config load #7171

Merged

bluwy closed this as completed in #7125 May 23, 2023

ezzle mentioned this issue May 30, 2023

Error: EMFILE: too many open files, open 'xxx.mdx' (x10) : Content and Pages at very large scale #7241

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Too many open files / UnknownContentCollectionError: site does not build #7073

Too many open files / UnknownContentCollectionError: site does not build #7073

fviolette commented May 12, 2023 •

edited by bluwy

This comment was marked as outdated.

ChrisChinchilla commented May 15, 2023

ezzle commented May 16, 2023

infovore commented May 16, 2023

ChrisChinchilla commented May 16, 2023

infovore commented May 16, 2023

bluwy commented May 16, 2023

ChrisChinchilla commented May 16, 2023 •

edited

ChrisChinchilla commented May 16, 2023 •

edited

fviolette commented May 18, 2023

ezzle commented May 19, 2023 •

edited by bluwy

ChrisChinchilla commented May 19, 2023

bluwy commented May 19, 2023 •

edited

infovore commented May 19, 2023

This comment was marked as outdated.

This comment was marked as outdated.

ezzle commented May 19, 2023 •

edited by bluwy

ezzle commented May 20, 2023 •

edited by bluwy

ChrisChinchilla commented May 20, 2023

bluwy commented May 22, 2023

ezzle commented May 22, 2023 •

edited by bluwy

bluwy commented May 22, 2023

ematipico commented May 22, 2023

ezzle commented May 22, 2023 •

edited by bluwy

ChrisChinchilla commented May 22, 2023

bluwy commented May 23, 2023

Too many open files / UnknownContentCollectionError: site does not build #7073

Too many open files / UnknownContentCollectionError: site does not build #7073

Comments

fviolette commented May 12, 2023 • edited by bluwy

What version of astro are you using?

Are you using an SSR adapter? If so, which one?

What package manager are you using?

What operating system are you using?

What browser are you using?

Describe the Bug

Link to Minimal Reproducible Example

Participation

This comment was marked as outdated.

ChrisChinchilla commented May 15, 2023

ezzle commented May 16, 2023

infovore commented May 16, 2023

ChrisChinchilla commented May 16, 2023

infovore commented May 16, 2023

bluwy commented May 16, 2023

ChrisChinchilla commented May 16, 2023 • edited

ChrisChinchilla commented May 16, 2023 • edited

fviolette commented May 18, 2023

ezzle commented May 19, 2023 • edited by bluwy

ChrisChinchilla commented May 19, 2023

bluwy commented May 19, 2023 • edited

infovore commented May 19, 2023

This comment was marked as outdated.

This comment was marked as outdated.

ezzle commented May 19, 2023 • edited by bluwy

ezzle commented May 20, 2023 • edited by bluwy

ChrisChinchilla commented May 20, 2023

bluwy commented May 22, 2023

ezzle commented May 22, 2023 • edited by bluwy

bluwy commented May 22, 2023

ematipico commented May 22, 2023

ezzle commented May 22, 2023 • edited by bluwy

ChrisChinchilla commented May 22, 2023

bluwy commented May 23, 2023

fviolette commented May 12, 2023 •

edited by bluwy

What version of `astro` are you using?

ChrisChinchilla commented May 16, 2023 •

edited

ChrisChinchilla commented May 16, 2023 •

edited

ezzle commented May 19, 2023 •

edited by bluwy

bluwy commented May 19, 2023 •

edited

ezzle commented May 19, 2023 •

edited by bluwy

ezzle commented May 20, 2023 •

edited by bluwy

ezzle commented May 22, 2023 •

edited by bluwy

ezzle commented May 22, 2023 •

edited by bluwy