Maximum call stack size exceeded

[aikar]

RangeError: Maximum call stack size exceeded
at node_modules/klaw/src/index.js:45:23
at go$readdir$cb (node_modules/graceful-fs/graceful-fs.js:187:14)
at FSReqWrap.oncomplete (fs.js:135:15)

version 2.1.1 but nothing in new version relates here.

I have a folder with 200k+ files in it that crashes klaw.

        const crawl = async (subdir: string, cb) => {
            const done = resolvable();
            klaw(path.join(dir, subdir), {
                filter: (item) => {
                    return item.endsWith(".job");
                }
            }).on('data', (item) => {
                cb(item.path);               
            }).on('end', () => done.resolve());
            await done;
        };

[aikar]

Ended up switching to https://www.npmjs.com/package/walk due to this bug.

[oorestisime]

Hey @jprichardson how would you feel changing this https://github.com/jprichardson/node-klaw/blob/master/src/index.js#L53 to not use apply? Either spreading the array or concat will avoid stack size range error and also probably would be faster?

i think concat might probably be the first solution but then array will need to be mutated

[jprichardson]

Certainly. Should modernize the code as well (separate PR)

[oorestisime]

Thanks for the quick response. PR opened :)

[jprichardson]

@oorestisime would you be willing to put a test in place verifying that this actually fixes the problem? Thanks.

[oorestisime]

I can take a look yeah. but this means adding a test with more than 200k files or so (haven't looked how the tests are )

[jprichardson]

Couldn't a test harness be used to generate the files?

[mhahn-sighthoundlabs]

We experienced this problem with around 60K files in a folder.

[oorestisime]

hey i am back at this. i completely forgot about it. what are our options here. change seems fairly trivial imho

[pea]

@oorestisime's fix it working for us with 260k pages.