Performance loss in version 8

[ernscht]

I noticed a performance loss when updating to version 8.0.1 in globby.sync().
I try to find a file in some defined folders in a tree of 500 documents.
Using the sync method in this example is about 10 times faster with version 7.1.1

Switching to "fast_glob" does not seem to be faster in any case,
or maybe you see potential to optimize the used patterns or options?

[sindresorhus]

What OS and Node.js version?

[sindresorhus]

// @mrmlnc

[mrmlnc]

I added this issue to my ToDo list to this week.

At first glance, the problem occurs at the read level of the file system. The values range from 8 to 15 milliseconds. I think we read more than we should.

[ernscht]

Thanks for the fast replies.

I use windows 10 pro, but the difference is also noticeable on macOS.
The example uses node version 8.10.0, but with node version 6.13.1 or 8.11.1, the difference between globby version 7.1.1 and 8.0.1 is about the same.

some tests on windows 10:

node version	globby 7.1.1	globby 8.0.1
6.13.1	39 - 44 ms	130 - 151 ms
8.10.0	9 - 11 ms	90 - 121 ms
8.11.1	9 - 13 ms	98 - 119 ms

[SBoudrias]

We're facing the same issue on Yeoman yeoman/yo#176 (comment)

I saved a trace in this gist https://gist.github.com/SBoudrias/3004406099a97f05478305311e5dc3a3 - if it can helps @mrmlnc debug the root cause.

[SBoudrias]

Specifically the 2 calls we do are:

globby.sync(
    ['generator-*', '@*/generator-*'],
    {cwd: root, onlyFiles: false, absolute: true}
)

globby.sync('*/index.js', {cwd: pattern, absolute: true})

from https://github.com/yeoman/environment/blob/master/lib/resolver.js

[sinedied]

Any news on this?

[yhatt]

The latest globby v10 is nearing v7 performance. v10 is about 3-4 times faster than v9, and still about twice slower than v7.

[ernscht]

That sounds good. Thank you.
Current measurements with node 10.16.0 on windows and 1000 Files

globby version	duration
7.1.1	12 - 17 ms
8.0.2	132 - 203 ms
9.0.0	129 - 189 ms
10.0.0	29 - 70 ms

[mrmlnc]

This issue will be fixed with the release of the next version of fast-glob. You can track progress in mrmlnc/fast-glob#156 or mrmlnc/fast-glob#249.

Windows 10
node.js 13.7.0

10 times

globby@7.1.1 — 6 - 16 ms
globby@8.0.2 — 85 - 103 ms
globby@9.2.0 — 81 - 95 ms
globby@10.0.1 — 16 - 33 ms
fast-glob@3.x.x — 5 - 11 ms

[mrmlnc]

The fast-glob@3.2.0 was published.

If the problem still remains, let me know.

Success: File y/y.json found in src/types/five/u/sub/y/y.json
[Time] for matching: 13 ms (globby version: 7.1.1)
Success: File y/y.json found in src/types/five/u/sub/y/y.json"
[Time] for matching: 124 ms (globby version: 8.0.2)
Success: File y/y.json found in src/types/five/u/sub/y/y.json"
[Time] for matching: 85 ms (globby version: 9.2.0)
Success: File y/y.json found in src/types/five/u/sub/y/y.json"
[Time] for matching: 15 ms (globby version: 10.0.1)
Success: File y/y.json found in src/types/five/u/sub/y/y.json"
[Time] for matching: 12 ms (globby version: 11.0.0)

[ernscht]

I can confirm that. Cool. Thank you very much.

globby version	duration	average
7.1.1	6 - 14 ms	9 ms
8.0.2	70 - 228 ms	88 ms
9.2.0	71 - 205 ms	85 ms
10.0.1	7 - 19 ms	9 ms
11.0.0	7 - 16 ms	9 ms