Performance optimization for simple patterns

[mrmlnc]

Environment

• OS Version: macOS High Sierra
• Node.js Version: 10.9.0

Prerequisites

$ ls node_modules/ | wc -l
     518

await fg('node_modules/*');

Problem

  fast-glob
    node_modules
       - readdir 0.886335ms [0.811814ms...2.300569ms]
       - python.glob (bridge) 3.079168ms [2.979289ms...3.263059ms]
       - glob 11.066838ms [10.789296ms...11.703257ms]
       - tiny-glob 117.77282ms [116.387603ms...120.094648ms]
       - globby 265.250122ms [259.802455ms...283.017961ms]
       - fast-glob 266.385609ms [260.734759ms...281.370003ms]

[mrmlnc]

We read more than we should.

node_modules/.bin (depth=0, maxDepth=2)
node_modules/@lerna (depth=0, maxDepth=2)
node_modules/@mrmlnc (depth=0, maxDepth=2)
...
node_modules/dateformat (depth=0, maxDepth=2)
node_modules/yargs (depth=0, maxDepth=2)
node_modules/yargs-parser (depth=0, maxDepth=2)
...
node_modules/@lerna/add (depth=1, maxDepth=2)
node_modules/@lerna/batch-packages (depth=1, maxDepth=2)
...
node_modules/vrsource-tslint-rules/test/rules/prefer-case-blocks (depth=3, maxDepth=2)
node_modules/vrsource-tslint-rules/test/rules/prefer-literal (depth=3, maxDepth=2)

So, this is related to the deep filter and the getMaxPatternDepth method inside it.

Estimated time

I have manually specified the maximum number of subdirectories to read.

  fast-glob
    node_modules/*
       - readdir 0.942ms [0.809ms...3.046ms]
       - python (bridge) 3.116ms [2.977ms...3.927ms]
       - glob 11.055ms [10.838ms...11.694ms]
       - tiny-glob 116.927ms [115.202ms...118.892ms]
       - globby 10.396ms [9.266ms...11.683ms]
       - fast-glob 10.268ms [8.630ms...11.676ms]

[mrmlnc]

Right now we can solve the problem, but with very bad code, because we're trying to guess the depth of the pattern from user input.

When we switch to fs.walk we will not need to use two different filters (deep and entry) because fs.walk will call the same function to filter both cases.

Perhaps we made an incorrect decision within issue #83.

I don't remember why we didn't use positive patterns to understand whether to read the directory now or not. I'll take a look at it again.

[mrmlnc]

Will be shipped at #136.