Add default concurrency limit

[sindresorhus]

Currently the concurrency limit is Infinity. I have a gut feeling it would generally be faster with some kind of concurrency limit. Probably a number between 4 and 10. This will need some benchmarking. Help wanted.

Context: 66f069f#commitcomment-21722984

[evocateur]

Do you have any preferred benchmarking tools, or would repeated runs of a shell script calling time del packages/*/node_modules in a bootstrapped repo suffice?

[sindresorhus]

I usually use https://github.com/logicalparadox/matcha

Example: https://github.com/sindresorhus/strip-json-comments/blob/254d84a3826b3a7acba8dc484660a97c43dca821/package.json#L17 https://github.com/sindresorhus/strip-json-comments/blob/254d84a3826b3a7acba8dc484660a97c43dca821/benchmark.js

[nexdrew]

Would a default of require('os').cpus().length (or some derivative thereof) be too naive?

[evocateur]

That was my first thought, but then considered that a single core only manages one process, as I understand it. Basically, cpus().length only really helps when using child processes.

…

On Jun 9, 2017, at 09:05, Andrew Goode ***@***.***> wrote: Would a default of require('os').cpus().length (or some derivative thereof) be too naive? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[nexdrew]

I took an initial stab at creating a matcha benchmark test. You can find it in the benchmark branch of my fork.

I'm probably doing something wrong, but I don't think these tests show an appreciable difference between different concurrency settings. Here's an example run on my machine:

Note that in the initial version of these tests, the first actual deletion has to find and delete half of the 1000 created files, and the last actual deletion deletes the other half.

Let me know if this looks helpful at all. Comments welcomed.

[frank-dspeed]

the benchmarks are fals and you should not optimize that with a concurrency limit as a engine author i can tell you that this is part of v8 already you simply fire your stack of actions in and it takes care of the rest everything else will not be the same as your tests include filesystem operations that take time too

and they depend on the write and delete methods that the filesystem uses and so on

to test the raw performance it would be enough to test how many actions you can fire so how many fsCalls you can put on the stack and again that is also optimized by v8 it self and nodejs abstracts that and maybe makes that even more buggy via its libuv integration the loop is bound to both v8 and libuv which does the I/O processing between the OS and the c code that calls v8