Skip to content

Commit

Permalink
Merge pull request #434 from mrmlnc/remove_concurrency_option
Browse files Browse the repository at this point in the history
Remove the concurrency option
  • Loading branch information
mrmlnc committed Dec 25, 2023
2 parents 4b71e29 + aad2a32 commit a1a07fa
Show file tree
Hide file tree
Showing 9 changed files with 0 additions and 48 deletions.
26 changes: 0 additions & 26 deletions README.md
Expand Up @@ -27,7 +27,6 @@ This package provides methods for traversing the file system and returning pathn
* [convertPathToPattern](#convertpathtopatternpath)
* [Options](#options-3)
* [Common](#common)
* [concurrency](#concurrency)
* [cwd](#cwd)
* [deep](#deep)
* [followSymbolicLinks](#followsymboliclinks)
Expand Down Expand Up @@ -307,31 +306,6 @@ fg.win32.convertPathToPattern('\\\\?\\c:\\Program Files (x86)') + '/**/*';

### Common options

#### concurrency

* Type: `number`
* Default: `os.cpus().length`

Specifies the maximum number of concurrent requests from a reader to read directories.

> :book: The higher the number, the higher the performance and load on the file system. If you want to read in quiet mode, set the value to a comfortable number or `1`.
<details>

<summary>More details</summary>

In Node, there are [two types of threads][nodejs_thread_pool]: Event Loop (code) and a Thread Pool (fs, dns, …). The thread pool size controlled by the `UV_THREADPOOL_SIZE` environment variable. Its default size is 4 ([documentation][libuv_thread_pool]). The pool is one for all tasks within a single Node process.

Any code can make 4 real concurrent accesses to the file system. The rest of the FS requests will wait in the queue.

> :book: Each new instance of FG in the same Node process will use the same Thread pool.
But this package also has the `concurrency` option. This option allows you to control the number of concurrent accesses to the FS at the package level. By default, this package has a value equal to the number of cores available for the current Node process. This allows you to set a value smaller than the pool size (`concurrency: 1`) or, conversely, to prepare tasks for the pool queue more quickly (`concurrency: Number.POSITIVE_INFINITY`).

So, in fact, this package can **only make 4 concurrent requests to the FS**. You can increase this value by using an environment variable (`UV_THREADPOOL_SIZE`), but in practice this does not give a multiple advantage.

</details>

#### cwd

* Type: `string`
Expand Down
1 change: 0 additions & 1 deletion src/benchmark/suites/product/async.ts
Expand Up @@ -33,7 +33,6 @@ class Glob {
cwd: this.#cwd,
unique: false,
followSymbolicLinks: false,
concurrency: Number.POSITIVE_INFINITY,
}));
}

Expand Down
1 change: 0 additions & 1 deletion src/benchmark/suites/product/stream.ts
Expand Up @@ -49,7 +49,6 @@ class Glob {
cwd: this.#cwd,
unique: false,
followSymbolicLinks: false,
concurrency: Number.POSITIVE_INFINITY,
});

const action = new Promise<string[]>((resolve, reject) => {
Expand Down
1 change: 0 additions & 1 deletion src/benchmark/suites/regression/async.ts
Expand Up @@ -20,7 +20,6 @@ class Glob {
this.#options = {
unique: false,
followSymbolicLinks: false,
concurrency: Number.POSITIVE_INFINITY,
...options,
};
}
Expand Down
1 change: 0 additions & 1 deletion src/benchmark/suites/regression/stream.ts
Expand Up @@ -20,7 +20,6 @@ class Glob {
this.#options = {
unique: false,
followSymbolicLinks: false,
concurrency: Number.POSITIVE_INFINITY,
...options,
};
}
Expand Down
1 change: 0 additions & 1 deletion src/providers/provider.spec.ts
Expand Up @@ -75,7 +75,6 @@ describe('Providers → Provider', () => {
const actual = provider.getReaderOptions(task);

assert.strictEqual(actual.basePath, '');
assert.strictEqual(actual.concurrency, settings.concurrency);
assert.strictEqual(typeof actual.deepFilter, 'function');
assert.strictEqual(typeof actual.entryFilter, 'function');
assert.strictEqual(typeof actual.errorFilter, 'function');
Expand Down
1 change: 0 additions & 1 deletion src/providers/provider.ts
Expand Up @@ -40,7 +40,6 @@ export abstract class Provider<T> {
return {
basePath,
pathSegmentSeparator: '/',
concurrency: this.#settings.concurrency,
deepFilter: this.deepFilter.getFilter(basePath, task.positive, task.negative),
entryFilter: this.entryFilter.getFilter(task.positive, task.negative),
errorFilter: this.errorFilter.getFilter(),
Expand Down
2 changes: 0 additions & 2 deletions src/settings.spec.ts
@@ -1,5 +1,4 @@
import * as assert from 'node:assert';
import * as os from 'node:os';

import Settings, { DEFAULT_FILE_SYSTEM_ADAPTER } from './settings';

Expand All @@ -26,7 +25,6 @@ describe('Settings', () => {
assert.ok(settings.globstar);
assert.ok(settings.onlyFiles);
assert.ok(settings.unique);
assert.strictEqual(settings.concurrency, os.cpus().length);
assert.strictEqual(settings.cwd, process.cwd());
});

Expand Down
14 changes: 0 additions & 14 deletions src/settings.ts
@@ -1,5 +1,4 @@
import * as fs from 'node:fs';
import * as os from 'node:os';

import type { FileSystemAdapter, Pattern } from './types';

Expand Down Expand Up @@ -38,13 +37,6 @@ export interface Options {
* @default true
*/
caseSensitiveMatch?: boolean;
/**
* Specifies the maximum number of concurrent requests from a reader to read
* directories.
*
* @default os.cpus().length
*/
concurrency?: number;
/**
* The current working directory in which to search.
*
Expand Down Expand Up @@ -153,7 +145,6 @@ export default class Settings {
public readonly baseNameMatch: boolean;
public readonly braceExpansion: boolean;
public readonly caseSensitiveMatch: boolean;
public readonly concurrency: number;
public readonly cwd: string;
public readonly deep: number;
public readonly dot: boolean;
Expand All @@ -177,11 +168,6 @@ export default class Settings {
this.baseNameMatch = options.baseNameMatch ?? false;
this.braceExpansion = options.braceExpansion ?? true;
this.caseSensitiveMatch = options.caseSensitiveMatch ?? true;
/**
* The `os.cpus` method can return zero. We expect the number of cores to be greater than zero.
* https://github.com/nodejs/node/blob/7faeddf23a98c53896f8b574a6e66589e8fb1eb8/lib/os.js#L106-L107
*/
this.concurrency = options.concurrency ?? Math.max(os.cpus().length, 1);
this.cwd = options.cwd ?? process.cwd();
this.deep = options.deep ?? Number.POSITIVE_INFINITY;
this.dot = options.dot ?? false;
Expand Down

0 comments on commit a1a07fa

Please sign in to comment.