Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify cpuCount utility and add limiting functionality #5673

Open
wants to merge 5 commits into
base: v2
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 2 additions & 2 deletions packages/core/workers/src/WorkerFarm.js
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import {
} from '@parcel/core';
import ThrowableDiagnostic, {anyToDiagnostic, md} from '@parcel/diagnostic';
import Worker, {type WorkerCall} from './Worker';
import cpuCount from './cpuCount';
import {getCoreCount} from './cpuCount';
import Handle from './Handle';
import {child} from './childState';
import {detectBackend} from './backend';
Expand Down Expand Up @@ -557,7 +557,7 @@ export default class WorkerFarm extends EventEmitter {
static getNumWorkers(): number {
return process.env.PARCEL_WORKERS
? parseInt(process.env.PARCEL_WORKERS, 10)
: cpuCount();
: getCoreCount();
}

static isWorker(): boolean {
Expand Down
68 changes: 8 additions & 60 deletions packages/core/workers/src/cpuCount.js
Original file line number Diff line number Diff line change
@@ -1,66 +1,14 @@
// @flow
import os from 'os';
import {execSync} from 'child_process';

const exec = (command: string): string => {
try {
let stdout = execSync(command, {
encoding: 'utf8',
// This prevents the command from outputting to the console
stdio: [null, null, null],
});
return stdout.trim();
} catch (e) {
return '';
let cachedCoreCount = 0;
const platform = os.platform();
export function getCoreCount(
limit: number = platform === 'win32' ? 4 : 8,
Copy link
Contributor

@aminya aminya Mar 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not an OS-relevant issue. The parcel doesn't use the threads it spawns. Why do you want to create so many threads which Parcel doesn't use?

#5072 (comment)

Other than workers 0 and 1, the others almost do nothing.
image

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought this was a filesystem limitation of windows

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a separate issue.

Copy link
Contributor

@aminya aminya Apr 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
limit: number = platform === 'win32' ? 4 : 8,
limit: number = 4,

Copy link
Contributor

@aminya aminya Apr 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The smarter way is to spawn threads proportional to the work that Parcel wants to do. For example:

min((fileNumbers)/20, coresAvailable)

So, if there are 100 files and we have 16 cores, it will spawn only 5 cores.

Copy link
Member

@mischnic mischnic Apr 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The smarter way is to spawn threads proportional to the work that Parcel wants to do. For example:

Yes, but we don't (always) know this beforehand.

Copy link
Contributor

@aminya aminya Apr 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use the information when it is available. When not known, we can just use a constant:

min(fileNumbers ? (fileNumbers)/20 : 4, coresAvailable)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is fileNumbers?
On an initial build (where there's the most work), we don't know.
For subsequent builds, we only know the filecount of the previous build, not how much changed (= how much work there is).

So I would say this isn't possible to know at all

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So maybe it is better to perform an initial analysis using all threads. Then after that, it can use a suitable number of workers for the actual build.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would probably slow it down more than it would speed up?

): number {
if (!cachedCoreCount) {
cachedCoreCount = Math.ceil(os.cpus().length / 2);
}
};

export function detectRealCores(): number {
let platform = os.platform();
let amount = 0;

if (platform === 'linux') {
amount = parseInt(
exec('lscpu -p | egrep -v "^#" | sort -u -t, -k 2,4 | wc -l'),
10,
);
} else if (platform === 'darwin') {
amount = parseInt(exec('sysctl -n hw.physicalcpu_max'), 10);
} else if (platform === 'win32') {
const str = exec('wmic cpu get NumberOfCores').match(/\d+/g);
if (str !== null) {
amount = parseInt(str.filter(n => n !== '')[0], 10);
}
}

if (!amount || amount <= 0) {
throw new Error('Could not detect cpu count!');
}

return amount;
}

let cores;
export default function getCores(bypassCache?: boolean = false): number {
// Do not re-run commands if we already have the count...
if (cores && !bypassCache) {
return cores;
}

try {
cores = detectRealCores();
} catch (e) {
// Guess the amount of real cores
cores = os
.cpus()
.filter((cpu, index) => !cpu.model.includes('Intel') || index % 2 === 1)
.length;
}

// Another fallback
if (!cores) {
cores = 1;
}

return cores;
return Math.max(Math.min(cachedCoreCount, limit), 1);
}
24 changes: 14 additions & 10 deletions packages/core/workers/test/cpuCount.test.js
Original file line number Diff line number Diff line change
@@ -1,19 +1,23 @@
import assert from 'assert';
import os from 'os';

import getCores, {detectRealCores} from '../src/cpuCount';
import {getCoreCount} from '../src/cpuCount';

describe('cpuCount', function() {
it('Should be able to detect real cpu count', () => {
// Windows not supported as getting the cpu count takes a couple seconds...
if (os.platform() === 'win32') return;
let cpus = os.cpus().length;

let cores = detectRealCores();
describe('cpuCount', function() {
it('getCoreCount should return more than 0', () => {
let cores = getCoreCount();
assert(cores > 0);
});

it('getCores should return more than 0', () => {
let cores = getCores(true);
assert(cores > 0);
});
if (cpus > 2) {
it('Should be able to limit coreCount', () => {
let allCores = getCoreCount();
let limitedCores = getCoreCount(1);

assert(allCores > limitedCores);
assert.equal(limitedCores, 1);
});
}
});