Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(global-options): Default concurrency to logical CPU count #1931

Merged
merged 2 commits into from Feb 14, 2019
Merged

feat(global-options): Default concurrency to logical CPU count #1931

merged 2 commits into from Feb 14, 2019

Conversation

ghost
Copy link

@ghost ghost commented Feb 13, 2019

Description

The PR makes little changes to set concurrency option to count of logical CPU cores by default.

Motivation and Context

--concurrency option is set to 4 by default what can be not the best value with different count of CPUs.
Setting the option to specific number isn't good options cause the same script can be run on different servers.

Fixes #1929

How Has This Been Tested?

Unfortunately, my computer have 4 logical CPU what was the value before my changes, so for now I didn't test what is effected. I'm going to run lerna tests in virtual environments with different count of CPU cores.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • I have updated the documentation accordingly.
  • My change requires a change to the documentation.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

Copy link
Member

@evocateur evocateur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, I'm concerned that os.cpus().length isn't always the appropriate value for concurrent operations. Not that 4 is much better, it's pretty arbitrary.

core/global-options/README.md Show resolved Hide resolved
core/command/index.js Show resolved Hide resolved
@evocateur evocateur changed the title Set concurrency option to count of logical CPU cores by default feat(global-options): Default concurrency to logical CPU count Feb 13, 2019
@evocateur
Copy link
Member

(My concerns aside, I'm in favor of this change)

@bdwain
Copy link
Contributor

bdwain commented Feb 13, 2019

Overall, I'm concerned that os.cpus().length isn't always the appropriate value for concurrent operations. Not that 4 is much better, it's pretty arbitrary.

I agree. But when it's not, my guess is they probably have a hard coded number in mind anyway, so they can just set the concurrency if they want.

If people wanted to use half of the cpus, or all but one, they'd need a node script to do that. But that's the case today today, and if it was that popular of a request, there could be options such as --use-half-cpus or --use-all-but-one-cpu. I doubt they'd be necessary though.

too long and it mangles the help output
@evocateur evocateur merged commit 2c487fe into lerna:master Feb 14, 2019
@fyodorvi
Copy link

fyodorvi commented Mar 30, 2019

FYI, it seems that this feature caused us some grief in CircleCI where running a command across 30+ packages would randomly fail on one of the packages with:

lerna ERR! yarn run build stderr:
error Command failed with signal "SIGKILL".

We fixed it by explicitly adding --concurrency 2 (2 is the default number of core CircleCI gives you).

@sidharthachatterjee
Copy link

CircleCI doesn't report number of cores correctly so this fails on it without concurrency being added explicitly 😞

@evocateur
Copy link
Member

Sounds like a Circle CI bug to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Change default concurrency from 4 to available number of cpus
5 participants