Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

benchmark: rewrite the startup performance benchmark #45746

Merged
merged 5 commits into from Dec 8, 2022
Merged

benchmark: rewrite the startup performance benchmark #45746

merged 5 commits into from Dec 8, 2022

Conversation

joyeecheung
Copy link
Member

benchmark: make benchmarks runnable in older versions of Node.js

Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.

benchmark: rewrite the startup performance benchmark

The previous benchmark was inaccurate in several ways:

  • It previously measured "how many workers/child processes can finish
    loading in parallel in 1 second" which is prone to wrong
    interpretations. It's also not typical for users to spin up >20
    workers/child processes in parallel, which was what the benchmark
    usually ended up doing. The parallelism can also be greatly affected
    by system configurations.
  • The time spent on loading a Node.js binary for the first time varies
    greatly depending on system configurations, but it was inevitable
    for the benchmark to load Node.js multiple times to reduce the
    influence of fluctuations.

This patch rewrites the benchmark so that:

  • It now measures "how long it takes to finish 30 workers/child processes
    in serial" which is generally more in line with how other benchmarks
    are written and with the figures one gets from doing something like
    time node index.js or hyperfine 'node index.js'
  • It now warms up the loading of the Node.js instance to reduce the
    statistical outliers, so that we get more meaningful figures when
    trying to compare the startup performance of different Node.js
    binaries.

@joyeecheung
benchmark: make benchmarks runnable in older versions of Node.js
996719f 
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.
@joyeecheung
benchmark: rewrite the startup performance benchmark
bc05a5f 
The previous benchmark was inaccurate in several ways:

- It previously measured "how many workers/child processes can finish
  loading in parallel in 1 second" which is prone to wrong
  interpretations. It's also not typical for users to spin up >20
  workers/child processes in parallel, which was what the benchmark
  usually ended up doing. The parallelism can also be greatly affected
  by system configurations.
- The time spent on loading a Node.js binary for the first time varies
  greatly depending on system configurations, but it was inevitable
  for the benchmark to load Node.js multiple times to reduce the
  influence of fluctuations.

This patch rewrites the benchmark so that:

- It now measures "how long it takes to finish 30 workers/child processes
  in serial" which is generally more in line with how other benchmarks
  are written and with the figures one gets from doing something like
  `time node index.js` or `hyperfine 'node index.js'`
- It now warms up the loading of the Node.js instance to reduce the
  statistical outliers, so that we get more meaningful figures when
  trying to compare the startup performance of different Node.js
  binaries.
@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/startup

@nodejs-github-bot nodejs-github-bot added the benchmark Issues and PRs related to the benchmark subsystem. label Dec 5, 2022
@joyeecheung
Copy link
Member Author

joyeecheung commented Dec 5, 2022 •
edited

New benchmark results from #45659, which is more in line with the results I see from hyperfine:

                                                                                     confidence improvement accuracy (*)   (**)  (***)
misc/startup.js count=30 mode='process' script='benchmark/fixtures/require-builtins'                -0.25 %       ±2.05% ±2.78% ±3.71%
misc/startup.js count=30 mode='process' script='test/fixtures/semicolon'                    ***     14.04 %       ±2.80% ±3.79% ±5.06%
misc/startup.js count=30 mode='worker' script='benchmark/fixtures/require-builtins'                  0.19 %       ±2.10% ±2.84% ±3.79%
misc/startup.js count=30 mode='worker' script='test/fixtures/semicolon'                     ***     16.86 %       ±2.67% ±3.61% ±4.82%

I'll recompile and see the new results from #45716

@joyeecheung joyeecheung mentioned this pull request Dec 5, 2022
Closed
@joyeecheung
Copy link
Member Author

New benchmark numbers from #45716


                                                                                     confidence improvement accuracy (*)   (**)  (***)
misc/startup.js count=30 mode='process' script='benchmark/fixtures/require-builtins'        ***     12.28 %       ±1.13% ±1.52% ±2.00%
misc/startup.js count=30 mode='process' script='test/fixtures/semicolon'                    ***     17.39 %       ±1.12% ±1.50% ±1.97%
misc/startup.js count=30 mode='worker' script='benchmark/fixtures/require-builtins'         ***     14.54 %       ±0.77% ±1.03% ±1.35%
misc/startup.js count=30 mode='worker' script='test/fixtures/semicolon'                     ***     18.02 %       ±1.02% ±1.36% ±1.79%

@joyeecheung joyeecheung mentioned this pull request Dec 5, 2022
Merged
@legendecas
Copy link
Member

Linter is still complaining.

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/48305/

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/48313/

@github-actions github-actions bot mentioned this pull request Dec 8, 2022
Open
8 tasks
@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/48341/

@joyeecheung joyeecheung added the review wanted PRs that need reviews. label Dec 8, 2022
@joyeecheung
Copy link
Member Author

@nodejs/performance Can I have some reviews please?

@anonrig anonrig added performance Issues and PRs related to the performance of Node.js. author ready PRs that have at least one approval, no pending requests for changes, and a CI started. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. commit-queue Add this label to land a pull request using GitHub Actions. labels Dec 8, 2022
@nodejs-github-bot nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Dec 8, 2022
@nodejs-github-bot nodejs-github-bot merged commit 2b9d3b2 into nodejs:main Dec 8, 2022
@nodejs-github-bot
Copy link
Collaborator

Landed in 2b9d3b2

@github-actions github-actions bot mentioned this pull request Dec 9, 2022
Open
5 tasks
This was referenced Dec 10, 2022
Open
Open
ErickWendel pushed a commit to ErickWendel/node that referenced this pull request Dec 12, 2022
@joyeecheung @ErickWendel
benchmark: make benchmarks runnable in older versions of Node.js
6dc3e2d 
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.

PR-URL: nodejs#45746
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
targos pushed a commit that referenced this pull request Dec 12, 2022
@joyeecheung @targos
benchmark: make benchmarks runnable in older versions of Node.js
6f0b1a1 
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.

PR-URL: #45746
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
@targos targos mentioned this pull request Dec 12, 2022
Merged
targos pushed a commit that referenced this pull request Dec 13, 2022
@joyeecheung @targos
benchmark: make benchmarks runnable in older versions of Node.js
102c2dc 
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.

PR-URL: #45746
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
danielleadams pushed a commit that referenced this pull request Dec 30, 2022
@joyeecheung @danielleadams
benchmark: make benchmarks runnable in older versions of Node.js
f292117 
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.

PR-URL: #45746
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
@danielleadams danielleadams mentioned this pull request Dec 30, 2022
Merged
danielleadams pushed a commit that referenced this pull request Dec 30, 2022
@joyeecheung @danielleadams
benchmark: make benchmarks runnable in older versions of Node.js
199b4f1 
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.

PR-URL: #45746
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
danielleadams pushed a commit that referenced this pull request Jan 3, 2023
@joyeecheung @danielleadams
benchmark: make benchmarks runnable in older versions of Node.js
5e167bd 
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.

PR-URL: #45746
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
danielleadams pushed a commit that referenced this pull request Jan 4, 2023
@joyeecheung @danielleadams
benchmark: make benchmarks runnable in older versions of Node.js
d6014bb 
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.

PR-URL: #45746
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
danielleadams pushed a commit that referenced this pull request Jan 5, 2023
@joyeecheung @danielleadams
benchmark: make benchmarks runnable in older versions of Node.js
1442010 
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.

PR-URL: #45746
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@addaleax addaleax addaleax approved these changes

@anonrig anonrig anonrig approved these changes

@legendecas legendecas legendecas approved these changes

Assignees
No one assigned
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. benchmark Issues and PRs related to the benchmark subsystem. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. performance Issues and PRs related to the performance of Node.js. review wanted PRs that need reviews.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@joyeecheung @nodejs-github-bot @legendecas @addaleax @anonrig