benchmark: add misc/startup-cli-version benchmark

[joyeecheung]

This benchmarks the startup of various CLI tools that are already checked into the source code. We use --version because the output tends to be minimal and fewer operations are done to generate these so that the startup cost is still dominated by a more indispensible part of the CLI.

[GeoffreyBooth]

You could add TypeScript too.

Are all of these CommonJS? Is there anything we can add to benchmark ESM startup?

[joyeecheung]

You could add TypeScript too.

We don’t have the TypeScript CLI checked in, only one library bundle is in the fixtures (so that's supposed to be require()d). Though that could probably be added to the other startup benchmark or as a CommonJS bundle loading benchmark of its own. Probably out of scope for this one though.

Are all of these CommonJS? Is there anything we can add to benchmark ESM startup?

I can’t find any existing CLI checked into the code base using ESM. If we have something in the future or if any one of them gets updated to use ESM then that can be covered. Otherwise I guess this is just what we have for now. This just serves as “since we already have some real-world CLIs checked in the code base, might as well use them for some benchmarking to ensure we don’t regress real-world CLIs”. Whether these real-world CLIs uses ESM is up to them.

If we really want to check ESM performance for CLIs that we don’t use, I’d suggest doing it in a different repo (the CLIs listed here are already making the checkout humongous, but then we at least have some proper use of them other than benchmarking). I'd say the same applies to the maintenance of this list - if for some reason we no longer use any of them (I doubt that though, maybe eslint can go away in the future? But the other two probably won't), we should just remove them from this benchmark and should't keep them in the codebase just for benchmarking.

[GeoffreyBooth]

I can’t find any existing CLI checked into the code base using ESM. If we have something in the future or if any one of them gets updated to use ESM then that can be covered. Otherwise I guess this is just what we have for now.

Yeah, this is something we need but I agree I don’t want to add unnecessary dependencies to the repo.

You could perhaps have another set of spawn calls that include --import 'data:text/javascript,' (a no-op --import) to trigger the handling of the entry point to be by the ESM loader. Everything after the initial entry would still be handled by the CommonJS loader, but at least we would be benchmarking the “startup to ESM entry” part of the flow.

[joyeecheung]

You could perhaps have another set of spawn calls that include --import 'data:text/javascript,' (a no-op --import) to trigger the handling of the entry point to be by the ESM loader.

That sounds like a separate benchmark than this one, which is about real world CLIs. I haven’t seen any CLIs that would be loaded this way in the real world (maybe there are, but I just haven’t seen any)

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/55600/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/55661/

[nodejs-github-bot]

Landed in 59b27d6