Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add async parallel helper and update benchmarks to use it #730

Open
alangpierce opened this issue Aug 1, 2022 · 0 comments
Open

Add async parallel helper and update benchmarks to use it #730

alangpierce opened this issue Aug 1, 2022 · 0 comments

Comments

@alangpierce
Copy link
Owner

Currently, the README benchmark measures the single-threaded/per-core performance of each tool. For Sucrase, Babel, and TypeScript, this is the easiest way to invoke the tools, but for swc and esbuild, I took some care to make sure they are invoked in a single-threaded way for the sake of a fair comparison. This approach has received some skepticism, e.g. https://twitter.com/JakeDChampion/status/1384919314755637248 , so it would probably be more convincing to build a (hopefully) simple orchestration layer that runs Sucrase, Babel, and TypeScript in parallel and post those numbers as the primary benchmark. I'm thinking that I'll also link to a more detailed write-up with an explanation of the caveats and some reproducible benchmark numbers to illustrate the effects of parallelism and JIT warm-up.

While I still think that per-core performance is a fine way to measure these tools (given that transpilation is inherently parallelizable without needing the tool to support it), it's a very fair criticism that swc and esbuild make it easy to parallelize work, while Sucrase expects you to set it up yourself. I'm hoping that part of this work can be to provide an async transform that runs Sucrase on a background thread (or process) with reasonable defaults. Of course, this mechanism isn't as portable as a plain JS function, so I'll want to make sure people can still use the regular transform function without expecting Node. This function will hopefully make it easier for people to do their own comparisons, and may be useful for a built-in Node ESM loader in the future as well.

As a very simple proof of concept, I tested the peak parallel performance of each tool by running each in multiple terminal windows on the same machine, continuously in a loop and reporting its speed at each iteration. With just one terminal window, parallel swc and parallel esbuild are a little faster than Sucrase, but Sucrase has a much faster combined speed across four terminal windows. On my laptop, Sucrase maxes out at about 3 million lines per second, swc maxes out at about 1.2 million lines per second, and esbuild maxes out at about 0.8 million lines per second, which is consistent with my single-core measurements allowing for JIT warm-up.

My current plan is to use either https://github.com/piscinajs/piscina or https://github.com/tinylibs/tinypool for parallel orchestration, though it looks like I may need to work through some performance issues of those libraries when handling many small tasks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant