Parse AST in worker #5211
Parse AST in worker #5211
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Thank you for your contribution! ❤️You can try out this pull request locally by installing Rollup via npm install rollup/rollup#parse-web-workerNotice: Ensure you have installed Rust nightly. If you haven't installed it yet, please first see https://www.rust-lang.org/tools/install to learn how to download Rustup and install Rust, then see https://rust-lang.github.io/rustup/concepts/channels.html to learn how to install Rust nightly. or load it into the REPL: |
Codecov Report
@@ Coverage Diff @@
## master #5211 +/- ##
==========================================
- Coverage 98.82% 98.76% -0.07%
==========================================
Files 231 233 +2
Lines 8849 8884 +35
Branches 2314 2319 +5
==========================================
+ Hits 8745 8774 +29
- Misses 43 49 +6
Partials 61 61
|
Also fix a type issue
lukastaegert
force-pushed
the
parse-web-worker
branch
from
a509b7b
to
b0706eb
Compare
lukastaegert
force-pushed
the
parse-web-worker
branch
from
b0706eb
to
265836d
Compare
|
worker_threads are not as efficient as native POSIX threads by design. Since the current use case is delightfully parallellisable, a unique native POSIX thread in rust with JS bindings and parsing task concurrency limit and a task FIFO will very likely out perform any worker_threads pool implementations available (I co-maintain one and be warned that: worker_threads communication are dead slow under load, worker_threads high memory usage needs tweaking to avoid OOM conditions, worker_threads startup time is quite slow compared to native POSIX thread, worker_threads memory sharing is way less convenient to work with than with native POSIX threads). |
|
Yes, this confirms my observations at #5202 (comment), which is why I just merged and released @sapphi-red's approach #5202. This PR currently is mostly for reference and as inspiration if we want to parallelize the WASM builds after all. |
This PR contains:
Are tests included?
Breaking Changes?
List any relevant issue numbers:
cf. #5202
Description
This will move regular parsing into a worker thread. It would be interesting to compare this approach with #5202. The advantage of using a worker thread as opposed to native threads in Rust is that this approach also parallelizes the
@rollup/wasm-nodebuild.As parsing is currently faster than analyzing and tree-shaking, I do not think it makes sense to use more than one additional thread. However, I set it up in a way so that we could easily switch to a worker pool with a configurable number of workers.
Originally, I wanted to use a singleton worker. However, then it became difficult to decide when to terminate the worker, as a running worker would prevent the process from completing. Instead, the worker will now terminate itself if inactive for one macro task tick. Another approach could have been to terminate the worker on build end. However in the long run, I hope to extend the plugin API to allow async parsing in generate hooks, in which case we would still need the worker.