[napi] Support WebAssembly

[yisibl]

By integrating with wasm_bindgen, we can get WASM support.

Once supported, we can automate what this PR does: parcel-bundler/lightningcss#1

[BasixKOR]

It won't make sense anyway. N-API is Node-js-only feature that doesn't work on the web and wasm-bindgen is already doing a good job in this field.

[yisibl]

You misunderstood, this issue actually expects to provide an ability to bind wasm-bindgen.

[daniel-brenot]

Why though? If you wish to call wasm rust from js you can do that with wasm-bindgen. This library exists to allow running specifically native code. Could you elaborate your intent if we have misunderstood?

[Brooooooklyn]

It's not possible for napi-rs 1.x to support wasm, but with the procedural macro introduced in 2.0, we can generate code for Rust WASM compiling. For example:

#[napi]
fn add(a: u32, b: u32) -> u32 {
  a + b
}

We can generate wasm bindgen code for the wasm compile target:

#[wasm_bindgen]
fn add(a: u32, b: u32) -> u32 {
  a + b
}

[daniel-brenot]

Unless i've missed something, I still don't quite understand the use case for this.

[Brooooooklyn]

Image you have a library like crc32:

use crc32;

#[napi]
fn crc32(input: Buffer) -> u32 {
  crc32::caclulate(input.as_slice())
}

With the WebAssembly support, you can just run napi build --release to build native addons. And run napi build --release --target wasm32-unknown-unknown to build wasm library.

The code will be totally reused.

[daniel-brenot]

So then the use case is to reuse the same code to produce a native bindgen or a wasm bindgen? If so, that sounds like a great feature.

[Brooooooklyn]

So then the use case is to reuse the same code to produce a native bindgen or a wasm bindgen

Yes, this will be released in 3.0

[nickbabcock]

I'm curious what the plan is for how Tasks will be supported in Wasm. I use napi-rs to run compute heavy tasks off the main thread and I'm not sure how well that translates to the browser. Would it be on the user's responsibility to ensure that the napi-rs produced Wasm is executed in a web worker (so tasks would only are computed off the execution thread in node.js) or would napi-rs produce an output that would manage a pool of web workers to run shuffle tasks around.

Anyways some food for thought as this issue is explored.

[janus-reith]

I use napi-rs to run compute heavy tasks off the main thread and I'm not sure how well that translates to the browser.

While the output should be able to run in a browser too, I believe the main target here would be server-side executed wasm, in general I assume implementation details like if the output should run in a worker would be up to the user?

[Kayshen-X]

So then the use case is to reuse the same code to produce a native bindgen or a wasm bindgen

Yes, this will be released in 3.0

When is version 3.0 planned to be released?

[devongovett]

I kinda got this working a different way - by reimplementing napi in JS so it can be used from WASM: https://github.com/devongovett/napi-wasm. I'm planning on using it in lightningcss to avoid having two separate implementations, which is getting to be more work as we add more features. With this, the existing napi implementation just works when recompiled for WASM.

I ran the napi-rs compat mode test suite using it and it mostly passes, minus some features that don't work in WASM (like threads and fs access). In addition, there are a few other features that don't work because napi-rs uses the #[ctor] attribute which is currently not supported in wasm targets (see rustwasm/wasm-bindgen#1216). To work around that, I manually implemented napi_register_wasm_v1 and registered my exports in there (see example in the readme). But that won't work for other features in the napi bindgen API which use ctors to register things for later use in napi_register_module_v1. Perhaps there is some other way of doing that though.

Anyway, maybe this approach is easier than modifying all of napi-rs to have two completely different implementations? Most of what's being done in napi-wasm is similar to what wasm-bindgen generates anyway.

[toyobayashi]

I created https://github.com/toyobayashi/emnapi for emscripten, well tested by using Node.js official test cases, and it support async work and thread-safe functions on modern browsers (based on emscripten pthreads). I'm not familiar with rust, not sure if it can help.

[toyobayashi]

Successfully make emnapi work with napi-rs and wasi-sdk

toyobayashi/emnapi#29

I compiled the rust code into a static library of the wasm32-unknown-unknown target, linked against emnapi C code compiled by clang. I'm new to rust, not sure if there's a better way to do this, but it works.

[devongovett]

Very cool! I'm curious what the advantage of using Emscripten for this is vs implementing it in JS? I see you have part implemented in C and a separate bridge layer in JS, which is very similar to how wasm-bindgen works. But why is that extra layer needed? Seems more straightforward to just implement napi in JS and inject it as an external library into the WASM module as I've done in https://github.com/devongovett/napi-wasm. Adding Emscripten and a whole extra layer seems like it would complicate the build and increase the binary size potentially.

[toyobayashi]

I see you have part implemented in C and a separate bridge layer in JS

Only the following things implemented in C:

• napi_*_async_work
• napi_*_threadsafe_function
• napi_*_cleanup_hook
• APIs those return static variable such as napi_get_last_error_info and napi_get_node_version

To make the behavior close to Node.js as much as possible, emnapi has ported libuv's threadpool which is required by async work related API, and this uses emscripten's pthread support, and the C implementation is rewrite according to Node.js source code. Other API implemented in JavaScript is also consistent with the behavior of the nodejs source code, I have separated JS runtime code to npm package to make the runtime reusable for non-emscripten environment.

Adding Emscripten

No, we do not need emscripten. But maybe we need wasi-sdk in the future due to emnapi C implementation use the C library and pthread. toyobayashi/emnapi#29 is working on non-emscripten support.

[toyobayashi]

Currently the latest version wasi-sdk 19 has no threads support yet, so pthread related APIs are unavailable. napi-rs cli could find emnapi precompiled static library to link, I plan to publish static library precompiled by clang in next version @tybys/emnapi package.

Two approach:

1. rust produce static library, and wasi-sdk compile emnapi C code and link the rust static library, This is feasible in my experiment.
2. emnapi provide precompiled static library, not sure if rust can successfully use it to link, due to C use the wasi-sdk sysroot.

[toyobayashi]

The real world use case of emnapi, though not many, it is functionally reliable.

lovell/sharp#3522
TryGhost/node-sqlite3#1674

[devongovett]

Ah I see. I think it would be nice to avoid relying on Emscripten for pthreads etc. Same with WASI – I don't think WASI is available in browsers (yet?). I was going to look into how wasm-bindgen-rayon implemented WASM threads (I think they are just web workers) and follow that for napi-wasm, but I haven't had time yet.

[toyobayashi]

I think it would be nice to avoid relying on Emscripten for pthreads

emnapi is design for emscripten at first, for convenience I use pthread to port libuv threadpool and async work / tsfn implementation of Node.js to pure C. It's a bit hard for me if replace the pthread part then still keep passing all nodejs official test.

I don't think WASI is available in browsers (yet?)

I did this in https://github.com/toyobayashi/wasm-util/tree/main/src/wasi, by rewriting uvwasi in JavaScript, optionally use mem-fs or any Node.js fs API compatible library as filesystem support, but currently have no thread yet, after wasi thread landed, I think it is also just web webworkers.

[toyobayashi]

Additionally i would like to explain, emnapi's goal is 1) help users port their or existing node-api native addons to wasm with code change as less as possible, 2) runtime behavior matches native Node.js as much as possible.

So I decided to port Node.js node-api source code implementation to JavaScript and C, if you have look into emnapi's source code, you could see most code (both TS and C) matches Nodejs's src/js_native_api_v8.cc and src/node_api.cc. Relying on pthread is reasonable and it is the best way to achieve 2) because I can port part of libuv, do tsfn mutex lock/unlock and cond wait/signal exactly same as nodejs. As most addons are written in C/C++, emscripten is still the first class support target, then wasi-sdk, napi-rs could also be supported if @Brooooooklyn approve emnapi's practice. WASI is the standard, I could not find some reason to avoid it.

[devongovett]

Is there a plan for getting rid of #[ctor] in napi? At the moment this blocks us from using the modern napi-rs API in Parcel. We instead have to fall back to the old API if we want to support WASM. Maybe there is a way we could disable ctor for wasm targets and somehow manually call the initialization functions?

[Brooooooklyn]

@devongovett I'm working on it on this branch: https://github.com/napi-rs/napi-rs/tree/emnapi

[devongovett]

Cool, thanks. Hopefully it won't require WASI though? Not sure that will work in browsers.

In the meantime, I got it working by patching rust-ctor: devongovett/rust-ctor@30aa204. All that does is mark the ctor functions as #[no_mangle] in wasm32 targets. Then, I just have to do this from the JS:

for (let key in instance.exports) {
  if (key.startsWith('__napi_register__')) {
    instance.exports[key]();
  }
}

As long as that is done before calling napi_register_module_v1 it seems to work. 🥳

[toyobayashi]

Hopefully it won't require WASI though?

Working in progress, no WASI required for async work and tsfn! Currently emnapi already has another TSFN implementation and single thread async work mock, both are implemented in pure JavaScript and passed node js official test suite. I'll bring real multithreaded async work implementation via WebWorkers in next release. emnapi users can decide themselves to use
async work / tsfn 's C or JS implementation.

According to @Brooooooklyn , napi-rs uses tokio, unfortunately they still need WASI, but emnapi itself can avoid WASI soon. My WASI polyfill is well tested in browser as well.

[devongovett]

For my use cases, I don't really need support for tokio or async work in WASM so I wouldn't really mind if there were some features of napi-rs that were only for non-WASM targets. It would be great if we could ensure that those features are optional and that large polyfills aren't required unless you explicitly include them. I mainly need the basics to work: calling Rust from JS, and JS from Rust. Once in Rust-land, I'm happy to manage things like async tasks myself.

[Brooooooklyn]

@devongovett I can split changes in abbea85 and merge it before WebAssembly implemented, so that you can call the register functions before napi_register_module_v1 with these changes.

[devongovett]

Thanks @Brooooooklyn, that would be helpful! I'm also happy to use the ctor patch in the meantime if it's too hard.

[Brooooooklyn]

@devongovett published napi-derive@2.12.1 for that, you can try it!

[AlexMikhalev]

I think basic webassembly support is nearly there:
I compiled code with #[napi] derive into wasm using cargo build --release --target=wasm32-unknown-unknown but yarn build fails:
yarn run build --target=wasm32-unknown-unknown Finished release [optimized] target(s) in 0.02s Type Error: Operating system not currently supported or recognized by the build script
with @napi-rs/cli/scripts/index.js:11540:31.
I am ok not having threads and tokio and others since even Wasi runtime doesn't support any of it.
But being able to hook into a browser or native is a killer feature.

[Brooooooklyn]

Closed in #1669