wasi: introduce initial WASI support #30258

cjihrig · 2019-11-05T03:23:34Z

This PR introduces experimental support for the WebAssembly System Interface (WASI). This PR adds a new core module, wasi, and a new C dependency, uvwasi, that implements the WASI system call API on top of libuv. uvwasi currently implements version wasi_unstable_preview0 of the WASI API. It's worth pointing out that WASI itself is still evolving, and uvwasi is still young. There will be bugs and breaking changes.

Up to this point, development has been happening in the uvwasi repository, and the nodejs/wasi fork. The commit metadata currently lists the individuals that have written and reviewed code in those two repositories.

I'm opening this PR because I can't realistically continue to dedicate so much time to this work (which started in July) indefinitely without confirmed buy in from the project. Because this introduces a new core module, cc: @nodejs/tsc. Also cc: @nodejs/wasi and @nodejs/libuv.

Refs: #27850

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

devsnek · 2019-11-05T03:42:43Z

Can you call the exposed module wasi_unstable_preview0 or wasi/unstable_preview0 instead of wasi?

guybedford · 2019-11-05T03:46:44Z

Given that this whole feature is behind a flag I think it makes sense to adopt a name that can be considered as the final name here.

@devsnek would you consider giving up your wasi package on npm and renaming your package to something else?

Otherwise, if it helps, I have wasi_unstable on npm and would be glad to give that up for this.

devsnek · 2019-11-05T04:19:59Z

i think the name should match the phase we're implementing. if we want to use wasi or wasi/unstable_preview0 i will cede my package.

guybedford

Amazing work @cjihrig, thanks so much for your efforts and it would be great to see this in Node.js continuing to be iterated upon further.

Since there is no wasi: subsystem, how about we use a generic wasm: subsystem for this PR to run against CI for now. I've posted nodejs/core-validate-commit#82.

@devsnek the reason for using a versioned import in WASI output is to ensure backwards compatibility can be maintained. That is allowing different WASI binaries to coexist in the wild.

Using the same versioning scheme for Node.js core conflates the WASI API version with the WASI Node.js API version, which are two very separate things. Node.js users tie their WASI version to the version of the Node.js overall - we don't version core modules for this reason. So there is no benefit to versioning any WASI builtin module in core, although an unstable marker could be considered certainly.

doc/api/wasi.md

lib/wasi.js

doc/api/wasi.md

devsnek · 2019-11-05T04:53:18Z

@guybedford i'm aware why the versions are the way they are, i just want to ensure maximum verbosity and breakage while this isn't stable.

mcollina

rubberstamp LGTM

Would this require some changes to our CI to be able to run the tests?

saghul

Excellent work!

cjihrig · 2019-11-05T13:52:31Z

@devsnek, would changing the CLI flag to --experimental-wasi-unstable-preview0 and using wasi as the module name work for you? We should avoid changing the top level module name because of the impact on the npm ecosystem.

Would this require some changes to our CI to be able to run the tests?

@mcollina no, as long as we check in the .wasm files. Recompiling the WASI tests from source would require the CI or user to have wasi-libc set up though.

mcollina · 2019-11-05T14:39:07Z

no, as long as we check in the .wasm files. Recompiling the WASI tests from source would require the CI or user to have wasi-libc set up though.

That would work for now, however I think we should aim for it in the future (maybe with just one env?), maybe after this land you might want to open a tracking issue on build.

devsnek · 2019-11-05T15:45:47Z

@cjihrig that works too I guess

galvesribeiro

Awesome! LGTM

addaleax

I think this needs to come with a license file checked into the repo and added to tools/license-builder.sh (and then have that script run)?

addaleax · 2019-11-05T17:37:58Z

deps/uvwasi/include/fd_table.h

+  uvwasi_fd_t id;
+  uv_file fd;
+  char path[PATH_MAX_BYTES];
+  char real_path[PATH_MAX_BYTES];


Is uvwasi ABI-stable at this point, or are breaking changes fine? I’d maybe open a PR against it to make these pointers, both to avoid artificial limitations on the max path lengths and to avoid the large memory overhead that this is likely to have?

Breaking changes are totally fine at this point 😄

src/node_config.cc

src/node_wasi.cc

src/node_wasi.h

src/node_wasi.cc

lib/wasi.js

src/node_wasi.h

lib/wasi.js

addaleax · 2019-11-05T19:26:27Z

Since there is no wasi: subsystem, how about we use a generic wasm: subsystem for this PR to run against CI for now. I've posted nodejs/core-validate-commit#82.

Fwiw, I’d prefer to just keep using wasi:. It’s somewhat annoying that core-validate-commit makes Travis CI fail, but ignoring it should be fine.

tniessen · 2019-11-05T21:04:36Z

If wasi was to become modular (which is one of the goals if I remember correctly), could selecting a subset of modules (or a specific version of wasi) be achieved through the options object passed to new WASI?

devsnek · 2019-11-05T21:13:56Z

there wouldn't be a need to select a subset of functionality or version, because the namespaces are distinct to the wasm module. we can just provide all of it and the module takes what it wants.

richardlau · 2019-11-06T00:03:23Z

Since there is no wasi: subsystem, how about we use a generic wasm: subsystem for this PR to run against CI for now. I've posted nodejs/core-validate-commit#82.

Fwiw, I’d prefer to just keep using wasi:. It’s somewhat annoying that core-validate-commit makes Travis CI fail, but ignoring it should be fine.

wasi and wasm have been added to core-validate-commit. I've rerun the failed Travis job and it's now green.

devsnek · 2019-11-06T05:23:54Z

test/wasi/test-wasi.js

+
+  runWASI({ test: 'cant_dotdot' });
+  runWASI({ test: 'clock_getres' });
+  runWASI({ test: 'exitcode', exitCode: 120 });


any reason you removed the stdin/stdout/exitcode from the file itself?

Not really, I just thought it was simpler to have the values as part of the JS tests vs. parsing values out of the C file.

doc/api/wasi.md

addaleax

I think all comments that I’d have considered blocking have been addressed 👍

cjihrig · 2019-11-06T13:58:37Z

wasi and wasm have been added to core-validate-commit. I've rerun the failed Travis job and it's now green.

Thank you for taking care of that!

I think all comments that I’d have considered blocking have been addressed

I'm going to spend more time on it today - I ran out of time last night before I could get everything else addressed.

src/node_wasi.cc

nodejs-github-bot · 2019-11-11T15:11:30Z

CI: https://ci.nodejs.org/job/node-test-pull-request/26501/

Co-authored-by: Gus Caplan <me@gus.host> Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com> Co-authored-by: Jiawen Geng <technicalcute@gmail.com> Co-authored-by: Tobias Nießen <tniessen@tnie.de> Co-authored-by: Chengzhong Wu <legendecas@gmail.com> PR-URL: #30258 Refs: #27850 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Saúl Ibarra Corretgé <saghul@gmail.com> Reviewed-By: Guy Bedford <guybedford@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Wyatt Preul <wpreul@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com>

Notable changes: * http: * Make maximum header size configurable per-stream or per-server (Anna Henningsen) #30570 * http2: * Make maximum tolerated rejected streams configurable (Denys Otrishko) #30534 * Allow to configure maximum tolerated invalid frames (Denys Otrishko) #30534 * wasi: * Introduce initial WASI support (cjihrig) #30258 PR-URL: #30774

Notable changes: * fs: * Reworked experimental recursive `rmdir()` (cjihrig) #30644 * The `maxBusyTries` option is renamed to `maxRetries`, and its default is set to 0. The `emfileWait` option has been removed, and `EMFILE` errors use the same retry logic as other errors. The `retryDelay` option is now supported. `ENFILE` errors are now retried. * http: * Make maximum header size configurable per-stream or per-server (Anna Henningsen) #30570 * http2: * Make maximum tolerated rejected streams configurable (Denys Otrishko) #30534 * Allow to configure maximum tolerated invalid frames (Denys Otrishko) #30534 * wasi: * Introduce initial WASI support (cjihrig) #30258 PR-URL: #30774

sam-github · 2020-01-08T18:11:28Z

test/wasi/c/symlink_loop.c

+int main() {
+  FILE* file = fopen("/sandbox/subdir/loop1", "r");
+  assert(file == NULL);
+  assert(errno == ELOOP);


@cjihrig would it be possible to create these recursive symlinks dynamically during the test?

The error that is occurring above, ELOOP, is also being found by tooling that recurses the files from a nodejs/node checkout. I'm working around it, but it broke part of my work flow:

$ git ls-files -z $(git rev-parse --show-cdup) | tr '\0' '\n' | entr -crx ninja -C out/Release entr: cannot stat 'test/fixtures/wasi/subdir/loop1': Too many levels of symbolic link

@sam-github I'll send a PR later this evening.

Co-authored-by: Gus Caplan <me@gus.host> Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com> Co-authored-by: Jiawen Geng <technicalcute@gmail.com> Co-authored-by: Tobias Nießen <tniessen@tnie.de> Co-authored-by: Chengzhong Wu <legendecas@gmail.com> PR-URL: #30258 Refs: #27850 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Saúl Ibarra Corretgé <saghul@gmail.com> Reviewed-By: Guy Bedford <guybedford@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Wyatt Preul <wpreul@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com>

Notable changes: New assert APIs The `assert` module now provides experimental `assert.match()` and `assert.doesNotMatch()` methods. They will validate that the first argument is a string and matches (or does not match) the provided regular expression This is an experimental feature. Ruben Bridgewater [#30929](#30929). Advanced serialization for IPC The `child_process` and `cluster` modules now support a `serialization` option to change the serialization mechanism used for IPC. The option can have one of two values: * `'json'` (default): `JSON.stringify()` and `JSON.parse()` are used. This is how message serialization was done before. * `'advanced'`: The serialization API of the `v8` module is used. It is based on the HTML structured clone algorithm. and is able to serialize more built-in JavaScript object types, such as `BigInt`, `Map`, `Set` etc. as well as circular data structures. Anna Henningsen [#30162](#30162). CLI flags The new `--trace-exit` CLI flag makes Node.js print a stack trace whenever the Node.js environment is exited proactively (i.e. by invoking the `process.exit()` function or pressing Ctrl+C). legendecas [#30516](#30516). ___ The new `--trace-uncaught` CLI flag makes Node.js print a stack trace at the time of throwing uncaught exceptions, rather than at the creation of the `Error` object, if there is any. This option is not enabled by default because it may affect garbage collection behavior negatively. Anna Henningsen [#30025](#30025). ___ The `--disallow-code-generation-from-strings` V8 CLI flag is now whitelisted in the `NODE_OPTIONS` environment variable. Shelley Vohr [#30094](#30094). New crypto APIs For DSA and ECDSA, a new signature encoding is now supported in addition to the existing one (DER). The `verify` and `sign` methods accept a `dsaEncoding` option, which can have one of two values: * `'der'` (default): DER-encoded ASN.1 signature structure encoding `(r, s)`. * `'ieee-p1363'`: Signature format `r || s` as proposed in IEEE-P1363. Tobias Nießen [#29292](#29292). ___ A new method was added to `Hash`: `Hash.prototype.copy`. It makes it possible to clone the internal state of a `Hash` object into a new `Hash` object, allowing to compute the digest between updates. Ben Noordhuis [#29910](#29910). Dependency updates libuv was updated to 1.34.0. This includes fixes to `uv_fs_copyfile()` and `uv_interface_addresses()` and adds two new functions: `uv_sleep()` and `uv_fs_mkstemp()`. Colin Ihrig [#30783](#30783). ___ V8 was updated to 7.8.279.23. This includes performance improvements to object destructuring, RegExp match failures and WebAssembly startup time. The official release notes are available at https://v8.dev/blog/v8-release-78. Michaël Zasso [#30109](#30109). New EventEmitter APIs The new `EventEmitter.on` static method allows to async iterate over events. Matteo Collina [#27994](#27994). ___ It is now possible to monitor `'error'` events on an `EventEmitter` without consuming the emitted error by installing a listener using the symbol `EventEmitter.errorMonitor`. Gerhard Stoebich [#30932](#30932). ___ Using `async` functions with event handlers is problematic, because it can lead to an unhandled rejection in case of a thrown exception. The experimental `captureRejections` option in the `EventEmitter` constructor or the global setting change this behavior, installing a `.then(undefined, handler)` handler on the `Promise`. This handler routes the exception asynchronously to the `Symbol.for('nodejs.rejection')` method if there is one, or to the `'error'` event handler if there is none. Setting `EventEmitter.captureRejections = true` will change the default for all new instances of `EventEmitter`. This is an experimental feature. Matteo Collina [#27867](#27867). Performance Hooks are no longer experimental The `perf_hooks` module is now considered a stable API. legendecas [#31101](#31101). Introduction of experimental WebAssembly System Interface (WASI) support A new core module, `wasi`, is introduced to provide an implementation of the [WebAssembly System Interface](https://wasi.dev/) specification. WASI gives sandboxed WebAssembly applications access to the underlying operating system via a collection of POSIX-like functions. This is an experimental feature. Colin Ihrig [#30258](#30258). PR-URL: #31691

nodejs/node#30258

* chore: bump node in DEPS to v12.16.0 * Fixup asar support setup patch nodejs/node#30862 * Fixup InternalCallbackScope patch nodejs/node#30236 * Fixup GN buildfiles patch nodejs/node#30755 * Fixup low-level hooks patch nodejs/node#30466 * Fixup globals require patch nodejs/node#31643 * Fixup process stream patch nodejs/node#30862 * Fixup js2c modification patch nodejs/node#30755 * Fixup internal fs override patch nodejs/node#30610 * Fixup context-aware warn patch nodejs/node#30336 * Fixup Node.js with ltcg config nodejs/node#29388 * Fixup oaepLabel patch nodejs/node#30917 * Remove redundant ESM test patch nodejs/node#30997 * Remove redundant cli flag patch nodejs/node#30466 * Update filenames.json * Remove macro generation in GN build files nodejs/node#30755 * Fix some compilation errors upstream * Add uvwasi to deps nodejs/node#30258 * Fix BoringSSL incompatibilities * Fixup linked module patch nodejs/node#30274 * Add missing sources to GN uv build libuv/libuv#2347 * Patch some uvwasi incompatibilities * chore: bump Node.js to v12.6.1 * Remove mark_arraybuffer_as_untransferable.patch nodejs/node#30549 * Fix uvwasi build failure on win * Fixup --perf-prof cli option error * Fixup early cjs module loading * fix: initialize diagnostics properly nodejs/node#30025 * Disable new esm syntax specs nodejs/node#30219 * Fixup v8 weakref hook spec nodejs/node#29874 * Fix async context timer issue * Disable monkey-patch-main spec It relies on nodejs/node#29777, and we don't override prepareStackTrace. * Disable new tls specs nodejs/node#23188 We don't support much of TLS owing to schisms between BoringSSL and OpenSSL. Co-authored-by: Shelley Vohr <shelley.vohr@gmail.com>

nodejs-github-bot added the lib / src Issues and PRs related to general changes in the lib or src directory. label Nov 5, 2019

guybedford approved these changes Nov 5, 2019

View reviewed changes

doc/api/wasi.md Show resolved Hide resolved

lib/wasi.js Show resolved Hide resolved

doc/api/wasi.md Outdated Show resolved Hide resolved

mcollina approved these changes Nov 5, 2019

View reviewed changes

saghul approved these changes Nov 5, 2019

View reviewed changes

galvesribeiro approved these changes Nov 5, 2019

View reviewed changes

addaleax reviewed Nov 5, 2019

View reviewed changes

richardlau mentioned this pull request Nov 5, 2019

lib: add wasm subsystem nodejs/core-validate-commit#82

Merged

devsnek reviewed Nov 6, 2019

View reviewed changes

legendecas reviewed Nov 6, 2019

View reviewed changes

doc/api/wasi.md Outdated Show resolved Hide resolved

addaleax approved these changes Nov 6, 2019

View reviewed changes

cjihrig force-pushed the wasi-pr branch from 1c6dae1 to 3ee6d25 Compare November 6, 2019 22:53

zenoplex mentioned this pull request Nov 7, 2019

2019/11/07 第159回フロントエンドMTG 議事録 feedforce/frontend-meeting#14

Open

addaleax reviewed Nov 8, 2019

View reviewed changes

src/node_wasi.cc Outdated Show resolved Hide resolved

tniessen approved these changes Nov 9, 2019

View reviewed changes

cjihrig force-pushed the wasi-pr branch from 372ffdc to 5165a82 Compare November 11, 2019 15:10

gengjiawen approved these changes Nov 11, 2019

View reviewed changes

legendecas approved these changes Nov 11, 2019

View reviewed changes

addaleax closed this Nov 30, 2019

cjihrig deleted the wasi-pr branch November 30, 2019 20:14

BridgeAR mentioned this pull request Dec 3, 2019

v13.3.0 proposal #30774

Merged

sam-github reviewed Jan 8, 2020

View reviewed changes

targos mentioned this pull request Jan 15, 2020

v12.15.0 release proposal #31368

Closed

MylesBorins mentioned this pull request Feb 8, 2020

v12.16.0 proposal #31691

Merged

codebytere added a commit to electron/electron that referenced this pull request Feb 12, 2020

Add uvwasi to deps

ad6a4cc

nodejs/node#30258

codebytere added a commit to electron/electron that referenced this pull request Feb 13, 2020

Add uvwasi to deps

665dc7b

nodejs/node#30258

codebytere added a commit to electron/electron that referenced this pull request Feb 13, 2020

Add uvwasi to deps

7a9142e

nodejs/node#30258

codebytere added a commit to electron/electron that referenced this pull request Feb 13, 2020

Add uvwasi to deps

f63bc3d

nodejs/node#30258

codebytere added a commit to electron/electron that referenced this pull request Feb 13, 2020

Add uvwasi to deps

089d49b

nodejs/node#30258

codebytere added a commit to electron/electron that referenced this pull request Feb 15, 2020

Add uvwasi to deps

d7a6e26

nodejs/node#30258

codebytere added a commit to electron/electron that referenced this pull request Feb 18, 2020

Add uvwasi to deps

85a77f8

nodejs/node#30258

codebytere added a commit to electron/electron that referenced this pull request Feb 21, 2020

Add uvwasi to deps

487acb0

nodejs/node#30258

codebytere added a commit to electron/electron that referenced this pull request Feb 21, 2020

Add uvwasi to deps

9f01bb2

nodejs/node#30258

codebytere added a commit to electron/electron that referenced this pull request Feb 24, 2020

Add uvwasi to deps

09e980f

nodejs/node#30258

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wasi: introduce initial WASI support #30258

wasi: introduce initial WASI support #30258

cjihrig commented Nov 5, 2019

devsnek commented Nov 5, 2019

guybedford commented Nov 5, 2019

devsnek commented Nov 5, 2019 •

edited

guybedford left a comment

devsnek commented Nov 5, 2019

mcollina left a comment

saghul left a comment

cjihrig commented Nov 5, 2019

mcollina commented Nov 5, 2019

devsnek commented Nov 5, 2019

galvesribeiro left a comment

addaleax left a comment

addaleax Nov 5, 2019

cjihrig Nov 5, 2019

addaleax commented Nov 5, 2019

tniessen commented Nov 5, 2019

devsnek commented Nov 5, 2019 •

edited

richardlau commented Nov 6, 2019 •

edited

devsnek Nov 6, 2019

cjihrig Nov 6, 2019

addaleax left a comment

cjihrig commented Nov 6, 2019

nodejs-github-bot commented Nov 11, 2019

sam-github Jan 8, 2020

cjihrig Jan 8, 2020

wasi: introduce initial WASI support #30258

wasi: introduce initial WASI support #30258

Conversation

cjihrig commented Nov 5, 2019

Checklist

devsnek commented Nov 5, 2019

guybedford commented Nov 5, 2019

devsnek commented Nov 5, 2019 • edited

guybedford left a comment

Choose a reason for hiding this comment

devsnek commented Nov 5, 2019

mcollina left a comment

Choose a reason for hiding this comment

saghul left a comment

Choose a reason for hiding this comment

cjihrig commented Nov 5, 2019

mcollina commented Nov 5, 2019

devsnek commented Nov 5, 2019

galvesribeiro left a comment

Choose a reason for hiding this comment

addaleax left a comment

Choose a reason for hiding this comment

addaleax Nov 5, 2019

Choose a reason for hiding this comment

cjihrig Nov 5, 2019

Choose a reason for hiding this comment

addaleax commented Nov 5, 2019

tniessen commented Nov 5, 2019

devsnek commented Nov 5, 2019 • edited

richardlau commented Nov 6, 2019 • edited

devsnek Nov 6, 2019

Choose a reason for hiding this comment

cjihrig Nov 6, 2019

Choose a reason for hiding this comment

addaleax left a comment

Choose a reason for hiding this comment

cjihrig commented Nov 6, 2019

nodejs-github-bot commented Nov 11, 2019

sam-github Jan 8, 2020

Choose a reason for hiding this comment

cjihrig Jan 8, 2020

Choose a reason for hiding this comment

devsnek commented Nov 5, 2019 •

edited

devsnek commented Nov 5, 2019 •

edited

richardlau commented Nov 6, 2019 •

edited