deno-esbuild: A request for WebAssembly support

[johnspurlock]

I'd like to be able to run esbuild transform in a Deno program, but using a loaded wasmModule (e.g. instantiated in Deno with WebAssembly.compileStreaming from a fetch response from https://unpkg.com/esbuild-wasm@0.14.43/esbuild.wasm).

tried:

await initialize({ wasmModule }); // does not throw
await transform(code, { target: 'es2019' }); // still requires run access to the esbuild binary in the deno cache, if denied, throws (does not use wasm)

It seems like this should be able to work, given Deno has the wasmModule loaded.

Thanks!
- John

[evanw]

The Deno API is not built to use WASM at all, so this isn't supposed to work anyway. The only reason doing this doesn't throw an error right now is that the wasmModule API option is a newer addition (#2155) and I forgot to add code to throw here when this option is passed. I'll fix this to make it throw.

Why do you want to use WebAssembly instead of native? It can easily be 10x slower.

[johnspurlock]

It would avoid the need for an additional --allow-run /path/to/deno/cache/esbuild Deno runtime permission.

I'm transpiling a single file, so the wasm version would be more than enough for my use case.

[jed]

This would also allow esbuild to run on Deno Deploy et al, which I'd love to see.

[evanw]

I tried poking at this and it seems relatively straightforward. Caveats:

• It wouldn't have file system support just like esbuild's browser-based API. Getting file system support is possible of course but would be additional work.

• There are two options for loading WASM in Deno: reading it from the file system and downloading it from the internet. One needs --allow-read and one needs --allow-net. I assume I should be downloading it from the internet since that's how Deno does dependencies? Not sure because I don't use Deno myself.

• Deno appears to have a severe WebAssembly performance bug where it hangs for 15 seconds on exit, spinning at 100% CPU. I'm assuming that this is Deno blocking exit on finishing WASM compilation even though it's irrelevant, since node has the same bug and they both use V8. In that case using esbuild's WASM implementation in Deno for short operations will be extremely inefficient due to this Deno bug.

  Long story is that V8 has a two-tiered JIT for WASM. WASM can start running with the low tier, and may even finish running completely. The high tier is supposed to only be for long-running applications that need optimal performance. You can read more about this architecture here: https://v8.dev/docs/wasm-compilation-pipeline. IMO after the application has exited Deno shouldn't be waiting for the high tier to finish compiling.

  Node has the same bug: tiered WebAssembly compilation broken nodejs/node#36616. My ugly hack to "fix" this in node for the CLI version of esbuild's WASM implementation is to load a native node extension on exit that invokes the _exit syscall directly, which bypasses node's lifecycle code:

  esbuild/npm/esbuild-wasm/bin/esbuild

  Lines 26 to 63 in 698898f

  	// Node has an unfortunate bug where the node process is unnecessarily kept open while a
  	// WebAssembly module is being optimized: https://github.com/nodejs/node/issues/36616.
  	// This means cases where running "esbuild" should take a few milliseconds can end up
  	// taking many seconds instead. To work around this bug, it is possible to force node to
  	// exit by calling the operating system's exit function. That's what this code does.
  	process.on('exit', code => {
  	// If it's a non-zero exit code, we can just kill our own process to stop. This will
  	// preserve the fact that there is a non-zero exit code although the exit code will
  	// be different. We cannot use this if the exit code is supposed to be zero.
  	if (code !== 0) {
  	try {
  	process.kill(process.pid, 'SIGINT');
  	} catch (e) {
  	}
  	return;
  	}
  	// Otherwise if the exit code is zero, try to fall back to a binary N-API module that
  	// calls the operating system's "exit(0)" function.
  	const nativeModule = `${process.platform}-${os.arch()}-${os.endianness()}.node`;
  	const base64 = require('../exit0')[nativeModule];
  	if (base64) {
  	try {
  	const data = zlib.inflateRawSync(Buffer.from(base64, 'base64'));
  	const hash = crypto.createHash('sha256').update(base64).digest().toString('hex').slice(0, 16);
  	const tempFile = path.join(os.tmpdir(), `${hash}-${nativeModule}`);
  	try {
  	if (fs.readFileSync(tempFile).equals(data)) {
  	require(tempFile);
  	}
  	} finally {
  	fs.writeFileSync(tempFile, data);
  	require(tempFile);
  	}
  	} catch (e) {
  	}
  	}
  	});

  I'm guessing there's nothing like that for Deno, and even if there was that you may not want to use it since it's quite the hack. I assume this bug doesn't matter if you're using Deno in a long-lived context, since the WASM compilation should be finished by the time you want to exit.

  Edit: It looks like calling Deno.exit(0) does successfully kill the background compilation job, unlike node. So this is not too much of a concern.

[johnspurlock]

Re: loading interface - the deno_emit wasm module has a way to provide a custom loader function, which is basically:

type Loader = (spec: string) => Promise<LoadResponseModule>;

interface LoadResponseModule {
    /** A module with code has been loaded. */
    kind: 'module';
    /** The string URL of the resource. If there were redirects, the final
     * specifier should be set here, otherwise the requested specifier. */
    specifier: string;
    /** For remote resources, a record of headers should be set, where the key's
     * have been normalized to be lower case values. */
    headers?: Record<string, string>;
    /** The string value of the loaded resources. */
    content: string;
}

I've been working on getting the emit wasm running on Deno Deploy (see bundle.deno.dev), and it appears to be working, although a much smaller library than esbuild.

To me though, while running a future esbuild wasm on an edge platform would be cool, just being able to run it locally without needing the process permission would be very useful as well.