Option to build the N-API native extension locally

[ryanprior]

Hi there! I've been maintaining the Guix package for esbuild. For purposes of security, portability and reproducibility we like to remove pre-built binary bits from package sources and replace them with something we can build from source using our own infrastructure.

What do you think about providing an option (or instructions would be fine) to build the N-API native extension locally?

[evanw]

Well for reproducibility ideally a good state to be in would be where all bits can be reproducible on all platforms. Unfortunately it doesn't look like that's currently the case. It'd be great to get there though.

If I were you I'd be much more worried about the WebAssembly binary since the N-API stubs are super tiny while the WebAssembly module is huge. Go's WebAssembly interface has a general-purpose API that should be able to access all node APIs including network methods. I just tried building the WebAssembly module on macOS and Linux, both with Go 1.15.5 (the Go version I'm using for releases), but the files are unfortunately different. Any tips for getting these to be the same would be appreciated. I'm not an expert with binary reproducibility.

Anyway the instructions for building the N-API native extensions locally are here. They aren't super robust because I wasn't intending for them to be rebuilt automatically so these instructions are more just for documentation. Each one was run on that platform (so there was no cross-compilation).

I have experimented with cross-assembling small executable stubs before and I briefly tried using that same technique here, but I couldn't immediately figure out how to get nasm to build dynamic libraries (I didn't try too hard) and I also wasn't confident that the dynamic libraries I produced would be as portable. For example, I think I remember that you're supposed to invoke syscalls via libSystem.dylib instead of directly on macOS for compatibility with future macOS versions. So I just went with the simpler approach of manually building them on each platform and checking them in for now. But I'd love to get them building from source in a cross-platform way if there's an easy way to do that without heavy dependencies.

Also an alternative for you could be to just omit these N-API extensions. The resulting build will be slower but it should still work. I'm not sure how you're building esbuild but if you just delete the .node files before the build script is run, the directory scan in the build script that includes these files will just not include them. This is probably the easiest way to solve your problem.

[ryanprior]

I'm interested in building the webassembly binary from source as well. I'm fairly new to wasm, whether via golang or otherwise, so I'm looking for examples of how that's typically done. I'll give this all another read later, thank you for providing the links and pointers!

[evanw]

The WebAssembly build happens here. Most of the complexity is building the various forms of the JavaScript companion code. The actual build command for the WebAssembly is just GOOS=js GOARCH=wasm go build -o esbuild.wasm ./cmd/esbuild.

[evanw]

One piece of the reproducible build puzzle was just solved: #1200 (see the commit message for details). Another piece is needing the -trimpath flag according to golang/go#16860, which I have also just added. With both of these, builds now appear to be reproducible, at least between my macOS and Linux computers.

[evanw]

Another update: I have added automated reproducibility verification for compiled Go and WASM binaries after every release: 96c6e35. This was done back in March but I forgot to update this issue. All major binary artifacts have been reproducible for quite a while (except for these small N-API native extensions, which is why this issue is still open).

[evanw]

I just discovered that node has finally fixed their bug: nodejs/node#36616 (comment). So it's not a problem anymore in node v18.3.0+. I think this means I can finally delete these N-API extensions, and then consider this issue to be fixed.

[ryanprior]

Let's go! Fully bootstrapped esbuild in sight.