Add support for Node native addons

[geekuillaume]

This adds the supports for Nodejs native addons.

How it works:

• When detecting a .node file, it will package it like an asset and not output any warning.
• On run-time, when requireing a .node file, it will use the same code as classic files to detect if the file is in the snapshot or not. If it is, it will write it to a file in the os.tmpdir() directory.
• The filename of the temporary file will be passed to the rest of the chain.

Potential future optimizations:

• Write a patch for the bindings module to detect the correct modules.
• Delete the temporary files on process exit

Related #329 #619 #749 #663

[btsimonh]

I do this exact thing in nexe.
Hints: Add support for all executable types (.exe, .dll, .so, etc). You MAY wish to write the file(s) to a specific subfolder, to avoid conflicts between dependencies.
With this style of native module use, I can use, for example, Opencv, or canvas, with all deps, and so far, so good.

[kamontat]

Any update on this?

[n1ru4l]

@geekuillaume The build is currently failing due to eslint errors.

[geekuillaume]

@n1ru4l I fixed the linting errors but the Travis CI process seems to be broken (same error on master).

[nklayman]

I tried this out and was able to bundle Theia pretty smoothly on Linux. This is really awesome, hopefully it can get merged soon. Thanks for the great work!

[nklayman]

However, on windows:

pkg/prelude/bootstrap.js:1244
      throw error;
      ^

Error: The specified module could not be found.
\\?\C:\snapshot\tauri-theia\node_modules\nsfw\build\Release\nsfw.node
    at tryImporting (pkg/prelude/bootstrap.js:1607:37)
    at process.dlopen (pkg/prelude/bootstrap.js:1633:5)
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:805:18)
    at Module.load (internal/modules/cjs/loader.js:651:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:591:12)
    at Function.Module._load (internal/modules/cjs/loader.js:583:3)
    at Module.require (internal/modules/cjs/loader.js:690:17)
    at Module.require (pkg/prelude/bootstrap.js:1225:31)
    at require (internal/modules/cjs/helpers.js:25:18)
    at Object.<anonymous> (C:\snapshot\tauri-theia\node_modules\nsfw\js\src\index.js:5:18)

[geekuillaume]

@nklayman is it occurring only on Windows and working without pkg? This error can also be caused if the .node file wasn't included in the package.

[nklayman]

The error is only on windows and the code works fine when just running with node, the issue is only present when running the pkg binary.

[noraibrahimi]

Any update on this? It seems like a high priority PR based on the opened issues.

[geekuillaume]

@noraibrahimi It seems the tests are broken on master. There is no activity on this repo, seems like the project has been abandoned by Vercel. @igorklopov do you have more information about the state of this project?

[noraibrahimi]

@geekuillaume I think some changes have been pushed yesterday to master branch (which might be a fix for why the build is failing), maybe you could try rebuilding it again?

[LoSunny]

Hello everyone, I'm trying to get pkg working with sharp, here is the error. (when using leafac/pkg fork)

(node:2428) UnhandledPromiseRejectionWarning: Error:
Something went wrong installing the "sharp" module

The specified module could not be found.
C:\Users\User\AppData\Local\Temp/ca4d31d82c70f9db56d44025114a7bf80842b881298974cd425a21369edaa38b_sharp.node

- Remove the "node_modules/sharp" directory then run
  "npm install --ignore-scripts=false --verbose" and look for errors
- Consult the installation documentation at https://sharp.pixelplumbing.com/install
- Search for this error at https://github.com/lovell/sharp/issues

    at Object.<anonymous> (C:\snapshot\SY-Sync\node_modules\sharp\lib\constructor.js:34:9)
    at Module._compile (pkg/prelude/bootstrap.js:1342:22)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1218:10)
    at Module.load (internal/modules/cjs/loader.js:1047:32)
    at Function.Module._load (internal/modules/cjs/loader.js:935:14)
    at Module.require (internal/modules/cjs/loader.js:1087:19)
    at Module.require (pkg/prelude/bootstrap.js:1247:31)
    at require (internal/modules/cjs/helpers.js:73:18)
    at Object.<anonymous> (C:\snapshot\SY-Sync\node_modules\sharp\lib\index.js:3:15)
    at Module._compile (pkg/prelude/bootstrap.js:1342:22)
(Use `app --trace-warnings ...` to show where the warning was created)
(node:2428) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:2428) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I've tried vercel/pkg, kkoomen/pkg and leafac/pkg, but all of them have an error. Anyone can get sharp working on a Windows 10 machine.

EDIT: Here is my package.json if needed

{
  ...
  "main": "app.js",
  "bin": "app.js",
  "dependencies": {
    "pkg": "git+https://github.com/leafac/pkg.git",
    "screenshot-desktop": "^1.12.3",
    "sharp": "^0.27.1",
    ...
  },
  "pkg": {
    "assets": [
      "views/**/*",
      "node_modules/sharp/build/Release/sharp.node"
    ]
  }
}

[leafac]

@LoSunny:

It seems like the patch in this pull request doesn’t with sharp. The same issue is happening for me on macOS.

I got it to work by using the official pkg package from npm (not a fork) and then following the instructions from the warnings that showed up when running pkg to produce the executable:

> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/build/Release
  %2: path-to-executable/sharp/build/Release
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/vendor/lib
  %2: path-to-executable/sharp/vendor/lib
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/build/Release
  %2: path-to-executable/sharp/build/Release
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/vendor/lib
  %2: path-to-executable/sharp/vendor/lib

I copied node_modules/sharp/ into <path-where-my-executable-ends-up>/sharp/ and then the executable worked.

Of course, this is annoying, and it suggests that the code in this pull request isn’t fixing the issue 100%.

So far the code in @leafac/pkg is just the code I found here and packaged for distribution. I’ll investigate alternatives to pkg (see comments above) and determine if this problem is worth solving or if there’s something better already out there. Depending on what I find I’ll return here and try to fix these issues…

Meanwhile, you can help me in the investigation by trying the tools I mentioned in the comment above on your project.

[bryiac]

Any updates on this whole situation I've tried nexe + nexe-natives with no luck I am experience the same issue as Winexcel where the patch moves my .node file to the temp folder but doesn't seem to be able to do anything with it @leafac

[leafac]

Very-Early Announcement of a New Approach to Packaging: caxa

I did my investigation and learned that there isn’t a satisfactory solution to my problem, so I’m cooking something up which I’m calling caxa. It isn’t ready for consumption yet, but I wanted to check in and let y’all in the loop. Here’s a few details (more to come in the near future):

My Requirements

• I just want to distribute Node.js applications easily.
• I don’t care about hiding the source code.
• I want builds to be fast.
• I want no surprises at all. If the application works in development, then it should work after packaged. Native modules and all.
• I don’t a patched version of Node.js. That’s bound to go stale (see the situation with pkg and fs/promises, which I had to patch in @leafac/pkg).
• I want the packaged application to be as small as possible.

Prior Art

Unfortunately, all packaging solutions I could find either require you to compile Node.js from scratch, which is slow (hours!) and consumes ~10GB of disk(!) (boxednode, Node Packer, and so forth), or they don’t support native extensions transparently (pkg, nexe, and so forth).

I decided not to invest in pkg anymore, and when I release caxa in the next few days I’ll deprecate @leafac/pkg. (Note that I’m far from being the maintainer of pkg, I’m just a random dude who passed by…)

How caxa Works

The root of the problem seems to be that Node.js insists on loading .node files from disk.

The workaround for native extensions in this PR and in other packaging solutions is to put those .node files in a temporary directory. Your packaged application becomes a sort of self-extracting archive.

caxa takes this approach to the next level and achieves blissful simplicity and robustness. It puts your project alongside the Node.js in a self-extracting archive.

Limitations

• Cross-compilation is out of the table. You can’t, for example, generate a Windows .exe from macOS. I learned that not even the best in the cross-compilation business, Go, supports cross-compiling native extensions (or, as they call it, CGO). I think that in the Go community they try to avoid CGO if they can, but here in Node.js there’s no way around them; think of SQLite, Sharp, and so forth. This isn’t a big deal, because GitHub Actions provides runners for Linux, macOS, and Windows.
• You must bundle the same version of node that you’re running. The reasoning is pretty much the same as on the point above: Native modules break when compiled against an incompatible version of node. Also, there’s no shenanigans of downloading different versions of node in the packaging process. And you don’t have to wait for anyone to release a new pre-compiled binary (or worse, compile it yourself). Minimal work; minimal surprises. And again, GitHub Actions are a good solution here: just use a matrix build.

What We Have so Far

• caxa produces self-extracting archives for Linux & macOS.
• The binaries are smaller than the competition (because they’re gzipped).
• Support for every Node.js version.
• Support for native modules.
• Fast builds.
• caxa can also produce macOS Application Bundles (.app). It’s a cute trick.

What’s Coming in the Next Few Days

• Windows support. (As far as I can tell the idea of self-extracting archives came from the Windows world, so I expect things to work great there, but I don’t have a Windows machine and VirtualBox with modern.ie is painful to work with because my development machine can barely handle it. So Windows support comes last… If you’re reading this, you have an old Windows machine laying around, and you want to support my work, please send the machine my way).
• Documentation.

What’s in the Name

caxa is a misspelling of caixa, which is Portuguese for box. I find it amusing to say that I’m putting my application in the caxa.

[kkoomen]

@leafac Nice. I hope it's a nice replacement for this repo. Also, please maintain it actively, because I just hate repositories and their owners like these ones where it's used a lot by many projects/people/companies, but no one is fixing issues or implementing new features.

Second, don't worry too much about cross-compiling on a single system. If it works, it'd be nice, but this is up to the user and I currently solve this problem myself using Github Actions. I use pkg to build on Windows in order to create a Windows package. I do the exact same for linux (using ubuntu) and for MacOS, so this is totally doable for your users to compile it on the system itself, but keep in mind not everyone is using Github Actions. Some people use GitLab, some use BitBucket. Both do not support the awesomely built Github Actions, so if you want to spent time on adding cross-compiling that'd be really nice and a big plus compared to this package, although it's Node, meaning it needs to be build per system, so I doubt whether you can achieve this or not.

[bryiac]

This is cool if you need someone to test builds on windows I'm willing too

[kkoomen]

@leafac Please notify us here when it's ready to be used in production including docs and windows support.

[david-mohr]

@leafac Sounds great! Watching with keen interest.

I have finished my first take on automated cross-platform support. It should work with any native node module that supports prebuild-install (which seems to be most of the major ones). You can see the code here: https://github.com/david-mohr/pkg/tree/feat/addon2

This also supports "manually" adding prebuilt files if required, but keen for everyone's feedback if this meets the basic needs. I can put a PR onto @leafac/pkg if that helps

[leafac]

@david-mohr: Thanks for work 😃

Please don’t send the pull request to @leafac/pkg. I’m about to deprecate it in favor of caxa (see comment above).

I’m not pursuing the idea of cross-compilation for the moment. I learned that’s really hard to get right in some cases and I think that packaging should be as unsurprising as possible. I dislike the idea that you’d add a dependency to your project and all of the sudden your packaging no longer works.

[david-mohr]

@leafac fair enough. If there's interest, I can publish my branch for those that still require cross-compilation. I'll keep caxa in mind for the next project.

[leerob]

Hey everyone! Sorry for the slow responses here from the maintainers. I'm Lee from Vercel and I'll be helping review / merge PRs. There's a lot of context and discussion in this PR, so I'm hoping to lean on the community and contributed here for recommendations going forward.

• Is this PR ready as is?
• Do we need to have more discussion?

Any and all context would be helpful!

[carlo-quinonez]

@leerob I've been using the code from the PR as-is with my project, and it works fine. From the conversation, there's edge-cases where it's not working because the .node addons require DLLs that can't be bundled.

And thank you for taking up maintainer duties!

[geekuillaume]

@leerob Hi Lee, it's been quite a while since I've worked on this PR. I'm not using it anymore for my usage so I've stop participating here but feel free to use what I already did as it seems to work for most use.

[kkoomen]

@leerob You can check my fork, because I've fixed another issue there in the master branch (based on this PR) in order to built for Windows properly, which works for me inside Github Actions, but some people also say that they run into issues, even with my fork.

[leerob]

It sounds like this solution has been working for those who have forked. I'm okay with going ahead and merging this, and feel free to contribute back edge cases that have been found from other forks.

Sorry again for the long wait here and thank you for everyone who has contributed to this discussion 🙏

[leafac]

Hello everyone,

caxa 1.0.0 is out 🙌 📦

It’s ready for y’all to try out and I’d love to hear your feedback.

Highlights

• Works on Windows, macOS, and Linux.
• Simple to use. npm install caxa and call caxa from the command line. No need to declare which files to include; no need to bundle the application into a single file.
• Supports any kind of Node.js project, including those with native modules (for example, sharp, @leafac/sqlite (shameless plug!), and others).
• Works with any Node.js version.
• Packages in seconds.
• Relatively small binaries. A “Hello World!” application is ~30MB, which is terrible if compared to Go’s ~2MB, and worse still if compared to C’s ~50KB, but best-in-class if compared to other packaging solutions for Node.js.
• Produces .exes for Windows, simple binaries for macOS/Linux, and macOS Application Bundles (.app).
• Based on a simple but powerful idea. Written in ~200 lines of code.
• No magic. No traversal of require()s trying to find which files to include; no patches to Node.js source.

---

@leerob: Thanks for taking up the mantle! pkg is awesome and I’m happy to see it receive some attention. Though I know how these things go: open-source is hard.

In any case, I recommend you go to my now-deprecated fork at https://github.com/leafac/pkg and grab the implementation of fs/promises.

Also, would be so kind as to mention caxa in pkg’s README? It’s a different approach that solves some of the issues with pkg, for example, support for all Node.js versions and ease of use without having to declare assets (while not solving some things that pkg can do, for example, hiding the JavaScript source). (More detailed comparison)

[leerob]

@leafac Please do not excessively advertise here 🙏 But thank you for contributing and caxa looks nice really 😄