Improve documentation about addons

[ibc]

The README says almost nothing about how to deal with native modules:

Native addons (.node files) use is supported, but packaging .node files inside the executable is not resolved yet. You have to deploy native addons used by your project to the same directory as the executable.

When a package, that contains a native module, is being installed, the native module is compiled against current system-wide Node.js version. Then, when you compile your project with pkg, pay attention to --target option. You should specify the same Node.js version as your system-wide Node.js to make compiled executable compatible with .node files.

Such a text does not explain how to run the generated executable when it's moved out of the project folder:

$ pkg -t macos server.js
pkg@4.3.7
Warning Cannot include addon %1 into executable.
The addon must be distributed with executable as %2.
/private/tmp/kk/node_modules/zeromq/build/Release/zmq.node
path-to-executable/zmq.node

Honestly no idea how to understand this warning. What is %1? What is %2?
So, assuming I have the zmq.node binary somewhere in my filesystem, how to invoke my generated server executable? which command line options?

$ ./server
pkg/prelude/bootstrap.js:1178
throw error;
^

Error: Cannot find module '../build/Release/zmq.node'

1. If you want to compile the package/file into executable, please pay attention to compilation warnings and specify a literal in 'require' call.

The compilation warning (above) does not help much IMHO.

[ibc]

I've got it working by creating a /snapshot folder in my filesystem and replicating there all the path tree:

/snapshot/kk1/node_modules/mediasoup/worker/out/Release/mediasoup-worker

Obviously I cannot force users to create a /snapshot folder in the fs root. But I see not doc regarding how to modify such a path.

[sharathbaddam]

I cannot comment on documentation or other native node_modules that you are using, but i can suggest a way that worked for me.

I have a gulp script that changes require code in node_module packages that has .node files before building a executable.

For example in zmq node_module, require code will be changed from '../build/Release/zmq.node' to './zmq.node'
This will allow me to package executable and all .node files along with it without having any specific folder structure.

[ibc]

Thanks @sharathbaddam. So such a gulp task should copy all the node_modules tree into a separate location, search for all require("**/*.node") into it, replace them with ./*.node, run the pkg command, and restore the original node_modules... wow :)

I'm also having problems with a specific Node module (written by me) that compiles and runs a separate binary (it's not a Node native module). In JS it does something like this:

const { spawn } = require('child_process');

const workerbinaryPath = path.join(__dirname, '..', 'worker', 'out', 'Release', 'mediasoup-worker');

spawn(workerbinaryPath, ...);

and I cannot have it working even if I copy node_modules/mediasoup/worker/out/Release/mediasoup-worker into the root folder close to the PKG generated binary. But I assume this is a separate problem.

Thanks.

[ibc]

Thanks @EscolanoGarnica for hickjacking my issue.

[EscolanoGarnica]

@ibc sorry

[ibc]

340 open issues and growing up, most of them without any feedback from developers. Maddening.

[CadsoftBrianW]

Having the same issue with .node files for bcrypt and sqlite3.

[RobLoach]

@ibc It works with bindings. As mentioned in the docs, it can't be in the package itself, but in the same directory as the executable.

const bindings = require('bindings')('binding.node')

[mohshraim]

face the same problem that i cant run the app without node_module folder
hope to have clear documents how to fix it

[designbyadrian]

@RobLoach Is your code a solution or an example? What's binding.node? Where is this line?

[RobLoach]

An example of loading a Node.js addon through the bindings module. Can be run within a pkg archive, but the compiled addon needs to be in the same directory as the executable.

[lselden]

I got around finding the addons to include by automatically detecting and copying them based on pkg's stdout. I never had issues with having to specify alternate directories, however...having the .node files next to the executable has worked so far.

const execa = require('execa');
const cpy = require('cpy');
const {stdout} = await execa('pkg' ['-o', './build'], { shell: true });
const lines = stdout.split(/[\r\n]+/);
const addons = [];
let i = 0;
while (i < lines.length - 1) {
	const [line, next] = lines.slice(i, i + 2).map(s => s && s.trim());
	i += 1;
	if (
		line && next &&
		line.startsWith('The addon must be distributed') &&
		next.endsWith('.node')
	) {
		addons.push(next);
		// already know the next was match, so skip 2
		i += 1;
	}
	continue;
}
if (addons.length) {
	await cpy(addons, './build');
}

[benjhess]

pkg is usually unable to determine native addons, because the respective require call is dynamic. Most if not all the time, the binary.find method of module node-pre-gyp is used like this, to load the addon:

var binding_path = binary.find(path.resolve(path.join(__dirname, './package.json')));
var bindings = require(binding_path);

This reads the module name and relative path from the modules package.json and builds an absolute path to the .node addon by dirname of the package.json file.

E.g:
package json path: ~/your-project/node_modules/bcrypt/package.json

{
  "binary": {
    "module_name": "bcrypt_lib",
    "module_path": "./lib/binding/"
    ...
  }
}

target path in binary: /snapshot/your-project/node_modules/bcrypt/lib/binding/bcrypt_lib.node

Unfortunately, it's not possible to set an absolute path for "module_path". It will always resolve relative to the path of the package.json file. But you can manipulate the source code in 2 ways.

1. Manipulate all "binary.find" calls

We can actually provide an options object as param, which will allow to set an absolute path.

E.g:

var binding_path = binary.find(
  path.resolve(path.join(__dirname, './package.json')), 
  {module_root: '/opt/node/'}
);
var bindings = require(binding_path);

target path in binary: /opt/node/lib/binding/bcrypt_lib.node

2. Manipulate the "binary.find" method in node-pre-gyp module

Here we can directly hardcode the desired module root path.

E.g:
./node_modules/node-pre-gyp/lib/pre-binding.js

from:

opts = opts || {};

to:

opts = opts || {};
opts.module_root = '/opt/node';

This way should be prefered, as we can not exclude the possibility of an already present options object.

With this knowledge, it should be fairly easy to write a script that:

• detects all .node addon files in node_modules
• copies them to a desired absolute path (incl. relative folder structure)
• manipulates the "binary.find" method, to resolve from the desired absolute path

[from-the-river-to-the-sea]

@benjhess A potential issue with solution #2 is that there may be multiple versions of node-pre-gyp throughout the dependency tree, and each may need a different patch. A library of patches would have to be maintained.

[benjhess]

@lange Yes, that is also not really an ideal solution. Maybe we could change pkg to detect require paths with .node addons in it, and then change the path relative to the binary or to a desired absolute path. I guess, we could even implement something like this in userland...

E.g:
from: /snapshot/your-project/node_modules/bcrypt/lib/binding/bcrypt_lib.node
to: ${absolute_path_to_binary}/bcrypt_lib.node

[benjhess]

This seems to work just fine:

const path = require('path');
const Module = require('module');
const originalRequire = Module.prototype.require;

Module.prototype.require = function() {
  arguments && arguments[0] && arguments[0].match(/^\/snapshot\/.*\.node$/)
    && (arguments[0] = path.join('/opt/node/bindings/', path.basename(arguments[0])));

  return originalRequire.apply(this, arguments);
};

It resolves all .node addons from /opt/node/bindings.
Using path.dirname(require('process').argv[0]) would resolve the .node addons relative to the binary...

[leerob]

Initial support has been merged with #837.

If there's anything else missing, please feel free to contribute back and I will take a look at the PR. Thank you! 🙏