Support injecting multiple files into the executable #68

mxschmitt · 2023-04-21T10:06:00Z

What is the problem this feature will solve?

Projects which include multiple files or non-javascript files would benefit from it and would not have to bundle everything.

What is the feature you are proposing to solve the problem?

Support injecting multiple files, since right now its limited to inject a single file as per here.

What alternatives have you considered?

Bundling

Please add the single-executables label to it or suggest a better place to file this feature request.

The text was updated successfully, but these errors were encountered:

joyeecheung · 2023-11-09T13:14:14Z

I am working on something along the lines. My current draft:

In the config file, we support a dictionary of assets, with ids that are later used in the API as keys, and path to the actual files as values:

{
  "assets": {
    "a.dat": "/path/to/a.dat",
    "b.txt": "/path/to/a.txt"
  }
}

At build time, we read these assets and add them to the preparation blob.

At runtime, users can get the assets back like this:

const { getAsset } = require('node:sea');
const a = getAsset('a.dat', 'buffer');  // Creates a copy
const b = getAsset('b.txt', 'utf-8');   // decode as an utf8 string, no copy

The API I have in mind is like this because I think it's probably not a good idea to return mutable data to users.

This can be a somewhat low level API as a first step - we can build VFS on top of this later (e.g. put some kind of archive as one of the assets)

Any thoughts?

@nodejs/single-executable-admins

mxschmitt · 2023-11-09T14:15:57Z

I think that would work for us, we would then create the VFS abstraction in the user land for now. Last-modified timestamp would be probably good to keep that stored inside as well. Not sure about rwx permissions / user permissions, makes probably not a lot of sense.

tony-go · 2023-11-13T09:46:35Z

Really great @joyeecheung 👏🏼

Looking forward to see the PR 🤩

RaisinTen · 2023-11-14T08:26:15Z

This sounds like a good first step for the VFS 👍

joyeecheung · 2023-11-20T18:22:02Z

I have a branch at https://github.com/joyeecheung/node/tree/sea-assets that contains an implementation of the proposal which is probably good to go, but I think I need to look into some of the SEA flaky tests issues before adding another SEA feature/test.

robertsLando · 2023-11-21T08:05:29Z

@joyeecheung thanks for all your efforts with nodejs SEA feature!

What about supporting also multiple js files? I mean for structured applications bundle them on a single file could be complex, is there any thougts about that?

joyeecheung · 2023-11-21T11:45:44Z

That should probably be layered on top of VFS, which can be layered on top of this lower-level thing.

lukaslihotzki · 2023-11-24T17:41:58Z

Exposing a native, dedicated API to read assets is a great idea. VFS patching can be done on top of this, possibly by including archives as assets. For this use case, it would make sense to (also) expose assets as Blob (not just as Buffer and String). Blobs can be sliced (and then be read asynchronously), which would allow reading specific files inside an archive on-demand. Also Blobs (and their slices) can be streamed, which would be nice when the archive is compressed.

Until such a native API is available, you can include (binary) files into a SEA by reading them with fs.readFileSync during snapshot creation (https://gist.github.com/lukaslihotzki/c60fef03d5a14d1c8723bc1251ede0ee).

joyeecheung · 2023-11-27T15:17:17Z

An interface that returns a Blob sounds like a good idea.

When it's a short string, print it inline, otherwise print it from a separate line. Also add the missing line breaks finally. PR-URL: #50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>

With this patch: Users can now include assets by adding a key-path dictionary to the configuration as the `assets` field. At build time, Node.js would read the assets from the specified paths and bundle them into the preparation blob. In the generated executable, users can retrieve the assets using the `sea.getAsset()` and `sea.getAssetAsBlob()` API. ```json { "main": "/path/to/bundled/script.js", "output": "/path/to/write/the/generated/blob.blob", "assets": { "a.jpg": "/path/to/a.jpg", "b.txt": "/path/to/b.txt" } } ``` The single-executable application can access the assets as follows: ```cjs const { getAsset } = require('node:sea'); // Returns a copy of the data in an ArrayBuffer const image = getAsset('a.jpg'); // Returns a string decoded from the asset as UTF8. const text = getAsset('b.txt', 'utf8'); // Returns a Blob containing the asset. const blob = getAssetAsBlob('a.jpg'); ``` Drive-by: update the documentation to include a section dedicated to the injected main script and refer to it as "injected main script" instead of "injected module" because it's a script, not a module. PR-URL: #50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>

This patch adds support for `sea.getRawAsset()` which is similar to `sea.getAsset()` but returns the raw asset in an array buffer without copying. Users should avoid writing to the returned array buffer. If the injected section is not marked as writable or not aligned, writing to the raw asset is likely to result in a crash. PR-URL: #50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>

When it's a short string, print it inline, otherwise print it from a separate line. Also add the missing line breaks finally. PR-URL: nodejs#50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>

With this patch: Users can now include assets by adding a key-path dictionary to the configuration as the `assets` field. At build time, Node.js would read the assets from the specified paths and bundle them into the preparation blob. In the generated executable, users can retrieve the assets using the `sea.getAsset()` and `sea.getAssetAsBlob()` API. ```json { "main": "/path/to/bundled/script.js", "output": "/path/to/write/the/generated/blob.blob", "assets": { "a.jpg": "/path/to/a.jpg", "b.txt": "/path/to/b.txt" } } ``` The single-executable application can access the assets as follows: ```cjs const { getAsset } = require('node:sea'); // Returns a copy of the data in an ArrayBuffer const image = getAsset('a.jpg'); // Returns a string decoded from the asset as UTF8. const text = getAsset('b.txt', 'utf8'); // Returns a Blob containing the asset. const blob = getAssetAsBlob('a.jpg'); ``` Drive-by: update the documentation to include a section dedicated to the injected main script and refer to it as "injected main script" instead of "injected module" because it's a script, not a module. PR-URL: nodejs#50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>

This patch adds support for `sea.getRawAsset()` which is similar to `sea.getAsset()` but returns the raw asset in an array buffer without copying. Users should avoid writing to the returned array buffer. If the injected section is not marked as writable or not aligned, writing to the raw asset is likely to result in a crash. PR-URL: nodejs#50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>

When it's a short string, print it inline, otherwise print it from a separate line. Also add the missing line breaks finally. PR-URL: #50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>

With this patch: Users can now include assets by adding a key-path dictionary to the configuration as the `assets` field. At build time, Node.js would read the assets from the specified paths and bundle them into the preparation blob. In the generated executable, users can retrieve the assets using the `sea.getAsset()` and `sea.getAssetAsBlob()` API. ```json { "main": "/path/to/bundled/script.js", "output": "/path/to/write/the/generated/blob.blob", "assets": { "a.jpg": "/path/to/a.jpg", "b.txt": "/path/to/b.txt" } } ``` The single-executable application can access the assets as follows: ```cjs const { getAsset } = require('node:sea'); // Returns a copy of the data in an ArrayBuffer const image = getAsset('a.jpg'); // Returns a string decoded from the asset as UTF8. const text = getAsset('b.txt', 'utf8'); // Returns a Blob containing the asset. const blob = getAssetAsBlob('a.jpg'); ``` Drive-by: update the documentation to include a section dedicated to the injected main script and refer to it as "injected main script" instead of "injected module" because it's a script, not a module. PR-URL: #50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>

This patch adds support for `sea.getRawAsset()` which is similar to `sea.getAsset()` but returns the raw asset in an array buffer without copying. Users should avoid writing to the returned array buffer. If the injected section is not marked as writable or not aligned, writing to the raw asset is likely to result in a crash. PR-URL: #50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>

joyeecheung · 2024-02-18T23:17:44Z

Closed via nodejs/node#50960

mxschmitt · 2024-02-19T08:22:45Z

Thats great, thank you, will try it out!

mxschmitt · 2024-02-19T11:26:50Z

Tried it out and seems to work very well for us! 🚀