build: enable loading internal modules from disk #31321

devsnek · 2020-01-11T22:04:41Z

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
commit message follows commit guidelines

nodejs-github-bot · 2020-01-11T22:24:22Z

CI: https://ci.nodejs.org/job/node-test-pull-request/28368/

addaleax

Still sad that this isn’t a runtime option, but thanks for picking this up again.

src/node_native_module.cc

devsnek · 2020-01-12T17:29:23Z

@addaleax I'd be happy to make it a runtime option. I'm not sure how I'd get data from the options parser to the internal module loader though.

tniessen · 2020-01-12T17:33:35Z

This is great! I am fine with it being a configure option (maybe just for now), given that we probably wouldn't want people to use it outside of development. (For example, the current PR seems to resolve the path relative to the current working directory, which seems dangerous outside of the repo.)

src/node_native_module.cc

configure.py

Trott · 2020-01-13T15:20:53Z

Should this print a warning telling end users not to use it?

jasnell · 2020-01-13T16:53:08Z

+1 to emitting a warning on this.

addaleax · 2020-01-13T17:45:03Z

I think if we turn this into a runtime option, the warning may be appropriate, but for a compile time option I don’t quite see the point – whoever built the Node.js binary explicitly opted into a developer feature, and adding the warning is going to make running the test suite infeasible, which makes it less useful as a developer feature.

targos · 2020-01-13T17:50:41Z

Maybe James was thinking about a build time warning?

jasnell · 2020-01-13T17:56:34Z

Maybe James was thinking about a build time warning?

Yes, thank you. That's what I meant.

Regarding the idea of a runtime option: I see the benefit but I also see significant risk. The fact that we have to expose --expose-internals is bad enough. I'd prefer we start with the configuration option and go from there.

joyeecheung

A couple of non-blocking nits. Happy to see this happening!

I think it's possible to add a test for it (at least by testing that adding a new module works), to feature-detect I think it makes sense to just use process.features. But that can be added later since this is just a developer feature after all.

src/node_native_module.cc

configure.py

devsnek · 2020-01-14T20:16:19Z

I'm not sure how to test this cc @nodejs/build

richardlau · 2020-01-14T20:58:18Z

I'm not sure how to test this cc @nodejs/build

If you want to do a one-off build you can run https://ci.nodejs.org/job/node-test-commit-custom-suites-freestyle/ with custom args passed to configure.

devsnek · 2020-01-14T20:59:43Z

@richardlau i mean continuously. I guess we'd need to add a new box/configuration?

configure.py

richardlau · 2020-01-14T21:08:18Z

@richardlau i mean continuously. I guess we'd need to add a new box/configuration?

You probably wouldn't need a new box but probably would need a new job since you'd have to build Node.js with different options passed to configure. The nearest to that we currently have are the various sharedlibs jobs but we do not test every option that can be passed to configure.

The Build WG is currently very stretched though so there may not be much enthusiasm for adding another build configuration for a developer feature.

richardlau · 2020-01-14T21:12:31Z

re. new build configurations nodejs/build#1982 / #30057 would be ideal for this sort of thing but unfortunately it's stuck (nodejs/build#1982 (comment)).

devsnek · 2020-01-14T22:19:29Z

cctest failures when this is enabled:

[----------] 6 tests from BaseObjectPtrTest
[ RUN      ] BaseObjectPtrTest.ScopedDetached
../test/cctest/test_base_object_ptr.cc:50: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 1
  0
../test/cctest/test_base_object_ptr.cc:53: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:55: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 1
  0
[  FAILED  ] BaseObjectPtrTest.ScopedDetached (33 ms)
[ RUN      ] BaseObjectPtrTest.ScopedDetachedWithWeak
../test/cctest/test_base_object_ptr.cc:66: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 1
  0
../test/cctest/test_base_object_ptr.cc:70: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:73: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 1
  0
[  FAILED  ] BaseObjectPtrTest.ScopedDetachedWithWeak (21 ms)
[ RUN      ] BaseObjectPtrTest.Undetached
../test/cctest/test_base_object_ptr.cc:87: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:83: Failure
Expected equality of these values:
  static_cast<Environment*>(arg)->base_object_count()
    Which is: 1
  0
[  FAILED  ] BaseObjectPtrTest.Undetached (20 ms)
[ RUN      ] BaseObjectPtrTest.GCWeak
../test/cctest/test_base_object_ptr.cc:104: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:111: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
[  FAILED  ] BaseObjectPtrTest.GCWeak (20 ms)
[ RUN      ] BaseObjectPtrTest.Moveable
../test/cctest/test_base_object_ptr.cc:129: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:140: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:145: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 1
  0
[  FAILED  ] BaseObjectPtrTest.Moveable (30 ms)
[ RUN      ] BaseObjectPtrTest.NestedClasses
../test/cctest/test_base_object_ptr.cc:175: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 4
  3
../test/cctest/test_base_object_ptr.cc:167: Failure
Expected equality of these values:
  static_cast<Environment*>(arg)->base_object_count()
    Which is: 1
  0
[  FAILED  ] BaseObjectPtrTest.NestedClasses (19 ms)
[----------] 6 tests from BaseObjectPtrTest (143 ms total)

rvagg · 2020-01-14T23:23:22Z

I don't mind putting in a bit of work to extend the sharedlibs set to include new build configurations if the build configuration is above some poorly defined level of significance. Basically, is this anticipated to be a commonly used build option? Do we think this is going to grow into a more significant thing? If it's a key feature then I could do some work with some pointers on how to compile and run the tests with this option unless this is just some obscure thing in the corner that serves the use of a couple of people with special needs.

devsnek · 2020-01-14T23:30:32Z

I'd imagine people working on node core would just run with this option enabled (unless they were working specifically on the internal module loader itself)

I don't think it's a huge priority to get testing infra for it though.

rvagg · 2020-01-14T23:32:36Z

(missed the button, sorry)

mmarchini · 2020-01-14T23:48:07Z

Can this be tested on GH Actions instead? It would give some coverage and should be lower maintenance than doing it on Jenkins right now.

src/node_native_module.cc

nodejs-github-bot · 2020-01-16T22:47:49Z

CI: https://ci.nodejs.org/job/node-test-pull-request/28455/

nodejs-github-bot · 2020-01-17T21:13:27Z

CI: https://ci.nodejs.org/job/node-test-pull-request/28467/

devsnek · 2020-01-17T21:15:18Z

this is not ready to land, see #31321 (comment)

devsnek · 2020-01-29T03:59:59Z

@nodejs/collaborators re #31321 (comment)

nodejs-github-bot · 2020-01-31T08:43:08Z

CI: https://ci.nodejs.org/job/node-test-pull-request/28804/

mcollina

lgtm

PR-URL: nodejs#31321 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

devsnek · 2020-01-31T09:24:25Z

landed in 43fb6ff

PR-URL: #31321 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

codebytere · 2020-03-15T04:27:45Z

@devsnek should this go into v12.x, and could you please manually backport it if so? if not, feel free to update the label.

PR-URL: nodejs#31321 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

PR-URL: #31321 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

nodejs-github-bot added build Issues and PRs related to build files or the CI. c++ Issues and PRs that require attention from people who are familiar with C++. labels Jan 11, 2020

devsnek requested a review from joyeecheung January 11, 2020 22:04

addaleax approved these changes Jan 12, 2020

View reviewed changes

tniessen approved these changes Jan 12, 2020

View reviewed changes

src/node_native_module.cc Outdated Show resolved Hide resolved

BridgeAR added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Jan 13, 2020

devsnek removed the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Jan 13, 2020

bnoordhuis reviewed Jan 13, 2020

View reviewed changes

src/node_native_module.cc Outdated Show resolved Hide resolved

src/node_native_module.cc Outdated Show resolved Hide resolved

cjihrig reviewed Jan 13, 2020

View reviewed changes

src/node_native_module.cc Outdated Show resolved Hide resolved

configure.py Outdated Show resolved Hide resolved

joyeecheung approved these changes Jan 14, 2020

View reviewed changes

src/node_native_module.cc Outdated Show resolved Hide resolved

configure.py Outdated Show resolved Hide resolved

devsnek force-pushed the load-internal-modules-from-disk branch from 52a14a6 to c44133d Compare January 14, 2020 21:00

joyeecheung requested changes Jan 14, 2020

View reviewed changes

configure.py Outdated Show resolved Hide resolved

devsnek force-pushed the load-internal-modules-from-disk branch 2 times, most recently from 6ecfc90 to dcc29cb Compare January 14, 2020 22:17

Trott approved these changes Jan 15, 2020

View reviewed changes

cjihrig approved these changes Jan 15, 2020

View reviewed changes

bnoordhuis reviewed Jan 16, 2020

View reviewed changes

src/node_native_module.cc Outdated Show resolved Hide resolved

src/node_native_module.cc Show resolved Hide resolved

devsnek force-pushed the load-internal-modules-from-disk branch from dcc29cb to fb0f519 Compare January 16, 2020 19:16

joyeecheung approved these changes Jan 17, 2020

View reviewed changes

richardlau mentioned this pull request Jan 31, 2020

cctest BaseObjectPtrTest failing on master #31585

Closed

mcollina approved these changes Jan 31, 2020

View reviewed changes

devsnek force-pushed the load-internal-modules-from-disk branch from fb0f519 to 43fb6ff Compare January 31, 2020 09:23

devsnek merged commit 43fb6ff into nodejs:master Jan 31, 2020

devsnek deleted the load-internal-modules-from-disk branch January 31, 2020 09:24

codebytere mentioned this pull request Feb 17, 2020

v13.9.0 proposal #31837

Merged

codebytere added the backport-requested-v12.x label Mar 15, 2020

targos removed the backport-requested-v12.x label Apr 25, 2020

leidegre mentioned this pull request Apr 30, 2020

Keepalive prevents garbage collection on unread responses #29394

Closed

targos mentioned this pull request May 2, 2020

v12.17.0 release proposal #33197

Merged

richardlau mentioned this pull request Nov 19, 2020

Any way to reload the built-in modules? nodejs/help#3087

Closed

RaisinTen mentioned this pull request Nov 21, 2020

build: enable loading builtin modules at runtime #36215

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: enable loading internal modules from disk #31321

build: enable loading internal modules from disk #31321

devsnek commented Jan 11, 2020 •

edited

nodejs-github-bot commented Jan 11, 2020

addaleax left a comment

devsnek commented Jan 12, 2020

tniessen commented Jan 12, 2020

Trott commented Jan 13, 2020

jasnell commented Jan 13, 2020

addaleax commented Jan 13, 2020

targos commented Jan 13, 2020

jasnell commented Jan 13, 2020

joyeecheung left a comment •

edited

devsnek commented Jan 14, 2020

richardlau commented Jan 14, 2020

devsnek commented Jan 14, 2020

richardlau commented Jan 14, 2020

richardlau commented Jan 14, 2020

devsnek commented Jan 14, 2020

rvagg commented Jan 14, 2020

devsnek commented Jan 14, 2020

rvagg commented Jan 14, 2020

mmarchini commented Jan 14, 2020

nodejs-github-bot commented Jan 16, 2020

nodejs-github-bot commented Jan 17, 2020

devsnek commented Jan 17, 2020

devsnek commented Jan 29, 2020

nodejs-github-bot commented Jan 31, 2020

mcollina left a comment

devsnek commented Jan 31, 2020

codebytere commented Mar 15, 2020

build: enable loading internal modules from disk #31321

build: enable loading internal modules from disk #31321

Conversation

devsnek commented Jan 11, 2020 • edited

Checklist

nodejs-github-bot commented Jan 11, 2020

addaleax left a comment

Choose a reason for hiding this comment

devsnek commented Jan 12, 2020

tniessen commented Jan 12, 2020

Trott commented Jan 13, 2020

jasnell commented Jan 13, 2020

addaleax commented Jan 13, 2020

targos commented Jan 13, 2020

jasnell commented Jan 13, 2020

joyeecheung left a comment • edited

Choose a reason for hiding this comment

devsnek commented Jan 14, 2020

richardlau commented Jan 14, 2020

devsnek commented Jan 14, 2020

richardlau commented Jan 14, 2020

richardlau commented Jan 14, 2020

devsnek commented Jan 14, 2020

rvagg commented Jan 14, 2020

devsnek commented Jan 14, 2020

rvagg commented Jan 14, 2020

mmarchini commented Jan 14, 2020

nodejs-github-bot commented Jan 16, 2020

nodejs-github-bot commented Jan 17, 2020

devsnek commented Jan 17, 2020

devsnek commented Jan 29, 2020

nodejs-github-bot commented Jan 31, 2020

mcollina left a comment

Choose a reason for hiding this comment

devsnek commented Jan 31, 2020

codebytere commented Mar 15, 2020

devsnek commented Jan 11, 2020 •

edited

joyeecheung left a comment •

edited