esm: bypass CommonJS loader under `--default-type=module` #49986

GeoffreyBooth · 2023-10-01T04:13:47Z

This PR implements the part of #49432 where we avoid using the CommonJS loader for ESM entry points in the new --default-type=module mode. This effectively completes some TODOs in the code that note that particular code paths should be deprecated.

The only user-observable effects of this are that, under --experimental-default-type=module:

Extension searching is no longer provided for the main entry point. Users need to type node entry.js, not node entry.
Libraries that were monkey-patching the CommonJS loader via code loaded by --require will no longer achieve anything. They weren’t achieving much before, as monkey-patching the CommonJS loader only affects CommonJS and not ES modules, so it’s not like the previous behavior was really working very well (if at all) for applications with an ESM entry point.

This inches us closer to actually deprecating the CommonJS loader and having the ESM loader handle all modules, which is a long term goal that will aid in maintainability and future functionality for the modules subsystem.

nodejs-github-bot · 2023-10-01T04:13:51Z

Review requested:

@nodejs/loaders

lib/internal/modules/run_main.js

GeoffreyBooth · 2023-10-01T17:45:11Z

So there are really two places where we resolve the input into a path: the resolveMainPath function, and this code within runMainESM:

    const main = path.isAbsolute(mainPath) ?
      pathToFileURL(mainPath).href : mainPath;

If we bypass or immediately return from resolveMainPath, then the mainPath input to runMainESM is the raw input from the CLI, like entry.js. And it usually won’t be absolute, so pathToFileURL isn’t getting called for it. So then we’re passing that input into esmLoader.import, which might work when the relative file path is parseable as a relative URL, which is most of the time—but not all of the time.

So I think either we undo the commit I just pushed, and always resolve the input into a path within resolveMainPath for both flows; or do something else. The more I think about this, the more it feels like runMainESM should always expect either an absolute file path or an absolute file URL, and code upstream of it is responsible for that conversion. We shouldn’t be resolving the input multiple times in multiple places. Thoughts @aduh95?

lib/internal/modules/run_main.js

GeoffreyBooth · 2023-10-02T00:08:22Z

@aduh95 I pushed some more commits, that undo the “streamline” commit, and add some tests and comments and improve an error message.

I’m open to being convinced otherwise, but it feels like a mistake to completely bypass resolveMainPath in ESM mode. Even if the esmLoader.import call within runMainESM somehow correctly parses a relative path as an URL (and please educate me on if that’s the case) we still should preserve the part of resolveMainPath that follows symlinks if --preserve-symlinks-main is not passed, right? We should presumably still support that in the new mode, right?

Assuming the above is all correct so far, I wanted a better error message for the “left off the file extension” case than just ENOENT: no such file or directory, lstat '/Users/geoffrey/Sites/node/test/fixtures/es-modules/package-without-type/index'. So I abstracted the “see if the resolution would’ve worked according to CommonJS rules” logic from esm/resolve.js and import it into the error path in run_main.js, so that node index prints an error message including Did you mean to import /Users/geoffrey/Sites/node/test/fixtures/es-modules/package-without-type/index.js? like we show for import specifiers that lack an extension. We were getting this message for free by passing through invalid input into esmLoader.import, but if we follow through with the toRealPath call in resolveMainPath then we need to handle it there.

JakobJingleheimer

Huzzah 🙂

A small nit: somewhere (I can't find the line now—on my mobile), it declares a typedef for default-type as a string: it's not any old string, it's an enum of specific strings. Ideally, those are numerated in the typedef.

ljharb · 2023-10-05T06:08:50Z

lib/internal/modules/esm/resolve.js

+    error.stack =
+      ArrayPrototypeShift(lines) + '\n' +
+      hint + '\n' +
+      ArrayPrototypeJoin(lines, '\n');


Suggested change

error.stack =

ArrayPrototypeShift(lines) + '\n' +

hint + '\n' +

ArrayPrototypeJoin(lines, '\n');

error.stack = `${ArrayPrototypeShift(lines)}\n${hint}\n${ArrayPrototypeJoin(lines, '\n')}`;

lib/internal/modules/run_main.js

GeoffreyBooth · 2023-10-05T14:44:27Z

I reverted the optimization for the legacy flow. I don’t want to make changes to the legacy flow in this PR, I’m too nervous about such changes introducing bugs since the changes don’t break any tests when I’d expect that they should. I want to just land the new feature intended by this PR, bypassing the CommonJS loader under --default-type=module, and we can refactor the legacy flow in a separate PR (and add more tests for it).

This reverts commit 6c6aefe.

…extension searching; fix lint

This reverts commit 6d1a6e5.

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

nodejs-github-bot · 2023-10-05T15:19:41Z

CI: https://ci.nodejs.org/job/node-test-pull-request/54560/

nodejs-github-bot · 2023-10-05T16:42:00Z

CI: https://ci.nodejs.org/job/node-test-pull-request/54564/

nodejs-github-bot · 2023-10-05T17:57:34Z

Landed in 19b470f

PR-URL: nodejs#49986 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

PR-URL: #49986 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

PR-URL: nodejs#49986 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

PR-URL: #49986 Backport-PR-URL: #50669 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

no code changes; patch just needed an update due to nearby upstream changes Xref: nodejs/node#49986

…patch Xref: nodejs/node#49986 minor manual changes needed to sync with upstream change

no code changes; patch just needed an update due to nearby upstream changes Xref: nodejs/node#49986

…patch Xref: nodejs/node#49986 minor manual changes needed to sync with upstream change

nodejs/node#49986

* chore: bump node in DEPS to v20.10.0 * chore: update feat_initialize_asar_support.patch no code changes; patch just needed an update due to nearby upstream changes Xref: nodejs/node#49986 * chore: update pass_all_globals_through_require.patch no manual changes; patch applied with fuzz Xref: nodejs/node#49657 * chore: update refactor_allow_embedder_overriding_of_internal_fs_calls Xref: nodejs/node#49912 no code changes; patch just needed an update due to nearby upstream changes * chore: update chore_allow_the_node_entrypoint_to_be_a_builtin_module.patch Xref: nodejs/node#49986 minor manual changes needed to sync with upstream change * update fix_expose_the_built-in_electron_module_via_the_esm_loader.patch Xref: nodejs/node#50096 Xref: nodejs/node#50314 in lib/internal/modules/esm/load.js, update the code that checks for `format === 'electron'`. I'd like 👀 on this Xref: nodejs/node#49657 add braces in lib/internal/modules/esm/translators.js to sync with upstream * fix: lazyload fs in esm loaders to apply asar patches * nodejs/node#50127 * nodejs/node#50096 * esm: jsdoc for modules code nodejs/node#49523 * test: set test-cli-node-options as flaky nodejs/node#50296 * deps: update c-ares to 1.20.1 nodejs/node#50082 * esm: bypass CommonJS loader under --default-type=module nodejs/node#49986 * deps: update uvwasi to 0.0.19 nodejs/node#49908 * lib,test: do not hardcode Buffer.kMaxLength nodejs/node#49876 * crypto: account for disabled SharedArrayBuffer nodejs/node#50034 * test: fix edge snapshot stack traces nodejs/node#49659 * src: generate snapshot with --predictable nodejs/node#48749 * chore: fixup patch indices * fs: throw errors from sync branches instead of separate implementations nodejs/node#49913 * crypto: ensure valid point on elliptic curve in SubtleCrypto.importKey nodejs/node#50234 * esm: detect ESM syntax in ambiguous JavaScrip nodejs/node#50096 * fixup! test: fix edge snapshot stack traces * esm: unflag extensionless ES module JavaScript and Wasm in module scope nodejs/node#49974 * [tagged-ptr] Arrowify objects https://chromium-review.googlesource.com/c/v8/v8/+/4705331 --------- Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com> Co-authored-by: Charles Kerr <charles@charleskerr.com> Co-authored-by: Shelley Vohr <shelley.vohr@gmail.com>

PR-URL: nodejs#49986 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

PR-URL: nodejs/node#49986 Backport-PR-URL: nodejs/node#50669 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

GeoffreyBooth added module Issues and PRs related to the module subsystem. cli Issues and PRs related to the Node.js command line interface. esm Issues and PRs related to the ECMAScript Modules implementation. labels Oct 1, 2023

GeoffreyBooth requested a review from aduh95 October 1, 2023 04:13

nodejs-github-bot added needs-ci PRs that need a full CI run. process Issues and PRs related to the process subsystem. labels Oct 1, 2023

GeoffreyBooth force-pushed the bypass-cjs-loader branch from a0fba47 to 4669340 Compare October 1, 2023 04:27

aduh95 reviewed Oct 1, 2023

View reviewed changes

lib/internal/modules/run_main.js Show resolved Hide resolved

GeoffreyBooth force-pushed the bypass-cjs-loader branch from 4669340 to 6c6aefe Compare October 1, 2023 17:32

aduh95 reviewed Oct 1, 2023

View reviewed changes

lib/internal/modules/run_main.js Outdated Show resolved Hide resolved

GeoffreyBooth force-pushed the bypass-cjs-loader branch from 6f61ca0 to 4cf437c Compare October 3, 2023 05:34

JakobJingleheimer approved these changes Oct 4, 2023

View reviewed changes

ljharb reviewed Oct 5, 2023

View reviewed changes

aduh95 reviewed Oct 5, 2023

View reviewed changes

lib/internal/modules/run_main.js Outdated Show resolved Hide resolved

aduh95 reviewed Oct 5, 2023

View reviewed changes

lib/internal/modules/run_main.js Outdated Show resolved Hide resolved

GeoffreyBooth and others added 11 commits October 5, 2023 07:59

esm: bypass CommonJS loader under --default-type

97b1966

Streamline

30821db

Revert "Streamline"

286c632

This reverts commit 6c6aefe.

Move/improve tests; decorate "not found" error to provide hint about …

4620842

…extension searching; fix lint

Update docs

bdafa66

Fix for Windows

83884e5

Move out main re-resolution

162b6fa

Optimize

22bddbd

Revert "Optimize"

6c97b9a

This reverts commit 6d1a6e5.

Update lib/internal/modules/run_main.js

32aaabc

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

Return early if possible

038dfb8

GeoffreyBooth mentioned this pull request Oct 5, 2023

Proposal: Set --experimental-default-type mode by detecting ESM syntax in entry point #50043

Closed

GeoffreyBooth added commit-queue Add this label to land a pull request using GitHub Actions. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. labels Oct 5, 2023

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Oct 5, 2023

nodejs-github-bot merged commit 19b470f into nodejs:main Oct 5, 2023
60 checks passed

GeoffreyBooth deleted the bypass-cjs-loader branch October 5, 2023 18:02

RafaelGSS mentioned this pull request Oct 9, 2023

2023-10-17, Version 21.0.0 (Current) #49870

Merged

alexfernandez pushed a commit to alexfernandez/node that referenced this pull request Nov 1, 2023

esm: bypass CommonJS loader under --default-type

6d366a9

PR-URL: nodejs#49986 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

targos pushed a commit to targos/node that referenced this pull request Nov 11, 2023

esm: bypass CommonJS loader under --default-type

3e4e72b

PR-URL: nodejs#49986 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

targos pushed a commit to targos/node that referenced this pull request Nov 11, 2023

esm: bypass CommonJS loader under --default-type

4f27091

PR-URL: nodejs#49986 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

targos mentioned this pull request Nov 11, 2023

[v18.x] Backport most ESM and customization hook changes #50669

Closed

targos pushed a commit that referenced this pull request Nov 11, 2023

esm: bypass CommonJS loader under --default-type

4de838f

PR-URL: #49986 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

targos pushed a commit to targos/node that referenced this pull request Nov 11, 2023

esm: bypass CommonJS loader under --default-type

151e812

PR-URL: nodejs#49986 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

targos mentioned this pull request Nov 12, 2023

v20.10.0 release proposal #50682

Merged

targos added the backported-to-v18.x PRs backported to the v18.x-staging branch. label Nov 23, 2023

targos pushed a commit that referenced this pull request Nov 23, 2023

esm: bypass CommonJS loader under --default-type

c3c945e

PR-URL: #49986 Backport-PR-URL: #50669 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

targos mentioned this pull request Nov 28, 2023

v18.19.0 release proposal #50953

Merged

juanarbol mentioned this pull request Dec 6, 2023

2023-12-06, N|Solid v5.0.1 Iron Release nodesource/nsolid#41

Merged

ckerr added a commit to electron/electron that referenced this pull request Dec 7, 2023

chore: update feat_initialize_asar_support.patch

bcd8239

no code changes; patch just needed an update due to nearby upstream changes Xref: nodejs/node#49986

ckerr added a commit to electron/electron that referenced this pull request Dec 7, 2023

chore: update chore_allow_the_node_entrypoint_to_be_a_builtin_module.…

88449ae

…patch Xref: nodejs/node#49986 minor manual changes needed to sync with upstream change

codebytere pushed a commit to electron/electron that referenced this pull request Dec 8, 2023

chore: update feat_initialize_asar_support.patch

943ccb7

no code changes; patch just needed an update due to nearby upstream changes Xref: nodejs/node#49986

codebytere pushed a commit to electron/electron that referenced this pull request Dec 8, 2023

chore: update chore_allow_the_node_entrypoint_to_be_a_builtin_module.…

3494aa6

…patch Xref: nodejs/node#49986 minor manual changes needed to sync with upstream change

codebytere added a commit to electron/electron that referenced this pull request Dec 8, 2023

esm: bypass CommonJS loader under --default-type=module

19f12ca

nodejs/node#49986

debadree25 pushed a commit to debadree25/node that referenced this pull request Apr 15, 2024

esm: bypass CommonJS loader under --default-type

672f793

PR-URL: nodejs#49986 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

esm: bypass CommonJS loader under `--default-type=module` #49986

esm: bypass CommonJS loader under `--default-type=module` #49986

GeoffreyBooth commented Oct 1, 2023 •

edited

nodejs-github-bot commented Oct 1, 2023

GeoffreyBooth commented Oct 1, 2023 •

edited

GeoffreyBooth commented Oct 2, 2023

JakobJingleheimer left a comment

ljharb Oct 5, 2023

GeoffreyBooth commented Oct 5, 2023

nodejs-github-bot commented Oct 5, 2023

nodejs-github-bot commented Oct 5, 2023

nodejs-github-bot commented Oct 5, 2023

esm: bypass CommonJS loader under --default-type=module #49986

esm: bypass CommonJS loader under --default-type=module #49986

Conversation

GeoffreyBooth commented Oct 1, 2023 • edited

nodejs-github-bot commented Oct 1, 2023

GeoffreyBooth commented Oct 1, 2023 • edited

GeoffreyBooth commented Oct 2, 2023

JakobJingleheimer left a comment

Choose a reason for hiding this comment

ljharb Oct 5, 2023

Choose a reason for hiding this comment

GeoffreyBooth commented Oct 5, 2023

nodejs-github-bot commented Oct 5, 2023

nodejs-github-bot commented Oct 5, 2023

nodejs-github-bot commented Oct 5, 2023

esm: bypass CommonJS loader under `--default-type=module` #49986

esm: bypass CommonJS loader under `--default-type=module` #49986

GeoffreyBooth commented Oct 1, 2023 •

edited

GeoffreyBooth commented Oct 1, 2023 •

edited