node should execute files without extension as ESM if "type": "module", is specified

[luxzeitlos]

• Version: v14.4.0
• Platform: The node:14 docker container

What steps will reproduce the bug?

I'm creating a file called foo (without extension) at /app/bin/foo. I have a /app/package.json file with "type": "module",.
The foo file has a #!/usr/bin/env node shebang and /app/bin is in the PATH.

Now I want to run foo.

What is the expected behavior?

I would expect node to execute the file as ESM because of the "type": "module", in the parents folder package.json.

What do you see instead?

I get the following error:

internal/modules/run_main.js:54
    internalBinding('errors').triggerUncaughtException(
                              ^

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for /app/bin/foo
    at Loader.defaultGetFormat [as _getFormat] (internal/modules/esm/get_format.js:65:15)
    at Loader.getFormat (internal/modules/esm/loader.js:113:42)
    at Loader.getModuleJob (internal/modules/esm/loader.js:243:31)
    at async Loader.import (internal/modules/esm/loader.js:177:17) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
}

I assume node wants to check the file extension to see if its a .cjs extension and would require to execute it as common js. However IMHO if there is no file extension it should just respect the package.json in the parent folder and run it as ESM.

[aalexgabi]

I can't believe it's been a year and this is still not fixed. How are you supposed to use simple executable with a shebang that you want to add to $PATH? You don't want to use myprogram.js in your terminal.

Not even --input-type works (not that it would be useful for $PATH executables):

$ node --input-type module  bin/program
internal/process/esm_loader.js:74
    internalBinding('errors').triggerUncaughtException(
                              ^

Error [ERR_INPUT_TYPE_NOT_ALLOWED]: --input-type can only be used with string input via --eval, --print, or STDIN
←[90m    at Loader.defaultResolve [as _resolve] (internal/modules/esm/resolve.js:804:13)←[39m
←[90m    at Loader.resolve (internal/modules/esm/loader.js:86:40)←[39m
←[90m    at Loader.getModuleJob (internal/modules/esm/loader.js:230:28)←[39m
←[90m    at Loader.import (internal/modules/esm/loader.js:165:28)←[39m
←[90m    at internal/modules/run_main.js:46:28←[39m
←[90m    at Object.loadESM (internal/process/esm_loader.js:68:11)←[39m {
  code: ←[32m'ERR_INPUT_TYPE_NOT_ALLOWED'←[39m
}
``

[wsehl]

I can't believe it's been a year and this is still not fixed. How are you supposed to use simple executable with a shebang that you want to add to $PATH? You don't want to use myprogram.js in your terminal.

Not even --input-type works (not that it would be useful for $PATH executables):

$ node --input-type module  bin/program
internal/process/esm_loader.js:74
    internalBinding('errors').triggerUncaughtException(
                              ^

Error [ERR_INPUT_TYPE_NOT_ALLOWED]: --input-type can only be used with string input via --eval, --print, or STDIN
←[90m    at Loader.defaultResolve [as _resolve] (internal/modules/esm/resolve.js:804:13)←[39m
←[90m    at Loader.resolve (internal/modules/esm/loader.js:86:40)←[39m
←[90m    at Loader.getModuleJob (internal/modules/esm/loader.js:230:28)←[39m
←[90m    at Loader.import (internal/modules/esm/loader.js:165:28)←[39m
←[90m    at internal/modules/run_main.js:46:28←[39m
←[90m    at Object.loadESM (internal/process/esm_loader.js:68:11)←[39m {
  code: ←[32m'ERR_INPUT_TYPE_NOT_ALLOWED'←[39m
}
``

I had exactly the same problem, try deleting repository folder and cloning it again, it helped me, or as a last resort, you can try wsl

[aalexgabi]

@wsehl I don't think proposed solutions would help.

In an empty folder this works:

touch empty && node empty

But not in a folder with package.json containing type: module which is worse since it's explicit

$ echo '{"type": "module"}' > package.json && touch empty && node empty
internal/process/esm_loader.js:74
    internalBinding('errors').triggerUncaughtException(
                              ^

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for ...\empty
←[90m    at Loader.defaultGetFormat [as _getFormat] (internal/modules/esm/get_format.js:71:15)←[39m
←[90m    at Loader.getFormat (internal/modules/esm/loader.js:102:42)←[39m
←[90m    at Loader.getModuleJob (internal/modules/esm/loader.js:231:31)←[39m
←[90m    at async Loader.import (internal/modules/esm/loader.js:165:17)←[39m
←[90m    at async Object.loadESM (internal/process/esm_loader.js:68:5)←[39m {
  code: ←[32m'ERR_UNKNOWN_FILE_EXTENSION'←[39m
}

[aduh95]

How are you supposed to use simple executable with a shebang that you want to add to $PATH?

Here's a workaround assuming bin/program is the ES module you want executed:

cd bin
mv program program.mjs
echo '{ "type": "commonjs" }' > package.json
printf '#!/usr/bin/env node\n"use strict";import("./program.mjs");\n' > program

Then you can add it to your $PATH and don't have to use the extension in your terminal – I think the only difference is you won't get exit code 13 in case of unfinished Top-Level Await (because you can't use TLA in CJS), other than that it should behave as if bin/program.mjs was executed directly.

[aalexgabi]

@aduh95 Thank you for the workaround. It worked 🎉

I had to add manual handling of rejections to make TLA errors in bin/program result in exit code 1:

#!/usr/bin/node

import('../src/myprogram.js').catch((err) => {
  console.error(err);
  process.exit(1);
});

However it's very sad to have to use this workaround in node and it's going to make backwards compatibility with already released node versions hard even if it's fixed.

I hope that node team will fix this soon before everyone starts using this workaround although some package maintainers that care about being compatible with all node versions will always have to.

[mizdra]

I also encountered this problem, but I was able to work around it by adding .js. The type field of package.json is still module.

• mizdra/eslint-interactive@a932cb1

[elibarzilay]

(This comment is more fitting for #37512, but that one was closed as a duplicate of this, so posting it here.)

Perhaps it is best to repost this as a new issue, since I haven't since any serious discussion about the plethora of bogus workaround suggestions.

I'm surprised at the fact that writing an esm script is still such a huge pita. But this is not new, since it seems that everyone and their cats are surprised at this too.

What's also surprising is the constant flood of "workaround" hacks that are fundamentally and utterly broken (like the above, or that one). Here's a list of what I can remember:

• Rename your file as *.mjs. This is not an option, since a script name should absolutely not include the language in its extension. If I give you a script called foo which is written in bash, I can later reimplement it in perl and you won't need to change your use. It should be possible to do the same with a node script.
• Add a "type": "module" to your package.json. This is not an option in the same scenario: now instead of dropping my script in your ~/bin or wherever, I give you too files?? But more than that: what if you want to drop my script into a directory that already has an existing package.json? Or if my ~/bin already has some existing node scripts, I certainly don't want to break them all to run the new script.
• Use --input-type=module < "$0". When you do this, it looks like it works. At some point you'll notice that error messages all refer to [eval1], but it's a small price, right? The you do a script that -- gasp -- actually uses stdin for whatever, and it goes down in flames.
• The hack that I settled on (see below) is also broken: a script should not create new files or symlinks. For example, it should work from a read-only directory.

A proper way (which some imaginary node-esm binary should do) to run a script should not depend on the filename of the script in any way. It should work for foo, foo.js, or foo bar.py, and still allow me to write esm code. It should also not require any other files --- specifically, not any package.json files that affect more than just my script. And it should certainly not do what my hack does: create new files when you run them.

---

For reference, here's the only sane prefix that I found so far. It's pretty horrible, but it works. I'd be very happy if there's a better way to make stuff run.

#!/usr/bin/env bash
/*.................................................... 2>/dev/null # -*- js -*-
[[ -h "$0" || "$0" = *.mjs ]] && exec node "$0" "$@"
[[ -e "$0.mjs" ]] && { echo "error: $0.mjs exists" 1>&2; exit 1; }
mv "$0" "$0.mjs"
ln -s "$0.mjs" "$0"
exec node "$0.mjs" "$@"
*/

[aduh95]

FWIW you could achieve the same thing with a smaller boilerplate using the experimental loader API (Node.js 16.12+):

#!/bin/sh
/*.................................................... 2>/dev/null # -*- js -*-
exec node --experimental-loader 'data:text/javascript,let%20t%3D!0%3Bexport%20async%20function%20resolve(e%2Co%2Cn)%7Bconst%20r%3Dawait%20n(e%2Co)%3Breturn%20t%26%26(r.format%3D%22module%22%2Ct%3D!1)%2Cr%7D' "$0" "$@"
*/

Un-minified loader code

let entryPoint = true;
export async function resolve(url, context, next) {
  const nextResolve = await next(url, context);
  if (entryPoint) {
    nextResolve.format = 'module';
    entryPoint = false;
  }
  return nextResolve;
}

A proper way (which some imaginary node-esm binary should do)

Having a separate binary comes with its own challenges and drawbacks, we wouldn't want to end up in a Python 2 / Python 3 scenario. If you feel strongly about this, you could create that binary yourself (which could be a simple Bash script that calls node with the above loader CLI flag), and either make it available on the npm registry and ask your users to npm install -g node-esm, either open a PR to this repo that makes such script auto-install alongside node.

[elibarzilay]

@aduh95: thanks for the suggestion. I saw this mentioned, but ignored it since it's very clearly experimental.

But it still fails:

1. It spits a warning. Is there a way to silence just this one?
2. More importantly, it also fails with Unknown file extension "".

BTW, the imaginary node-esm point is mostly irrelevant. Yes, if there's a script then such a wrapper can be made in the same way. However, as long as it relies on experimental features, it should live in node itself, as part of the experiment. The inescapable point here is that there should be some sane way to achieve it, something that doesn't involve experimental features or obscure incantations. (But for my needs, I'll be happy to still try them out.)

[elibarzilay]

@aduh95, Update: looks like the value is a promise rather than a value, so the following is more likely what you wanted to do (I added the node: test just in case, since it seemed to make it angry). The only bit that is missing now is silencing the warning. Any idea how to do that?

#!/usr/bin/env bash
/*.................................................... 2>/dev/null # -*- js -*-
exec node --experimental-loader='data:text/javascript,
let fst = true;
export function resolve(url, context, next) {
  const res = next(url, context);
  if (url.startsWith("node:") || !fst) return res;
  fst = false;
  return res.then(r => (r.format="module", r));
}' "$0" "$@"
*/

[aduh95]

1. It spits a warning. Is there a way to silence just this one?

I guess you could add the --no-warnings flag, but that would silence all warnings, not just this one.

2. More importantly, it also fails with Unknown file extension "".

My bad, I forgot the await as you said in your other comment. Here's a snippet that works (I've tested with v16.14.2, and v17.9.0):

#!/bin/sh
/*.................................................... 2>/dev/null # -*- js -*-
exec node --experimental-loader 'data:text/javascript,let%20t%3D!0%3Bexport%20async%20function%20resolve(e%2Co%2Cn)%7Bconst%20r%3Dawait%20n(e%2Co)%3Breturn%20t%26%26(r.format%3D%22module%22%2Ct%3D!1)%2Cr%7D' "$0" "$@"
*/

something that doesn't involve experimental features

The loader API is your best hope to make node behaves as you want it to behave with ESM content. It's still experimental because there are not enough contributors to do the work; if you want this to change, please contribute or convince a big tech company to invest in this part of Node.js. (to be clear, I think your comments are helpful because you share a (hacky) workaround that can help others, and thank you for that. However pestering about the current situation is not welcome if you are not willing to make something to improve that situation.)

[aalexgabi]

@elibarzilay I admire your persistence. I have lost way too much time due to this issue. I'm disappointed that there needs to be a discussion about this after such a long time and it was not addressed from day one of es6 modules introduction in node. Seems such a basic requirement to consider for introducing modules in node.

@aduh95 Clever but still a workaround. We should provide a workaround in the meantime but most importantly we should actually fix this issue.

We should be able to have an isolated es6 script in a system bin folder work out of the box without a file extension as it works on most systems. You don't type cp.exe or composer.php in a terminal to invoke them. Also package.json should not be a requirement since a package.json is not required to make a node program work.

Some long term solutions that I see:

• do syntax detection for files without extension
• some type of explicit statement that enables parsing files as esm, maybe only usable in files without extension. Think of debugger; or 'use strict'; or require.enableModules()
• respect the type in package.json module type if a package.json is present for files without extension
• introduce another explicit executable for node and use different shebang #!/usr/bin/env node-esm (probably not cleanest but would work)

The main value is simple for me: provide a very straightforward way to use es6 modules as first class citizen in the future of js world and keep the scripting standards from python, perl, bash etc. of making executable files that can change language without needing to rewrite all the scripts that call those executables. Also make sure that stdin, stdout, stderr, signals and exit code are handled transparently.

It would be such a pity to have to write a bash wrapper for every executable exposed by a packages for the years to come 😢 Some projects have many executable entry points.

Of course the best solution should be discussed but I would love to see a focus on fixing the root problem once and for all the years to come.

[elibarzilay]

@aduh95: Bah. I tried an await, but forgot to try and make the function async, and instead assumed that the loader thing should be defined as a simple function. So if only there's a way to silence only a specific warning I'd be happy to even make it an npm package or whatever (though I'd need to test it beyond my simple need, like whether it withstands running in a directory with a random package.json etc).

And I think that you misunderstood me mumbling about an experimental feature. The thing is that as long as it's not publicly available (without an "experimental" name and without a warning), then it's essentially a private api, and therefore should not be used by external code. But it could be part of node somehow (some contrib script maybe), since then using private functionality would be perfectly fine.

@aalexgabi, FWIW, I do see the argument for avoiding a second binary, even if it's as mild as some node-esm symlink to node that simply has the esm default if it's invoked under the symlinked name. Just an example; I'm not really suggesting a symlink. Instead, if this thing works fine, and if it indeed doing the job of changing the default only for the "main" file on the command line, then just make --input-type do this too (or add a similar flag) and that will make writing esm scripts easy enough that there won't be any need for a second binary.

[aalexgabi]

@elibarzilay from my understanding some kernels ignore everything after the first argument to executable (/usr/bin/env) in shebangs

#!/usr/bin/env node --input-type module

Would end up in the kernel exec call as:

#!/usr/bin/env node

That's the reason for which I proposed a different executable and not another argument for node. But I guess it can be a symlink if node can check through which name it's called indeed.

More info here https://stackoverflow.com/a/46674720

[elibarzilay]

Heh, yes, I know that such problems can happen, though I think that a more obvious breakage (at least on linux) is that it treats the whole rest of the line as the name of the executable and will therefore complain that there is no file named node --input-type module (literally).

But as long as there is some simple way to do this, then going around such problems falls under the usual kind of problems which are easy to solve. (For example, with the kind of multi-language shebang lines as in what I'm using in the above).

[aduh95]

I quite like the symlink idea, but unfortunately it wouldn't work on Windows I don't think 😕

• do syntax detection for files without extension
• some type of explicit statement that enables parsing files as esm, maybe only usable in files without extension. Think of debugger; or 'use strict'; or require.enableModules()

FYI these have already been discussed, and would need change at the language level, so that's not something that can happen in Node.js (see "use module" proposal, or this blog post that explains how the syntax detection is not a realistic approach).

We should be able to have an isolated es6 script in a system bin folder work out of the box without a file extension as it works on most systems. You don't type cp.exe or composer.php in a terminal to invoke them.

There are already working solutions for this today, e.g. if you define "bin": { "myJavascriptApp": "./bin/entry.mjs" } in your package.json, npm will create for you a myJavascriptApp executable (and even a myJavascriptApp.CMD for Windows) and add it on your PATH if you install that package globally.

[aduh95]

And I think that you misunderstood me mumbling about an experimental feature. The thing is that as long as it's not publicly available (without an "experimental" name and without a warning), then it's essentially a private api, and therefore should not be used by external code. But it could be part of node somehow (some contrib script maybe), since then using private functionality would be perfectly fine.

Yep, sorry, written communication is hard. FYI the goal is to expose the loader API to end users once it's stable. There are discussions on how to make the UX more seemless to not require the use of a CLI flag, and some function signatures may still change at some point, and some features are still missing, which is why it's still experimental. But that workaround snippet is not using anything that's likely to change.

So if only there's a way to silence only a specific warning

Yep that'd be great, it's a long requested feature: #30810.

[zdm]

Any news?
It is possible to add new command line option to specify file type?

node --type=module file-without-extension

It can be used in shebang also.
This will solve the problem.

[dmohs]

This is my current workaround:
https://gist.github.com/dmohs/13e2ea044707b77ff5d2af1a1c8585f4
It's a C program because OSX does not allow scripts in a shebang line.

[trusktr]

Please fix this. This is a pain when migrating from CJS to ESM with a package that has bin/s.

[francisashley]

Any resolution for this?

[pinko-fowle]

@elibarzilay from my understanding some kernels ignore everything after the first argument to executable (/usr/bin/env) in shebangs

#!/usr/bin/env node --input-type module

Would end up in the kernel exec call as:

#!/usr/bin/env node

That's the reason for which I proposed a different executable and not another argument for node. But I guess it can be a symlink if node can check through which name it's called indeed.

Quite a wide wide variety of env commands support a -S argument that works around this behavior (I'm not aware of any that don't support this):

#!/usr/bin/env -S node --input-type module

--input-type module support already exists & works in many places. Supporting it here would be a colossal boon & alleviate a lot of frustration. Currently if you try to use it (as above), it errors/dies:

Error [ERR_INPUT_TYPE_NOT_ALLOWED]: --input-type can only be used with string input via --eval, --print, or STDIN

Adding --input-type support might not be a total solution, but it ought to be supported, please. It makes logical sense as a user to be able to use the existing mechanism to specify what type of script a file is.

Te current status quo of being unable to write ESM scripts that fit in with everything else on the computer is so painful & ugly. A lot of the discssion in this thread is around exposing better loader APIs, & hacking out shims, but I think most users just want to be able to write scripts in ESM & have a Node.js that can run those. This sure seems like a direct & simple approach, from the user perspective.

[jirutka]

Quite a wide wide variety of env commands support a -S argument that works around this behavior (I'm not aware of any that don't support this):

busybox env does not (the default variant on Alpine Linux).

[GeoffreyBooth]

@GeoffreyBooth can you please tell the full story:

We are working on solving this feature request via #49432 and #49629.

[GeoffreyBooth]

Resolved by #49974.