Loader Configuration

[tubbo]

With chained loaders being considered as part of this effort, it might become pretty unruly for large apps to benefit from using loaders to help shore up some of the hairy and error-prone sides of the app.

A real-world example here is a TypeScript app that uses GraphQL, Pug, and SQL queries that all need to be loaded using readFile(). When building this app, special care must be taken for these files which will not be included in the build from TS since it has no awareness of their existence. But if we could import those files, this would no longer be a problem. Tools can statically analyze the imports and determine which files need to be included in the build artifact, and that's a huge win over not just the current strategy, but other languages as well.

This poses a problem, though. How are these loaders going to be maintained for all the various commands you need to run? Does every developer need to have a huge $NODE_OPTIONS defined to ensure the app loads correctly?

I think this could be solved by allowing developers the option to define what loaders are to be run within a package.json. This would allow someone to communicate how loading should work in code, which is transferable to anyone else working on the project.

In keeping with the convention of other JS tools, the configuration should probably be nested in an object of its own rather than be defined on the top level:

{
  "node": {
    "loaders": []
  }
}

This array would be anything importable: a module, file path, etc.

Just a suggestion, and I'm willing to help work this out. But this is very interesting to me as I think loaders in node would open the door to a lot of quality of life improvements, like more sophisticated package management, lint tooling, and all kinds of stuff.

[GeoffreyBooth]

Yes, it’s been on the list of use cases for a while to have some way of specifying loaders that didn’t involve the command line --loader flag. Earlier today I opened #97 to make this more visible by adding it to the roadmap in the readme.

The complication is that if we’re going to allow configuring Node itself via package.json, why stop at loaders? Perhaps all CLI flags should be available in there. So this is a broader conversation with the larger Node team that will have to happen, which is why we haven’t prioritized it. But now that we’re getting closer to the end of our top priority tasks, its time may be near. cc @aduh95

To your specific use case, though, the way I’ve generally seen it handled today is via the "scripts" field in package.json, where you could have something like "start": "node --loader foo.js app.js" and so on, and the particular flags the app is expecting are defined that way.

[targos]

Another thing that I'm not sure I have seen before is the question of how to configure the loaders themselves. Since a loader is just an imported module, there is no way to pass it configuration other than relying on the file system or environment variables.
Let's say for example that I make a TypeScript loader and I'd like the user to be able to specify the location of the tsconfig file to use.

[JakobJingleheimer]

Another thing that I'm not sure I have seen before is the question of how to configure the loaders themselves. Since a loader is just an imported module, there is no way to pass it configuration other than relying on the file system or environment variables.

Let's say for example that I make a TypeScript loader and I'd like the user to be able to specify the location of the tsconfig file to use.

@targos this is possible via CLI args. See https://github.com/JakobJingleheimer/demo-css-loader, specifically https://github.com/JakobJingleheimer/demo-css-loader/blob/6f068d5efc684ea3d4e5f957b030f6ef145e6d64/package.json#L7

[cspotcode]

This is another thing that might get better with nodejs/node#43408. A clear bootstrapping phase for config discovery, with access to information about the entrypoint. This bootstrapper can discover and pass configuration to loaders.

Using CLI args seems like a hack since those are meant for the entrypoint, not the loader.

A pain point today is that loaders aren't passed an isMain context for the entrypoint. If you want a single entrypoint-relative config, it gets messy. If you want many configs, relative to each script that gets loaded, then you don't care which one isMain. But that might be a perf hit.
EDIT linking to past discussion about isMain: nodejs/node#41711

Relative to cwd isn't great for shell scripting scenarios.

[cspotcode]

I realized after posting that I can explain more clearly why CLI args are not suitable for configuring loaders. "Seems like a hack" is unnecessarily vague.

If I do node --foo it will fail with an error: "bad option." As a CLI tool, node has decided that unrecognized flags should be rejected. This is sensible. Supposing that a tool or script is implemented in JS to be run on node, the author might take the same approach: reject unrecognized options to avoid mistakes. For example, given the typo --dryrn instead of --dryrun, the script will err on the side of caution. So if someone does node --loader loader-takes-flags.js ./tool.js --flag-for-loader --flag-for-tool, tool.js will rightly reject --flag-for-loader with an error akin to "bad option."

[GeoffreyBooth]

Loaders support top-level await. So a loader could await import() or await readFile() a config file as part of the loader’s initialization. Loader authors could implement however complicated a “config search” algorithm they want.

[cspotcode]

Yeah but node should probably be passing them a resolved entrypoint path instead of asking them to figure it out themselves (is it an entrypoint? REPL? Code piped to stdin?) or exposing a flag like --loader-configuration-blob or making guarantees around access to URL query params in --loader ts-node/esm?configuration=yaddayadda.

If node implements a "loaders" field in package.json, then probably the loader wants to know about which package.json files have asked it to be there, in case it wants to pull config from them.

[GeoffreyBooth]

Relevant to this, I was just reading https://aws.amazon.com/blogs/compute/node-js-18-x-runtime-now-available-in-aws-lambda/ and noticed this paragraph:

Experimental features in Node.js can be enabled/disabled via the NODE_OPTIONS environment variable. For example, to stop the experimental fetch API you can create a Lambda environment variable NODE_OPTIONS and set the value to --no-experimental-fetch.

So I guess now it is possible to use --loader in an AWS lambda?

[cspotcode]

From my experience, NODE_OPTIONS has been supported on AWS Lambda for years. Maybe others were referring to similar environments with different restrictions? Does Lambda@Edge have different rules? Were they talking about Heroku?

[merceyz]

This issue is listed under Milestone 2: Usability improvements but for parity with CommonJS it needs to be possible to register loaders at runtime since in CommonJS you can add support for, for example, TypeScript in the current process but there isn't something similar for ESM as far as I can tell.