Publish as modules (instead of compiling to cjs)

[hzoo]

Progress tracked at https://github.com/babel/babel/projects/16

Feature Request

Is your feature request related to a problem?

Would like to be able to consume babel packages directly as modules (as they are written), instead of just cjs. (In my case, may want to import Babel itself into something like snowpack/vite which tries to only support esm). This would be for anything trying to use babel for tooling like the repl/etc so not a common usecase.

Also good that we dogfood what other packages/libraries are doing, even though we aren't really a tool for the browser it would be good to attempt to understand the issues with doing all this anyway.

Describe the solution you'd like

⚠️ EDIT: See #11701 (comment)

Use package exports. When building we change our config to do modules: false in preset-env. Currently we do "main": "lib/index.js", for most packages.

Could output a whole folder like dist or module instead of lib or output .mjs in the same lib folder.

{
  "main": "./lib/main.js",
  "module": "./lib/module.mjs",
  "exports": {
    ".": {
      "import": "./lib/module.mjs",
      "default": "./lib/main.js"
    },
    "./other": {
      "import": "./lib/other.mjs",
      "default": "./lib/other.js"
    }
  }
}

Describe alternatives you've considered.

Don't think there's an alternative other than consuming the source via github which is difficult because this is a monorepo. We only publish cjs.

[nicolo-ribaudo]

I'm working on this, but it won't happen sooner than Babel 8.

[ak06121991]

@nicolo-ribaudo Have you already started doing the changes or is this still up for grab ?

[ExE-Boss]

@ak06121991 Well, there’s #12632, which does this for the @babel/runtime packages.

[nicolo-ribaudo]

@ak06121991 I'd prefer to work myself on this issue because it deeply affects our build, release and testing process.

[nicolo-ribaudo]

Update

We are thinking about releasing Babel 8 as an ESM-only package, for the following reasons:

• starting from Babel 8, every Node.js version that we'll support will have native support for ES modules;
• it avoids the pain of managing dual-node packages, where basically everything would still be CJS + manually maintained ESM entrypoints;
• maintainers of popular packages on npm are starting to ship ESM-only versions of their libraries. For example, Sindre Sorhus is going to re-release all of its packages as ESM-only in 2021 (https://blog.sindresorhus.com/get-ready-for-esm-aa53530b3f77). Using CJS means that we are stuck on old dependency versions, without the ability to update them.
• ESM make it impossible to synchronously require() and babel.transformSync in CJS. However, it's already not always possible to safely do it (a plugin/preset or a config could contain an async step, forcing consumers to use babel.transformAsync) (*).

We currently got feedback from @devongovett that Parcel runs Babel in a Node.js vm context to track what files Babel requires internally. If we move to ESM, Parcel would probably be forced to use something like @vercel/nft to get Babel's internal dependencies AOT rather than at runtime.

(*) The webpack, rollup, browserify, gulp, and grunt integrations already use Babel asynchronously. We still have to figure out how it will work for ESLint. Jest issue: jestjs/jest#11081

[mischnic]

We currently got feedback from @devongovett that Parcel runs Babel in a Node.js vm context to track what files Babel requires internally.

(Parcel currently patches require to achieve that. With ESM, we'd either have to use vm (if ESM vm support doesn't get removed while it's still marked experimental) or fallback to some kind of static analysis (like the one you linked or https://github.com/guybedford/es-module-lexer))

[devongovett]

As a related topic, has babel considered bundling its internal dependencies? For example, @babel/core is currently shipped as many individual files. In fact, requiring babel and commonly used plugins results in loading over 400 unique modules. Bundling them into a single file per package would have the following benefits:

• Much faster to load. require (and likely import) are quite slow in Node, especially the resolution algorithm and FS operations. Bundling into a single file would alleviate this.
• Fewer files also helps out tools that perform caching (like Parcel), as it would reduce the number of files we need to track to invalidate our builds when babel itself changes.
• Prevents consumers from using unsupported internal APIs directly. This may affect some existing integrations, but should result in fewer broken integrations in the future as babel's internals evolve. It will give you more freedom to change things without doing a major version, and force you to provide officially supported APIs for more use cases, which I think is a good thing. A major version is the perfect time to do this.

[nicolo-ribaudo]

Yes! We already do that for some packages, but not for all of them due to backward compatibility.

From the performance side, we even had to introduce a "lazy requires" option in the cjs transform for ourselves, otherwise requiring @babel/core was too slow.

PS. For the "people cannot access your internal files", that's what "exports" in package.json is for 😛

[devongovett]

That's an interesting point about lazy requires. I guess that's not possible with ESM, unless everything is made async. But perhaps bundling would help alleviate it anyway.

[coreyfarrell]

@nicolo-ribaudo what does this mean for the future of @babel/register and other modules which use babel to perform live transformation of code that is being loaded via require?

[nicolo-ribaudo]

I'm still exploring solutions, but in the worst case when using @babel/register you would have to do

import("@babel/register").then(() => require("./entrypoint.js"));

instead of

require("@babel/register");
require("./entrypoint.js")

I'm also exploring moving @babel/register compilation to a Worker, which is similar to what has been proposed to Node.js to align the CJS and ESM loader hooks.

[coreyfarrell]

@nicolo-ribaudo thanks for your reply. The situation I'm working with is more of:

export NODE_OPTIONS='--require=@babel/register'
node test.js

The way nyc works is to injects a --require option into NODE_OPTIONS. Older versions of nyc would replace the entrypoint.js with a wrapper that initialized nyc but this was brittle (especially on Windows).

[nicolo-ribaudo]

Thanks for reminding me about that use case. FYI, I'm experimenting with compiling in workers at https://github.com/nicolo-ribaudo/babel-synchronized. For @babel/register, this also fixes problems caused by accidentally compiling (or not compiling) our own dependencies.

[nicolo-ribaudo]

@coreyfarrell I drafted a solution at #12814. We could re-use it also for babel-eslint, because ESLint doesn't support async parsers yet.

[overlookmotel]

@nicolo-ribaudo A few random questions picking up on your previous comments:

1. In comment above you said "ESM make it impossible to ... babel.transformSync in CJS". Why is that? I understand that you couldn't synchronously import an ESM @babel/core, but once you have it imported, why does ESM preclude running transformSync synchronously? Is this due to the "lazy requires"?

2. You said an approach similar to babel-synchronized "has been proposed to Node.js to align the CJS and ESM loader hooks". I would like to see this happen. Is this in an issue somewhere? I couldn't find any mention in module: ESM loaders next steps nodejs/node#36396. Or does "has been proposed" really mean "I have proposed"? :)

3. Any idea yet if the overhead of passing large blocks of code/ASTs back and forth to a worker affects performance? In particular I'm wondering if serializing+deserializing ASTs would be slow.

By the way, 1st question isn't a gripe, I'm just interested. Personally I think going ESM is all good, especially if there's something like babel-synchronized as a fallback for people stuck in require-land, and it's not terribly slow.

[devongovett]

Any idea yet if the overhead of passing large blocks of code/ASTs back and forth to a worker affects performance? In particular I'm wondering if serializing+deserializing ASTs would be slow.

Yes it is very slow. Parcel has a lot of experience with this. We found it slower in some cases than generating code + source map and re-parsing. 😬

[ExE-Boss]

This comment is probably no longer accurate, given that ES Modules will be the default, and CommonJS will be the exception:

babel/babel.config.js

Lines 30 to 32 in 792672e

	// The vast majority of our src files are modules, but we use
	// unambiguous to keep things simple until we get around to renaming
	// the modules to be more easily distinguished from CommonJS

As such, it should be the CommonJS files that get renamed to use the .cjs file extension.

[overlookmotel]

@devongovett Thanks for the info. Disappointing.

Was that using JSON as serialization format? I wonder if an alternative (streaming?) serialization format would make any difference?

And, when passing back and forth code only (not ASTs), are you saying that the performance isn't so bad?

[nicolo-ribaudo]

@overlookmotel

1. In comment above you said "ESM make it impossible to ... babel.transformSync in CJS". Why is that? I understand that you couldn't synchronously import an ESM @babel/core, but once you have it imported, why does ESM preclude running transformSync synchronously? Is this due to the "lazy requires"?

It will be impossible to both import synchronously and .transformSync, and you'll need at least one asynchronous step.

This will be possible:

const babel = await import("@babel/core");
const { code } = babel.transformSync("foo;");

However, if

1. Your configuration file is an ES module
2. You are asking Babel to load an ESM plugin/preset
   you cannot call babel.transformSync (this is already true in Babel 7).

i.e. this won't work because @babel/preset-env will be an ES module:

const babel = await import("@babel/core");
const { code } = babel.transformSync("foo;", {
  presets: ["@babel/preset-env"]
});

but you'll have to do this:

const babel = await import("@babel/core");
const env = await import("@babel/preset-env");
const { code } = babel.transformSync("foo;", {
  presets: [env]
});

Note that I said "it will be impossible to import and transform synchronously" and not just "it will be impossible to import synchronously" because we could easily expose a CJS entry point that only contains the *Async methods (but probably it's not needed).

2. You said an approach similar to babel-synchronized "has been proposed to Node.js to align the CJS and ESM loader hooks". I would like to see this happen. Is this in an issue somewhere? I couldn't find any mention in module: ESM loaders next steps nodejs/node#36396. Or does "has been proposed" really mean "I have proposed"? :)

There is a draft PR to move ESM loaders to a worker: nodejs/node#31229.
I can't find much anymore about CJS, except for this comment from a Node.js maintainer and this tweet from another Node.js maintainer.

3. Any idea yet if the overhead of passing large blocks of code/ASTs back and forth to a worker affects performance? In particular I'm wondering if serializing+deserializing ASTs would be slow.

Yes, it affects performance a lot. Using Babel in a worker is only an escape hatch for when it's not possible to do otherwise.

For example, when compiling this file three times with babel.transformAsync and three with babelSynchronized.transformSync I get these results (in ms):

babel.transformFileAync (first run) 4.706201999913901
babel.transformFileAync (second run) 0.19897600007243454
babel.transformFileAync (third run) 0.17150900000706315
sync.transformFileSync (first run) 325.1950079998933     // NOTE: This is a one-time cost to spin up the worker
sync.transformFileSync (second run) 48.88951799995266
sync.transformFileSync (third run) 39.04019799991511

Note that you do not need to manually serialize+deserialize ASTs (for example using JSON), because workers use the Structured Clone algorithm to pass data back and forth.

---

@ExE-Boss I'm opening different PRs to gradually migrate; that one is one of my next steps!
I'll setup a project so that it's easier to keep track of what I'm doing.

[ExE-Boss]

but you'll have to do this:

const babel = await import("@babel/core");
const env = await import("@babel/preset-env");
const { code } = babel.transformSync("foo;", {
  presets: [env]
});

That should be using Promise.all to start both imports in parallel, which more closely approximates how import statements work:

const {
	0: babel,
	1: env,
} = await Promise.all([
	import("@babel/core"),
	import("@babel/preset-env"),
]);

const { code } = babel.transformSync("foo;", {
	presets: [env],
});

[overlookmotel]

Sorry for slow reply.

One question: If you go ESM, how will you replace the "lazy requires" optimization? You'd need to either (1) abandon the laziness and switch to static import statements or (2) use lazy import() instead. If the latter, every function which relies on a lazy import becomes async, and that will cascade through the code base. I imagine that'd be a pain in terms of readability, and I wonder if there'd be a performance impact of introducing a microtick every function call for functions which are on a hot path.

Or does the non-blocking/parallel nature of the ESM loader remove the performance problem which the lazy requires were introduced to fix in the first place?

[nicolo-ribaudo]

If startup performance will be a problem again, we can:

1. Use import() to load @babel/core in @babel/cli, @babel/node, babel-loader, @rollup/plugin-babel
2. Bundle @babel/core