Feedback: ES Package Case Study

[evanplaice]

Introduction

I joined this group during it's inception for one reason; greasing the path to ship ES packages. The basis of that goal is rooted in exploring concrete implementations, tooling, and standards.

Essentially...

• what is available now?
• not what may be.
• not what could be.
• not what will eventually be.

Here are my findings of building a package from a ESM-first (ie "type": "module") perspective

The Package

I wanted to create a library that masquerades as a utility library. It would be published to NPM immediately and updated as progress is made in this group. It should work as a good example but presented it in a way that it isn't taken seriously; so -- in its experimental state -- it doesn't get adopted into an actual production codebase.

What I came up with is Absurdum

A collection of Lodash-esque operators implemented using only reduce and only ES modules.

Evolution

Modules support is a moving target so this library has been updated to keep up-to-date with each milestone.

Phase 1: No Module Support

I loathe C++ so compiling the Node.js source was off the table. Instead, I tapped the same old pattern that the FrontEnd ecosystem has been stuck with for the past 5 years. Transpiling.

Implementation:

• the source is ESM
• ESM is transpiled down to CJS using Babel
• testing is provided by Tape.js
• tests are implemented using CJS
• chokidir is used to watch for changes and re-transpile the source

Observations:

This was by-far the worst DX. As in, an order-of-magnitude more painful than anything used since. Transpiling presents a higher barrier of entry for contributors, transpiled code is harder to debug, setup is painful, shipped code is bloated, browser compat requires a 'kitchen sink' build, and context switching between CJS <-> ESM is not fun.

Phase 2: @std/esm

A few months after this group formed @jdalton shipped the @std/esm package. Which was nothing short of awesome. For the first time, ESM could be used w/o transpilation.

Benefits:

• Tape.js 'just works' w/ the module
• tests could now be written in ESM

Drawbacks:

• the extra layer of abstraction still exists (but is hidden)
• requires an additional dependency

Observations:

Big step in the 'right direction' in terms of DX. Lowered barrier-of-entry is a huge win. Unfortunately, it's still not actually standard JS.

Phase 3: --experimental-modules

This phase includes 2 significant changes. Rudimentary ESM support shipped in both Node and Chrome.

Benefits:

• finally, no extra layer of abstraction
• no additional dependencies required
• debugging ESM (ie via VSCode) is fantastic
• no more 'kitchen sink' build
• ESM 'just works' in browsers

Drawbacks:

• testing/tooling unexpectedly breaks
• browsers only work w/ relative imports

Observations:

DX is beautiful. I can't describe how awesome it was to finally have immediate feedback and full debugging support w/o a complicated build stack. After some research, it appears that Tape.js tests work fine but the test runner eats the experimental flag. I managed a workaround by manually implementing a glob-matching runner using linux commands.

Phase 4: Dev Velocity++

Changes:

• tests were co-located w/ the source (ie no more test/ dir)
• lots more operators added
• process improved to include documentation
• API simplified following a V8 release
• official support means I can automate testing/publishing via CI/CD

DX is not only nice but fast. Whatever effort I wasted previously trying to work around a complicated build process could be focused instead on adding actual value to the project. This is the win I've been anticipating for years.

Phase 5: Approaching Prod-Ready

Frustrated by the continual delay of ESM being unflagged, I decided that -- if I'm going to present this -- I should at least make it look like a prod-ready package.

Changes:

• add 'compatibility' bundles
• add JSDoc strings w/ types
• automate documentation creation (ie from JSDoc)
• add .d.ts (ie typings) support¹
• transition CI/CD from CircleCI to GH Actions²

¹Still experimental, can only be done w/ the latest Typescript RC release
²Not necessary, but I managed pick up Beta access and was itching to try it

Bundling:

Since the source is 100% browser-compatible, node-specific module resolution isn't used. While great for DX, the package doesn't play nice with older patterns (ie pre-esm node and bundling). To address this, I create and ship 2 compatibility bundles using Rollup.js. An ESM bundle mapped to pkg.module, and a CJS bundle that can be deep imported in old versions of Node.

JSDoc:

Turns out JSDoc is the oft-overlooked 'secret sauce' of vanilla JS. JSDoc strings not only serve as inline documentation but with the help of tooling can be used for much more.

Documentation:

The doc generation step kind of sucks, the best option I found was DocDown (ie used by Lodash). But I had to modify it to support one-doc-per-module documentation creation. The JS ecosystem has been bundle-focused for so long that even a lot of the tooling is still stuck on that pattern.

Type Checking:

VSCode supports typechecking via JSDoc types out-of-the-box. Nothing more to say, this is incredible.

Typings:

Supposedly not required. I can't say, in the past I've only used Typescript for typed JS. I figure, if I'm going to ship a typed JS it should follow the usual TS 'best practices' for packaging.

Automation:

After months of practice w/ CI/CD I have a well-defined set of workflows. Every push gets verified (ie test/lint/types), every tagged push gets published (ie verify/build/bump/publish).

Observations:

This phase transcends just DX. Typed vanilla JS is incredible. Automatic documentation generation is great but there's a ton of room for improvement in this space. Automating 'all the things' is such a massive time saver, I loathe to think how much time on non-value-add processes. No lie, if I'm 'in the zone' I could easily ship a dozen-or-more releases in a single day.

Phase 6: Unflag ESM (Current)

Not much left

Changes:

• remove all --experimental-modules flags
• use tape-es in place of janky test script
• update node version in CI/CD

Testing:

Contrary to my initial assumptions, the Tape.js test runner does not 'just work' with ESM. As a result I created tape-es to replace the sketchy shell-based test runner I've been using.

The test runner is simple, it glob matches to locate the test files and spawns subprocesses to run the tests concurrently with a default max of 10 threads. This runs the tests 3x faster than the previous strategy.

In theory, if the subprocesses run in a separate context then this runner should be capable of running both CJS and ESM. The one downside is the '-r' flag used to pre-import a dependency will never work with this.

CI/CD:

Remarkably, bumping the node version just worked. Now that the tests run 3x faster, CI/CD is fast; like, really fast.

Debug:

For whatever reason VSCode doesn't respect the Node version specified by nvm. This could be user error. Either way, I'll leave the --experimental-modules flag in the debug config for now.

Observations:

ESM as a universal module format works beautifully in both browsers and Node. I'm really looking forward to the day when jank workarounds are the exception. ESM landing unflagged in LTS will be key.

This message ExperimentalWarning: The ESM module loader is experimental. really muddies the output. I can't wait until it's removed.

On an unrelated note. Is tape-es the first pure ESM-based CLI?

---

Appendix A - Entry Points

I glossed over this b/c it's hard enough to work on the 'bleeding edge' without trying to hit a constantly moving target. While not optimal, here's what I use.

• pkg.main - points to the public API (ie index.js)
• pkg.module - points to the ESM build
• legacy - CJS requires a deep import

It's not that I dislike CJS, I just like ESM imports/exports so much better. By leveraging the capabilities of ESM it's finally possible to build an actual public API.

By comparison, deep imports are really bad. They unnecessarily expose implementation specifics of the package to users. As general rule, if users can see it some will inevitably depend on it. This makes major refacors much more painful than they need to be.

Ideally, I would prefer that (non-contributing) users will never have to open the 'src' directory.

Appendix B - Bundling

Fact, converting ESM->CJS is easier than CJS->ESM. To put it simply, CJS is a 'lesser' format. Meaning, it has fewer features/capabilities than ESM.

The transition path discussed in this group has been backward all along. Not only is the CJS produced by down-conversion less bloated than the opposite, it's also tree-shake-friendly for consumption by bundling tools.

Yes, doing a full refactor to ESM on a large+ scale project is going to be painful (can this be automated?). The silver lining is, once it's done providing backward compat -- CJS, or even ES5 -- build requires very little additional effort.

Appendix C - Dependencies

What about dependencies? This package doesn't include any but -- long story short -- they 'just work'¹. Relative importing from node_modules sucks but it's only a minor inconvenience.

*¹ I know this from other ES packages I've built for the FrontEnd like wc-markdown

Appendix D - Tooling

Tools that depend heavily on Node/CJS-specific patterns are going to suffer. I have already addressed this in ESLint but that is only a fix for side-loading CJS across package boundaries. Tools that rely on 'magic globals' for convenience are going to transition to ESM.

Also, take this with a grain of salt based on very limited experience. IMO, there's no way to accurately judge the impact ESM will have on the existing tooling ecosystem until support is rolled out at scale.

Appendix E - Obsolete Module Formats (ie IIFE/AMD/UMD)

Unlike CJS -- which integrates relatively well w/ ESM -- older formats really do not. ESM runs in strict mode by default. So, all the packages that bind to globals and include conditional require statements will break.

Speaking from experience, finding and patching these issues is a major PITA. Getting maintainers to merge fixes on these really old projects is nearly impossible.

Finding viable replacements for these really old packages will be a necessary requirement of building an ES package. If ESM achieves ubiquitous adoption, it will likely obsolete a not-insignificant chunk of the package ecosystem.

---

This should go without saying but this write up is nothing more than a snapshot of 'what is possible' considering the current state of standards and ES module support in both Node and browsers.

What it is not is a qualitative judgement on any debates/decisions made by this group. I'm here strictly as an 'observer'. Opinions and observations stated here are just that, opinions and observations.

[ljharb]

Is there a way for pre-ESM node to require your package? In post-ESM node, what happens if one dep requires you and another imports you?

[evanplaice]

I mentioned this in Appendix A

Is there a way for pre-ESM node to require your package?

For pre-ESM users there is a CJS compatibilty bundle that can be used via deep require

const test = require('tape');
const arrays = require('absurdum/dist/absurdum.cjs').arrays;

test('arrays.chunk(array) - should return a chunk for each item in the array', t => {
  const expect = [[1], [2], [3], [4]];
  const result = arrays.chunk([1, 2, 3, 4]);

  t.equal(Object.prototype.toString.call(result), '[object Array]', 'return type');
  t.equal(result.length, 4, 'output length');
  t.deepEqual(result, expect, 'output value');

  t.end();
});

In post-ESM node, what happens if one dep requires you and another imports you?

I'd assume the 2 caches contain 2 different copies. I don't attempt to provide a solution for this because there is none.

• if Node marries its ESM implementation to CJS, it breaks spec screwing FE devs
• if CJS was compatible w/ browsers, it would have become the standard

---

There is one solution to universal package support. Follow the spec and make everything ESM. Maybe, at some point in the future that'll be an option.

[ljharb]

Thanks, i must have missed it.

(ftr, CJS is quite compatible with browsers; that’s got nothing to do with why JS Modules were different)

[evanplaice]

It's all good. I tried my best to cover all the bases.

I know CJS->ESM is super common but checkout the bundle created by ESM->CJS. The conversion is nearly 1:1 with practically no overhead.

[GeoffreyBooth]

I’d assume the 2 caches contain 2 different copies. I don’t attempt to provide a solution for this because there is none.

For what it’s worth, loading two copies of this package isn’t really an issue (besides the performance hit of double loading) since the package is stateless like Underscore/Lodash. Dependents need to not treat it as a singleton, e.g. attaching more functions to it, but that’s just a good practice in general.

Perhaps the way Express’ middleware gets loaded should be a model for best practices for plugins?

const express = require('express');
const app = express();
const cookieParser = require('cookie-parser');
app.use(cookieParser());

[evanplaice]

Derp. I thought package duplication was the issue. Now, I see how that could be problematic.

From a lib/tooling dev perspective I can think of 2 strategies:

1. Eliminate coupling across package boundaries by dependency injecting the plug-in

Defining something like a function signature for middleware the passing a middleware function in by value would work.

2. Leverage the ES-specific nature of modules.

With ES exports you can define a public API via a single-file entry-point (ie how I use index.js). This does nothing more than remap internal modules into user-friendly API.

To leverage this, make each module use named exports specifically, then define all default exports to throw.

As long as CJS can never require named exports this should effectively block CJS from access to ES modules. With the added benefit that the stack trace will show what failed to call the module.

This could be used to block CJS from loading ES modules via import().

Aside: If this works it can also be used as a strategy to block deep path imports within a package.

---

As for a general purpose solution, I don't know.

On a positive note, large scale breaking updates aren't totally uncharacteristic in JS. The JS community is pretty resilient to change when it's necessary.

[jkrems]

Thanks a lot for working on this! These kinds of experiments are super valuable. :)

[evanplaice]

I ran an experiment. It turns out that it is -- in fact -- possible to throw on a default ESM exports.

export default (() => { throw Error('Default export is not supported, if you are trying to require this module, use the CommonJS bundle instead'); })();

IIFEs are completely valid values for exports. This can be used to block-and-inform users from mistakenly using import() to load an ES module in their CommonJS based package.

[ljharb]

@evanplaice default exports are values, not bindings or lazily evaluated code; that should block imports of any kind from the module since it should throw upon module evaluation.

[evanplaice]

Oops, should've tested it w/ non-default exports. Thanks for the heads up.

[giltayar]

@evanplaice FYI, regarding Mocha: there's light at the end of the tunnel! I submitted a PR that adds Node ESM support to Mocha. It works very nicely: just create ESM test files and you can use ESM to your hearts content: mochajs/mocha#4038. Hopefully, now that ESM is going to be unflagged, it will be reviewed and accepted.

You specified a problem with Mocha globals (describe et al). I didn't see any problems there, could you perhaps elaborate?

[evanplaice]

@giltayar I removed all mentions of issues w/ Mocha since this is apparently no longer an issue. Nice work, I look forward to using it in the future.

I mentioned magic globals b/c I was under the impression, that was the issue /w ESM compat. From what I gathered when I last looked into it, it looks like Mocha's test runner glob matches files, loads the test runner, then requires the test files into the context of the runner so they have access to the globals.

I didn't attempt an actual fix so my understanding is superficial and incomplete at best.

[jkrems]

I mentioned magic globals b/c I was under the impression, that was the issue /w ESM compat.

Since this comes up from time to time: mocha is using real globals so there's no issues in ESM. Actual globals work just like they always did.

The "magic globals" that are sometimes mentioned in the context of ESM aren't globals at all. They are things like require, __dirname, etc. and are local identifiers in CommonJS that are "magically" present. And since people usually don't see the function wrapper where they are declared, many think of them as "globals".

[giltayar]

@evanplaice FYI, the problem is the problem most tools that need to deal with running ESM will probably have: require is a synchronous function, and import is asynchronous. Which should be pretty simple, except that the call stack in Mocha leading up to the require is synchronous too, as is the public API.

I had to change the whole call stack to async, including the API, which is probably why it will be a SEMVER_MAJOR change in Mocha (hopefully in v7).

[evanplaice]

Note: Phase 6 has been updated with details about updating the package to unflagged Node

[MylesBorins]

Want to throw my repo in here as well

https://github.com/MylesBorins/node-osc/tree/next

Features

• module completely rewritten using ESM
• Fully tested using tap
  • coverage is not working though 😢
  • coverage works with c8!
• uses rollup to generate CJS both for tests + publishing
• uses experimental conditional exports for CJS + ESM entry points
• uses self-reference for examples + tests

[GeoffreyBooth]

Another one: https://gitlab.com/ericlathrop/silly-food-generator

[giltayar]

@MylesBorins - it seems that self-reference of modules made it in? I couldn't find any mention of it in the documentation. (I looked in the EcmaScript page and in the module page)

[evanplaice]

@giltayar It's not documented. I brought it up at the last meeting b/c its inclusion in the latest batch of ESM festures was mentioned in the previous meeting.

Long story short, it's not documented anywhere. I read through all the issues again, there's no obvious decision about the functionality. I read the spec, which cuts off at a TBD.

I had to read the source to figure out that there is no sigil. It works by matching the package name. AFAIK, if pkg.exports is defined it should rely on the exports dedined there. But, I haven't actually tried using conditional exports in combination with self referencing specifiers yet.

[ljharb]

I have; that’s how they work. “exports” only implies for otherwise-relative imports when using the package’s own name as a bare specifier.

That it’s undocumented seems like an oversight; a PR or issue about that would likely be appreciated.

[evanplaice]

Just to reiterate my previous disclaimer. I'm strictly here as an Observer.

My role is to provide a first-hand user perspective on developing libraries and CLIs using ESM packages (ie where type:module is specified). Nothing more.

[giltayar]

@ljharb - I would gladly contribute a PR for this, but it's not exactly clear where to put it, as this is a feature of both CJS and ESM. I could duplicate this information, but where would I put the CJS text?

Also—this is unflagged, right? I have a talk about ESM in Node.js coming up, and I'd like to know whether I can talk about this or not.

[MylesBorins]

I'd put it in the es modules docs for now, that's where we have exports documented. I agree with you that we should likely include something in the "modules" docs, but that can likely be a follow up that covers both features (maybe just a link)

…

On Mon, Feb 3, 2020, 6:09 AM Gil Tayar ***@***.***> wrote: @ljharb <https://github.com/ljharb> - I would gladly contribute a PR for this, but it's not exactly clear where to put it, as this is a feature of both CJS and ESM. I could duplicate this information, but where would I put the CJS text? Also—this is unflagged, right? I have a talk about ESM in Node.js coming up, and I'd like to know whether I can talk about this or not. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#415?email_source=notifications&email_token=AADZYV2HNZV6JUILCRB26C3RA73NHA5CNFSM4JHCNCTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKTNUYQ#issuecomment-581360226>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADZYV4U6X37ITEE7K3Z6ZTRA73NHANCNFSM4JHCNCTA> .

[giltayar]

@ljharb @MylesBorins - will try to create a PR this week sometime. Besides scouring the issues, is there any place this is documented? I've been following along, but it's sometimes hard to figure out what finally went in.

[MylesBorins]

You can use the name of the package and you will have the exact same interface as external consumer (e.g. package.exports is respected) Afaik that is the entire scope of the feature rn

…

On Mon, Feb 3, 2020, 6:15 AM Gil Tayar ***@***.***> wrote: @ljharb <https://github.com/ljharb> @MylesBorins <https://github.com/MylesBorins> - will try to create a PR this week sometime. Besides scouring the issues, is there any place this is documented? I've been following along, but it's sometimes hard to figure out what finally went in. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#415?email_source=notifications&email_token=AADZYV4TJH6QJWEAKCFIULDRA74G5A5CNFSM4JHCNCTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKTOJFI#issuecomment-581362837>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADZYVY7IK4T3DIT7F4MT73RA74G5ANCNFSM4JHCNCTA> .

[giltayar]

Just to be clear: so that means that it's not an alias for the package root, but rather a way to require/import yourself? If so, then gotcha!

[MylesBorins]

One additional note. If you do not have exports defined in the package json the feature will not work.

…

On Mon, Feb 3, 2020, 8:48 AM Gil Tayar ***@***.***> wrote: Just to be clear: so that means that it's not an alias for the package root, but rather a way to require/import yourself? If so, then gotcha! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#415?email_source=notifications&email_token=AADZYVZLBQL23ULBQZNTMB3RBAOCXA5CNFSM4JHCNCTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKT4WHA#issuecomment-581421852>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADZYV4YJSRZYR2Z7PCO6X3RBAOCXANCNFSM4JHCNCTA> .