Meta: Native support for ES Modules

[SimenB]

EDIT: quick guide for getting started: https://jestjs.io/docs/ecmascript-modules

ESM support will be unflagged in a future release of Node 12 (maybe not before April nodejs/node#29866 (comment)) and it is already unflagged in Node 13.2, so I think it's time to evaluate how we can add native support in Jest. I'll try to list which features Jest currently provides that are impacted by ESM support, and how we can solve/investigate them.

There is issue #4842, but I think that's more of a discussion issue, while this issue will be geared towards actually implementing support and more suitable to track for those who just want to get the current implementation status. Any comments added to this issue not related to how we can implement support for the below enumerated features will be marked as spam - please direct any workarounds/discussions to separate issues. Also feel free to tell us if anything related to ESM features is missing from the list!

Please note that Jest will use the vm API (https://nodejs.org/api/vm.html) and as of writing (node v13.6 v16.10) the ESM parts of this API is still flagged (--experimental-vm-modules). So saying ESM is unflagged is a bit of a misnomer at the moment. But I think we should start experimenting and potentially provide feedback to the Modules WG.

EDIT: Tracking issue for stabilization in Node: nodejs/node#37648

Lastly, I'm writing this issue mostly for people who will implement support, so it'll be somewhat low-level and specific to how Jest works. For people who just want to know whether support has landed or not, I recommend using GH's wonderful "custom notification" and only subscribe to notifications on closing/reopening.

---

• Running the module in the correct context

We achieve sandboxes by running a script within a given vm.Context (either provided by JSDOM or node core APIs). We need to do the same for ESM, but we'll need access to the context during construction of the module, not just when executing the module. I've opened up #9428 which adds the necessary APIs to JestEnvironment.

• Globals

expect, test, beforeEach etc will still be added as globals, nothing should change here. jasmine global will also still be here.

• jest "global" property

This is not really a global - it's injected into the module scope. Since the module scope is gone in ESM, we need to move it somewhere. Adding it to import.meta seems natural - there's an option called initializeImportMeta which we can use.

EDIT: Solution here is to fetch it via import {jest} from '@jest/globals'. We might still add it via import.meta in the future, but this should be enough for now.

• jest.(do|un)mock

Since ESM has different "stages" when evaluating a module, jest.mock will not work for static imports. It can work for dynamic imports though, so I think we just have to be clear in the docs about what it supports and what it doesn't.

jest.mock calls are hoisted, but that doesn't help in ESM. We might consider transforming import 'thing' to import('thing') which should allow hoisting to work, but then it's async. Using top-level await is probably a necessity for such an approach. I also think it's invasive enough to warrant a separate option. Something to discuss - we don't need to support everything jest.mock can for for an initial release.

PR: #10976

• jest.requireActual

Not sure if how it should behave in ESM. Should we provide a jest.importActual and let requireActual evaluate in CJS always?

• import.meta

Node has url as its only property (for now, at least). We need to make sure it's populated in Jest as well. We provide identifier instead of filename when constructing the module so I don't think it'll happen automatically, but url is essentially filename passed though pathToFileURL.

There's also an open PR for import.meta.resolve: nodejs/node#31032

• import thing from 'thing'

This should actually be fairly straightforward, we just need to implement a linker where we can also transform the source before returning it, meaning we don't need the loader API (which doesn't exist yet). This allows us to return mocks as well (albeit they'll have to come from a __mocks__ directory).

• import('thing')

Essentially the same as above, but passed as importModuleDynamically when constructing the module. Will also support jest.mock, jest.resetModules etc more cleanly, so likely to be used quite a bit.

This can also be done for vm.Script via the same option.

• Handling errors during evaluation

Right now it's a runtime error (e.g. module not found), but that's not necessarily true with ESM. Does it matter for us? We should verify errors still look nice.

• module.createRequire

We need to deal with this for people wanting to use CJS from ESM. I've opened up #9426 to track this separately as implementing it is not really related to ESM support.

EDIT: Implemented in #9469

• module.syncBuiltinESMExports

https://nodejs.org/api/modules.html#modules_module_syncbuiltinesmexports. Do we care about it, or is just making it a no-op enough? Not sure what the use case in Jest would be. Messing with the builtins is already breaking the sandbox and I don't think this should matter.

EDIT: #9469 made this into a no-op. I think that's fine?

• Detect if a file is supposed to be ESM or CJS mode

Inspecting type field in a module's package.json seems reasonable: https://nodejs.org/api/esm.html#esm_enabling. Should we also have our own config flag? Also needs to respect file endings.

nodejs/node#49446

• moduleNameMapper

Not sure if this impacts anything. I think not since we'll be linking the modules together ourselves. Needs investigation, though.

EDIT: This is all resolution logic, which we control. So no changes here.

• jest.config.mjs

Through #9291 we support jest.config.cjs - do we need to do anything special for .mjs? Probably use import('path/to/configFile.mjs') which means it'll have to be async. Is this an issue? Might be worth making config resolution async in Jest 25 so it's not a blocker for incremental support of ESM in Jest 25.

EDIT: #9431

• Package Exports

Node supports package exports, which sorta maps to Jest's moduleNameMapper, but also provides encapsulation features. Hopefully resolve will implement this, but if they do not we'll need to do something. Might be enough to use the pathFilter option? Unsure.

EDIT: #9771

• JSON/WASM module

https://nodejs.org/api/esm.html#esm_experimental_json_modules. Do we need to care? Probably, especially for json. It's trivial for us to support import thing from './package.json' since we control the linking phase, but we probably shouldn't do it by default as it'll differ from default node. Should we force people to define a transform for it?

WASM: #13505

• Code coverage

Does it matter? I don't think it's affected as we can still transform the source with babel (maybe it'll be confused by import statements, probably not) and V8 coverage definitely shouldn't care. We should verify though.

• Async code resolution

This is absolutely no blocker as sync resolution will work just fine. But we can use async resolution now, which is great. I wonder if we should look into just using the resolve module off of npm again, as it already supports async. See #9505.

• Async code transformation

Similar to above, not blocking, but would be nice to support it. Might make @jest/transformer more usable in other environments as well. See #9504.

EDIT: #9889 & #11191

• Bad performance when accessing globals

Due to #5163 we have the extraGlobals option as a workaround - that workaround is no longer viable in ESM. I've opened up and issue with node here: nodejs/node#31658

• Import assertions

https://nodejs.org/api/esm.html#import-assertions

[SimenB]

I've landed very basic support with #9772. I've only tested the simplest cases, and there are many known limitations (most notably no jest object support and broken semantics when mixing CJS and ESM), but at least it's something. It'll go out in the next release of Jest (hopefully soon, only blocked by #9806)

[SimenB]

25.4.0 has been released with the first pieces of support. In addition to #9772 mentioned above, I've also included #9842. In theory mixing CJS and ESM should work correctly now (🤞).

The one main missing feature is supporting the jest object. I haven't decided if we should stick it to import.meta or require people to import it through import {jest} from '@jest/globals'. Feedback appreciated!

I haven't written docs for this yet, but to activate it you need to do 3 things

1. make sure you don't run transform away import statements (set transform: {} in config or otherwise ensure babel doesn't transform the file to CJS, such as avoiding the modules option to preset-env)
2. Run node@^12.16.0 || >=13.2.0 with --experimental-vm-modules flag
3. Run your test with jest-environment-node or jest-environment-jsdom-sixteen

Please try it out and provide feedback! If reporting bugs, it'd be wonderful if you can also include how running the same code (minus any test specific code) runs in Node. I've read https://nodejs.org/api/esm.html a lot over the last few weeks, but I've probably missed something.

[just-boris]

The one main missing feature is supporting the jest object. I haven't decided if we should stick it to import.meta or require people to import it through import {jest} from '@jest/globals'.

For the typescript use-case it is better to have an explicit import.

[SimenB]

Yup, I've added (and the temporarily reverted) a @jest/globals package that supports this, so that will be available regardless. I'm wondering if it makes sense to also expose it on import.meta. Currently leaning towards not doing so, mainly since it's easier to add than remove later (and I'm personally no fan of globals)

[IlCallo]

+1 for the explicit import, it's a bit more verbose but simpler to understand

[ziedHamdi]

I'm stuck here as if I don't use transform: I get the error: Must use import to load ES Module (So I don't know if moisting would have happened)
And if I use it, moisting doesn't occur (I think): as the stripe module I'm trying to mock is loaded. Which might be "normal" as described in the docs

So I don't know how to configure that. If anybody could help I'd be very happy. The project is an open source, you can check the branch I struggled with here:
https://github.com/ziedHamdi/user-credits/tree/core-as-dependency (revision: 78270b1193d25c496f7cda1b2291b96ffa588a9a)

I solved it by declaring the Stripe structure types in an interface and injecting that through ioc (putting mocks in tests). But that was a lot of work for a single module, so having mocks working easily would be a great advancement.

My jest config is as follows:

export default {
  extensionsToTreatAsEsm: [".ts"],

  globalSetup: "<rootDir>/test/config/jest/globalSetup.js",

  // Add this line
  moduleFileExtensions: ["ts", "tsx", "js", "jsx"],

  // Other Jest configuration options...
  preset: "ts-jest",

  // Specify your global setup file
  setupFilesAfterEnv: ["<rootDir>/test/config/jest/setupBeforeAll.js"],

  testEnvironment: "node",

  testMatch: ["**/test/**/*.ts?(x)", "**/?(*.)+(spec|test).ts?(x)"],

  testPathIgnorePatterns: ["extend", "mock", "config"],
  transform: {
    ".*\\.(j|t)sx?$": ["@swc/jest"],
  },
  transformIgnorePatterns: [],
};

[nlwillia]

Sorry if this is the wrong place to ask this, but it seems relevant to the point about the jest global.

I'm finding that import {jest} from '@jest/globals' has a different type for mock instances than the implicit one.

Given:

// package.json - note type:module and jest config for ts-jest's ESM support
{
  "name": "jest-mock-unknown",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "test": "node --experimental-vm-modules node_modules/jest/bin/jest.js"
  },
  "license": "ISC",
  "devDependencies": {
    "@types/jest": "^29.5.10",
    "jest": "^29.7.0",
    "ts-jest": "^29.1.1",
    "typescript": "^5.3.2"
  },
  "jest": {
    "extensionsToTreatAsEsm": [
      ".ts"
    ],
    "moduleNameMapper": {
      "^(\\.{1,2}/.*)\\.js$": "$1"
    },
    "transform": {
      "^.+\\.tsx?$": [
        "ts-jest",
        {
          "useESM": true
        }
      ]
    }
  }
}

// tsconfig.json
{
  "compilerOptions": {
    "target": "ESNext",
    "module": "NodeNext",
    "esModuleInterop": true,
    "strict": true,
    "types": ["jest"]
  }
}

// __tests__/mock.spec.ts
import {jest} from '@jest/globals'

test('example', () => {
    const fn = jest.fn()
    fn({answer: 42})
    expect(fn.mock.calls[0][0].answer).toBe(42)
})

npm run test will succeed. However, the IDE (WebStorm, VSCode) reports an error on the expect line that the type of fn.mock.calls[0][0] is unknown. Casting jest.fn() as any circumvents the error, but this is not necessary in conventional use of jest with TypeScript.

If I remove the jest import, then the IDE error goes away, but the test fails because of the missing global.

[SimenB]

@nlwillia TS questions don't belong in this issue. But to answer; the implicit/global one comes from @types/jest in the DefinitelyTyped project - @jest/globals comes from Jest itself (and are maintained by the Jest project instead of outside contributors) which is more strictly typed (in this case - defaulting to unknown instead of any).

[ChristophP]

After almost 4 years. What is still blocking this?

Quite simply put: NodeJS. Since Jest is using the vm module of NodeJS for test isolation. That API itself is stable, but for ESM scripts it is not (see here).

Jest does support running native ESM as long as you do it with NODE_OPTIONS=--experimental-vm-modules jest src.
In order to remove that flag, NodeJS would have to stabilize its VM API for ESM scripts.

[Prabhakar-Poudel]

Thank you for the answer @ChristophP
It was a little annoying to have to set the flag manually every single time.

Did we consider auto-adding the flag e,g when package.json has type: 'module'? Or is it too risky to auto-add flags even when we are confident enough?

[Systerr]

Thank you for the answer @ChristophP It was a little annoying to have to set the flag manually every single time.

Did we consider auto-adding the flag e,g when package.json has type: 'module'? Or is it too risky to auto-add flags even when we are confident enough?

This is part of nodejs, not part of jest. Jest team can't do something there to enable that runtime feature without flag.

But i hope that vm.modules will be stable on a next nodejs release - i saw a lot of movements there by avoiding v8 runtime problem by node team

[ChristophP]

This is part of nodejs, not part of jest. Jest team can't do something there to enable that runtime feature without flag.

Provided you are using jest via CLI and not @jest/core directly jest could theoretically ship something like a jest-esm executable that sets this env var to a child Node process, right?

But i hope that vm.modules will be stable on a next nodejs release - i saw a lot of movements there by avoiding v8 runtime problem by node team

This is good news

[alesmenzel]

@SimenB fyi. the import.meta.dirname and import.meta.filename was backported to Node 20.11. Those should be added to jest as well as right now, jest is missing those.
https://nodejs.org/docs/latest-v20.x/api/esm.html#importmetadirname

[krutoo]

Can we add to Jest support for "main/module" fields in package.json?

For example overlayscroolbars-react doesn't have type field in package.json but has module and main fields

[ChristophP]

The module field is non-standard. Many bundlers (parcel, vite, etc) respect it but NodeJs for example completely ignores it.
Since Jest is running in Node I assume this is why the module field is ignored.

The new standard way of defining CJS and ESM endpoints is the exports field. Newer Node versions respect it and prefer over main if exports is present.
https://nodejs.org/api/packages.html#exports

(Note however that neither the exports or module field would replace the type field. Since the first two control which file is loaded, while type the controls whether files with a .js extensions are treated as ESM or CJS. Using .cjs or .mjs forces CJS/ESM treatment regardless of the value of the type field)

[krutoo]

@ChristophP Thank you for your answer, we discovered that Jest, despite the presence of "exports" field, does not understand the overlayscrollbars-react package in Node.js v20.11.0: KingSora/OverlayScrollbars#604

[ChristophP]

Great sounds good. Sadly a bunch of packages do not specify the exports field correctly and need to update. :-/