xdm webpack loader doesn’t work with webpack cli

[remcohaszing]

The issue can be reproduced from within this repository. I use Webpack 4, but the same issue can be reproduced using Webpack 5.

In this repository, add the following files:

webpack.config.cjs:

module.exports = {
  module: {
    rules: [
      {
        test: /\.mdx?$/,
        loader: require.resolve('./webpack.cjs')
      }
    ]
  }
};

src/index.js:

import './foo.md'

src/foo.md:

<Hello />

Now running the following command yields an error:

$ yarn webpack --mode production

The first time the following error is thrown:

(node:619531) UnhandledPromiseRejectionWarning: TypeError [ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING]: A dynamic import callback was not specified.
    at exports.importModuleDynamicallyCallback (internal/process/esm_loader.js:34:9)
    at Object.module.exports (/home/remco/Projects/xdm/webpack.cjs:14:3)
    at LOADER_EXECUTION (/home/remco/Projects/xdm/node_modules/loader-runner/lib/LoaderRunner.js:132:14)
    at runSyncOrAsync (/home/remco/Projects/xdm/node_modules/loader-runner/lib/LoaderRunner.js:133:4)
    at iterateNormalLoaders (/home/remco/Projects/xdm/node_modules/loader-runner/lib/LoaderRunner.js:250:2)
    at Array.<anonymous> (/home/remco/Projects/xdm/node_modules/loader-runner/lib/LoaderRunner.js:223:4)
    at runCallbacks (/home/remco/Projects/xdm/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:27:15)
    at /home/remco/Projects/xdm/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:200:4
    at /home/remco/Projects/xdm/node_modules/graceful-fs/graceful-fs.js:123:16
    at FSReqCallback.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:63:3)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:619531) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:619531) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

After a retry, this error is thrown:

(node:620590) UnhandledPromiseRejectionWarning: TypeError: Invalid host defined options
    at Object.module.exports (/home/remco/Projects/xdm/webpack.cjs:14:3)
    at LOADER_EXECUTION (/home/remco/Projects/xdm/node_modules/loader-runner/lib/LoaderRunner.js:132:14)
    at runSyncOrAsync (/home/remco/Projects/xdm/node_modules/loader-runner/lib/LoaderRunner.js:133:4)
    at iterateNormalLoaders (/home/remco/Projects/xdm/node_modules/loader-runner/lib/LoaderRunner.js:250:2)
    at Array.<anonymous> (/home/remco/Projects/xdm/node_modules/loader-runner/lib/LoaderRunner.js:223:4)
    at runCallbacks (/home/remco/Projects/xdm/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:27:15)
    at /home/remco/Projects/xdm/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:200:4
    at /home/remco/Projects/xdm/node_modules/graceful-fs/graceful-fs.js:123:16
    at FSReqCallback.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:63:3)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:620590) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:620590) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Afterwards the errors seem to alternate randomly.

I did not run into this issue when using koa-webpack. This may explain why this issue isn’t caught by any tests.

[wooorm]

Are you on node 12+, as noted in the readme?

It seems to work for this user. And in the tests here. Although both don’t use the webpack CLI

[wooorm]

The enhanced-resolve in your stack trace looks a bit weird. Where’s that from?

[remcohaszing]

I’m using Node v14.16.0.

enhanced-resolve is a webpack dependency.

$ yarn why enhanced-resolve     
yarn why v1.22.5
[1/4] 🤔  Why do we have the module "enhanced-resolve"...?
[2/4] 🚚  Initialising dependency graph...
[3/4] 🔍  Finding dependency...
[4/4] 🚡  Calculating file sizes...
=> Found "enhanced-resolve@4.5.0"
info Reasons this module exists
   - "webpack" depends on it
   - Hoisted from "webpack#enhanced-resolve"
info Disk size without dependencies: "276KB"
info Disk size with unique dependencies: "468KB"
info Disk size with transitive dependencies: "1004KB"
info Number of shared dependencies: 10

Since this is a dependency of webpack, not webpack-cli, I don’t think this is the issue.

I suppose I could call webpack using A NodeJS script instead of webpack-cli, but using webpack-cli is the most basic usage IMO.

webpack/webpack-cli#1274 and webpack/webpack-cli#1622 seem related, but they apply to usage of ESM in webpack configs, not loaders.

[wooorm]

Can you try running this with the DISABLE_V8_COMPILE_CACHE=1 flag?

[wooorm]

It’s mention in that thread and it worked for me.

I’m also guessing that this might be a problem in this project, but not when xdm is in node_modules?

[wooorm]

This seems to be the root: zertosh/v8-compile-cache#30

[remcohaszing]

I initially tried this in a separate project, but I figured reproducing it inside the repository might be convenient. The issue exists there as well.

Setting DISABLE_V8_COMPILE_CACHE=1 seems to work.

[wooorm]

Quite perplexing to me that these tools focussed on ESM, such as webpack and snowpack, do not support ESM 🙈

[remcohaszing]

I actually expect the situation to get even worse in the JavaScript ecosystem as more and more packages will start using ESM, whilst others won’t even support it yet. It will be better in the end though. 😄

[chenxsan]

Or rewrite lib/webpack-loader.js in CommonJS? I'm afraid It would take a while for webpack to support ESM in loaders. DISABLE_V8_COMPILE_CACHE is not a good idea in my opinion, it would be my last resort.

[remcohaszing]

The loader itself is using CommonJS. The issue is that Webpack loaders can’t use any dependencies that use ESM if they are used with webpack-cli.

In other words: to fix this, would mean all of xdm would need to be converted to CommonJS. The real issue is upstream in v8-compile-cache.

[wooorm]

Thanks for chiming in @chenxsan! But it does seem to be somewhere in what’s webpack using (#11 (comment)) that the real issue is. Because import() should be allowed in a CJS loader imo, but that issue prevents it

[chenxsan]

Seems v8-compile-cache is widely used by famous tools like eslint, webpack-cli, parcel, gatsby etc., so all of them will likely to have problem like this if we want to deal with ESM. I know Node.js 10 will be end of life soon, but the world doesn't seem be to ready for ESM yet even it's out for many years.

[ChristianMurphy]

Seems v8-compile-cache is widely used by famous tools like eslint, webpack-cli, parcel, gatsby etc., so all of them will likely to have problem like this if we want to deal with ESM

Yes, they likely will, which is another reason why this should be addressed in v8-compile-cache.

I know Node.js 10 will be end of life soon, but the world doesn't seem be to ready for ESM yet even it's out for many years.

The transition to ESM is happening across the JS community, not just here (for example sindresorhus/meta#15)

[wooorm]

I know Node.js 10 will be end of life soon, but the world doesn't seem be to ready for ESM yet even it's out for many years.

I don’t think we’d be ready in two years either. I feel like the push and having to move is what’s surfacing these things, whether that’s now or another moment in time!

Still, all the issue trackers are going to be on fire soon 😅

[wooorm]

Closing as it’s not something we can solve here 🤷‍♂️

[devmattrick]

Sorry for the bump, but if you added a bundler like rollup as a build step, wouldn't it change it to a CJS compatible format? I understand this would be a major architectural change, but I think it'd work?

[ChristianMurphy]

See unifiedjs/unified#121, not just xdm but all layers of unified are moving to pure ESM.
Most tools have already added support for ESM or are in the process of doing so https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c

We've experimented with mixed ESM and CJS in the past, it didn't go so well remarkjs/react-markdown#518
Tools are better equipped to handle pure ESM packages, than CJS and ESM fusion ones.

[devmattrick]

Gotcha, I appreciate the quick response. ESM is definitely the future, so I suppose tooling should adjust to support it. Thanks for your help!