Specifying plugins via the CLI

[andreialecu]

aws-cdk currently uses esbuild to bundle code used for lambdas: https://docs.aws.amazon.com/cdk/api/latest/docs/aws-lambda-nodejs-readme.html

Unfortunately, because of internal implementation details in aws-cdk, the esbuild nodejs API cannot be used directly. Ref: aws/aws-cdk#13179 (comment)

So they can only use the CLI to spawn esbuild.

The problem is that when used in conjunction with Yarn v2 and PnP, this plugin needs to be added: https://github.com/yarnpkg/berry/tree/master/packages/esbuild-plugin-pnp

Currently esbuild does not seem to allow specifying plugins via a CLI option, so Yarn PnP cannot work with the aws-cdk esbuild implementation. Ref: aws/aws-cdk#13257 (comment) (there's a workaround possible, but it makes things pretty complicated)

Would it be possible to allow specifying the plugin via the CLI? Or via some configuration file perhaps? (some file like .esbuildrc could be an option)

[evanw]

Sorry, esbuild's plugin API is intentionally designed to be accessible via JavaScript instead of the CLI. IMO using JavaScript is much more intuitive and ergonomic than a CLI for complex configuration scenarios. I don't really have any plans to change this at the moment.

I'm completely unfamiliar with what AWS CDK is, so my apologies if something I'm saying doesn't make sense. With that caveat in mind, some possible workarounds for you off the top of my head:

• Disable Yarn v2 PnP mode
• Use Yarn v1 instead of Yarn v2
• Use another package manager instead of Yarn
• Use Yarn v2 with PnP but run esbuild yourself in your own build script before the AWS CDK sees your code

Do any of those work for you?

I may in the far future explore a plugin API for binary plugins. If I do that, there's a possibility that they might be enabled for CLI usage. See #111 (comment) for more information. But right now I am not planning on taking esbuild's plugin API in this direction.

[fazouane-marouane]

A bit of a hack but, does creating an alias to npx solve the issue ? Maybe aliasing npx to "yarn" or "yarn dlx" would do ?

[andreialecu]

@evanw in the PnP resolver plugin's case, it just needs to be imported. No other configuration for it is required.

There is a workaround possible, by disabling PnP, but that's not a good solution for bigger monorepos that rely on it.

This is also related: #237 (comment)

A way to simply allow specifying a plugin, but for the CLI would be great. This is all that is needed:

import {pnpPlugin} from '@yarnpkg/esbuild-plugin-pnp';

await build({
  plugins: [pnpPlugin()],
  // ...
});

I noticed that the possibility of an esbuildrc file was discussed before: #39

An .esbuildrc.js could look like:

module.exports = {
  plugins: ['@yarnpkg/esbuild-plugin-pnp']
}

or

import {pnpPlugin} from '@yarnpkg/esbuild-plugin-pnp';
module.exports = {
  plugins: [pnpPlugin]
}

There are various tools already written that use the esbuild CLI, and they're all blocked from supporting PnP due to this omission, unfortunately. 🙁

[fregante]

IMO using JavaScript is much more intuitive and ergonomic than a CLI for complex configuration scenarios

This kind of makes sense from a "complexity within esbuild" standpoint but unfortunately the API isn't equivalent to the CLI. I just used it successfully and the first thing I notice is that I no longer get a list of outputted files, stats, and watching status.

I also have to deal with running the file myself, e.g. I have to use ts-node:

node --loader ts-node/esm esbuild.ts

[hyrious]

@fregante You can get output stats with logLevel: 'info', or try parse the metafile by yourself (an example of mine).

---

Talk back to the issue, I have made --plugin:xxx in my esbuild-dev, which is just a wrapper of esbuild to build & run files like ts-node. An example of using that is I made a svelte ssr plugin and with that I can import svelte file right in the node side.

Other solutions like antfu/esno use require.extensions / loaders to not pollute the file system and provide better experience at __filename and import.meta. I believe this will be a better solution than mine because node loaders are the future. However esbuild plugins won't work in transform mode, I don't know what will they do.

---

One thing I'd like to have is loading plugins from the global scope like rollup. But I haven't got the trick to implement that.

[shellscape]

Another vote for specifying plugins via CLI. CDK has us limited to builtin loaders, for which there isn't one for .graphql files, and forces us to abandon imports.

[VanTanev]

We're in a similar situation - aws-cdk now uses esbuild cli for bundling lambda functions. Because of that, there is no way to specify plugins when bundling with CDK, which means we have to have 2 separate bundling steps - one outside CDK, and then one in CDK that consumes the output of the first.

It would be much appreciated if we could add plugins by pointing to a .js file with module.exports = ... whatever plugin.

[piotrmoszkowicz]

It would be awesome to have that +1 to @VanTanev proposal.

[klarkc]

+1 for this. When using nix expressions, the derivation only exposes esbuild CLI, to use plugins with esbuild in this context, one need to build a js-backed derivation to inject plugins, it just does not worth the added complexity.

[thefat32]

+1 for this need this to simplify bundling using CDK aws-lambda-nodejs

[PatrykMilewski]

The same issue here with aws-cdk

[IARI]

similar issue here
(Although im building my lambda with esbuild manually, not using CDK, so I can workaround by not using the cli tool ....)

[evanw]

I'm closing this issue because I'm not planning on doing this. For what it's worth esbuild has since added native support for Yarn PnP, so the concern in the original issue should be resolved by just updating esbuild.

[shellscape]

Disappointing, as even Rollup has pulled this off without headache.

[klarkc]

FWIW, vite might be a good replacement for esbuild in this case, given that it support more features without the need for plugins.

[shellscape]

guess what vite uses under the hood? 😉

[VanTanev]

Hey @evanw, thank you for the amazing work on esbuild!

I wanted to ask, could you reconsider this issue? aws-cdk has really put us in a bind, by not allowing a js config for esbuild. The only thing we can configure is cli options. (They run it through a docker container too, further constraining options for configuring it other than outright replacing the esbuild container just to be able to configure esbuild).

I know that is of no fault of yours, and you have no obligation to take on the burden of supporting extra features in esbuild.

Still, it would be really helpful for us, and other teams that need to bundle through cdk. Having the option to either specify a config file, or a file with a plugin to be loaded from the cli would be a God sent.

Thank you!

[evanw]

This isn’t straightforward on esbuild’s side because the CLI is native code and doesn’t contain a JavaScript interpreter. This feels like something that sounds be fixed on AWS’s side instead. If they want to support JavaScript plugins, they should be using esbuild’s JavaScript API instead of the CLI. Please file an issue with them about it.

[aslakhellesoy]

FWIW, I worked around this issue with https://github.com/mrgrain/cdk-esbuild

[rjmackay]

This is the relevant CDK issue FYI aws/aws-cdk#18470