When TypeScript project is configured to ESM, projen commands gets ERR_UNKNOWN_FILE_EXTENSION

[giseburt]

When the project is configured to use ESM, the projen command usually maps to "exec":

ts-node --project tsconfig.dev.json .projenrc.ts

Due to TypeStrong/ts-node#2094 when using node >= 18.19 (18.18 works fine, 20 does not) yields an error like:

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for blah/.projenrc.ts

Session here show using node 20.10.0, node 18.19.0, and node 18.18.2:

By default, projen does not configure a TypeScript project for ESM (that's a separate issue), but when it does, we run into this.

Can be recreated with the following command using node 18.19 or newer:

npx projen new --from @10mi2/tms-projen-projects tms-typescript-app --package-manager=npm --ts-node-unknown-file-extension-workaround=false

# once done, simply
npx projen

# you'll see the errors as shown in the screen shot

Setting --ts-node-unknown-file-extension-workaround=true (or leaving it out) will apply the workaround (found mostly from the ts-node issue)

EDIT 2: I recommend the following on projects that run into this, or might in the future:

// add `tsx` to avoid the many issues with `ts-node`
project.addDevDeps("tsx");

const PROJEN_TSCONFIG_FILENAME = "tsconfig.projenrc.json";
if (project.defaultTask) {
  // add a secondary projenrc-specific tsconfig file that doesn't emit JS
  const projenTsconfig = new TypescriptConfig(project, {
    fileName: PROJEN_TSCONFIG_FILENAME,
    include: [
      ".projenrc.ts",
      "projenrc/**/*.ts", // added by projen for tsconfig.dev - gives a place for projenrc included files
    ],
    extends: TypescriptConfigExtends.fromTypescriptConfigs([
      project.tsconfigDev,
    ]),
    compilerOptions: {
      ...RESET_COMPILER_OPTIONS,
      noEmit: true,
      emitDeclarationOnly: false,
    },
  });

  // adjust the projen command to:
  // 1. run tsc to typecheck and syntax check the file
  project.defaultTask.reset(`tsc --project ${projenTsconfig.fileName}`);

  // 2. use the projenrc-specific tsconfig and tsx
  project.defaultTask.exec(
    `tsx --tsconfig ${projenTsconfig.fileName} .projenrc.ts`,
  );
  project.defaultTask.description =
    "Run projen with ts-node/esm (workaround for Node 18.19+ applied)";
}

This solution is better than the other suggestions I've made since it will work in CJS and ESM mode, avoids all the other ts-jest issues, and does full type-checking with clear (normal) typescript errors. See explanation here

See complete RFC for ESM mode here: #3447

NOTE ON EDIT: Previous version of this comment had tsc .projenrc.ts but I missed that when you pass tsc a file list, it ignored tsconfig.json files and only uses default configs. The second edit switch from ts-node to tsx to fix even more issues.

[mrgrain]

Thanks for reporting this! Sounds like an important thing to resolve one way or the other!

When the project is configured to use ESM, the projen command usually maps to "exec":

For purpose of reproducing this, how do you configure your project to use ESM?

tsc .projenrc.ts && node --loader ts-node/esm --no-warnings=ExperimentalWarning .projenrc.ts

For the fix, this seems to do quite a bit more than making ts-node use ESM? Is all of this needed? What am I missing?

[giseburt]

Ignoring ts-jest and esbundle configs, this is precisely what’s happening in the example above:

https://github.com/10mi2/tms-projen-projects/blob/9e552b1f75e40dd92994bc4f615db7aa619ff780/src/tms-typescript-app-project.ts#L376-L389

I believe the minimum to reproduce is the first of those lines:

this.package.addField("type", "module");

And what’s all happening: The ts-node command is a smart shortcut if you will to the second form, mostly. It does some other stuff to make the error messages clean that it’s easier for us to do by just running tsc with noEmit. In that “other stuff” is where the bug is.

# instead of
ts-node --project tsconfig.special.json src/index.ts

# use
tsc .projenrc.ts && \
  TS_NODE_PROJECT=tsconfig.special.json node --loader ts-node/esm --no-warnings=ExperimentalWarning src/index.ts

If there are any type errors, the node --loader ts-node/esm yields a difficult-to-read error message, so we run tsc first separately to get the type errors before running the node command.

The tsc command assumes the correct tsconfig file where the target is included has noEmit set to true. If not, add --noemit to the tsc command.

We don’t emit from things compiled with tsconfig.dev.json so that last bit is handled.

There are other options. We could use tsx or swc-loader - I’ve had great luck with tsx but haven’t tried swc-loader. Neither do type checking so the tsc pre-pass is still recommended.

[mrgrain]

Okay, so it seems like ts-node --esm is what's needed.

This is the minimal change I did come up with:

const project = new typescript.TypeScriptProject({
  // ...
  tsconfigDev: { compilerOptions: { module: 'ESNext' } },
});
project.package.addField('type', 'module');
project.tsconfigDev.file.addOverride('esm', true);

I don't really see how we can resolve this without addressing ESM in a wider context.

[giseburt]

That does indeed reproduce it for me, but it also breaks it more, sadly.

Last I looked, and it's a moving target, ts-node --esm and ts-node-esm both had the same issue as ts-node auto-detecting ESM mode.

It doesn't appear to fix it when I add --esm to the command manually or use ts-node-esm:

Running the command as follows gets past that but, but makes another:

TS_NODE_PROJECT=tsconfig.dev.json npx \
  node --loader ts-node/esm --no-warnings=ExperimentalWarning .projenrc.ts

To get it to work with ESM, minimally (not the project, just .projenrc.ts, for anyone reading this later):

const project = new typescript.TypeScriptAppProject({
  //...
  tsconfigDev: { compilerOptions: { module: 'node16' } }, // ← had to use 'node16'
});
project.package.addField('type', 'module');
project.tsconfigDev.file.addOverride('ts-node.esm', true); // ← 'tsnode.esm'  vs 'esm'

So, as a workaround, add this before the synth:

if (
  project.defaultTask
) {
  project.defaultTask.reset(
    'tsc .projenrc.ts && node --loader ts-node/esm --no-warnings=ExperimentalWarning .projenrc.ts',
  );
  project.defaultTask.env('TS_NODE_PROJECT', project.tsconfigDev.fileName);
  project.defaultTask.description =
        'Run projen with ts-node/esm (workaround for Node 18.19+ applied)';
}

And you can still run npx projen when in ESM mode.

And you're right, @mrgrain, more work is needed for ESM to work fully. But I thought I'd make smaller issues (with smaller PRs) that are easier to tackle to pave the way.

Another option, if we're willing to move away from ts-node, is tsx with tsc ran first (since esbuild that tsx uses doesn't do type checking):

const project = new typescript.TypeScriptAppProject({
  //...
  devDeps: ['tsx'],
  tsconfigDev: {
    compilerOptions: {
      module: 'node16',
    },
  },
});
project.package.addField('type', 'module');

if (
  project.defaultTask
) {
  project.defaultTask.reset(
    `tsc .projenrc.ts && tsx --tsconfig ${project.tsconfigDev.fileName} .projenrc.ts`,
  );
}

[giseburt]

I don't really see how we can resolve this without addressing ESM in a wider context.

I just retested and the --loader ts-node/esm mechanism doesn't work with the module not in ESM mode. 😞

However, the tsx option does work in both modes:

const project = new typescript.TypeScriptAppProject({
  //...
  // No tsconfig overrides here
  devDeps: ['tsx'],
});
// optional - works with or without type: "module", and no other tsconfig changes are needed
project.package.addField('type', 'module');

if (
  project.defaultTask
) {
  project.defaultTask.reset(
    `tsc .projenrc.ts && tsx --tsconfig ${project.tsconfigDev.fileName} .projenrc.ts`,
  );
}

[mrgrain]

Ha you are right, apologies! I think I misread the ts-node docs.

I can conform this works for me now:

const project = new typescript.TypeScriptAppProject({
  //...
  tsconfigDev: { compilerOptions: { module: 'nodenext' } }, // ← or 'node16'
});
project.package.addField('type', 'module');
project.tsconfigDev.file.addOverride('ts-node.esm', true);

---

  // No tsconfig overrides here
  devDeps: ['tsx'],

It feels odd to me to say that no tsconfig changes are required. You still need to change module for ESM to work, right? The override is only needed because the tsconfig types don't support ts-node as of now.

I'm not super keen on changing ts-node. I'm aware tsx is on the rise, but it feels to early. Have you looked at yet what it takes to create your own version ProjenRcTSX? 'Cause I'm not sure it's super easy to swap that out at the moment, but ultimately projen should allow that.

[giseburt]

It feels odd to me to say that no tsconfig changes are required. You still need to change module for ESM to work, right? The override is only needed because the tsconfig types don't support ts-node as of now.

I'm not sure I follow completely, but I'll take a swing at it: I was just saying that tsx works in my testing in both ESM (future) and non-ESM (current) contexts. In ESM context, you'd want to change tsconfig.dev,json somewhat, yes.

I'm not super keen on changing ts-node. I'm aware tsx is on the rise, but it feels to early. Have you looked at yet what it takes to create your own version ProjenRcTSX? 'Cause I'm not sure it's super easy to swap that out at the moment, but ultimately projen should allow that.

I've been using tsx and esbuild (that it's based on) for a while. I've only run into a couple of issues:

1. It doesn't type-check, so code that TypeScript wouldn't pass will pass - solution is to run tsc with noEmit first
2. It doesn't support experimental features like decorators - I don't expect anyone to be using those in the Projen tooling, but if they NEED to, the solution is to run tsc with emit enabled and then run the JS (not terribly pretty, since you have to put the temporary javascript somewhere)

That said, I don't think we necessarily need to switch - they do provide the same basic features.

Referencing Projenrc.addDefaultTask()

Here's an untested rough sketch to make tsx an opt-in:

export interface ProjenrcOptions {
//  ...
  /**
   * Whether to use `tsx` instead of `ts-node`.
   *
   * Ignored `swc` option if `tsx` is set to `true`.
   *
   * @default false
   */
  readonly tsx?: boolean;
}

// ...

export class Projenrc extends ProjenrcFile {
// ...
  private addDefaultTask() {
    // this is the task projen executes when running `projen` without a
    // specific task (if this task is not defined, projen falls back to
    // running "node .projenrc.js").

    // we use "tsconfig.dev.json" here to allow projen source files to reside
    // anywhere in the project tree.

    if (this._tsx) {
      // Use tsx
      this._tsProject.addDevDeps("tsx");
      this._tsProject.defaultTask?.exec(
        `tsc ${this.filePath} && tsx --tsconfig ${this._tsProject.tsconfigDev.fileName} ${this.filePath}`
      );
    } else {
      // Use ts-node
      const deps = [];
      if (this._swc) {
        deps.push("@swc/core");
      }
      deps.push("ts-node");
      this._tsProject.addDevDeps(...deps);

      const tsNode = this._swc ? "ts-node --swc" : "ts-node";

      this._tsProject.defaultTask?.exec(
        `${tsNode} --project ${this._tsProject.tsconfigDev.fileName} ${this.filePath}`
      );
    }
  }
//...
}

Note that this wouldn't be replacing or effecting other uses of ts-node, such as that inside the cdk projects.

[giseburt]

You can verify if tsx work easily with:

npx tsx --tsconfig tsconfig.dev.json .projenrc.ts

In fact, when ts-node hits an error, that's what I've been running to execute projen again and rebuild the package.json and tsconfig*.json files.

[mrgrain]

I'm not sure I follow completely, but I'll take a swing at it: I was just saying that tsx works in my testing in both ESM (future) and non-ESM (current) contexts. In ESM context, you'd want to change tsconfig.dev,json somewhat, yes.

My point really just is that for ESM to work you need to make changes to your tsconfig. One change more doesn't seem like a big issue to me.

[giseburt]

Have you looked at yet what it takes to create your own version ProjenRcTSX?

That confused me for a while, since tsx (the tool) just executes TypeScript files by compiling them on the fly (with esbuild) and running them with node (same as ts-node, etc.). (I'm aware I greatly simplified what those tools do.)

Then I realized that there's confusion due to poor naming: *.tsx files (used by React, etc.) have nothing to do with tsx the tool I was referring to ☝️. (You can use them together, since *.tsx is effectively an extension of TypeScript, but otherwise they're unrelated.)

With the ProjenRcTSX reference were you thinking I was suggesting to make a .projenrc.tsx option, perhaps? If so, that's not the case.

On a slightly different note, how should I proceed? My eventual goal is to chip away at ESM support with several smaller Issue→PR steps. I have several projects that use ESM mode already (optionally), and projen itself works fine in/with ESM-based projects, so no changes are needed in projen itself, just how it configures the tsconfig, package, ts-node, and ts-jest.

Those projects are here: https://github.com/10mi2/tms-projen-projects

[mrgrain]

Oh no, I see the confusion now. Sorry for being unspecific.

What I meant is:

• You are proposing to extend the existing typescript.Projenrc component with a new option (e.g. tsx: true)
• I am proposing you build a completely new tms.ProjenrcUsingTsx component that does exactly what you want.

Reason being that there is a in almost infinite permutation of ways how we can deal with .projenrc.ts. And even when we are taking out the clearly niche uses cases, that's still a lot of possible ways that we should support inside typescript.Projenrc.

Now saying that, I'm not sure if it's currently possible to write this tms.ProjenrcUsingTsx without using hacks. So I'm curious to find out where the gaps are and what paper cuts you encounter.

[mrgrain]

On a slightly different note, how should I proceed? My eventual goal is to chip away at ESM support with several smaller Issue→PR steps. I have several projects that use ESM mode already (optionally), and projen itself works fine in/with ESM-based projects, so no changes are needed in projen itself, just how it configures the tsconfig, package, ts-node, and ts-jest.

Please start with writing an RFC issue on ESM support. It's really hard to judge smaller PRs without seeing the bigger pictures. Once we have Community agreement on the bigger picture, I'm happy to accept smaller PRs.

[giseburt]

I'm not sure how 'ProjenrcUsingTsx' would work. Or why, really. Nothing in the .projenrc.ts file would change, and nothing in the output would change other than the command in the default task, and the dev dependency going from ts-node to tsx.

I guess I could extend Projenrc class and override the addDefaultTask() (which is currently private). Is that what you had in mind?

On the ESM score, I'll work something up. Will likely take a few days to get the time.

[mrgrain]

I'm not sure how 'ProjenrcUsingTsx' would work. Or why, really. Nothing in the .projenrc.ts file would change, and nothing in the output would change other than the command in the default task, and the dev dependency going from ts-node to tsx.

typescript.Projenrc and other Projenrc components are only partly about the contents of the rcfile. After the initial creation, the content is written by users after all. The Projenrc components are mostly about "how to execute a program, so that it produces the desired output" (i.e. project.synth()). But yes, they also take care of the initial generation.

I guess I could extend Projenrc class and override the addDefaultTask() (which is currently private). Is that what you had in mind?

That would probably be the best approach. You could write it from scratch, but like you've said some parts would end up being copy and paste.

On the ESM score, I'll work something up. Will likely take a few days to get the time.

Thank you!

[dj-rabel]

I can conform this works for me now:

const project = new typescript.TypeScriptAppProject({
  //...
  tsconfigDev: { compilerOptions: { module: 'nodenext' } }, // ← or 'node16'
});
project.package.addField('type', 'module');
project.tsconfigDev.file.addOverride('ts-node.esm', true);

Just wanted to add that we ran into the same issue and solved it very similar to what you did. This worked for us:

const project = new typescript.TypeScriptAppProject({
  //...
  tsconfigDev: { compilerOptions: {
    module: 'ES2022', // <-- starting from ES2015, everything works
    moduleResolution: TypeScriptModuleResolution.NODE,
  } },
});
project.tsconfigDev!.file.addOverride('ts-node', {
  esm: true,
  experimentalSpecifierResolution: 'node',
});

For us it didn't work without node as moduleResolution and epxerimentalSpecifierResolution. Not 100% sure why (I think it was something about file extensions in import statements or something like this), but I wanted to mention this.

[giseburt]

@dj-rabel This is very dependent on node version. Were you on node 18.18 or older, or something newer?

[dj-rabel]

@giseburt sure. I just wanted to contribute our experience to the discussion. For the project I'm referring to, we're on 18.14.

[giseburt]

@dj-rabel 👍. Watch out for upgrading to node 18.19 or newer, you'll run into the TypeError [ERR_UNKNOWN_FILE_EXTENSION] bug in ts-node

[dj-rabel]

@giseburt thanks for the hint. This error sounds very familiar to me. I'm not 100% sure, but this might be the reason why we ended up adding this experimentalSpecifierResolution setting 🤔🧐 will check with the colleges tomorrow.

[giseburt]

EDITED: I'm working on a new RFC for general ESM support, but for those running into this issue, here's the escape hatch I'm using ATM.

See edit in original comment above, I put the code there.

What's happening here is, since ts-node when run as --loader ts-node/esm has terrible error messages if there were type errors, and tsx doesn't do type checking at all, we want to call tsc before executing the code. We can't have it compile just one file and have it use settings from a tsconfig file (not that I've found, at least), so we have to have it compile a whole project.

If we use tsconfig.dev.conf it'll include the project source and the testing source code as well, which means if the code itself is malformed, it'll error and nor run .projenrc.ts. This happens in a case like where the code is referring to a module that hasn't been installed yet because it's in deps and you can't run projen because it keeps erroring on that missing dependency.

So, we make a new tsconfig.projenrc.json that extends tsconfig.dev.json and only includes ".projenrc.ts", "projenrc/**/*.ts" (the latter if you want to move some code out of .projenrc.ts into a library).

Also, tsx has far fewer issues than ts-node - it just doesn't do type-checking.

[giseburt]

@mrgrain:

Please start with writing an RFC issue on ESM support.

Here it is: #3447