TypeScript refuses to emit files from node_modules with getEmitOutput()

[s-panferov]

Hi. In out company we use workflow when all packages are written in pure TypeScript and we publish and consume them as-is, without any pre-compilation. I know that it's slower, but it's very convenient for developers, because they don't need to run several parallel tsc --watch processes.

We used webpack and awesome-typescript-loader for this and everything worked fine. But I see that now TypeScript refuses to emit files from node_modules directory using getEmitOutput(), because of isSourceFileFromExternalLibrary check here.

Can we make this configurable?

[mhegazy]

getEmitOutput follows the same rules the compiler follows when emitting a file. if the file is not expected to be emitted in a batch compilation scenario it is not going to be emitted by getEmitOutput. otherwise, the output generated from different builds vary, and bad things happen.

Files coming from "node_modules" are considered "external", and are not emitted. you can see why writing output to you "node_modules" is generally a bad thing, as you could overwrite package files.

However, If the files are included in your compilation, they should be emitted. This does not seem to work correctly today, so we need to fix this.

[mhegazy]

Just to clarify, this issue is tracking emitting node_module files iff they are listed in the "include" list in a tsconfig.json.

[s-panferov]

@mhegazy I see several different use-cases here:

1. When you use tsc from a command line, you don't expect node_modules to be emitted, this is understandable because you work with a real file system and bad things can happen.

2. In our case we build a single bundle file using WebPack and work with in-memory file system without any intermediate IO, so we want these files to be emitted by TypeScript (if we explicitly ask to) so we can pipe the emitted output directly to WebPack.

I'm not quite sure that adding node_modules folder to include is a good thing in our case because it will work really slow trying to traverse all the folders inside node_modules. Adding libraries one-by-one is not good enough too because there can be transitive dependencies, and we don't know the exact path of the modules on a file system.

I think the best solution for (2) is to allow to override the check somehow.

[mhegazy]

I think the best solution for (2) is to allow to override the check somehow.

The two code paths cannot produce different output. I understand this makes sense to your scenario but it sure will be odd that you run tsc on the commandline, and get different out than running it through webpack extension.

I'm not quite sure that adding node_modules folder to include is a good thing in our case because it will work really slow trying to traverse all the folders inside node_modules. Adding libraries one-by-one is not good enough too because there can be transitive dependencies, and we don't know the exact path of the modules on a file system.

Just to put things in prescriptive, we used to emit files in node_modules (as a result of a bug), and users have complained about this repeatedly. it is unexpected when compiling your code to change your dependencies. there are numerous packages out there that ship with their sources as well. and i would argue you need to know what files you are touching at any point of time, otherwise running npm install can include some package you did not know about, or give you errors in code you do not own.

[jjrv]

Outside node_modules TypeScript emits .js files for any .ts files I import, whether they're listed under files in tsconfig.json or not. This is very nice.

I have a monorepo structured like boennemann/alle. It makes the monorepo very easy to work with, requiring minimal tooling. The trick is to place all code under node_modules so Node's module lookup strategy will find them without having to create any symlinks.

Unfortunately I'm running into this issue... Now it's necessary to either omit files, list all of them or use a wildcard. In this case there would be a reasonable strategy: when a .ts file included in files is located under node_modules then for any further .ts files it imports from the same directory or a subdirectory (without entering any deeper node_modules directories), a .js file should still be emitted.

Alternatively a configuration option in tsconfig.json listing paths where .js files should be emitted, whether under node_modules or not, should fix this issue for everyone's possible use cases.

[philkunz]

whats the status on this one?

[alfaproject]

I also have the same mono repo structure as @jjrv and accidentally landed up in this issue from the reference in ts-jest. ):

[ahnpnl]

@jjrv said it right. At the moment in ts-jest, either ts-jest has to use Program as fallback or read jest config to solve the issue. I hope TypeScript can solve this natively.