Bug With Symlinked Packages in Monorepo

[zaripych]

Bug report

I'm using dts-bundle-generator in my experimental monorepo. I found it irreplaceable, especially if I bundle in some of my devDependencies and they would not exist as packages in published npm package, but their TypeScript declarations need to be inlined. I don't know any other tool that does this. Well done 👍. And thank you 🙏.

I use pnpm to install packages. So I have following directory structure which describes pnpm behaviour the best:

packages/
  packageA
    node_modules
      packageB -> ../../../packageB
    *.ts
  packageB
    *.ts

We can see that packageB is going to be a symbolic link.

When using dts-bundle-generator I get a compiler error, saying that I cannot import this or that. Example:

Additional context

The test that reproduces this issue is here:
zaripych@49ad3a6

I fixed it by making these changes:
zaripych@794a1a3#diff-5a014ac4d87f713ea8b9c8dd8966ce3b77df4ecc8e263b4baf9b9469bf9f2917

Would it be possible to incorporate these changes in the main repo?

[timocov]

Hey @zaripych, is it expected that in your test case you're restricting the list of files that might be referenced in the sub-project is the only "exports": "./index.ts"? And also, since your package is referenced via node_modules, it makes sense to compile a dependency before compiling a project so the consumer's will use d.ts files only. Have you tried that? Does it help?

[zaripych]

Hey @timocov I understand that it's quite unusual that the imported code is uncompiled TypeScript. The monorepo setup I'm experimenting with is to allow development (not runtime when the package is already published and installed as dependency) without having to compile anything at all and without having to wait for those bundles to be generated/created. Imagine a complex project with multitude of packages in a single repo - each of which is TypeScript that needs to be bundled before it can be used - this is quite a headache. When developers just use "main": "src/index.ts" as their entry points - everything just works - including the TypeScript compiler which is able to work it out and does produce .d.ts unlike dts-bundle-generator which fails with above error.

However, when the code bundled and released - those src/index.ts will properly turn into dist/main.es.js.

Running TypeScript is now possible with tools like ts-node which can take advantage of swc to compile it super quickly, or tsx which uses esbuild. So basically - I don't transpile anything in my monorepo for development purposes explicitly - it all happens on demand (like in unit test frameworks).

E.g. if I have following dependency tree:
packageA -> packageB -> packageC
I don't want to be building and bundling packageA and packageB explicitly to bundle packageC. It's best if that bundling happens implicitly, especially if we consider that packageA and packageB were never supposed to be published and maybe inlined via bundler in packageC and are actually removed from dependencies in package.json of packageC.

Does this make sense?

[zaripych]

E.g. I do the same with rollup as bundler. It never fails bundling/transpiling packageC if I didn't bundle packageA first. It just works.

[zaripych]

Just to clarify, dts-bundle-generator would actually work just fine if that node_modules module was a regular non-symlinked module. I've added another test case to clarify what I mean:
zaripych@bacf243

I've added node_modules/sub-package, deleted the symlink from export-from-sub-package, and created export-from-linked-sub-package with the symlink. So now we have two test cases that point to same sub-package with non-transpiled TypeScript.

If we revert the proposed change - the non-symlinked test case still works, while symlinked test case stops working.

Trying to highlight that this is not about whether I transpile things or not, but about the symlink. We are getting resolvedModule.isExternalLibraryImport === true in the symlinked test case from TypeScript itself.

[timocov]

Btw, did you try to disable followSymlinks option? It seems that it might help since you're inlining your local packages anyway.

Imagine a complex project with multitude of packages in a single repo - each of which is TypeScript that needs to be bundled before it can be used - this is quite a headache

I'd use composite projects in this case and make a relation between projects explicitly. But yeah, in this case you have to compile a dependency project to make it work.

Your current approach looks a bit hacky tbh, because for example the compiler won't compile the dependencies from node_modules simply because it doesn't know where to output the generated files from node_modules.

E.g. I do the same with rollup as bundler. It never fails bundling/transpiling packageC if I didn't bundle packageA first. It just works.

I didn't dig into this, but basically if you won't compile a dependency and keep ts files as in in node_modules it might not work, because as I said the compiler doesn't generate output for files from node_modules. But I'd look at followSymlinks option to give it a try, I believe it was created for similar purposes.

[zaripych]

Thank you for bearing with me. Appreciate your time.

Btw, did you try to disable followSymlinks option? It seems that it might help since you're inlining your local packages anyway.

I will try it and let you know what this does 👍. Would the bundler be able to inline my subpackage though. Will see.

I'd use composite projects in this case and make a relation between projects explicitly. But yeah, in this case you have to compile a dependency project to make it work.

I considered that. Composites are configuration bloat from my perspective. Package.json is enough. And it worked so far - surprisingly 😀 Just trying to keep it DRY.

Your current approach looks a bit hacky tbh, because for example the compiler won't compile the dependencies from node_modules simply because it doesn't know where to output the generated files from node_modules.

I've had experience with monorepo setups with TypeScript before composites were a thing and people would use "baseUrl" and "paths" with relative values to make things work even without symlinks. They still do that. Just look at some monorepo projects (some very modern projects). Whether it was called hacky or not - didn't matter - it worked for multiple teams and allowed them to progress. 👍 So I understand that this is non-idiomatic TypeScript use, I care more if this helps developers deal with less configuration bloat and keep it DRY.

So, I'm not interested much in what tsc outputs as I only use it for type checking only.

I didn't dig into this, but basically if you won't compile a dependency and keep ts files as in in node_modules it might not work

This already works. Worked on multiple projects for me. It's just I never got to fixing declarations for inlined packages.

[timocov]

This already works. Worked on multiple projects for me. It's just I never got to fixing declarations for inlined packages.

Have you tried to publish a package that consists of ts files only (without prior compilation/transpilation) and build an app that depends on this project? I'd like to see how it works 🙂 This is basically what I see when you say that "I link other packages in node_modules via symlinks and it works fine". Yes, it might work in some cases and I do understand why exactly people do it, but the problem is wider that you might think and the solution is still hacky is relies that tools resolve symlinks and don't treat them as part of node_modules. For exactly this reasons followSymlinks option has been introduced a long time ago and I hope it helps with your scenario well 🙂

[zaripych]

Sorry I still haven't got to trying. 👍 Will do this evening.

Have you tried to publish a package that consists of ts files only

If by publishing you mean to npm - I wouldn't consider doing something like that. Consumers do not have dev tools that you have in your monorepo. We talking about development environment vs what is published - those are different. Before publishing a monorepo package you need to generate a proper package.json (and get rid of pnpm workspace: * references), build all ts files to be cjs/mjs, prepare declarations and assets that you include into package - at this moment there is going to be no src files left. 👍

However, at some point I did consider not to bother and generate declarations and just include source code along with bundled js to save time on declarations generation.

[zaripych]

@timocov the config didn't help, and it prevented from other npm packages mounted by pnpm from working as well. To clarify, with following config:

{
  "compilationOptions": {
    "preferredConfigPath": "./tsconfig.json",
    "followSymlinks": false
  },
  "entries": [
    { "filePath": "src/index.ts", "outFile": "dist/dist/main.es.d.ts" },
    { "filePath": "lint.ts", "outFile": "dist/dist/lint.es.d.ts" }
  ]
}

dts-bundle-generator gives me error:

Error: Cannot find symbol for node: Plugin

But with this it works 👍

{
  "compilationOptions": {
    "preferredConfigPath": "./tsconfig.json",
    "followSymlinks": true
  },
  "entries": [
    { "filePath": "src/index.ts", "outFile": "dist/dist/main.es.d.ts" },
    { "filePath": "lint.ts", "outFile": "dist/dist/lint.es.d.ts" }
  ]
}

The Plugin in the error message probably refers to the rollup's Plugin type that I meant to expose in the interface of my package.

Basically the dts-bundle-generator doesn't work with pnpm unless it follows symlinks and doesn't work with my monorepo setup, unless I pre-bundle the entire dependency tree manually first. If you insist that this is an ok behaviour for this product - it's unfortunate for me.

[zaripych]

@timocov

UPD: I've tried rollup-plugin-dts today to see if it fits my needs. It breaks my type definitions and also incapable of generating .d.ts directly from TypeScript source code. So far the forked/patched version of dts-bundle-generator is a winner. master...zaripych:dts-bundle-generator:master

I wonder if these somewhat naive code changes I've made (that basically make dts-bundle-generator tsc-compile anything it references on demand when needed) could be a start of a solution to support composite projects? Basically all we need is to call getDeclarationFiles for every referenced project, right?

Then I could just generate project references automatically from package.json just for the dts-bundle-generator. Or maybe even allow dts-bundle-generator to have a parameter similar to references in it's config.

[zaripych]

@timocov if we agree on a solution that would allow users to generate bundles via single command, whether it is explicit tsconfig.json with composite references or modified dts-config.json with references (which would be more robust and free people from having to generate tsconfig.json) I'd be very happy to contribute.

Currently I see the solution somewhat like this:

1. Load config with references (or multiple tsconfig.json)
2. Build a graph of reference projects via https://graphology.github.io/
3. Ensure no cycles, walk the graph topologically and ensure we have all .d.ts for all explicit dependencies; (OR)
4. Alternatively to two points above, still compile on demand like I did in my fork, because it's possible that not all dependencies would be used to generate the requested .d.ts - but only compile if the file belongs to a referenced project - this should safeguard us from trying to compile things in node_modules.
5. Done.

Thoughts on this solution:

1. The above would capitalise on on-demand in-memory .d.ts generation that already exists in dts-bundle-generator.
2. I'm not sure about adding a dependency to graphology and doing unconditional walk through entire graph and build everything - that could slow things down too much. I think we could use resolveModuleNames hook and build on demand only if a file belongs to a referenced project.
3. dts-bundle-generator already slow because it has on demand compilation, because now it compiles more projects - it might be slower - but I think it all comes down to how large the projects are
4. We could start supporting incremental builds and instead of generating in memory .d.ts allow TS to cache results

[timocov]

@zaripych sorry for late reply.

After reading this thread again I think that I'm not going to implement compiling files from the node_modules folder, this is not how the compiler works, this is not how the other tools are supposed to work and by adding this "feature" we might allow and "support" to use an incorrect and a kind of bad practice of developing and maintaining packages (think about this in wide, not your particular example that you'd like to solve). Once it is added, it won't be possible to remove it. This is just not right.

But the good news, it looks like the issue might be related to the way how the linking of packages via pnpm works and how pnpm links dependencies inside the node_modules folder and I think that this is something that the tool can add and maintain (and probably should, if there is a problem).

Is it possible for you to create a repro for your issue (with pnpm usage and getting Error: Cannot find symbol for node: Plugin error) so I can take a look what's is going on there? I think it would be much helpful to understand why exactly the current solution doesn't work for pnpm's workspaces.

---

Below some replies to your messages and questions.

I wonder if these somewhat naive code changes I've made (that basically make dts-bundle-generator tsc-compile anything it references on demand when needed) could be a start of a solution to support composite projects? Basically all we need is to call getDeclarationFiles for every referenced project, right?

For supporting composite projects there is an issue, see #93. But IIRC there were some issues with supporting this, because the compiler's API for composite projects is different it it might not be possible to get the output for everything. Also, there might be a question how to treat your composite parts - as an "externals" or "internals" (but it is possible that I'm messing it up with other tool https://github.com/timocov/ts-transformer-properties-rename).

dts-bundle-generator already slow because it has on demand compilation, because now it compiles more projects - it might be slower - but I think it all comes down to how large the projects are

If you run it against d.ts files, it won't compile any ts files and will use these files as it. It is not perfect and blazing fast, but at least something and if you already compiled your project, you don't need to compile it once again.

We could start supporting incremental builds and instead of generating in memory .d.ts allow TS to cache results

If you're talking about incremental compilation that the compiler supports this might be tricky. From what I've seen, the compiler doesn't store anything about the compilation besides the deps tree, last modified dates and some other meta information that helps the compiler to say "nothing has changed from the last compilation so nothing to do". But once you change a file, the compiler will compile the project anyway. So from my observations the compiler doesn't store or cache any results in that way.

The way how the compilation of composite projects works is that the compiler for a dependency doesn't use ts file but use .d.ts compiled file (that's why declaration: true is required for dependencies) (it is looking similar to what we were talking about above, isn't it? 😄).

On the other side, the tool requires the files to be compiler and type checked because the tool relies on that and this cannot be omitted.

[zaripych]

@timocov No worries and thank you for getting back to me. Appreciate it.

I think that I'm not going to implement compiling files from the node_modules folder

That makes total sense. I already got the vibe, so was trying to solve my problem somehow differently. I would rather not maintain a broken fork myself or let people use it and suffer later.

Just to clarify my end goal - is to reduce amount of configuration that we need to do in a monorepo. We don't necessarily need "compiling from node_modules folder" (which really sounds bad). I just need dts-bundle-generator to see other dependencies in a monorepo. To my view - we can accomplish this by declaring "baseUrl": "." and adding "path": { "depencendy": "../../dependency/src" } parameter and introducing TypeScript aliases that resolve outside directory of the package (in a monorepo, not node_modules). Another native option is to use composites, but both of these options require changing and maintaining tsconfig.json ... which leads to configuration bloat or having to regenerate and commit tsconfigs ... which will be my last resort, I guess.

it looks like the issue might be related to the way how the linking of packages via pnpm works

I think I highlighted in the above screenshot in this comment that TypeScript itself treats sub-package/keyValuePair.ts as non-external module. And that's where dts-bundle-generator will replace .ts suffix with .d.ts and then it fails to resolve - as it was never dts-compiled because it's outside of the input project.

I'm still not really sure about pnpm - maybe it's more about what the return value is from this internal TypeScript function - specifically resolvedModule.isExternalLibraryImport.

I think it would be much helpful to understand why exactly the current solution doesn't work for pnpm's workspaces.

As you pointed out - I'm doing it hacky way. I'm using "types": "src/index.ts". And if I point to .d.ts - it should just work. So if you still want to look at why this happens - I've created a branch for you here and repro steps:

git clone https://github.com/zaripych/repka.git -b dts-bundle-generator-repro
cd repka
npm install -g pnpm@7.5.0
pnpm install
pnpm run turbo run declarations --filter @tooling-tests/todo-list-cli --force

Have a look at ./packages/tooling-tests/todo-list-cli/src/store and the package it depends on ./packages/tooling-tests/todo-list-store.

---

On the side note - if dts-bundle-generator allowed to override tsconfig.json options, like rollup-plugin-dts allows it to:

https://github.com/Swatinem/rollup-plugin-dts/blob/master/src/index.ts#L22

which uses TypeScript native API to parse and load it - but then injects those options in:

https://github.com/Swatinem/rollup-plugin-dts/blob/master/src/program.ts#L58

Then I could have just generated aliases (via paths and baseUrl) dynamically without writing to tsconfig.json and pass them to dts-bundle-generator. Then I could just use the API of the generator with custom TypeScript options ...

[zaripych]

I was able to have a proof of concept working for the approach with specifying path compiler option. 👍 The changes I've made to the dts-bundle-generator here are only about how tsconfigs are loaded. We can polish them to your satisfaction - but I believe that it shouldn't burden you with any extra maintenance and should be completely backward compatible for all existing consumers.

Summary of changes:

1. Move compiling, saving and validating output from bin to the API surface - this allows users to consume the API and benefit the automatic validation of the generated output
2. Provide means for consumers to increase log output levels when using the API
3. Add TypeScript compiler options to CompilationOptions
4. Merge new compilerOptions config field with compiler options that TypeScript loads for us in get-compiler-options.ts

The API is used like so in my repository:
https://github.com/zaripych/repka/blob/6c2cd969aa7091b1745dee25079c6882e39890fb/packages/build-tools/ts/src/tsc/declarationsViaDtsBundleGenerator.ts#L34 - this is just a POC - I would be generating proper aliases from devDependencies and dependencies that match packages in monorepo.

UPD: Obviously the alternative to this is to generate tsconfig.dts.json ... specify paths there ... 😅 ... along with dts-config.json ... both ... then no changes are required for dts-bundle-generator ... but it's quite clunky.

[timocov]

@zaripych Apologies for late reply (again 🙁).

Add TypeScript compiler options to CompilationOptions

Have you looked into #137 ? Were able to find a proper types for the compiler options? I'd like to have compiler options in the options too, but we cannot expose internal interfaces, but public ones aren't provided in the types (at least from my understanding).

which leads to configuration bloat or having to regenerate and commit tsconfigs

Wait, don't you have tsconfigs in your repo committed? What is the main difference between your "normal" tsconfig and the one for dts-bundle-generator? Keeping in mind that you can use extends feature, having a few more options for the tool doesn't look like something bad from my experience (e.g. you might have composite projects but want to use the tool, so you extract "options" into a separate config file and then use extends to create another configs for composite parts and the tool; also, you might not want to have something enabled by default for your compilation, e.g. declaration or something else, but for the tool is it required).

On the side note - if dts-bundle-generator allowed to override tsconfig.json options, like rollup-plugin-dts allows it to:

I like this, but the difference is that you are are to write a config file in json file where you don't have an access to typescript module to specify a configuration (ts.CompilerOptions uses compiler's enumerations inside so it is not the same as you might specify in a tsconfig.json file.

So if you still want to look at why this happens - I've created a branch for you here and repro steps:

Is there anything changed there? I'm trying to run the commands you've provided but getting the error:

$ pnpm run turbo run declarations --filter @tooling-tests/todo-list-cli --force

> @repka-kit/repository@0.0.0-development turbo C:\projects\repka
> turbo "run" "declarations" "--filter" "@tooling-tests/todo-list-cli" "--force"

node:internal/errors:464
    ErrorCaptureStackTrace(err);
    ^

Error: spawn /C:/projects/repka/packages/build-tools/ts/node_modules/.bin/turbo ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19)
    at onErrorNT (node:internal/child_process:477:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -4058,
  code: 'ENOENT',
  syscall: 'spawn /C:/projects/repka/packages/build-tools/ts/node_modules/.bin/turbo',
  path: '/C:/projects/repka/packages/build-tools/ts/node_modules/.bin/turbo',
  spawnargs: [
    'run',
    'declarations',
    '--filter',
    '@tooling-tests/todo-list-cli',
    '--force'
  ]
}
 ELIFECYCLE  Command failed with exit code 1.

[zaripych]

No worries. I think I have a good solution - even though I maintain my own fork - the changes are in safe zone of just configuration reading and easy to merge with upstream. No rush at all - would appreciate it if we eventually land with something useable by everyone though 👍

Wait, don't you have tsconfigs in your repo committed? What is the main difference between your "normal" tsconfig and the one for dts-bundle-generator? Keeping in mind that you can use extends feature, having a few more options for the tool doesn't look like something bad from my experience (e.g. you might have composite projects but want to use the tool, so you extract "options" into a separate config file and then use extends to create another configs for composite parts and the tool; also, you might not want to have something enabled by default for your compilation, e.g. declaration or something else, but for the tool is it required).

The main difference is that there are no references or paths that developers have to maintain. The tsconfig.json is simple bare-bones that never changes and is there for VSCode to pickup type auto-completions and that's all. E.g. if I add a packageA into packageB dependencies (where packageA is one of the packages in monorepo), developers already have to change package.json in packageB - and that's automatic via pnpm add. Now in addition to that they will have to hardcode location of the packageA into tsconfig.json as well now. Multiply this by the number of tools each of which requires their own bit of tsconfig.json tweaking and we get ourselves a bunch of tsconfig.this.json and tsconfig.that.json with half developer already forgetting why the first one was needed. Multiply that by the number of packages in monorepo as well. People used to this kind of stuff ... but try to imagine all these little things in bigger repos getting out of hand. I'm trying to make things "just work".

It's all good - I can generate the part of tsconfig.json that I need - I just don't want commit it. I'm actually already generating it here:
https://github.com/zaripych/repka/blob/main/packages/build-tools/ts/src/tsc/declarationsViaDtsBundleGenerator.ts#L43

Have you looked into #137 ? Were able to find a proper types for the compiler options? I'd like to have compiler options in the options too, but we cannot expose internal interfaces, but public ones aren't provided in the types (at least from my understanding).

I see what you mean. That means my changes are actually maybe somewhat buggy. ts.CompilerOptions while unparsed in JSON file is not the same thing as after parsing. If we allow passing compilerOptions to override the standard options via API (e.g. in generateDtsBundle) - then the problem is solved - we don't need the type for JSON.

But if we also want to allow compilerOptions to be specified in BundlerConfig - then yeah - we don't have the official type, I think doing best effort and replacing enums with string is OK solution. Then that value can be passed in to TypeScript by overriding sys.readFile ... and let the TypeScript validate it. 👍 Those strings would nicely turn into enums as if they were read from file.

This means my current code has typing bugs I need to fix ...

Is there anything changed there? I'm trying to run the commands you've provided but getting the error:

This looks like dependencies weren't installed. Maybe because I'm using canary build of turbo. You can try changing it from canary to whatever is latest in packages/build-tools/ts/package.json - I will test this out on my windows machine and get back to you. 👍 I've never tried on windows yet. 😅

[timocov]

Ok I was able to run this on wsl, here is my observations:

• by default (if you don't change the config file) you get bunch errors like error TS2307: Cannot find module or its corresponding type declarations., by digging into the issue I found out that this is the behaviour of the compiler - it treats files in node_modules as something that shouldn't have an emit so it doesn't generate any output for them (that's what we've discussed above, I agree with the compiler and having a code that handles this correctly is kind of strange and I wouldn't like to do this). Even though I don't quite understand why these files are treated as files from node_modules, I'd expect them to be treated as local ones, because after resolve they don't have node_modules in their paths. But I think this is the questions to the compiler team:
  

• if you add followSymlinks: false to the config, then it warns with... the same issue actually but different errors - now the tool complains about files that are in the compilation but they are not d.ts ones. But later the building ended with an error Error: Cannot find symbol for node: Plugin. After debugging this problem I figured out, that for some reason the compiler decides to load types for jest package and it causes this error (it comes directly from its types). If you add "types": ["node"], to tsconfig for the tool the problem goes away. But then of course the compilation fails because in the declaration bundle it finds something that shouldn't be there, for example a function's body, and it comes directly from the warning.

So I'd say probably the best solution here is to use paths. Alternatively you can ask the compiler team why the compiler doesn't generate an output for files that were resolved from node_modules to a local folder (outside of node_modules; so basically the case is the following: node_modules/@internal-package/src/index.ts -> ../internal-package/src/index.ts, but the compiler treats this file as "comes from node_modules" and doesn't generate an output for it).

Another solution could be set up the right order of building packages and compile them first and then bundle the declarations.

[zaripych]

@timocov on the side note - I don't think it's just tsc here - it's how dts-bundle-generator setups it internally. If you try to run tsc --build tsconfig.json in the packages/tooling-tests/todo-list-cli - it works without errors. The only difference is that composite: false for the generator ...

[timocov]

it's how dts-bundle-generator setups it internally. If you try to run tsc --build tsconfig.json in the packages/tooling-tests/todo-list-cli - it works without errors. The only difference is that composite: false for the generator ...

Because it switches the compiler to another way of working, which is a build mode or know as composite projects (or referenced projects) that aren't supported right now (see #93).

[zaripych]

@timocov Thank you for your time so far! So would you be willing to look at PR for compilerOptions override? So I can approach this programmatically via paths?

[timocov]

@zaripych sorry for late reply. For #137 - if you will find a way to getting the types for compiler options from the compiler's types (so they would be up to date and related to the current compiler's version - that is installed locally) - feel free to share it and create a PR. Otherwise I don't think that it makes sense to have something like any or unknown there tbh.

As for this ticket, is there anything that left unresolved so far?