Caching for ts-node (deprecation of typescript-cached-transpile and usage of swc)

[CMCDragonkai]

Specification

Currently we use a third party package https://www.npmjs.com/package/typescript-cached-transpile to do caching for our ts-node. This is important in PK tests because we test alot of program executions of polykey and without caching the compilation, the test times blow out.

However this third party package isn't very stable and it has become out of date with respect to the new ts-node release. This caused some timeout problems in MatrixAI/Polykey#379. For now we had to fix our ts-node version to exactly 10.7.0.

Upstream ts-node has an official caching solution TypeStrong/ts-node#1364, that we should switch to when it is available as that will be more stable.

At the same time, we should explore the usage of swc instead of tsc. Note that swc is only transpilation, not type checking. The tsc is still needed when checking types. However our current usage of tsc in ts-node is to ignore type checking and only do transpilation anyway, so it should be fine for this usecase. The only issue would be automatic conversion from tsconfig to swcrc... It could also be used for jest tests itself, however I think jest testing is fine to keep using tsc.

Either way, it's not maintainable to keep using typescript-cached-transpile.

Additional context

• Disk cache prototype TypeStrong/ts-node#1364
• ci: merge staging to master Polykey#379 (comment)

Tasks

1. Try out swc and see how fast it is
2. See how swc can be integrated
3. See what happens if swc is only used by ts-node, we don't want to maintain another .swcrc
4. Consider incremental typescript compilation build mode with project references

@emmacasolin

[CMCDragonkai]

I've tested out swc. It is indeed quite quick. But the main reason is that it doesn't do any typechecking. So when it's only doing transpiling, it's able to do it in 20ms.

Now if we tell tsc to only do typechecking, and only emit type declaration files with emitDeclarationOnly: true, this doesn't actually save any time. It takes 6 seconds to compile everything, it still takes 6 seconds after not requiring emitting any JS code.

So you don't really save any time when doing a proper build. You still want to check the types, and you still want to emit the type declaration files, which swc cannot do.

The only place where it would save time, is 2 places:

1. Where ts-node is used, it may be better to avoid any typechecking in the first place, and thus swc makes sense here.
2. Where jest is used, it may be better to not require type checking when running jest. This is not such a gain due to jest caching, which is likely to cached compiled typescript code. At the same time, when running tests, it may actually be a good idea for the types to be checked.

The benefits of running ts-node faster is great though... but it does require additional configuration and additional dependencies. Without a usable typescript-cached-transpile, this can be quite problematic.

I'm not convinced bringing in swc right now would be a great benefit. The usage of ts-node is sufficient now with the caching. Furthermore, there's already an option applied to ts-node to only transpile, and not type check, and even then it's still slow.

If ts-node is an issue, and we have to upgrade it which makes typescript-cached-transpile not work, then we may need to bring in swc...

If we can use swc but only because of ts-node... and not require us to maintain .swcrc or any swc-specific configuration, then we may be able to do this: TypeStrong/ts-node#908 (comment)

---

In another world, the tsc incremental compilation could work without needing us to delete the dist directory. It would maintain correctness and ensure any files deleted is also deleted in the target dist directory. It looks like this is actually possible with a new mode of tsc: https://www.typescriptlang.org/docs/handbook/project-references.html#tsc--b-commandline