Improve how Typescript compiles across the monorepo #1085

JaapRood · 2021-08-05T14:58:01Z

Why

Optic is structured as a monorepo, with multiple interdependent packages individually published and built in one repo. However, making changes across Typescript packages has been very painful so far, basically sourcing from every package being compiled in isolation by a separate Typescript instance.

Generically, when working on package A, that depends on B, which depends on C, making a change in one of the dependencies doesn't automatically propagate up to package A: you have to manually make sure everything is compiled in the right order.

Often, to make sure things are done correctly, you end up rebuilding everything, which is super slow. And even then you have to reload the VSC window, or restart the Typescript language server.

What

In essence, we're connecting the isolated packages and making the Typescript compiler aware of dependencies living locally and are compilable as well.

It should enable :

Compiling all Typescript from the project root
Watching and compiling all the Typescript from the project root
Compiling a single package with all it's dependencies auto-compiling when necessary
Support TypeScript tooling across packages (e.g. Go to definition, refactor, etc.)
Allow for source maps debugging across all packages
Shared configurations and setup to consolidate setup.
making ui-v2 webpack-dev-server automatically recompile when one of the dependencies changes (might not work out through a lack of support: Need support for Typescript Project references facebook/create-react-app#6799)

It manages to do this through:

Typescript Project References, providing the links between packages.
TypeScript Path Mapping, to allow use to import using @useoptic/*, while allowing the compiler to still find the local versions it could compile.

It adds a task:

workspaces:watch, to watch the source of all Typescript and WASM targets and recompile them automatically.

Additional todos:

Reconsider building from root with one TSC instance for workspaces:build (possibly more efficient): not necessary. As all Typescript packages are now connected, even building them one by one causes incremental builds, only recompiling what's necessary.

Validation

CI passes
All tasks around compiling packages work in various states

niclim

🎉

JaapRood · 2021-08-06T10:46:39Z

Taskfile.yml

+  workspaces:watch:
+    desc: Watch source files and build as they change
+    deps: [workspaces:watch:typescript]
+    cmds:
+      - echo "Watching ended"
+
+  workspaces:watch:typescript:
+    desc: Watch all the source of Typescripts workspaces and automatically recompile (with a single watcher)
+    cmds:
+      - yarn tsc -b --watch
+
+  workspaces:watch:wasm:
+    desc: Watch the source of WASM targets and automatically recompile
+    cmds:
+      - yarn wsrun --parallel --exclude-missing ws:watch-wasm
+


@notnmeyer this is the best way I could imagine doing watchers with Taskfile. Do you see any problems with this / know of a better way?

sorry, been out. this looks fine to me though. i dont think its a better solution in this case, but taskfile also supports fingerprinting source files and watching a task too,

https://github.com/go-task/task/blob/master/docs/usage.md#watch-tasks
https://github.com/go-task/task/blob/master/docs/usage.md#by-fingerprinting-locally-generated-files-and-their-sources

JaapRood · 2021-08-06T10:52:40Z

I've added watching of all the assets, except for optic-engine-native, where changes to the underlying Rust code don't trigger any recompiles up the chain (and aren't necessary for the Javascript).

Cargo doesn't have any watching abilities, and while the cargo-watch exists, the preferred method is to run your development binaries through cargo run instead of compiling and calling the binary manually. The latter would actually be the preferred and simplest way to make sure optic-engine-native always runs against a up-to-date binary in development, and prevents us from having to watch the optic-engine Rust code twice (already happens for optic-engine-wasm).

However, that would touch the production code path for spawning optic-engine-native, so that's better done in a separate PR with separate review and testing.

JaapRood · 2021-08-06T10:55:22Z

workspaces/optic-engine-wasm/scripts/watch.js

+// We provide a mock for the webpack compiler, so we don't have to reimplement all the watching and can
+// leverage the existing plugin.
+
+let beforeCompileHook;
+let thisCompilationHook;
+
+const compiler = {
+  options: {
+    mode: 'development',
+  },
+
+  watchMode: true,
+
+  hooks: {
+    beforeCompile: {
+      tapPromise(pluginName, hook) {
+        beforeCompileHook = hook;
+      },
+    },
+    thisCompilation: {
+      tap(pluginName, hook) {
+        thisCompilationHook = hook;
+      },
+    },
+  },
+};
+
+let wasmPack = createWasmPackPlugin();
+
+wasmPack.apply(compiler);
+
+beforeCompileHook().catch((err) => {
+  throw err;
+});


How hacky is this? Pretty hacky! But given how incredibly small the integration between the wasm-pack watcher and webpack is, there being no alternatives and the complexity of forking / writing it ourselves being more complicated than this, I say it's worth it (for now, at least).

…to local workspace packages

…k dependencies

…pescript packages efficiently

…thing

niclim · 2021-08-09T13:49:20Z

Seems reasonable - watching on optic-engine-native would be nice, but this solves a large painpoint with the current TS compilations

JaapRood · 2021-08-10T08:47:54Z

Developed against this for two days, all seems to working smoothly, passing CI as well. Lets get this out to develop!

JaapRood · 2021-08-10T09:43:26Z

@LouManglass these changes are definitely an added risk for the next release train. Perhaps we should get a side channel build out there to verify it's all functional?

LouManglass · 2021-08-10T13:01:50Z

@LouManglass these changes are definitely an added risk for the next release train. Perhaps we should get a side channel build out there to verify it's all functional?

Yeah, I'll check that today. Better to find out sooner.

JaapRood requested a review from devdoshi as a code owner August 5, 2021 14:58

JaapRood requested a review from niclim August 5, 2021 14:58

github-actions bot added the chore label Aug 5, 2021

niclim approved these changes Aug 5, 2021

View reviewed changes

JaapRood force-pushed the chore/monorepo-typescript branch from 54b8f74 to e1d06f3 Compare August 6, 2021 10:45

JaapRood commented Aug 6, 2021

View reviewed changes

JaapRood requested a review from niclim August 6, 2021 10:55

JaapRood added 10 commits August 9, 2021 11:14

Setup new TS base file at root, including path mapping for @useoptic …

a6a0ee5

…to local workspace packages

Add project references to all typescript packages to allow TS to trac…

a453e34

…k dependencies

Add central tsconfig.json for workspaces to conveniently build all ty…

9cfe267

…pescript packages efficiently

Annotate some Promises that didn't strictly specify resolving with no…

d9aab30

…thing

Extend the base config, not the project wide config

5aad4e4

Remove the incremental ts stores when cleaning up a workspace

3c8ee27

Fix compiling ui-v2 from scratch

3be4d2c

Allow building of all typescript from root

7e4fe0b

Add workspaces:watch and watch all typescript with a single instance

31bc2de

Implement watching for optic-engine-wasm

53c4d4f

JaapRood force-pushed the chore/monorepo-typescript branch from e1d06f3 to 53c4d4f Compare August 9, 2021 09:14

JaapRood merged commit bba68be into develop Aug 10, 2021

JaapRood deleted the chore/monorepo-typescript branch August 10, 2021 08:47

JaapRood mentioned this pull request Aug 10, 2021

Fix Typescript compilation for ui-v2 #1094

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve how Typescript compiles across the monorepo #1085

Improve how Typescript compiles across the monorepo #1085

JaapRood commented Aug 5, 2021 •

edited

niclim left a comment

JaapRood Aug 6, 2021

notnmeyer Aug 10, 2021

JaapRood commented Aug 6, 2021

JaapRood Aug 6, 2021 •

edited

niclim commented Aug 9, 2021

JaapRood commented Aug 10, 2021

JaapRood commented Aug 10, 2021

LouManglass commented Aug 10, 2021

Improve how Typescript compiles across the monorepo #1085

Improve how Typescript compiles across the monorepo #1085

Conversation

JaapRood commented Aug 5, 2021 • edited

Why

What

Validation

niclim left a comment

Choose a reason for hiding this comment

JaapRood Aug 6, 2021

Choose a reason for hiding this comment

notnmeyer Aug 10, 2021

Choose a reason for hiding this comment

JaapRood commented Aug 6, 2021

JaapRood Aug 6, 2021 • edited

Choose a reason for hiding this comment

niclim commented Aug 9, 2021

JaapRood commented Aug 10, 2021

JaapRood commented Aug 10, 2021

LouManglass commented Aug 10, 2021

JaapRood commented Aug 5, 2021 •

edited

JaapRood Aug 6, 2021 •

edited