Better support for monorepos

[wesleytodd]

I believe that for monorepos we can greatly improve the DX by building a matrix of the packages and running each in parallel in on it's own. I also think that we can run lint in the same way. The main benefit of this in actions is that you can see specifically which one failed without having to open the logs.

[ljharb]

In a monorepo, things should be run in topological order - only things that aren’t dependent on each other should run parallel. Unfortunately no workspace tools get this right yet.

[wesleytodd]

Technically turborepo does depending on the setup. That said, what I am talking about here is more about a UX affordance than doing it strictly "correctly". I could be convinced, but I think that most situations would be better served by just doing it in parallel in a matrix and letting app devs worry about the interdependency of their tests than have the "managed workflow" try and get too fancy.

[ljharb]

I mean, if we're letting app devs worry about it then they can do their own parallelization too :-p

[wesleytodd]

I was hoping they don't need to worry about it? My goal was to add a better baseline for monorepos. When I said "letting them worry about it" meant not trying to get fancy with topological order when 90% of test failures will not be because of an in-repo-dep failing in a hard to debug way. Just run it in parallel and that alone with satisfy the mass majority of cases.

[wesleytodd]

And I guess also lastly I should address that publish is the only step that actually benefits from topological ordering, and this does not cover that.

[ljharb]

Build steps do as well, as do if one of the monorepo packages is a dev tool that requires building, used in another one. It comes up quite a bit in my experience.

[wesleytodd]

Hm, this is a typescript thing isn't it? I really should probably adopt a TS monorepo and try to actually get it into a state I can stand working in so I catch these things earlier. Anyway, I was thinking that this repo should not have opinions on how folks build, but just run npm run test and have the repo configure how it wants to build. I guess though that does mean npm run -w pkg test would need to know how to build the whole monorepo, so maybe that is too tough to achieve in a non opinionated way.

[dominykas]

Is there even a common pattern in the ecosystem for how monorepo CI is supposed to work? Conventions, guidelines, etc? Most of the time I've heard devs say "just us nx" or smth along these lines, but that does not really do anything that's native to actions (i.e. it does not build a matrix - it just executes things in parallel).

And yeah, compiling TS ahead of the matrix and caching is a pre-requisite for any monorepo of decent size, by the looks of it. I don't know how monorepo people afford the Actions minutes 🤷‍♂️

While there's definitely lots of scope for discussions and collecting best practices, my gut is that code wise this might need to start as some experiments.

[wesleytodd]

No I dont think there is. That said, I think it would be fair to say that it is common enough that if we had a basic opinionated one that it might work for a lot of people.

my gut is that code wise this might need to start as some experiments.

I have a bunch of internal work which does this. It relies a bit on turborepo to make the magic happen, but that is easily swapped to something less opinionated. Maybe I should work toward a state where I can open source some of that and then we can discuss with that as an example.