branching policy

[phaer]

We've seen quite a few regressions on main in the history of this project, and I don't think it will be feasible to test all platforms and subsystem for each commit anytime soon.

So the following snippet is a draft, which can be improved in many ways, but I'd like to propose it for discussion here. It's a PR so you can comment inline, feel free to use the suggestion feature or just leave a comment.

One thing that's missing here right now is a version scheme.
At least medium term i would try to circa match nixos releases, so do dream2nix-23.11 or so. We could do more regular releases, i.e. monthly as long as changes are more likely and we don't have to many v1-subsystems to test? Would that work?

[DavHau]

I like the direction of the proposal. Some thoughts:

We've seen quite a few regressions on main in the history of this project, and I don't think it will be feasible to test all platforms and subsystem for each commit anytime soon.

I think to prevent regressions in the future, the primary focus should lie on unit tests. Having a high test coverage should prevent the usual bugs that we fight with. These tests are also cheap enough to execute for every single commit, no compromises needed. Right now we are in a situation where we don't have a lot of unit tests (even none at all for nix code). I think we should come up with a strategy for how we can dramatically increase our test coverage quickly.

Once the unit tests are there, they introduce almost zero delays to our development workflow. Other means of fighting bugs, like for example, expensive integration tests, or async code reviews, or more complex branching models, do all increase delays, therefore focusing on unit tests should have the highest priority of all, I believe.

Changelogs

I think maintaining the changelog should be integrated into the development workflow instead of being done retrospectively. Whenever a noteworthy change is implemented, an entry should be added to the changelog right away. This gets us closer to the ideal where the current state is always releasable, making releasing almost a noop. We could do this by keeping an unreleased section on the top of the changelog.

Fast and Slow tests

It would be ideal if we can get rid of our slow tests and replace them with fast ones. I'm not saying that we should not do integration tests, but our current integration tests are unnecessarily huge. For example, we currently build large expensive packages like prettier. Those could be replaced with more minimal packages. Ideally different kinds of hello world projects for all languages / frameworks etc.

Assuming we don't get rid of slow tests, we can still run them continuously on main, just maybe not for every single commit. Quicker feedback loops are always better. If we break something, it's better if we learn about it as soon as possible instead of once every few weeks or months when releasing. We might want to have a policy, that if our tests ever fail, fixing that should be prioritized over any other work. Any fix of an integration tests should be delivered with a cheap test that tests for exactly that breakage (ideally a unit test). Doing that should gradually decrease our reliance on the slow tests and shift it towards the fast tests.

Even if we decide to build a large package set and have that as a release blocker, we should be possible to build that at least once a day. Probably even more often.

Releasing

I think we can release a lot more often than nixpkgs does. Potentially we could automate the release, to always release whenever all of our tests are green (including the slow ones, if there are any).
We could still have tags like nixos-22.11 to indicate our latest version compatible with that nixpkgs release.

Once we do stable releases, I think following branch for release might be a good option which seems to be common for trunk based style.

As long as we don't guarantee any API stability, then every release is like a bugfix release and there will never be the case where one would want to cherry-pick a bug fix to an existing release. The fix is simply a new release. Therefore, we could just release from trunk via tags.