[Tracking] Use Changesets to manage releases

[vanessayuenn]

Tracking progress for the implementation of #22182

Main Deliverables

Beta Give feedback

1. Create prototype repository to implement trial Changesets setup - https://github.com/storybookjs/monorepo-release-tooling-prototype
2. Add Changesets CLI
3. Add Changesets GitHub Action
4. Create release command to that compiles and publishes
5. Add Changeset GitHub bot Currently doesn't work because of feat: check for changed files not only in root folder changesets/bot#71
6. Define the prerelease flow
7. Define the setup and process needed to be able to backport fixes to previous minor and major releases
8. Define how to ensure only the core team can cut releases
9. Fix addon-storyshots-X causing changesets to require a major version bump: Add a new bump strategy for peers bump changesets/changesets#1132
10. Format main changelog, and remove them from packages
11. Investigate how this can interface with GH environments and environment secrets https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#about-environments
12. Fix the issue with the Changeset Bot. Maybe forking this idea: Consider moving the bot back to Github Actions changesets/bot#54 (comment)
13. Migrate script to automatically cherry pick PRs with patch label
14. Migrate script to automatically update versions.ts in CLI
15. Set up canary releases for all PRs
16. Investigate how to go from 8.0.0-alpha.6 to 8.0.0-beta.0
17. A video tour of the workflow from A-Z that clearly demonstrates how we work with it, to gather feedback on if we want to change the main workflow
18. Update Contribution documentation detailing the new workflow

Migrating prototype to monorepo

Beta Give feedback

1. Add Changesets CLI, configuration, bot, GitHub Action and anything else necessary to the monorepo with a PR to next.
2. Remove all the deprecated bits that aren't necessary for releases anymore.
3. Merge to next and hopefully see the "Version Packages" PR
4. Merge the "Version Packages" PR and hopefully trigger a new 7.1.0-alpha version release
5. Ensure monitoring, status page and scheduled CI still works with the branches

[JReinhold]

Progress update

Summary

🎉

• The CLI is working with releases, versions and publishing
• We've defined a workflow for prereleases that works in our current branching setup
• We've defined a workflow for backporting patches to older releases

🤔

• addon-storyshots* is causing trouble
• Prereleases bump versions in a suboptimal way - alpha.8 -> beta.9
• We need to port over the existing script for semi-automatically cherry-picking fixes back to older versions
• The Changeset bot is not working in our use case, we want to fork it

Tasks

Prototype repo

We've created a prototype repo where we're experimenting with all this stuff at https://github.com/storybookjs/monorepo-release-tooling-prototype . We forked the SB monorepo and renamed all packages from @storybook/X to @junk-temporary-prototype/X - which was just as hard as you'd imagine. 😰

It works on the main, new-next and release-7.0 branches. I used new-next instead of next so I didn't have to do the whole package renaming twice - this won't be a limitation in the final solution. 💪

Changesets CLI

Everything is up and running, correctly creating versions, releases and publishing. 🎉

Changesets Action

Also works, creating correct "Version Packages" PRs, and when merged, releases new versions. 🎉

Prereleases

I've tried out a workflow that supports our current branching with main and next, and it works. 🎉 Here's the high level workflows - which will be documented in more detail later:

Starting a prerelease

We'll do this when we release eg. 7.1.0 and we want to start working on 7.2.0-alpha.0 from next.

1. Be on next branch, in code dir
2. yarn changeset pre enter alpha
3. in code/.changeset/config.json: set baseBranch to next if it isn't already
4. Commit and push (to next or in a PR, doesn't matter)

Promoting from one prerelease to another - eg. alpha to beta

1. Be on next branch (or in a PR), in code dir
2. yarn changeset pre exit
3. yarn changeset pre enter beta
4. There needs to be changes to actually release for beta.0. There most likely will be, otherwise create anything with yarn changeset add choose all or any package
5. Commit and push
6. Merge "Version Packages" PR

See limitation in Current obstacles section below.

Promoting from prerelease to stable

1. Be on next branch (or in a PR), in code dir
2. yarn changeset pre exit
3. Commit and push
4. Merge "Version Packages" branch
5. Merge next to main
6. On main branch: in code/.changeset/config.json: set baseBranch back to main
7. On next branch, enter pre mode again as described above.

Releasing patches to older major/minors

I also have a workflow here that works 🎉 - but I'm unsure how compatible it is with our current branching strategy since I don't know much about that for backports.
High level:

Backport patch from 7.1-alpha to 7.0:

Say we have released 7.0.7 on main and 7.1.0-alpha.2 on next.

1. Be on main
2. Cherry-pick merge commit of the patch - the commit that merged the patch into next. It should contain the full feature and a matching changeset.
3. Make sure the changeset is set to a patch bomp. If it is not, change it, and commit
4. push
5. Merge “Version Packages” to main, will release 7.0.8.

Backport patch from 7.1 to 7.0:

Say we have released 7.0.7 and 7.1.0 on main and 7.2.0-alpha.8 on next.

1. Checkout the tag for 7.0.7 - should be an old commit on the main branch
2. Create a new branch from there, named release-7.0
3. On that branch, cherry pick the merge commit from the patch. It should include a changeset
4. Make sure the changeset is set to patch. If it is not, change it, and commit
5. push
6. Merge “Version Packages” to release-7.0, will release 7.0.8.

The release-7.0 branch can contiously be used to release new patches to 7.0.

Current obstacles

addon-storyshots* causing issues

For some reason, Changesets always insists on doing a major bump for addon-storyshots and addon-storyshots-puppeteer, even though they don't have any changesets associated. And given all our packages are fixed together, this causes all packages to get a major version bump.
We're currently working around this by ignoring those two packages, but we need to figure out a permanent fix for this.

Promoting from alpha to beta

I've confirmed a limitation that we initially knew about, the fact that bumping from one prerelease tag to another will not recount the release from 0. So if we want to promote 7.1.0-alpha.16 to beta, it will become 7.1.0-beta.17 and not 7.1.0-beta.0. This is a known issue changesets/changesets#933, with a workaround changesets/changesets#933 (comment) that we should investigate. I also want to investigate how other OSS projects get around this.

Automatic cherry-picking fixes

The current strategy for backporting fixes is to label PRs with "patch". We then have a script that picks up all these PRs, cherry picks them, releases them to the old version and adds a "picked" label.
This is a semi-manual process, because roughly 25 % of cherry picks have merge conflicts that needs to be resolved.

We want to port this workflow + script over to this new release strategy, because I think it's the best we can do.

Changesets bot only works in root

The Changesets bot is used to comment on PRs to tell if the PR has a changeset or not, which is valuable to both new contributors that doesn't understand the process, and to maintainers to quickly add missing changesets.
However it only works of changesets are stored in the root .changeset/ dir, but ours are in code/.changeset/. So for us, it will never detect changesets, rendering it useless. This is a reported limitation changesets/bot#60 with a PR open changesets/bot#71, but it's unclear when that PR will get merged, if ever.

We're considering forking the bot, to make it work better for our use case. Not only forking it, but re-implementing it as a GitHub Action instead (which it was initially), because that would make it easier to customize its behavior - and Actions are generally easier to develop on than bots. This may sound like a lot of work, but the bot/action is only about 100 LOC, so it's not crazy, and I think it would help us a lot.
Maybe our fork could get released back upstream at some point.

[JReinhold]

I've investigated how to lock down the release pipeline, to ensure no bad actors can release anything without approval from core. Here's the solution:

By experimenting with GitHub Environment protection and some clever hacks, I've managed to make it work in a very secure way.

• The NPM token will only be available in a GitHub Environment called release
• That release environment will only be accessible from actions running on the main, next and release-* branches
• When a "Version Packages" PR is merged to any of those branches (to publish a new release), the release GH action will enter that release environment - no other PRs or commits will enter that environment.
• Before the action starts, it will have to be approved manually by anyone from the core team

This means that:

1. No random GitHub Action has access to the NPM token, only those actions running on the correct branches
2. Even if any of our outside contributors merge in a PR to next in bad faith (eg. to extract the token), a core member still have to approve the action running, which they won't.

🎉🔒

[JReinhold]

Changelogs and GitHub Releases

I've been exploring Changesets' ability to format changelogs and GitHub releases. Our "one release to rule them all (91 packages)" approach to releasing doesn't work very well with Changesets in this regard.

Changelog

Changesets only supports writing a changelog per package, in the package's directory. I don't think that's very useful to us given our versioning strategy - but it's also not totally out of the question. It would make it easier for us to see which changes and versions affected which packages, but it would make it harder for users to get a clear overview of what is changing between each Storybook version. Changesets do support basic customization of the changelog, but only on a per-line basis, and not how or where the changelog is written.

There is currently a PR open changesets/changesets#1062 to support producing a single changelog for a group of packages, and because the author (thanks @VirtuaBoza!) published it on npm I've been playing around with it. I think it does an okay job of creating a nice output, as seen here: https://github.com/storybookjs/monorepo-release-tooling-prototype/blob/new-next/code/CHANGELOG.md
It does have downsides however:

1. A minor thing, but there's a limitation in Changesets that means it can't write to CHANGELOG.md in the root, but only in code/CHANGELOG.md, which is annoying.
2. As the PR currently isn't merged, we're relying on an untested, unmerged feature with @abizzle/changesets-cli@2.27.0-next.4

Point 2 makes me uncomfortable, I don't think we want that as part of our main release pipeline. It is a promising PR however.

That leaves us with two options:

1. Use the default mechanism and write separate changelogs per package
2. disable Changesets' changelog mechanism and roll our own

I'm leaning towards option 2 - I'm hopeful that we can leverage Changesets' internals (they are neatly published as separate packages) to generate changelogs based on changesets, maybe using our current release-script as a starting point.

GitHub releases

There is no way to make Changesets generate a single GutHub release, it will generate separate releases for each package published. As you can see here, that is not an option for us at all: https://github.com/storybookjs/monorepo-release-tooling-prototype/releases

We need to disable Changesets' GitHub release generation (it's an option in the GitHub Action), and generate our own as part of the process. Again, I'm hopeful we can piggy bag off of the changelog generated above, and just add a subset of that to a GitHub release.

Is Changesets still the best option?

These obstacles here makes it valid to ask ourselves, if Changesets is still the best tool for us, given our rather unique monorepo setup. We're effectively:

1. Disabling Changesets' changelog and GH release generation
2. Disabling individual package releases
3. Forking the bot

So we're left with:

1. Using Changesets' to have a clear process for how changes are documented (adding changesets to every PR)
2. A possibility to write clearer and more elaborate changelogs, because changesets support any markdown, and is not just a single line PR title
3. A setup for a maybe/maybe-never future where packages are versioned individually
4. From a contributor's perspective, a release/changelog workflow recognized in the ecosystem

There is a universe where we abandon the idea of Changesets now and then roll our own release tool based on the existing scripts - or find another tool. I personally still think Changesets is the right setup for us given the gains listed above, but I'm very open to have that discussion. I'm not married to a tool, I want the best solution for us.

[JReinhold]

We've concluded that trying to integrate Changesets into our pipeline isn't feasible at the moment, for a few key reasons:

1. Changesets is primarily built to release and version packages indivudually, where we always release all packages at the same time, with the same version.
2. It doesn't support aggregating changelogs and GH releases, meaning it will default to an entry per package, which in our case would create 90 releases every time we released a new version. We would have to rebuild the changelog functionality ourselves.
3. The GitHub bot doesn't support .changesets dir not in the root directory. This is a very minor bug, but a blocker for adopting it at the moment.
4. Even with changesets, we would still need a few custom scripts to support backporting releases and more.

In summary, we would need to disable most of Changesets core functionality to make it work for us, which would mean that we wouldn't gain as much from using it.

I'm in the process of rethinking the release tooling proposal, merging concepts from Changesets (like release PRs) with our existing process and scripts. Closing this tracking issue, will update the RFC and create a new Tracking issue.

[JReinhold]

Superseded by #22532