Release schedule

[romanlutz]

One thing that @riedgar-ms and I have been discussing for a while is the release schedule. Right now we don’t have a regular schedule (e.g., every 2 weeks), but rather release whenever there’s “something worth releasing” (which remains to be clearly defined).

Concretely, Richard is adding support for the control features in mitigation right now ( #631 ), so once that’s checked in we could theoretically create another release. It would be nice to get an idea of what everyone considers to be release criteria.

For me any additional feature or bugfix that could be useful to users is sufficient, as long as that doesn’t mean we release more than once a week. Of course, quality needs to be there, but that should be true even for every single PR. Additionally, we wouldn't want a lot of breaking changes, so if it's possible to avoid them by slowing releases down a little bit and consolidating changes (e.g., release only once a month as opposed to bi-weekly) then I'm all for that. [Note that we have a page that describes how to migrate from one version to the next ( https://fairlearn.github.io/master/user_guide/migrating_versions/index.html ).]

@MiroDudik @adrinjalali @hildeweerts @riedgar-ms

[hildeweerts]

Do you know how other OS projects do this? A monthly release seems very reasonable to me, although I suppose you'd like to get the new release out ASAP for bug fixes. But I have zero experience in this area, so very curious to hear other people's thoughts.

[riedgar-ms]

Different projects have different release schedules - I'm sure that @adrinjalali can fill us in about the reasoning behind SciKit-Learn's one :-)

Doing regular releases is good, since it means that you're testing the release process frequently (I was very nervous about the v0.5.0 release, since I'd created the pipeline four or five months before, and it hadn't been used for a Prod release). It also reduces pressure to get something in for a particular release - if you miss the cutoff, then you know how much delay there will be (and it won't be very long). The flip side of that is that releases don't wait for 'one last fix' either. Emergency bugfixes would be outside this, but hopefully we'd keep those to a minimum (if not, then we need to rethink our testing process).

I think that fortnightly releases would be excessive though - I agree with @hildeweerts that monthly feels more reasonable.

[adrinjalali]

We may be okay with a monthly release now that things are changing fast, but in general that sounds way too often to me. On the scikit-learn side we try to do a release per 6 months, and the minor releases are whenever there's a few bug fixes which are worth pushing out fast. That means the first minor release usually goes out a few weeks after the first release, since things come up that we hadn't tested for, and the next ones can be after two months or so.

In the long run, I don't think we really need to be pushing out releases too frequently once the project is at a stable state.

[romanlutz]

[Wanted to respond to all of these comments in one response, but since everyone created a new thread I can't :-( ]

All of this makes sense to me. We should acknowledge that we're nowhere near as stable & mature as scikit-learn at the moment, so more often is certainly a good way to get new functionality to people to try out and provide feedback (as long as it's not excessive and everything breaks from version to version).

One thing I was curious about is how people feel about releases "on a schedule". Our friends over at interpret-community are doing bi-weekly releases (unless there were no changes) even if there's only been a tiny change in that period. Not sure how I feel about that despite the advantages @riedgar-ms mentioned in his thread.

Getting bug fixes out, as @hildeweerts mentions, should always be possible since we can simply cut a branch off of that particular release and do a release with the fix. I think that applies in any of the potential release modes.

[hildeweerts]

Maybe I've just not given it enough thought, but IMO there's no need to spend too much mental energy on this. I'd suggest to just try the monthly release and evaluate after a few months (3?) how we feel about it.

[romanlutz]

Related to releases: as we accumulate more and more the webpage versioning gets crowded and the build takes a longer time. I'd vote for removing old versions where there are multiple patch versions of the same minor. For example, right now we have

and I'd propose to do the same as scikit-learn and cut out 0.6.0 and 0.6.1, but keep 0.6.2 (the latest patch version within minor 0.6).

Eventually we can also discuss getting rid of old versions, but I'd wait a while for that. 0.4.6 is almost exactly a year old, but perhaps at the 2 or 3 year mark we can remove versions from the documentation. Wdyt @fairlearn/fairlearn-maintainers ?

[hildeweerts]

Sounds good to me!

[riedgar-ms]

There are two separate issues here:

1. What our support policy is for old versions?
2. We should improve our build system for doc versions

Only supporting the latest patch is a reasonable approach, but in general, we shouldn't let (2) drive(1).

[romanlutz]

I agree that the build system is a problem but why shouldn't it drive the decision on which versions to show? It's a real constraint, and I doubt we'll have anyone spend the considerable time to fix the build system anytime soon.

[riedgar-ms]

Hence "only supporting the latest patch is a reasonable approach, but in general, ..."