Release Celery v5.3.5

[auvipy]

Celery Patch Release : v5.3.5

This issue will summarize the status and discussion in preparation for the new release. It will be used to track the progress of the release and to ensure that all the necessary steps are taken. It will serve as a checklist for the release and will be used to communicate the status of the release to the community.

⚠️ Warning: The release checklist is a living document. It will be updated as the release progresses. Please check back often to ensure that you are up to date with the latest information.

Checklist

• Codebase Stability
• Breaking Changes Validation
• Compile Changelog
• Release
• Release Announcement

Release Details

The release manager is responsible for completing the release end-to-end ensuring that all the necessary steps are taken and that the release is completed in a timely manner. This is usually the owner of the release issue but may be assigned to a different maintainer if necessary.

• Release Manager: Asif Saif Uddin
• Release Date: 10.11.2023
• Release Branch: main

Release Steps

The release manager is expected to execute the checklist below. The release manager is also responsible for ensuring that the checklist is updated as the release progresses. Any changes or issues should be communicated under this issue for centralized tracking.

1. Codebase Stability

• The main branch build passes

  

Release Blockers

• added python 3.12 to the CI and drop py3.7 py-amqp#423
• added Python 3.12 in the CI kombu#1812
• test initial support of python 3.12.x #8549
• intial schema migrations setup for sqlalchemy models  #8532
• Release https://github.com/celery/vine
• Release https://github.com/celery/py-amqp
• Release https://github.com/celery/billiard
• Release https://github.com/celery/kombu
• updated new versions to fix CI #8607
• Fixed CI Support with Python 3.12 #8611

2. Breaking Changes Validation

A patch release should not contain any breaking changes. The release manager is responsible for reviewing all of the merged PRs since the last release to ensure that there are no breaking changes. If there are any breaking changes, the release manager should discuss with the maintainers to determine the best course of action if an obvious solution is not apparent.

3. Compile Changelog

The release changelog is set in two different places:

1. The Changelog.rst that uses the RST format.
2. The GitHub Release auto-generated changelog that uses the Markdown format. This is auto-generated by the GitHub Draft Release UI.

⚠️ Warning: The pre-commit changes should not be included in the changelog.

To generate the changelog automatically, draft a new release on GitHub using a fake new version tag for the automatic changelog generation. Notice the actual tag creation is done on publish so we can use that to generate the changelog and then delete the draft release without publishing it thus avoiding creating a new tag.

• Create a new tag

• Generate Markdown release notes

• Copy the generated release notes.

• Delete the draft release without publishing it.

3.1 Changelog.rst

Once you have the actual changes, you need to convert it to rst format and add it to the Changelog.rst file. The new version block needs to follow the following format:

.. _version-x.y.z:

x.y.z
=====

:release-date: YYYY-MM-DD HH:MM P.M/A.M TimeZone
:release-by: Release Manager Name

Changes list in RST format.

These changes will reflect in the Change history section of the documentation.

3.2 Changelog PR

The changes to the Changelog.rst file should be submitted as a PR. This will PR should be the last merged PR before the release.

4. Release

4.1 Prepare releasing environment

Before moving forward with the release, the release manager should ensure that bumpversion and twine are installed. These are required to publish the release.

4.2 Bump version

The release manager should bump the version using the following command:

bumpversion patch

The changes should be pushed directly to main by the release manager.

At this point, the git log should appear somewhat similar to this:

commit XXX (HEAD -> main, tag: vX.Y.Z, upstream/main, origin/main)
Author: Release Manager
Date:   YYY

    Bump version: a.b.c → x.y.z

commit XXX
Author: Release Manager
Date:   YYY

    Added changelog for vX.Y.Z (#1234)

If everything looks good, the bump version commit can be directly pushed to main:

git push origin main --tags

4.3 Publish release to PyPI

The release manager should publish the release to PyPI using the following commands running under the root directory of the repository:

python setup.py clean build sdist bdist_wheel

If the build is successful, the release manager should publish the release to PyPI using the following command:

twine upload dist/celery-X.Y.Z*

⚠️ Warning: The release manager should double check that the release details are correct (project/version) before publishing the release to PyPI.

⚠️ Critical Reminder: Should the released package prove to be faulty or need retraction for any reason, do not delete it from PyPI. The appropriate course of action is to "yank" the release.

Release Announcement

After the release is published, the release manager should create a new GitHub Release and set it as the latest release.

Add Release Notes

On a per-case basis, the release manager may also attach an additional release note to the auto-generated release notes. This is usually done when there are important changes that are not reflected in the auto-generated release notes.

OpenCollective Update

After successfully publishing the new release, the release manager is responsible for announcing it on the project's OpenCollective page. This is to engage with the community and keep backers and sponsors in the loop.

Update v5.3 for back port

Update the https://github.com/celery/celery/tree/v5.3 branch after release from main branch and in future back port only hot fix or bug fix releases.

[Nusnus]

@auvipy hey there, any updates on the release 5.3.5?

Updates on my part:
Approximately by the end of November I plan to get the pytest-celery v1.0.0 released including full code/Sphinx documentation. I already fixed the read-the-docs to not break in the CI with autogenerated empty docs (merged to main already).

By then I'll finish the initial batch of the smoke tests as well.

After you finish releasing this version I'll create an issue for the 5.4 release to communicate well the advancement to the upgraded testing infrastructure and we can also merge waiting PRs (if there are any) that were waiting for the patch release first.

[auvipy]

I forgot to update the release date! I added python 3.12 to some dependencies+ some tests needed to fix in the dependencies. 1 in kombu. then CI in celery with python 3.12 will look more decent. got caught with some high prio personal issues so got less time for celery. but will clear them up within next 10 days hopefulllyy

[ThiefMaster]

Hopefully everything's fine again!

(If yes, any news on the release? :))

[Nusnus]

@auvipy is there any way I can assist moving forward with the release?

I can spare some time to help, let me know what I can do 🙏

[auvipy]

I am back and started with pending release of vine 5.1.0. we need to check/fix the remaining 1 integration failure in kombu with python 3.12

[Nusnus]

I am back and started with pending release of vine 5.1.0. we need to check/fix the 1 integration failure in kombu with python 3.12

Awesome!
Tag me where you need help, I'm at your service @auvipy

[auvipy]

two more package released! next I will try to fix the integration test of kombu on python 3.12

[auvipy]

with the release of python 3.12 supported core dependencies, the Celery CI is away from only one test assertion fix. after that we will see if any integration tests fail or not. #8607

[auvipy]

I will open a PR for changelog update today for review hopefully. thanks a lot @Nusnus for fixing the remaining test failure.

[Nusnus]

I will open a PR for changelog update today for review hopefully. thanks a lot @Nusnus for fixing the remaining test failure.

Sure! And thank you for leading this release!
I am sure many would be grateful for the new Python 3.12 support which wasn't so easy to add! Good job 👏

I think we can stop merging new PRs for today unless it makes total sense, so you'll have a code-freeze to work on the changelog and the release, knowing nothing would change under your nose during that time. If I'll wish to merge something I'll ask for confirmation first until you're done @auvipy

[auvipy]

exactly! agree with you, didn't thought python 3.12 will need that much effort!! I'm grateful for your continuous support. I will start generating and editing the changelog PR as per the templates suggestion. and no more code change. until tomorrow

[ThiefMaster]

Once the release is out, I think it'd be nice to add 3.13-dev as "allowed to fail" to the CI so any failures are visible early enough

[auvipy]

Once the release is out, I think it'd be nice to add 3.13-dev as "allowed to fail" to the CI so any failures are visible early enough

I think that should be added after 3.13beta1 only. because celery depends on so many dependencies. without those adding python 3.13 support, it will be just wastage of CI resources.

[Nusnus]

Once the release is out, I think it'd be nice to add 3.13-dev as "allowed to fail" to the CI so any failures are visible early enough

Good point but I agree with Asif.
We'll see how it goes when we're closer to 3.13.

[auvipy]

We might need to tweak the codcov stuff to prevent its intermittent outage

[Nusnus]

@auvipy I suspect I found a critical bug with the broker reconnection flow which I'll be investigating in the coming days top priority (first define the bug, then propose a fix), but I won't make it in time before the 5.3.5 release.

Just giving you a heads up on my focus from now on until I fix it.

P.S
Was found by mistake* during the smoke tests development.

*Seems the smoke tests are working in finding production bugs not possible to test with the integration tests due to the requirement to restart the broker which is possible with the upcoming release of pytest-celery 🙂

[auvipy]

No problem. Thanks for the update

[auvipy]

@Nusnus if you don't have any major objection regarding the release notes / changelog except the conversion to rst, I will release 5.3.5 tomorrow. You can leave any suggestions on the changelog PR

[Nusnus]

@Nusnus if you don't have any major objection regarding the release notes / changelog except the conversion to rst, I will release 5.3.5 tomorrow. You can leave any suggestions on the changelog PR

Cool np!
I'll try to get to it before you move forward

[Nusnus]

Can we close this issue @auvipy ?