Release 5.3

[thedrow]

Version: 5.3
Release PR:

Checklist

• Release PR drafted

• Release PR reviewed

• The main branch build passes

  

• Release Notes

• What's New

Process

Alphas

• Alpha 1

Betas

• Beta 1
• Beta 2

Release Candidates

• RC 1
• RC 2

Release Blockers

• Serialization/Deserialization UUID issue kombu#1590
• Celery Import Error #7783

Potential Release Blockers

[thedrow]

@auvipy Please allow me to manage the release. There are a couple of bugs in stamping I need to fix before we reach GA.
In any case, we can't release without a What's New document which I'm currently writing.

[auvipy]

FYI, I am not going to release the final version. just the alphas/beta's

[thedrow]

Please don't release another beta before we finalize the Header Stamping API.
By the time we release RC1, the API has to be stable.

Also note that documentation points to the beta instead of the latest GA version.

[auvipy]

OK no worries. I will be focusing on other stuffs

[dpdoughe]

This is just a comment on timelines as I know the project is very busy and as always doing great work. I was looking at releases and noticed that the due date for this milestone for 5.3 is past now. It looks like there are a good deal more tasks in this 5.3 milestone that previous releases. According to the progress meter this milestone is 50% complete and it seems like it could be (guessing) ~6 months before the release.

I'm wondering if it is possible for the team to reduce the scope of 5.3 to make a release? Alternatively, might you change the due date of this milestone?

[auvipy]

I think we might release 5.3 by December 2022. And yes we already covered lots of stuff. what is halting another release is some bug in newly introduced Stamping API in the canvas. they are regularly ironed out. I will check and re milestone some issues to next version to reduce the remaining stuff.

[auvipy]

@thedrow @Nusnus should we expect a new beta by the end of December? or that will take little longer? Another thing I want your opinion is, python 3.7 is going to be EOL by june 2023, pypy already dropped support for python 3.7. should we keep python 3.7 for 5.3? or python 3.8-3.11 should be the supported version? I don't have problem if you want to keep 3.7 support for final release. just want to know your opinions/views on this.

[Nusnus]

should we expect a new beta by the end of December? or that will take little longer?

Before the 5.3 release, I want to improve the testing to deliver a stable version with many bugs crushed.
I will focus very soon on the pytest-celery plugin to add more sophisticated tests that validate edge cases that are impossible to test right now, like multiple workers edge cases, worker killed edge cases, etc.
In addition, there are possibly a few more bugs I'm investigating that I want to be fixed before 5.3.

So, in my opinion, until the plugin provides new capabilities that are used for adding more tests to Celery, it would be unwise to release. We cannot even attend attention now due to the need for testing facilities for production-like edge cases. We want to increase our quality where real production environments are failing.
So it's essential for >SBMs / Enterprise scale Celery users, which require higher trust in open-source systems, and that's a very important focus to prioritize over the last announced deadline, IMHO.

Another thing I want your opinion is, python 3.7 is going to be EOL by june 2023, pypy already dropped support for python 3.7. should we keep python 3.7 for 5.3? or python 3.8-3.11 should be the supported version? I don't have problem if you want to keep 3.7 support for final release. just want to know your opinions/views on this.

I think 5.3 should come with Python 3.7 support.
>5.3 - let's talk. But for now lets leave the 3.7 support that is already available.

[Nusnus]

should we expect a new beta by the end of December? or that will take little longer?

Before the 5.3 release, I want to improve the testing to deliver a stable version with many bugs crushed. I will focus very soon on the pytest-celery plugin to add more sophisticated tests that validate edge cases that are impossible to test right now, like multiple workers edge cases, worker killed edge cases, etc. In addition, there are possibly a few more bugs I'm investigating that I want to be fixed before 5.3.

So, in my opinion, until the plugin provides new capabilities that are used for adding more tests to Celery, it would be unwise to release. We cannot even attend attention now due to the need for testing facilities for production-like edge cases. We want to increase our quality where real production environments are failing. So it's essential for >SBMs / Enterprise scale Celery users, which require higher trust in open-source systems, and that's a very important focus to prioritize over the last announced deadline, IMHO.

Another thing I want your opinion is, python 3.7 is going to be EOL by june 2023, pypy already dropped support for python 3.7. should we keep python 3.7 for 5.3? or python 3.8-3.11 should be the supported version? I don't have problem if you want to keep 3.7 support for final release. just want to know your opinions/views on this.

I think 5.3 should come with Python 3.7 support. >5.3 - let's talk. But for now lets leave the 3.7 support that is already available.

I'd suggest a new tentative deadline: Feb 2023
@auvipy @thedrow

[auvipy]

OK fair enough. will plan accordingly. in the mean time we can improve CI and other stuff as well.

[Nusnus]

Awesome, I also want to attend celery/kombu#1590 which is listed as a release blocker on the checklist above. I don't know when I'll get to it, but I got it on my mind.

[auvipy]

OK. also ping on pytest-celery works as well.

[Nusnus]

@auvipy It would be useful to stabilize the "contribution pipeline" meaning all of the noise when opening a new PR.
Linter, Redis, and anything that should be working and is currently broken should be on high priority.
Imagine someone opening a PR and starts wasting time or worse getting scared for many errors and abandon whatever idea he had due to "dirty environment" from our side.

That + CI adjustments/upgrades (like I see you're doing recently 💯 ) would be efficient improvements as they are relatively small and infra-related so fixes are "searchable" (in opposed to features impl etc.)

If you could prioritize the above I would appreciate it a lot!
Reducing noise increases efficiency so I believe it would be widely useful.

[Nusnus]

OK. also ping on pytest-celery works as well.

I'll start by stabilizing the mvp and getting the GitHub project set-up to make it useful and informative.
The first priority is to reach a merge of mvp and main, leaving the current impl and "skipping" the new addition via config, just for the sake of merging MVP to main to allow standard SDLC for the project which is now damaged due to the gravity of the mvp branch of pytest-celery and the distance from main.

To keep on topic, the effort on pytest-celery would be centrelized on the project, so we'll chat later on over there when it gets relevant @auvipy 🤙

[jackokring]

celery 5.2.7 has requirement billiard<4.0,>=3.6.4.0, but you have billiard 4.1.0.

[dpdoughe]

Can you comment on the possibility to reduce or re-milestone this 5.3 to achieve a release? Since Dec when the new date was set to Feb (now passed) there have only been a few percent of the current tasks (now at 64% complete). It seems like an additional 6 month wait time till some release is not unreasonable unless some things can be restructured/removed from this scope.

[Nusnus]

@dpdoughe

Can you comment on the possibility to reduce or re-milestone this 5.3 to achieve a release? Since Dec when the new date was set to Feb (now passed) there have only been a few percent of the current tasks (now at 64% complete). It seems like an additional 6 month wait time till some release is not unreasonable unless some things can be restructured/removed from this scope.

I agree, we should update the official 5.3 to match the actual delta for release.
That being said, there aren't any more new stuff planned for the 5.3 release.
The current version main isn't stable enough for release so the main effort now is improving the testing so we can deliver a reliable stable release. That means bug fixes + improvements in tests are the main priority and once we get to a quality level that can be released to the public we will move on with the release steps.

To be more specific, the current main doesn't play along with different versions of Celery (e.g Celery 4), so hybrid production environments will suffer the most from an early release. We have to make sure we don't break legacy systems and allow migrating to Celery 5 with ease. I apologize for the implicit delay and the lack of clear communication. I hope this helps clear a bit of the fog.

[auvipy]

it would be great if we at least can release another beta ASAP

[thedrow]

I released a new beta version.

[Nusnus]

Thank you @thedrow !
I will adjust the 5.3 roadmap soon as well 💪

[Nusnus]

@dpdoughe

Can you comment on the possibility to reduce or re-milestone this 5.3 to achieve a release? Since Dec when the new date was set to Feb (now passed) there have only been a few percent of the current tasks (now at 64% complete). It seems like an additional 6 month wait time till some release is not unreasonable unless some things can be restructured/removed from this scope.

I've updated the milestones and 5.3 project.
It does not reflect 100% the upcoming plans, but it's much more accurate and updated than before.

Missing Steps from the official 5.3 Milestone:

• I am currently performing testing on hybrid celery versions (main vs Celery 4) to confirm stability with legacy versions.
• After that, I am going to develop pytest-celery v1.0 which is a blocker for 5.3 release.
• Then, I plan on developing a new smoke testing layer on top of the existing unit/integration, using the new capabilities I'll develop in pytest-celery to implement production-sanity smoke tests that will run in the CI with every new PR.

Once we have a new passing smoke testing layer with new tests for edge cases that we cannot test in the existing infrastructure, we'll be able to sign the 5.3 as "Stable" and allow a proper final release.

None of the above is listed in the 5.3 Milestone, but its my top-priority at the moment, including fixing any bug I find on the way.

During the time I'll be invested in the above, other contributors might address more issues from which we will focus on bug fixes and enhancements. No breaking API changes, no complicated additions, though common sense would be applied per case.

So the new milestone might be very close to 100%, but it (partially) lacks the above and as the pytest-celery plugin release will take a toll in time, we'll have to stay with the beta releases so far, having new version released today with many new contributions and fixes!

P.S
I am less aware of the situation in kombu so I did not take it into consideration.

[Nusnus]

@thedrow @auvipy
I’ve cleaned a lot of the 5.3 milestone, going almost one-by-one over the 100-200 issues that were assigned the 5.3 Milestone. I've removed the milestone from some of them, the rest I've moved to the Future milestone. I did not touch the 5.3.x Milestone so it requires syncing as well later on.

@auvipy - Let's focus on marking 5.3 more closely. I've updated the project so the columns should mean what they say with better accuracy to what's under it. Let's use the Project property on new PRs to mark their progress/status.

I've also updated the 5.3 milestone date to May 2023. I believe this makes sense per my previous comment.
Let's try to clean up the issues, close old/irrelevant stuff, ping recent/waiting PRs.

@auvipy - The main purpose is to do some cleanup for better understanding of actual critical open tickets for 5.3, improve transparency with the community and organize the 5.3 project per actual real status for improved clarity on our current state.
The goal is to keep the May 2023 date so we have to clean up noise to allow focusing on the highest priority issues.

@auvipy , if you may, please invest some time cleaning up per the above, I will also try to monitor new issues and mark them accordingly.

[Nusnus]

@thedrow @auvipy of course if there are any objections or conflicting ideas, let's talk!

[Nusnus]

celery 5.2.7 has requirement billiard<4.0,>=3.6.4.0, but you have billiard 4.1.0.

@auvipy btw was this handled? I think you fixed something regarding billiard not long ago right?

[Nusnus]

@rgangopadhya

First of all, thank you everyone for your work on Celery!

Thank you ❤️

My question is more specific: as far as a user of Celery 5.2.7 is concerned (i.e., someone who isn't looking to incorporate new features like the stamp feature), are the beta releases of 5.3.0 production ready?

Short answer: Yes.
Long answer: But it's not verified.

I understand we want to improve the test coverage using the new plugin, is that because we believe that previously working functionality may no longer be working

There are many production-relevant use-cases that have never been automatically tested due to the limitations of the current testing infrastructure, so the time has come to cover that aspect of our testing coverage.

or more because 5.3.0 is also trying to target an enterprise audience looking for that additional level of verification in their open source software?

The benefits of improved testing apply to all celery users, but yes, enterprise audience may have a higher level of acceptance standards, and releasing 5.3 with improved QA coverage, specifically for production-relevant cases, may improve engagement and motivate upgrading to 5.3 by a wider range of users.

I'm wondering at what level you're talking about here - are there critical issues in Celery as a whole, including features available in 5.2.7 and earlier, or instead only with new features? If it's the latter, the choice to publish beta releases in the first place is interesting.

Basically, besides the stamping feature, main is pretty much stable and there aren't any critical known issues AFAIK.
As @auvipy said, there are tons of critical bug fixes, improvements and upgrades and we're starting to close-in on the latest stuff (trying not to focus on complicated new stuff) to be able to push the release as soon as we can.
The reason we want to publish beta releases in this situation, is to allow the community to start migrating and also provide some time for receiving feedback on the latest version.
The stamping feature is a release-blocker because it breaks Celery 4/Celery 5 support (multi-version multi-worker environment), besides being still buggy by itself. So the beta releases allow the community to start migrating (dev/staging env) as we stabilize the feature towards the RC and final release.

Lastly, the new pytest plugin will allow adding smoke tests that might discover actual real critical new bugs that we also want to fix before the final release, to the best effort we can under a reasonable time frame.

--

I hope I answered all of your questions @rgangopadhya 🙏

[Nusnus]

There are many issues of type "I have this setup, using this config/running this canvas, and I get XYZ".
XYZ May be a connection error, the worker being killed, some configuration misbehaving, etc. (just see one of the many similar issues). It is possible that >90% isn't relevant to anyone else besides the person reporting the issue, but these reports might be caused by a real bugs that we cannot reproduce without the new plugin which makes it hard to assess and handle.

We don't know how critical are the existing unknown bugs, but as we progress into reproducing such architectures in the (upcoming) smoke tests, we'll be able to have a better definition of our stability to allow releasing 5.3 based on automatic tests and not a hunch / deadline.

[rgangopadhya]

Thanks for the clarifications @Nusnus and @auvipy!

[Nusnus]

As this thread has evolved into an updates feed for the 5.3 release, I'd like to share some recent developments.

1. Per supervision by @thedrow, I've released a new RC1 version: v5.3.0rc1. Aside from the issue with the stamping feature, everything else should be ready for release. We are aware of numerous important open issues, but they are not considered release-blockers.
2. Pytest Celery v1.0.0 has reached an alpha development level and is currently in the integration phase with Celery as part of the smoke testing effort.
3. I am making progress on both fixing bugs in the stamping feature and adding smoke tests to a growing PR, which acts as acceptance tests for the v1.0.0 release of pytest-celery. This is a concerted effort to finalize known bugs and identify critical production-blocking bugs before the final 5.3 release.

Finally, as per @thedrow's instructions, we can only release 5.3 once the stamping feature functions as expected.
Our expected release date was May (technically 1st), but I am aware of critical production blocking bugs in the stamping feature, which means we'll have to be more patient (again, I know 🙏 ).

In retrospect, we'll have to avoid reaching a single release blocker that is long and complicated to finalize. Still, for 5.3, we'll use this buffer to accept additional contributions, refinements, and bug fixes that will go into 5.3.

Lastly, from 5.3 forward, it will be much easier and simpler to advance, and our release cycles will be able to shrink significantly.

[auvipy]

@Nusnus lets re target at may 30. I also need to cross check some stuff and release kombu

[Nusnus]

@Nusnus lets re target at may 30. I also need to cross check some stuff and release kombu

Sounds great @auvipy
Go ahead 🚀

[jbergler]

Thanks for all the work that's going into the 5.3 release - we've been running 5.3.0rc1 on python3.11 in production for ~10 days now and things seem mostly fine. The biggest challenge is this regression breaks error tracking with Sentry, Datadog, etc

I'm not sure where to look for status of the stamping feature, so depending on how much feedback you'd like from folks testing this release - I'd suggest maybe considering an rc2 if the stamping feature ends up taking longer than this week (based on May 30 estimate)

[Nusnus]

Thanks for all the work that's going into the 5.3 release - we've been running 5.3.0rc1 on python3.11 in production for ~10 days now and things seem mostly fine. The biggest challenge is this regression breaks error tracking with Sentry, Datadog, etc

I'm not sure where to look for status of the stamping feature, so depending on how much feedback you'd like from folks testing this release - I'd suggest maybe considering an rc2 if the stamping feature ends up taking longer than this week (based on May 30 estimate)

Hey @jbergler, thanks for the feedback.
I'll discuss this with @thedrow and see what direction he suggests taking.

Regarding RC2 - I am currently investigating #8105 & #8086 which I wish to get fixed before we release RC2 as this would be a significant fix that I want to include in the next release candidate, as I expect challenges with this task.

Lastly, per @thedrow last given instruction, we cannot release without the stamping feature working, but as the release date has been postponed too many times, I will discuss with @thedrow the possibility of making cuts and allowing releasing 5.3.0 with known issues and having 5.3.1 to close the known gaps which are just slow to close but are actively being worked on.

This also means the smoke tests will also be ready only for 5.3.1, but I think it's bearable.

@auvipy does the above sit well with you?

[auvipy]

@auvipy does the above sit well with you?

it does.

[Nusnus]

Ok, so I've fixed #8105 and #8086 and also talked with @thedrow regarding 5.3 release.
He accepted my suggestion above and instructed me to manage the release.

Kombu
@auvipy - Can we release an official version or do we have a critical blocker?
If we can, please proceed with releasing Kombu officially + adjust the req.txt files in Celery main to be using the last version of Kombu after releasing it.

Celery
I've updated v5.3 Project/Milestone and I have only one open issue #8256
I am not sure if it's a release blocker - can you please comment @auvipy ?
Assuming it's not, I will release Celery 5.3 in the next 72h, assuming Kombu's part above is finished.

Lastly

1. I've created a 5.4 Project/Milestone and moved the issues so they would match their status from the 5.3 project/board. During the coming days, we need to see which 5.3.x issue needs to be included in 5.4 and move the issues around accordingly.
2. I see you've just released Celery RC2 - I am not sure what went wrong, but it appears it overwrote previous release messages (for example, I had a message in the RC1 mentioning the instability of the stamping feature in the version which is now gone, including Markdown formatting for improved readability of the release notes, etc.).
   Please review what happened @auvipy, it's not the end of the world as its RC1 and we're going to release 5.3.0 by EOW, but we need to avoid having it happen in the future because @thedrow has asked me to prepare a proper release announcement in the 5.3.0 release and we don't want it to be deleted in the future (5.3.x / 5.x.x) releases.

P.S
I will edit the RC1 message as I have a way to get my original message back so no worries for now, just for the future :)

[auvipy]

@Nusnus the thing you are not seeing are still on https://github.com/celery/celery/releases/tag/v5.3.0rc1. we had some missing dependencies issues in both celery & kombu and redis + sqlalchemy upgrade issues fixed so for compatibilty check and early feedback I just released another RC/RC2. I keep checking remaining kombu issues and release a stable release and test it in celery CI. so that you can move to stable release in next 72 hours.

[Nusnus]

@Nusnus the thing you are not seeing are still on https://github.com/celery/celery/releases/tag/v5.3.0rc1. we had some missing dependencies issues in both celery & kombu and redis + sqlalchemy upgrade issues fixed so for compatibilty check and early feedback I just released another RC/RC2. I keep checking remaining kombu issues and release a stable release and test it in celery CI. so that you can move to stable release in next 72 hours.

Ok cool, thanks for the update @auvipy !

Let's aim for getting Kombu released today if possible - we also need to see if there are any remaining Python 3.7 in the docs/readme etc. before we release Celery/Kombu 5.3.0 so as to avoid confusion (I will announce in the GitHub release we've officially dropped Python 3.7 support).

Once you're done with Kombu @auvipy , I will open a PR for 5.3.0 for Celery with the version bump + changelog etc., so we can properly review the last step before release and avoid "oops we forgot this and that".

LMK what you think @auvipy !
And thank you so much for your effort and time - it is much appreciated and very helpful 💚

[auvipy]

Tomorrow morning and will ping you. [Updated due date to 5th June 2023]

[Nusnus]

[Updated due date to 5th June 2023]

Just noticed that - ok, thanks for the update.

I'll move on with the Celery 5.3.0 release (GitHub / PyPI) once you let me know you're ready from your side @auvipy.

[Nusnus]

Hey @auvipy , unfortunately I am dealing with an urgent personal issue and have had to get any work on hold.

I will get back to partial availability tomorrow (6/June) and hopefully as the urgent issue passes, to full capacity.

Family first ❤️

[Nusnus]

I will be available to get Celery 5.3.0 out and finish the PR we have open during the day (6/June) @auvipy

I don't want to postpone the release any further if we can.

[auvipy]

OK I will try to improve the doc now. don't worry.

[auvipy]

@Nusnus I have updated the whats-new part & release notes. also tagged https://github.com/celery/celery/releases/tag/v5.3.0 and released the stable version as you are busy with emergencies. I tried to cover as much detail as i can. I completed it as I got some free time and not sure when again get that in 1 weeks. we can improve the release highlights and related docs as needed. thanks for all you effort and heads up.

[auvipy]

backport branch created for when we need that https://github.com/celery/celery/tree/v5.3

[Nusnus]

@Nusnus I have updated the whats-new part & release notes. also tagged https://github.com/celery/celery/releases/tag/v5.3.0 and released the stable version as you are busy with emergencies. I tried to cover as much detail as i can. I completed it as I got some free time and not sure when again get that in 1 weeks. we can improve the release highlights and related docs as needed. thanks for all you effort and heads up.

I have been a bit off these days, but I'm coming back slowly.
Thank you so much for making it happen, my timing was "perfect" for having personal stuff to attend.

I'll catch up on what I missed and move forwards.

P.S
I've removed/applied this issue's milestone by mistake, you may ignore it if you notice :)

P.P.S
@auvipy - how can I contact you privately? Twitter fits?

[auvipy]

you can ping me on twitter / email whatever easy for you.