Release v3.5 patch releases discussion

[terrytangyuan]

This issue tracks commits for 3.5 patch releases.

[terrytangyuan]

#12025 needs to be fixed for this

[terrytangyuan]

I am thinking about releasing v3.5.1 next week. The most important fix would be #12068 since it blocks people from upgrading to v3.5.

[terrytangyuan]

Security fix: #12111

[terrytangyuan]

#12130

[terrytangyuan]

Released: https://github.com/argoproj/argo-workflows/releases/tag/v3.5.1

[agilgur5]

Would be good to get my UI code-splitting / bundle reductions changes into the next patch release once they're all merged in (#12061, #12097, #12150). They are primarily refactors in content, but their overall intent is to fix long wait times for the UI on slow networks or slow devices as described in #11970

[terrytangyuan]

#12133

[terrytangyuan]

#12203

[terrytangyuan]

TODO: @terrytangyuan to post a list of commits to cherry-pick here

[terrytangyuan]

The plan is to cherry-pick the following commits (unless there's conflict) for v3.5.2 release

Fixes:

• 9eadf2d fix: leak stream (fix: leak stream #12193)
• 206c901 fix: Fix for missing steps in the UI (fix: Fix for missing steps in the UI #12203)
• 939ce40 fix(server): allow passing loglevels as env vars to Server (fix(server): allow passing loglevels as env vars to Server #12145)
• 222d53c fix: Resource version incorrectly overridden for wfInformer list requests. Fixes Controller: High RAM Usage and Frequent Crashes on Large Scale Workflows #11948 (fix: Resource version incorrectly overridden for wfInformer list requests. Fixes #11948 #12133)
• f95a5fc fix: retry S3 on RequestError. Fixes saving log artifact to s3 missing retries #9914 (fix: retry S3 on RequestError. Fixes #9914 #12191)
• 6805c91 fix: ArtifactGC Fails for Stopped Workflows. Fixes Artifacts not set in Status.Nodes | ArtifactGC prevents Workflow deletion #11879 (fix: ArtifactGC Fails for Stopped Workflows. Fixes #11879 #11947)
• 873bacb fix(ui): Cost Opt should only apply to live Workflows (fix(ui): Cost Opt should only apply to live Workflows #12170)

Tests:

• test: add functional tests for memoization without outputs #12215
• test: add test for missing child node #12214

Chore(deps):

• 75d7eb1 chore(deps): bump github.com/aliyun/credentials-go from 1.3.1 to 1.3.2 (chore(deps): bump github.com/aliyun/credentials-go from 1.3.1 to 1.3.2 #12227)
• 63f53ea chore(deps): bump github.com/gorilla/websocket from 1.5.0 to 1.5.1 (chore(deps): bump github.com/gorilla/websocket from 1.5.0 to 1.5.1 #12226)
• b659002 chore(deps): bump golang.org/x/term from 0.13.0 to 0.14.0 (chore(deps): bump golang.org/x/term from 0.13.0 to 0.14.0 #12225)
• bce96ab chore(deps): bump cronstrue from 2.41.0 to 2.44.0 in /ui (chore(deps): bump cronstrue from 2.41.0 to 2.44.0 in /ui #12224)
• c95d9a1 chore(deps): bump golang.org/x/sync from 0.4.0 to 0.5.0 (chore(deps): bump golang.org/x/sync from 0.4.0 to 0.5.0 #12185)
• 877a2e7 chore(deps): bump github.com/go-jose/go-jose/v3 from 3.0.0 to 3.0.1 (chore(deps): bump github.com/go-jose/go-jose/v3 from 3.0.0 to 3.0.1 #12184)
• 460b30a chore(deps): bump golang.org/x/time from 0.3.0 to 0.4.0 (chore(deps): bump golang.org/x/time from 0.3.0 to 0.4.0 #12186)
• a47e462 chore(deps): bump github.com/aliyun/aliyun-oss-go-sdk from 2.2.9+incompatible to 3.0.1+incompatible (chore(deps): bump github.com/aliyun/aliyun-oss-go-sdk from 2.2.9+incompatible to 3.0.1+incompatible #12140)
• 69b47b9 chore(deps): bump monaco-editor from 0.43.0 to 0.44.0 in /ui (chore(deps): bump monaco-editor from 0.43.0 to 0.44.0 in /ui #12142)
• 664569b chore(deps): bump cronstrue from 2.32.0 to 2.41.0 in /ui (chore(deps): bump cronstrue from 2.32.0 to 2.41.0 in /ui #12144)
• 77173a5 chore(deps): bump github.com/creack/pty from 1.1.18 to 1.1.20 (chore(deps): bump github.com/creack/pty from 1.1.18 to 1.1.20 #12139)
• 789c608 chore(deps): bump cloud.google.com/go/storage from 1.33.0 to 1.34.1 (chore(deps): bump cloud.google.com/go/storage from 1.33.0 to 1.34.1 #12138)
• 2f0094c chore(deps): bump react-datepicker and @types/react-datepicker in /ui (chore(deps): bump react-datepicker and @types/react-datepicker in /ui #12096)
• 8f09108 chore(deps): upgrade swagger-ui-react to latest 4.x.x (fix(deps): upgrade swagger-ui-react to latest 4.x.x #12058)
• 5cc8484 chore(deps): bump google.golang.org/api from 0.147.0 to 0.148.0 (chore(deps): bump google.golang.org/api from 0.147.0 to 0.148.0 #12051)
• 7a25c05 chore(deps): bump github.com/coreos/go-oidc/v3 from 3.5.0 to 3.7.0 (chore(deps): bump github.com/coreos/go-oidc/v3 from 3.5.0 to 3.7.0 #12050)
• 35f7208 chore(deps): automatically audit fix UI deps (chore(deps): automatically audit fix UI deps #12036)
• 32d31ab chore(deps): bump google.golang.org/api from 0.143.0 to 0.147.0 (chore(deps): bump google.golang.org/api from 0.143.0 to 0.147.0 #12001)

Others:

• UI code-splitting PRs mentioned in Release v3.5 patch releases discussion #11997 (comment)
• ci: Fix lint check #12245

[Joibel]

Could you add #12215 and #12214 to 3.5.2 please @terrytangyuan, just to ensure those tests are happening.

[agilgur5]

Could we add the UI code-splitting PRs as I mentioned above? I don't think they have a big impact to that many users, so I'm fine with it going in 3.5.3 instead if you'd prefer to keep 3.5.2 smaller/more stable.

Otherwise commit list LGTM ✅

[terrytangyuan]

@Joibel Good idea.

@agilgur5 I'd like to focus on fixes and chore(deps) in patch releases.

[agilgur5]

As I wrote above:

They are primarily refactors in content, but their overall intent is to fix long wait times for the UI on slow networks or slow devices as described in #11970

I would definitely not call them features at least

[agilgur5]

We probably want #12245 as well since #12225 and #12227 are in the list and may break lint again on the 3.5.x branch

[terrytangyuan]

I would definitely not call them features at least

Sounds good. In the future, let's make sure the PR titles can tell us that.

[agilgur5]

Yea each PR individually is more of a refactor, but as a whole they are a fix -- in other words, a bunch of refactors that enable a fix.
Naming things is a hard problem 😅 ; as always feel free to change the titles if that makes things easier for you to release

[terrytangyuan]

Working on branch https://github.com/argoproj/argo-workflows/tree/release-3.5.2

Note that I also included some merged dependabot PRs from master branch and some of the dependabot PRs cannot be cherry-picked.

[terrytangyuan]

v3.5.2 tag created https://github.com/argoproj/argo-workflows/releases/tag/v3.5.2

[juliev0]

Request to include #12353

[terrytangyuan]

This needs to be included #12470

[agilgur5]

#12421 is also a fix for a 3.5.0 feature

[sarabala1979]

Hi
I am working on release v3.5.3 which includes below PRs
#12421
#12470
#12353

[agilgur5]

Updating here that 3.5.3 was released

[agilgur5]

#12397 for the next release

[isubasinghe]

Good news, it is done now, I want to go over the changes carefully again though.
I should probably go to sleep soon so I will stop now.

Should be finished tomorrow evening or by Tuesday evening.

[agilgur5]

I want to go over the changes carefully again though.

Do you want to post a branch or something for others to review?

[isubasinghe]

The changes included in patch 3.5.5 are:

Fixes

• a927379 fix: find correct retry node when using templateRef. Fixes: inline task in DAG makes workflow fail: failed to get a template #12633 (fix: find correct retry node when using templateRef. Fixes: #12633 #12683)
• 5c8062e fix: Add limit to number of Workflows in CronWorkflow history (fix: Add limit to number of Workflows in CronWorkflow history #12681)
• 4ad0db9 fix: Patch taskset with subresources to delete completed node status.… (fix: Patch taskset with subresources to delete completed node status. Fixes #11489 #12620)
• 57a078f fix(typo): fix some typo (fix(typo): fix some typo #12673)
• 030d581 fix(ui): ListWatch should not both set and depend on nextOffset (fix(ui): ListWatch should not _both_ set and depend on nextOffset #12672)
• 23f8c35 fix(controller): re-allow changing executor args (fix(controller): re-allow changing executor args #12609)
• 148252a fix(controller): add missing namespace index from workflow informer (fix(controller): add missing namespace index from workflow informer #12666)
• 0acb435 fix: retry node with expression status Running -> Pending (fix: retry node with expression status Running -> Pending #12637)
• 66680f1 fix(build): check for env vars in all dirs (fix(build): check for env vars in all dirs #12652)
• 130417b fix(docs): remove workflow-controller-configmap.yaml self reference (fix(docs): remove workflow-controller-configmap.yaml self reference #12654)
• ae0973a fix: pass through burst and qps for auth.kubeclient (fix(server): pass through burst and qps for auth.kubeclient #12575)
• 5f4b235 fix: controller option to not watch configmap (fix: controller option to not watch configmap #12622)
• 6c8a715 fix: artifact subdir error when using volumeMount (fix: artifact subdir error when using volumeMount #12638)
• fbd70aa fix: Allow valueFrom in dag arguments parameters. Fixes valueFrom not allowed for argument parameters in dag templates (but in steps) #11900 (fix: Allow valueFrom in dag arguments parameters. Fixes #11900 #11902)
• 09edbf7 fix(resources): improve ressource accounting. Fixes Resource duration is overestimated for cpu #12468 (fix: improve resource duration accuracy. Fixes #12468 #12492)
• e771bde fix: make the process of patching pods exclusive (fix: make the process of patching pods exclusive #12596)
• 4226269 fix: upgrade expr-lang. Fixes Dependencies and Enhanced Depends is broken if you call a task 'split' #12037 (fix: upgrade expr from 1.15.5 to 1.16.0. Fixes #12037 #12573)
• 8f2746a fix: make sure taskresult completed when mark node succeed when it has outputs (fix: make sure taskresult completed when mark node succeed when it has outputs #12537)
• a155877 fix: Mark resource && data template report-outputs-completed true (fix: Mark resource && data template report-outputs-completed true #12544)
• 8d27a9f fix: make etcd errors transient (fix: make etcd errors transient #12567)
• af2cacb fix(ui): clone the ListWatch callback array in WorkflowsList (fix(ui): clone the ListWatch callback array in WorkflowsList #12562)
• c46986e fix: Global Artifact Passing. Fixes The global output cannot be used correctly after upgrade v3.5.3 #12554 (fix: Global Artifact Passing. Fixes #12554 #12559)
• 1dbc856 fix: update minio chart repo (fix(docs): update minio chart repo #12552)
• 46c1324 fix: cache configmap don't create with workflow has retrystrategy. Fixes: Memoized didn't created when there is retryStrategy #12490 Memoization doesn't work in templates that are dag or use retries #10426 (fix: cache configmap don't create with workflow has retrystrategy. Fixes: #12490 #10426 #12491)
• 1ab7cd2 fix: add resource quota evaluation timed out to transient (fix: add resource quota evaluation timed out to transient #12536)
• 1202ae4 fix: prevent update race in workflow cache (Fixes Parent node not initialized #9574) (fix: prevent update race in workflow cache (Fixes #9574) #12233)
• e20f312 fix: Fixed mutex with withSequence in http template broken. Fixes Workflow Hangs Indefinitely with 'mutex' and 'withSequence' in 'http' Template #12018 (fix: Fixed mutex with withSequence in http template broken. Fixes #12018 #12176)
• 3931e59 fix: SSO with Jumpcloud "email_verified" field SSO with Jumpcloud "email_verified" causes failure to authenticate #12257 (fix: SSO with Jumpcloud "email_verified" field #12257 #12318)
• b290e51 fix: wrong values are assigned to input parameters of workflowtemplat… (fix: wrong values are assigned to input parameters of workflowtemplat… #12412)
• b447951 fix: Add missing 'archived' prop for ArtifactPanel component. Fixes Artifact in workflow diagram not working when workflow is archived and deleted from cluster #12331 (fix: Add missing 'archived' prop for ArtifactPanel component. Fixes #12331 #12397)
• 44b33fa fix: merge env bug in workflow-controller-configmap and container. Fixes ENV Merge Bug  #12424 (fix: merge env bug in workflow-controller-configmap and container. Fixes #12424 #12426)
• bcf5672 fix(docs): handle fields examples with md_in_html (fix(docs): handle fields examples with md_in_html #12465)
• 96f25af fix(docs): exclude docs/requirements.txt from docs build (fix(docs): exclude docs/requirements.txt from docs build #12466)
• 1968436 fix(docs): render Mermaid diagrams in docs (fix(docs): render Mermaid diagrams in docs #12464)
• d1cae63 fix: documentation links (fix: documentation links #12446)

chore(deps)

• 873d3de chore(deps): fixed medium CVE in github.com/docker/docker v24.0.0+incompatible (fix(deps): fixed medium CVE in github.com/docker/docker v24.0.0+incompatible #12635)

[isubasinghe]

Please help me check the correctness here: https://github.com/isubasinghe/argo-workflows/tree/release-3.5.5

[tczhao]

Would it be a good idea to make a PR to v3.5.4 and trigger git actions to make sure all tests pass?

[isubasinghe]

I've doubled checked it's fine and CI passes. I don't have the permissions to create a tag however. @terrytangyuan

[terrytangyuan]

Can you update the release branch? Once updated, I can also help create a tag: https://github.com/argoproj/argo-workflows/tree/release-3.5

[agilgur5]

The changes included in patch 3.5.5 are:

I did notice that two missing from this list were mentioned above in this issue: #12556 and #12581

Please help me check the correctness here: https://github.com/isubasinghe/argo-workflows/tree/release-3.5.5

I checked a few of the individual commits that LGTM / match their counterparts. The notable difference is the reversed order of the commits (that you had mentioned offline due to the cherry-pick.sh script).

Also ran a git diff release-3.5.5 main and and there were only a few things missing outside of features. The vast majority were build, CI, and docs, but some were deps changes and there were a few fixes that seemingly never made it in / were not explicitly listed before. We should perhaps try to get some of those in as well.

[agilgur5]

I did notice that two missing from this list were mentioned above in this issue: #12556 and #12581

I added these two:

• baef485 chore(deps): bump github.com/cloudflare/circl to 1.3.7 to fix GHSA-9763-4f94-gfch (chore(deps): bump github.com/cloudflare/circl to 1.3.7 to fix GHSA-9763-4f94-gfch #12556)
• d9caa75 docs: consistently use "> v{version} and after" (docs: consistently use "> v{version} and after" #12581)

and another patch that was recently added:

• 9bec114 fix: make WF global parameters available in retries (fix: make WF global parameters available in retries #12698)

https://github.com/argoproj/argo-workflows/tree/release-3.5

[isubasinghe]

Thanks @agilgur5, I suppose let's ship it

[terrytangyuan]

You should be able to create a tag now.

[terrytangyuan]

https://github.com/argoproj/argo-workflows/releases/tag/v3.5.5

Thanks @agilgur5 and @isubasinghe!

[z63d]

Would you consider cherry-picking #12312 into the v3.5 release branch? You can find more details in #12829.

[agilgur5]

Would you consider cherry-picking #12312 into the v3.5 release branch?

Completed in #12312 (comment)

[yulin-li]

could we make a patch release for recent CVE fixes?

[terrytangyuan]

Yes, we should release a patch for security fixes.

[terrytangyuan]

#12900 and #12901

[SlaterByte]

@isubasinghe, I see that #12413 was originally included as a feature addition for v3.5.5, but the was removed from your final comment. Is this feature being bumped to v3.5.6 or to a v3.6.x release? I see a few followup PRs ( #12596, #12632, #12862) as well, so maybe there was a stability concern? Looking forward to getting this feature released. Thanks!

[agilgur5]

I see that #12413 was originally included as a feature addition for v3.5.5, but the was removed from your final comment. Is this feature being bumped to v3.5.6 or to a v3.6.x release?

It's a feature, so it will be released in the next minor, v3.6.

[agilgur5]

#12900 and #12901

Backported both to release-3.5

[agilgur5]

Released v3.5.6

[terrytangyuan]

Any objections to cherry-pick #12736 to the next 3.5 patch release?

[agilgur5]

Nope, had been planning to do so since it's a feature that fixes substantial 3.5 regressions (so long as it didn't break anything). Had also added the 3.5.x milestone to the respective issues