Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update PVCs WITHOUT disrupting service - using sync option #8501

Open
KlavsKlavsen opened this issue Feb 15, 2022 · 4 comments · May be fixed by argoproj/gitops-engine#561
Open

Update PVCs WITHOUT disrupting service - using sync option #8501

KlavsKlavsen opened this issue Feb 15, 2022 · 4 comments · May be fixed by argoproj/gitops-engine#561
Labels
enhancement New feature or request

Comments

@KlavsKlavsen
Copy link

Summary

Argocd cannot handle sync'ing of pvc changes in statefulsets, because k8s cannot.

There is a solution for this (recreate statefulset without cascade delete) - but as I understand it - recreate is unfortunately done WITH cascade=true (default) as far as I can see.. (the docs here does not even mention recreate option though: https://github.com/argoproj/argo-cd/blob/master/docs/user-guide/sync-options.md

I propose recreate option either be changed to have cascade=false by default - or a new cascade - without cascade be added.

Motivation

I have a statefulset with a defined pvc size of 500GB.
I want to resize the resulting PVCs WITHOUT disrupting my service.
Unfortunately k8s does NOT allow for editing that part of the statefulset definition (yet - kubernetes/kubernetes#68737 and ).
The solution I have found so far, is do to this:

kubectl delete sts --cascade=false <statefulset>

and then the normal "sync" procedure will work - as it would recreate the statefulset - and my pods would NOT be down'ed.

Proposal

Argocd should have a sync option called:
"recreate - without cascade" or the current "recreate" sync option should be with cascade=false (which IMHO is the safer default)

p.s. and the current sync options are already plentiful and dificult to figure out WHEN to use - so there should be a "popup" guide on when to "tick which box" on sync options. And possibly some sync options could atleast be automaticly "recommended" for a sync operation - to help the users "do the right thing" - based on a check of "resources in sync and their state" ? should not be too difficult once the cases are documented - to also start automating a "recommender of options" - per sync - which in the end could turn into something making automatic sync much less dangerous :)

@KlavsKlavsen KlavsKlavsen added the enhancement New feature or request label Feb 15, 2022
@jessesuen
Copy link
Member

I propose recreate option either be changed to have cascade=false by default

When you say "recreate" do you mean Argo CD's "replace" sync option?

If this workaround works: kubectl delete sts --cascade=false <statefulset>, couldn't deleting the object in Argo CD UI with cascade=false also work for you as well?

@KlavsKlavsen
Copy link
Author

Yes - I meant replace sorry :)

Definetely - its just difficult to find the right object manually on a big application and a very manual process - and I know several of my colleagues can't figure out which "option" is the correct one to tick, when they want to sync an application - so argocd really should make it easier..

I would really think "replace" - should by default do this "delete without cascade and then recreate" - as that is the "safest route" when you need to do this- and explanation on the option could say that it does this, which SHOULD avoid the problems it lists now.

@hobti01
Copy link

hobti01 commented Sep 5, 2022

We also do not want manual steps required when rolling out an application.

What about adding a SyncOption so that we could declare replacement with the kubernetes cascade option?

metadata:
  annotations:
    argocd.argoproj.io/sync-options: Replace=true, Cascade=orphan
    # Cascade=[background (default) | foreground | orphan]

@sthomson-wyn
Copy link
Contributor

Hopefully addressed via argoproj/gitops-engine#561

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
4 participants