-
Notifications
You must be signed in to change notification settings - Fork 459
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Is there a way to only restart pods once, with both configmap and secrets change applied? #508
Comments
@ppb-ludekdolejsky This is an interesting situation indeed. Could you please elaborate on the "dangerous situation" you are ending up with these changes? Is it a case that the old secret and new configmap incompatibility will cause your application to crash? If that is the case, could you explain how Reloader can help here? |
@bnallapeta dangerous in a way that unless your healthcheck somehow catches this invalid combination and does not let the new version to roll out, you might end up with app that is running but not functional Example: you switch to a different DB host using configmap, and update credentials using secrets So no that the app crashes, but more like it rolls out successfully, but then does not work, causing a major incident |
@ppb-ludekdolejsky Ack. Do you have a solution in mind for this problem? |
we are going to test a workaround == improving our healthcheck to prevent such misconfigured pods from running, hoping that it will remove that interim (misconfigured) period (where only configmap got updated, but not yet secrets), but still allow the pods to restart once secrets get updated, too it would be still nice if Reloader could restart the pods only once, after both configmap & secrets are updated - is it achievable? |
With the default implementation in place, k8s still creates two events - one for each change (configmap & secret). One possibility is to introduce a time delay before the restart so that both the configmap and secret are updated. But this comes with challenges:
We will discuss this within the team and see if this can be solved to serve a general usecase. |
@ppb-ludekdolejsky We discussed this and plan to implement the following:
|
@bnallapeta Is there any ETA available for the proposed 'reloader.stakater.com/delay' annotation? Thanks. |
@scartledge @ppb-ludekdolejsky Stakater does not have any internal use case for the enhancement request, so this work would be done by Stakater if you buy our Enterprise support. Otherwise, it will be a community effort to add it. You can email sales@stakater.com for our Enterprise support. |
We have noticed that when we change both configmap and secrets, first, new configmap is applied and pods restarted, and some time later, new secrets are applied and pods are restarted again.
So there is a period in between those two restart where pods are running with old secrets but new configmap, which is potentially a dangerous situation for us.
We are using
"reloader.stakater.com/auto": "true"
The text was updated successfully, but these errors were encountered: