Protect load balancer from being deleted

[Kavantix]

Currently load balancers are always deleted when the service is deleted unless the disown annotation is specified.
The disown annotation however disables all other features as well.

So what I would like to have is an annotation protect: "true" that leaves all features intact but prevents deletion such that the ip address of that load balancer can be reused if someone manages to delete the service.

[timoreimann]

Hey @Kavantix, one question for clarification: do you expect a delete request for a LoadBalancer-typed Service to just silently not carry out the LB deletion against the API or rather return an error? That is, should the Service still be deleted or not? What is your use case?

[timoreimann]

@rawkode interested in your feedback on the above question as well since you submitted a PR.

[Kavantix]

@timoreimann preventing the service from being deleted would also be fine.

My main concern is that normally deleting and recreating a service in kubernetes will not break anything but when it is mapped to a do load balancer it now does break since the ip changes.

We managed to accidentally delete a load balancer once while thinking it was disowned

[timoreimann]

Got it. I think a validating web hook may be an equally good if not better solution for you assuming that deleting the Service without the LB would still be a bit of a hassle to you (i.e., if you accidentally deleted the Service, you would still have the LB but would also have to craft a new Service object with the right annotation to re-own the orphaned LB).

The validating web hook is something that our CCM could also provide and it would presumably be controlled by an annotation. So same mechanism, but different implementation.

[Kavantix]

So basically an 'are you sure you want to delete this service' if it has the 'protect: true' annotation?

[timoreimann]

Sort of: If the protective annotation was set, then a validation web hook would reject DELETE requests to the Service object right away.

FWIW this can already be done with a custom web hook unrelated to CCM (and frameworks like OPA / GateKeeper / Kyvero should allow to do so without a custom implementation). I'd normally refer to one of these solutions but I also see the value in making it a built-in feature of our CCM given that the loss of the LB and its public IP address is quite impactful.

[rawkode]

For my use-case, preventing deletion of the service is a no go.

I provision the LB and service independently and this is already supported by providing the load-balancer-id tag. For DOKs customers that don’t want to go down the external-dns route, this is probably pretty common.

[timoreimann]

At this point, I'm more strongly leaning towards suggesting to use kyverno or similar out-of-band tools as they offer a more generalized solution to the given problem. The built-in CCM approach also has the issue that it may fail due to a bug or become non-effective if, say, a faulty annotation breaks the protection mechanism in ways we have observed similarly. Finally, annotations (being the only configuration means we have today) are all string-based so it's easy to get wrong without realizing (through, e.g., a typo), whereas something like a webhook offers proper typing and validation.

Hope this makes sense. Closing the issue.