KEP-4212: Declarative Node Maintenance

[atiratree]

• One-line PR description: introduce a NodeMaintenance API to solve various node drain issues

• Issue link: Declarative Node Maintenance #4212

• Other comments:

TODO:

• explore permission model for NodeMaintenance, Node, Pods (RBAC, ValidatingAdmissionPolicy) and all of the actors
• explore whether NodeMaintenance can be used as a replacement or superset of the Graceful Node Shutdown feature
• explore possible interactions with Kubelet
• explore the descheduling aspects of this feature
• explore the DaemonSets and StatefulSets use case
• figure out the best format/implementation for the EvacuationRequest

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: atiratree
Once this PR has been reviewed and has the lgtm label, please assign janetkuo for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/sig-apps/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sftim]

Thank you very much for opening the issue and PR!

I know this is draft; I've got some feedback already, and I hope it's helpful.

[sftim]

🤔 should this be SIG Node rather than SIG Apps?

[atiratree]

🤔 should this be SIG Node rather than SIG Apps?

Not sure yet, the API itself is SIG Node I guess, but it has a lot of implications for SIG- Apps. Let's delay this decision after I redo the KEP and we have additional rounds of discussions.

[kwilczynski]

Would the shutdown cancellation work in the same way as it currently is under Graceful Node Shutdown? I assume, yes, it would.

[sftim]

This KEP also supports node maintenance for Windows nodes, where graceful shutdown handling is (AIUI) not implemented.

[kwilczynski]

No, it's not—very much Linux, not even other *nix, centric indeed. Would this be a blocker then for this KEP to move forward?

[sftim]

IMO you can mark a node as needing maintenance even if it doesn't run systemd (either Linux without systemd, or Windows which doesn't really have an init process in the POSIX sense). So this KEP can move forward whether or not your nodes gracefully handle a shutdown signal.

[atiratree]

Cancelation by kubelet

Note:
There are cases when Node termination was cancelled by the system (or perhaps manually by an administrator). In either of those situations the Node will return to the Ready state. However, Pods which already started the process of termination will not be restored by kubelet and will need to be re-scheduled.

Is the node termination cancellation reliable? Can we use it for deletion of the NodeMaintenance?

---

Also the behavior is application specific, when the the NodeMaintenance is cancelled/deleted (please see the Evacuation KEP). The application/evacuator can terminate the pod or it can also recover.

[kwilczynski]

It might be too late to fallback to something else. We should attempt to add NodeMaintenance object, should it be missing, but it might be too late to stop Graceful Node Shutdown, which might be too far gone, so to speak. This would be a one-way? hard? fallback.

[kwilczynski]

Alternatively, perhaps then doing nothing or refusing to carry-out a shutdown would be a safer option?

That said, depends on how the shutdown was triggered: if it was due to external factors (which would be the default modus operandi for which Graceful Node Shutdown has been designed), then it would be potentially unsafe to interrupt the ongoing shutdown, or there might be not enough time, etc.

[atiratree]

It would be too late for the pods that are already terminating. But it might be useful for the running non-terminated pods, right?

It seems to me that by default it would be good to carry out the Graceful Node Shutdown in the disconnected scenario. At least to have a softer landing in bad situations. I wonder if it would make sense to make this behavior configurable?

[tuibeovince]

we would fall back to the original behavior of Graceful Node Shutdown feature

does this imply that Graceful Node Shutdown will be used the way it is now? or does it mean DNM will internally reproduce the same original GracefulNodeShutdown and just perform that behavior?

[sftim]

AIUI, DNM is about marking the node as needing maintenance and barely overlaps with node shutdown handling.

The only overlap is that both things trigger a node drain; but other things could also trigger that, such as node problem detector deciding the node is irreparably faulty.