New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Restarted kubelet should not evict pods due to the node affinity #124586
Comments
/sig node |
kubernetes/staging/src/k8s.io/api/core/v1/types.go Lines 3458 to 3464 in 7d880fd
^ |
We can fix the behavior or documents. The fix seems simple but may break many existing use cases. We may need some discussion about this. |
/cc |
/triage accepted In hindsight, the labelling part (xref: #123980 (comment)) deserves its own issue. |
/cc |
1 similar comment
/cc |
/assign @gjkim42 gjkim42 |
since we had a similar discussion before in #101218 (comment) What do you think about this? |
I guess the definition of The verification is happening both in the scheduler and in kubelet, at startup, but not during execution. Yes, we can update the documentation to be more clear, but we can't change the behavior. |
Do you mean you can not change the behaviour to support |
We are ignoring during execution. In other words, we are honoring what the API says. But there is an intermediate stage that the API doesn't say anything about: the time between pod scheduled and before it starts executing. That's what needs to be clarified in the documentation, and it should match the existing behavior that has been there since this feature first launched. We cannot change that behavior or it would be backwards incompatible. |
Ok, trying to rephrase with my own words to be sure I understand:
That sounds fair to keep. |
if pod use |
Yes |
@alculquicondor, @Homura222 did some experiements (kubevirt/kubevirt#11843 (comment)) and it turns out that
and I would expect that it "ignores" the label changes during the execution phase of the pod. Any thoughts on that? |
Then, is it ok to evict the executing(running) pod only after restarting the kubelet? If we have to document the behavior as it works now, |
@SergeyKanzhelev can you chime in the kubelet details? |
What happened?
Currently, a running pod is being evicted if the assigned node, which does not meet the pod's node affinity requirements anymore, restarts its kubelet.
xref: #123980 (comment)
What did you expect to happen?
https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity
According to the documentation, node affinity should only affect the scheduling phase. Therefore, the running pod should continue to run without interruption.
How can we reproduce it (as minimally and precisely as possible)?
kubectl label node TARGET_NODE foo=bar --overwrite
so that the pod is scheduled in the TARGET_NODE.kubectl label node TARGET_NODE foo=nonbar --overwrite
Anything else we need to know?
previous discussion: #101218 (comment)
Kubernetes version
Cloud provider
OS version
Install tools
Container runtime (CRI) and version (if applicable)
Related plugins (CNI, CSI, ...) and versions (if applicable)
The text was updated successfully, but these errors were encountered: