New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
upgrade-health-check Job fails on a single control plane node cluster after drain #3050
Comments
thanks for testing @ilia1243 this is tricky problem. but either way there is a regression in 1.30 that we need to fix. the only problem here seems to be with the CreateJob logic.
^ this completes the upgrade of a single node CP and addons are applied correctly. one option is to skip this check if there is a single CP node in the cluster. |
another option (for me less preferred) is to make the CreateJob health check return a warning instead of an error. |
+1 for skip I need to check it again. I think I faiI for another reason that I did not install CNI for control plane and the pod failed for no CNI(not sure if this is a general use case, in this case the job should be run in hostNetwork.). I will do some test tomorrow. |
My suggestion is to print a warning and skip Job creation when there are no nodes to schedule. WDYT? |
IIUC, the only way to test if a job pod can schedule somewhere is to try to create the same job? the problem is that this preflight check's purpose is exactly that - check if the cluster accepts workloads. i don't even remember why we added it, but now we need a fix it right away. perhaps later we can discuss removing it. |
we could look at Unschedulable taints on nodes, which means they were cordoned. but listing all nodes on every kubeadm upgrade command will be very expensive in big scale clusters with many nodes. so i am starting to think we should just convert this check to a preflight warning. |
I'm not sure whether it is an expected patch for this issue? i.e. add a new toleration into job like
FYI:
Or just convert this check to a preflight warning? |
i have a WIP PR for this.
i don't know... ideally a node should be drained before upgrading kubelet. we do upgrade coredns and kube-proxy for a single node cluster while the node is drained with kubeadm upgrade apply, but we do ignore daemon sets anyway and the coredns pods will remain pending if the node is not schedulable. so technically for the addons we don't schedule new pods IIUC. |
please see kubernetes/kubernetes#124503 |
@carlory came up with a good idea how to catch the scenario. more reviews are appreciated. |
fix will be added to 1.30.1 |
1.30.1 is out with the fix |
Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT
Versions
kubeadm version (use
kubeadm version
): v1.30.0Environment:
kubectl version
): v1.30.0uname -a
): 5.15.0-50-genericWhat happened?
kubeadm upgrade apply v1.30.0
fails withIt seems that previously the pod was Pending as well, but this was ignored, because the jobs was successfully deleted in
defer
and return value was overridden with nil.https://github.com/kubernetes/kubernetes/blob/v1.29.1/cmd/kubeadm/app/phases/upgrade/health.go#L151
Similar issue #2035
What you expected to happen?
There might be no need to create the job.
How to reproduce it (as minimally and precisely as possible)?
See What happened?
The text was updated successfully, but these errors were encountered: