-
Notifications
You must be signed in to change notification settings - Fork 355
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pods not syncing quickly: pod-syncer Error updating pod: context deadline exceeded #1765
Comments
Let me summarise: the pods get scheduled on the host, but the vcluster api server doesn’t reflect the status correctly? Do you observe significant load (i.e. request latency, total requests increased) on the host api server when this happens? |
Looks similar to #1589 to me |
That's correct!
I do not. The vcluster pods on the host cluster are given a good amount of resources. 3 vCPUs, 4Gi of memory. Usually the node this vcluster pod is on is no where near these limits.
Yes, indeed! I suppose this issue can be marked as a duplicate. Thanks for catching that! |
@alextricity25 would you mind trying virtual Kubernetes version v1.29.4 as there was a pretty significant bug in v1.29.0 that caused issues (kubernetes/kubernetes#123448), which could be the problem for this |
@FabianKramm The default is v1.29.0. I'll override this to 1.29.4 to see if that does anything. It's difficult to iterate and test to see if this certain changes fixes this issue, because it doesn't happen all the time. I'll let you know! |
@FabianKramm I tested 1.29.4 (Server Version: v1.29.4+k3s1, I think embedded db/sqlite) with helm chart v0.20.0-beta.5 and still observed the issue once the metrics server was present in the host cluster. (host is 1.18.2) |
@FabianKramm I also observed the issue again on 1.29.5. Screen shot below. The pods in the vCluster context eventually did report the correct status, but it took about 1-2 minutes before they did. |
What happened?
I'm experiencing this rather strange bug where some pods are reporting to be stuck in the "Init: 0/1" status, but only when connected to the vcluster context. When connected to the host cluster context, the pod status reports correctly, however, the pod status in the vcluster occasionally is stuck in either "PendingCreate" or the "Init: 0/1" status, which causes downstream issues with my helm chart installation flow. The pod events while connected to the vcluster context report the following:
The below image shows the bug in action with two panes. The left pane is K9s connected to the vcluster context, and the right connected to the host cluster context. As you can see, the pods in the host cluster are "Running", but the same pods in the vcluster context are suck in the "Init:0/1" status.
Looking at the pod events, I see the following:
The pod-syncer error only appears when connected to the vcluster.
The only "error" that I noticed in the vcluster syncer logs is:
What did you expect to happen?
I expect the pods status while connected to the vcluster context to reflect the correct status.
How can we reproduce it (as minimally and precisely as possible)?
Anything else we need to know?
The pods while connected to the vcluster context will eventually report the correct status, but sometimes it takes 5 or 10 minutes before it does.
Host cluster Kubernetes version
Host cluster Kubernetes distribution
vlcuster version
v0.20.0-beta.5
Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)
OS and Arch
The text was updated successfully, but these errors were encountered: