You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Registry pods under container-registry namespace is stuck in pending state. The microk8s vm was created on an aws instance using packer and converted to an AMI. Images were added to the image with registry service before AMI was created. When new instance is created using the AMI, the registry service gets stuck in pending state.
journalctl shows the issue as follows
May 11 13:58:42 ip-172-31-22-250 microk8s.daemon-kubelite[44501]: E0511 13:58:42.338492 44501 schedule_one.go:154] "Error selecting node for pod" err="nodeinfo not found for node name "ip-172-31-32-250"" pod="container-registry/registry-6c9fcc695f-lf4fc"
May 11 13:58:42 ip-172-31-22-250 microk8s.daemon-kubelite[44501]: E0511 13:58:42.338528 44501 schedule_one.go:1004] "Error scheduling pod; retrying" err="nodeinfo not found for node name "ip-172-31-32-250"" pod="container-registry/registry-6c9fcc695f-lf4fc"
When checking the nodes, the node IP is different as the new VM was created from the AMI image. Microk8s scheduler tries to connect to registry hostname from the original VM which was used to create the AMI.
Original IP : ip-172-31-32-250
New VM IP : ip-172-31-22-250
Refereces to the old ip exist in the following files inside /var/snap
Summary
Registry pods under container-registry namespace is stuck in pending state. The microk8s vm was created on an aws instance using packer and converted to an AMI. Images were added to the image with registry service before AMI was created. When new instance is created using the AMI, the registry service gets stuck in pending state.
journalctl shows the issue as follows
May 11 13:58:42 ip-172-31-22-250 microk8s.daemon-kubelite[44501]: E0511 13:58:42.338492 44501 schedule_one.go:154] "Error selecting node for pod" err="nodeinfo not found for node name "ip-172-31-32-250"" pod="container-registry/registry-6c9fcc695f-lf4fc"
May 11 13:58:42 ip-172-31-22-250 microk8s.daemon-kubelite[44501]: E0511 13:58:42.338528 44501 schedule_one.go:1004] "Error scheduling pod; retrying" err="nodeinfo not found for node name "ip-172-31-32-250"" pod="container-registry/registry-6c9fcc695f-lf4fc"
When checking the nodes, the node IP is different as the new VM was created from the AMI image. Microk8s scheduler tries to connect to registry hostname from the original VM which was used to create the AMI.
Original IP : ip-172-31-32-250
New VM IP : ip-172-31-22-250
Refereces to the old ip exist in the following files inside /var/snap
/var/snap/microk8s/6750/credentials/kubelet.config: user: system:node:ip-172-31-32-250
/var/snap/microk8s/6750/credentials/kubelet.config:- name: system:node:ip-172-31-32-250
/var/snap/microk8s/6750/var/tmp/csr-dqlite.conf:DNS = ip-172-31-32-250
root@ip-172-31-22-250:~# kubectl get nodes -o yaml|kubectl neat
apiVersion: v1
items:
kind: Node
metadata:
annotations:
node.alpha.kubernetes.io/ttl: "0"
projectcalico.org/IPv4Address: 172.31.22.250/20
projectcalico.org/IPv4VXLANTunnelAddr: 10.1.183.196
volumes.kubernetes.io/controller-managed-attach-detach: "true"
labels:
beta.kubernetes.io/arch: amd64
beta.kubernetes.io/os: linux
kubernetes.io/arch: amd64
kubernetes.io/hostname: ip-172-31-22-250
kubernetes.io/os: linux
microk8s.io/cluster: "true"
node.kubernetes.io/microk8s-controlplane: microk8s-controlplane
name: ip-172-31-22-250
kind: List
metadata:
resourceVersion: ""
What Should Happen Instead?
registry pods should initialize
Reproduction Steps
This is reproducible in all recent versions of microk8s I believe post 1.22. I have verified against 1.24-1.29
Introspection Report
Done
Can you suggest a fix?
No
Are you interested in contributing with a fix?
No
inspection-report-20240511_140434.tar.gz
The text was updated successfully, but these errors were encountered: