You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
$ kubectl -n kube-system logs kube-vip-ds-68ldv
time="2024-04-29T08:28:59Z" level=info msg="Starting kube-vip.io [v0.8.0]"
time="2024-04-29T08:28:59Z" level=info msg="namespace [kube-system], Mode: [ARP], Features(s): Control Plane:[true], Services:[true]"
time="2024-04-29T08:28:59Z" level=info msg="prometheus HTTP server started"
time="2024-04-29T08:28:59Z" level=info msg="Using node name [k3s-03]"
time="2024-04-29T08:28:59Z" level=info msg="Starting Kube-vip Manager with the ARP engine"
time="2024-04-29T08:28:59Z" level=info msg="beginning services leadership, namespace [kube-system], lock name [plndr-svcs-lock], id [k3s-03]"
I0429 08:28:59.926436 1 leaderelection.go:250] attempting to acquire leader lease kube-system/plndr-svcs-lock...
time="2024-04-29T08:28:59Z" level=info msg="Beginning cluster membership, namespace [kube-system], lock name [plndr-cp-lock], id [k3s-03]"
I0429 08:28:59.926733 1 leaderelection.go:250] attempting to acquire leader lease kube-system/plndr-cp-lock...
time="2024-04-29T08:28:59Z" level=info msg="Node [k3s-02] is assuming leadership of the cluster"
time="2024-04-29T08:28:59Z" level=info msg="new leader elected: k3s-02"
I0429 08:29:01.419796 1 leaderelection.go:260] successfully acquired lease kube-system/plndr-cp-lock
time="2024-04-29T08:29:01Z" level=info msg="Node [k3s-03] is assuming leadership of the cluster"
time="2024-04-29T08:29:01Z" level=info msg="Gratuitous Arp broadcast will repeat every 3 seconds for [192.168.100.100/enp1s0]"
I0429 08:29:05.139854 1 leaderelection.go:260] successfully acquired lease kube-system/plndr-svcs-lock
time="2024-04-29T08:29:05Z" level=info msg="(svcs) starting services watcher for all namespaces"
time="2024-04-29T08:31:25Z" level=info msg="(svcs) adding VIP [192.168.100.201] via enp1s0 for [default/blog]"
time="2024-04-29T08:31:25Z" level=info msg="[service] synchronised in 15ms"
time="2024-04-29T08:31:25Z" level=warning msg="(svcs) already found existing address [192.168.100.201] on adapter [enp1s0]"
time="2024-04-29T08:31:28Z" level=warning msg="Re-applying the VIP configuration [192.168.100.201] to the interface [enp1s0]"
Find out where the leader kube-vip Pod is and SSH into the node to check whether the load balancer IP address is actually assigned to the network interface that kube-vip is working on
$ ip addr show enp1s0
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 96:27:05:1e:11:d0 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.140/24 brd 192.168.100.255 scope global dynamic enp1s0
valid_lft 31355738sec preferred_lft 31355738sec
inet 192.168.100.100/32 scope global enp1s0
valid_lft forever preferred_lft forever
inet 192.168.100.201/32 scope global enp1s0
valid_lft forever preferred_lft forever
inet6 fe80::9427:5ff:fe1e:11d0/64 scope link
valid_lft forever preferred_lft forever
Delete the LB-type Service
kubectl delete svc blog
There's only one line of log generated by the kube-vip leader Pod:
time="2024-04-29T08:34:46Z" level=info msg="(svcs) [default/blog] has been deleted"
Go back to the node and see the IP address is still there
$ ip addr show enp1s0
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 96:27:05:1e:11:d0 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.140/24 brd 192.168.100.255 scope global dynamic enp1s0
valid_lft 31355589sec preferred_lft 31355589sec
inet 192.168.100.100/32 scope global enp1s0
valid_lft forever preferred_lft forever
inet 192.168.100.201/32 scope global enp1s0
valid_lft forever preferred_lft forever
inet6 fe80::9427:5ff:fe1e:11d0/64 scope link
valid_lft forever preferred_lft forever
Expected behavior
The load balancer IP address should be removed from the network interface of the node after the corresponding LB-type Service object is deleted.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
This was first found in a Harvester deployment (RKE2-based cluster) validating the load balancer functionality when upgrading kube-vip from v0.6.0 to v0.8.0 (details were recorded in harvester/harvester#5682), but the same issue is also spotted on a newly created K3s cluster, as described in the reproduction section. I hope this will help reduce the noise.
The text was updated successfully, but these errors were encountered:
I enabled the debug level and saw the same Service object was reconciled again and again.
time="2024-04-30T06:43:26Z" level=debug msg="(svcs) [blog] has been added/modified with addresses [[192.168.100.201]]"
time="2024-04-30T06:43:26Z" level=debug msg="[STARTING] Service Sync"
time="2024-04-30T06:43:26Z" level=debug msg="init enable service security: false"
time="2024-04-30T06:43:26Z" level=info msg="(svcs) adding VIP [192.168.100.201] via enp1s0 for [default/blog]"
time="2024-04-30T06:43:26Z" level=debug msg="(svcs) will update [default/blog]"
time="2024-04-30T06:43:26Z" level=debug msg="(svcs) broadcasting ARP update for 192.168.100.201 via enp1s0, every 3000ms"
time="2024-04-30T06:43:26Z" level=info msg="[service] synchronised in 15ms"
time="2024-04-30T06:43:26Z" level=warning msg="(svcs) already found existing address [192.168.100.201] on adapter [enp1s0]"
time="2024-04-30T06:43:26Z" level=debug msg="(svcs) [blog] has been added/modified with addresses [[192.168.100.201]]"
time="2024-04-30T06:43:26Z" level=debug msg="[STARTING] Service Sync"
time="2024-04-30T06:43:26Z" level=debug msg="isDHCP: false, newServiceAddress: 192.168.100.201"
time="2024-04-30T06:43:26Z" level=debug msg="(svcs) [blog] has been added/modified with addresses [[192.168.100.201]]"
time="2024-04-30T06:43:26Z" level=debug msg="[STARTING] Service Sync"
time="2024-04-30T06:43:26Z" level=debug msg="isDHCP: false, newServiceAddress: 192.168.100.201"
time="2024-04-30T06:43:29Z" level=warning msg="Re-applying the VIP configuration [192.168.100.201] to the interface [enp1s0]"
It seems that the Service's UID was not put into the activeSerivce map, which is why the peculiar behavior occurred.
Describe the bug
In ARP mode, kube-vip does not remove the load balancer IP address from the node's network interface when the LB-type Service object is deleted.
To Reproduce
k3sup
blog
withnginx
imageblog
Deploymentkubectl expose deployment blog --port 80 --type=LoadBalancer --overrides='{"metadata": {"annotations": {"kube-vip.io/loadbalancerIPs": "192.168.100.201"}}}'
kube-vip
Pod is and SSH into the node to check whether the load balancer IP address is actually assigned to the network interface that kube-vip is working onExpected behavior
The load balancer IP address should be removed from the network interface of the node after the corresponding LB-type Service object is deleted.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
Kube-vip.yaml
:Additional context
This was first found in a Harvester deployment (RKE2-based cluster) validating the load balancer functionality when upgrading kube-vip from v0.6.0 to v0.8.0 (details were recorded in harvester/harvester#5682), but the same issue is also spotted on a newly created K3s cluster, as described in the reproduction section. I hope this will help reduce the noise.
The text was updated successfully, but these errors were encountered: