Problem with the label kube-vip.io/has-ip

[aevenot]

Describe the bug
I want to use kube-vip to have a vip address used by my cilium policy (for firewalling reason). Then my cilium policy uses the label kube-vip.io/has-ip set by kube-vip but the label is set on the wrong node.

To Reproduce
Steps to reproduce the behavior:
Apply the following yaml and wait for the label kube-vip.io/has-ip with the correct IP and on the right node. Actually, the vip is set correctly on my node on the ens192 interface but the label on the node sometimes disappears or is set on the wrong node.

Expected behavior
The label is set on the node which have the vip address et any label on the other node

Screenshots

In my screenshot, the first three zones display a ssh connection on my three masters. We can see that the IP address .51 (set by kube-vip) is present but the label kube-vip.io/has-ip is set on the node .170 (the third) instead of the node .166 (the first).

Environment (please complete the following information):

• OS/Distro: RedHat RHEL 9.3
• Kubernetes Version: 1.24.17
• Kube-vip Version: 0.6.4 and 0.7.0

Kube-vip.yaml:
If Possible add in your kube-vip manifest (please remove anything that is confidential)

apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: '80'
    meta.helm.sh/release-name: kube-vip
    meta.helm.sh/release-namespace: kube-system
  generation: 80
  labels:
    app.kubernetes.io/managed-by: Helm
  name: kube-vip
  namespace: kube-system
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: kube-vip
      app.kubernetes.io/name: kube-vip
  template:
    metadata:
      annotations:
        cattle.io/timestamp: '2024-02-07T10:07:39Z'
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: kube-vip
        app.kubernetes.io/name: kube-vip
      namespace: kube-system
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node-role.kubernetes.io/etcd
                    operator: Exists
              - matchExpressions:
                  - key: node-role.kubernetes.io/control-plane
                    operator: Exists
      containers:
        - args:
            - manager
          env:
            - name: vip_leaseduration
              value: '5'
            - name: vip_renewdeadline
              value: '3'
            - name: vip_retryperiod
              value: '1'
            - name: address
              value: 10.x.x.51
            - name: svc_election
              value: 'false'
            - name: vip_leaderelection
              value: 'true'
            - name: vip_interface
              value: ens192
            - name: cp_namespace
              value: kube-system
            - name: vip_leasename
              value: plndr-cp-lock2
            - name: egress_withnftables
              value: 'true'
            - name: svc_enable
              value: 'true'
            - name: lb_enable
              value: 'false'
            - name: cp_enable
              value: 'true'
            - name: vip_arp
              value: 'true'
            - name: vip_cidr
              value: '32'
            - name: vip_leaderelection
              value: 'true'
            - name: vip_loglevel
              value: '5'
            - name: enable_node_labeling
              value: 'true'
          image: >-
            ghcr.io/kube-vip/kube-vip-iptables:v0.7.0
          imagePullPolicy: Always
          name: kube-vip
          resources: {}
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
                - NET_RAW
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      hostNetwork: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: kube-vip
      serviceAccountName: kube-vip
      terminationGracePeriodSeconds: 30
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists
        - effect: NoExecute
          key: node-role.kubernetes.io/etcd
          operator: Exists

Additional context
Cluster RKE2 1.24 (on Rancher)with cilium cni

My cilium Policy
apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
annotations:
name: egress-sample
spec:
destinationCIDRs:
- 0.0.0.0/0
egressGateway:
egressIP: 10.xx.xx.51
nodeSelector:
matchLabels:
kube-vip.io/has-ip: 10.xx.xx.51
selectors:
- podSelector:
matchLabels:
app: nginx

[lechugaletal]

I'm currently facing the same issue, there are two things to have into account:

1. This function only works when kube-vip is configured with service election flag set to false:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  name: kube-vip
  namespace: kube-system
spec:
  containers:
  - args:
    - manager
    env:
    - name: svc_election
      value: "false"
[...]

I'm not sure if this behaviour is intended, but this is defined at pkg/manager/manager_arp.go l121.

2. The label value is always the value from address config parameter, which can lead to confusion if you plan to use more than one IP at your kube-vip instance. Also this makes impossible to track where is a certain IP assigned by using the node label.

Expected behavior

When using multiple addresses, kube-vip should label the node with a list of the addresses that this same node holds.

Additional thoughts

While troubleshooting this issue i've also noticed that in the case where a node (i.e.: node01) looses connectivity to the controlplane, and this node has a VIP assigned, another node (i.e.: node02) assumes the leadership (as expected) and adds the kube-vip.io/has-ip label to itself without removing the label from node01. The label remains "duplicated" until node01 reconnects to te cluster.

[jisnardo]

I've checked this behaviour too https://github.com/kube-vip/kube-vip/blob/main/pkg/manager/manager_arp.go#L121 any suggestion @thebsdbox? Thanks in advance.

Related info #698 #617

May be @mjtrangoni or @fibbs could help.