DHCP for control plane VIP fails: got an error while the discovery request: no matching response packet received

[Alphadelta14]

Describe the bug
When using kube-vip in just --controlplane node with --ddns and a DNS entry for VIP, DHCP does not succeed in my case.
I am just using this mode for testing (where I don't have a static set of VIPs available) but would like to be able to rely on this in other automated configuration environments.
I've tracked this down to a protocol level issue where a DHCP Offer packet is never received (despite being sent by the DHCP server). I filed a bug upstream to the dhcp library this uses: insomniacslk/dhcp#532.

To Reproduce
This was done on a single-node debian bookworm kubelet with a Calico eBPF CNI to narrow down conditions

1. Run an isc-dhcp-server within this subnet (192.168.96.1 is my gateway with no DHCP, i run the dhcp server on a debian bookworm node at 192.168.96.4)
2. Generate a kube-vip config as a static manifest: ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:v0.8.0 vip /kube-vip manifest pod --interface eth0 --address kube-api.cluster.internal --controlplane --ddns --arp --leaderElection | tee /etc/kubernetes/manifests/kube-vip.yaml
3. Observe kube-vip pod running in a kube-system namespace (kubectl get pods -n kube-system)
4. Logs for the service indicate that it failed to acquire the VIP via DHCP (kubectl logs -n kube-system kube-vip-7e61f5b45882)

Expected behavior
kube-vip would acquire a DHCP lease (e.g., 192.168.96.150) and use that as the VIP for the control plane.

Screenshots

time="2024-05-08T13:27:08Z" level=info msg="Starting kube-vip.io [v0.8.0]"
time="2024-05-08T13:27:08Z" level=info msg="namespace [kube-system], Mode: [ARP], Features(s): Control Plane:[true], Services:[false]"
time="2024-05-08T13:27:08Z" level=info msg="Using node name [7e61f5b45882]"
time="2024-05-08T13:27:08Z" level=info msg="prometheus HTTP server started"
time="2024-05-08T13:27:08Z" level=info msg="Starting Kube-vip Manager with the ARP engine"
time="2024-05-08T13:27:08Z" level=info msg="Beginning cluster membership, namespace [kube-system], lock name [plndr-cp-lock], id [7e61f5b45882]"
I0508 13:27:08.843180       1 leaderelection.go:250] attempting to acquire leader lease kube-system/plndr-cp-lock...
I0508 13:27:08.873045       1 leaderelection.go:260] successfully acquired lease kube-system/plndr-cp-lock
time="2024-05-08T13:27:08Z" level=info msg="Node [7e61f5b45882] is assuming leadership of the cluster"
time="2024-05-08T13:27:08Z" level=info msg="waiting for ip from dhcp"
time="2024-05-08T13:27:43Z" level=error msg="request failed, error: unable to receive an offer: got an error while the discovery request: no matching response packet received (waiting 10s)"
time="2024-05-08T13:28:28Z" level=error msg="request failed, error: unable to receive an offer: got an error while the discovery request: no matching response packet received (waiting 16.068956075s)"
time="2024-05-08T13:29:20Z" level=error msg="failed to get an IP address after 3 attempts, error unable to receive an offer: got an error while the discovery request: no matching response packet received, giving up"

Environment (please complete the following information):

• OS/Distro: Debian Bookworm
• Kubernetes Version: v1.28
• Kube-vip Version: v0.8.0

Kube-vip.yaml:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  name: kube-vip
  namespace: kube-system
spec:
  containers:
  - args:
    - manager
    env:
    - name: vip_arp
      value: "true"
    - name: port
      value: "6443"
    - name: vip_nodename
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    - name: vip_interface
      value: eth0
    - name: vip_log
      value: "5"
    - name: vip_cidr
      value: "32"
    - name: dns_mode
      value: first
    - name: cp_enable
      value: "true"
    - name: cp_namespace
      value: kube-system
    - name: vip_ddns
      value: "true"
    - name: vip_leaderelection
      value: "true"
    - name: vip_leasename
      value: plndr-cp-lock
    - name: vip_leaseduration
      value: "5"
    - name: vip_renewdeadline
      value: "3"
    - name: vip_retryperiod
      value: "1"
    - name: address
      value: kube-api.cluster.internal
    - name: prometheus_server
      value: :2112
    image: ghcr.io/kube-vip/kube-vip:v0.8.0
    imagePullPolicy: IfNotPresent
    name: kube-vip
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
        - NET_RAW
    volumeMounts:
    - mountPath: /etc/kubernetes/admin.conf
      name: kubeconfig
  hostAliases:
  - hostnames:
    - kubernetes
    ip: 127.0.0.1
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/admin.conf
    name: kubeconfig
status: {}

Additional context
Running isc-dhclient in the network namespace of the kube-vip pod (or just on the node) works just fine, as i described in insomniacslk/dhcp#532. I made sure to clear state and stopping the client and reset IPs between tests to avoid disturbing the state.