fix: mTLS does not work when kubelet does not listen on 127.0.0.1

[weizhoublue]

bug description

with latest version v1.14.1, when kubelet listens on a specified local ip address but not 0.0.0.0, spire agent will fail to contact local kubelet with 127.0.0.1:10250, and mTLS feature does not work at all.

some log on issue enviroment:

when node has multiple network interface, kubelet listens on a specified local address 172.16.1.11 but not 0.0.0.0

[root@master1 ~]# cat /etc/kubernetes/kubelet-config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
nodeStatusUpdateFrequency: "10s"
failSwapOn: True
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: True
  x509:
    clientCAFile: /etc/kubernetes/ssl/ca.crt
authorization:
  mode: Webhook
staticPodPath: /etc/kubernetes/manifests
cgroupDriver: systemd
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
maxPods: 110
podPidsLimit: -1
address: 172.16.1.11
readOnlyPort: 0
healthzPort: 10248
healthzBindAddress: 127.0.0.1
kubeletCgroups: /system.slice/kubelet.service
clusterDomain: cluster.local
protectKernelDefaults: true
rotateCertificates: true
clusterDNS:
- 172.26.0.3
resolvConf: "/run/systemd/resolve/resolv.conf"
eventRecordQPS: 5
shutdownGracePeriod: 60s
shutdownGracePeriodCriticalPods: 20s

[root@master1 ~]# ss -lntp
LISTEN          0               4096                         172.16.1.11:10250                          0.0.0.0:*              users:(("kubelet",pid=1625,fd=22))

the spire agent fails to contact kubelet of its local node

[root@master1 ~]# kubectl logs -n cilium-spire   spire-agent-pjpsb
Defaulted container "spire-agent" out of: spire-agent, init (init)
time="2023-08-18T08:49:12Z" level=error msg="Failed to collect all selectors for PID" error="workload attestor \"k8s\" failed: rpc error: code = Internal desc = workloadattestor(k8s): unable to perform request: Get \"https://127.0.0.1:10250/pods\": dial tcp 127.0.0.1:10250: connect: connection refused" pid=8491 subsystem_name=workload_attestor
time="2023-08-18T08:49:13Z" level=error msg="Failed to collect all selectors for PID" error="workload attestor \"k8s\" failed: rpc error: code = Internal desc = workloadattestor(k8s): unable to perform request: Get \"https://127.0.0.1:10250/pods\": dial tcp 127.0.0.1:10250: connect: connection refused" pid=6845 subsystem_name=workload_attestor
time="2023-08-18T08:49:13Z" level=error msg="no identity issued" method=SubscribeToX509SVIDs service=spire.api.agent.delegatedidentity.v1.DelegatedIdentity subsystem_name=debug_api


[root@master1 ~]# kubectl logs -n kube-system    cilium-l5dk4
level=error msg="Error in delegate stream, restarting" error="rpc error: code = PermissionDenied desc = no identity issued" subsys=spire-delegate

it could not find any SPIFFE ID with following command , and mTLS could not work at all

kubectl exec -n cilium-spire spire-server-0 -c spire-server -- /opt/spire/bin/spire-server entry show -selector cilium:mutual-auth

how to fix

referring to spire doc and spire code , spire support to specify the kubelet address with an enviroment named "MY_NODE_NAME"

no matter what the local address listened by the kubelet, the spire agent could succeed to contact its local kubelet with status.hostIP

with this fix, it works well

Fix behavior where SPIRE doesn't work when kubelet does not listen on 127.0.0.1

[squeed]

Nice fix!

[squeed]

Just marking as do-not-merge until @meyskens has had a chance to review.

[weizhoublue]

@meyskens done

[meyskens]

LGTM

[meyskens]

/test

[weizhoublue]

it looks like the PR has no business making the 'ConformanceGatewayApi' CI fail ?

https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819

I will try to rebase the main branch

=== CONT  TestConformance/HTTPRouteRequestMirror/0_request_to_'/mirror'_should_go_to_infra-backend-v1
[1624](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1625)
    httproute-request-mirror.go:69: Making GET request to http://172.18.255.200/mirror
[1625](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1626)
    http.go:222: Response expectation failed for request: {URL: {Scheme:http Opaque: User: Host:172.18.255.200 Path:/mirror RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}, Host: , Protocol: HTTP, Method: GET, Headers: map[X-Echo-Set-Header:[]], UnfollowRedirect: false, Server: , CertPem: <truncated>, KeyPem: <truncated>}  not ready yet: expected status code to be 200, got 404 (after 3.1µs)
[1626](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1627)
    http.go:228: Request passed
[1627](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1628)
    mirror.go:41: Searching for the mirrored request log
[1628](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1629)
    mirror.go:42: Reading "gateway-conformance-infra/infra-backend-v2" logs
[1629](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1630)
panic: runtime error: invalid memory address or nil pointer dereference
[1630](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1631)
[signal SIGSEGV: segmentation violation code=0x1 addr=0xc0 pc=0x2087919]
[1631](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1632)

[1632](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1633)
goroutine 1756 [running]:
[1633](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1634)
k8s.io/client-go/kubernetes.(*Clientset).CoreV1(...)
[1634](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1635)
	/home/runner/work/cilium/cilium/vendor/k8s.io/client-go/kubernetes/clientset.go:309
[1635](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1636)
sigs.k8s.io/gateway-api/conformance/utils/kubernetes.DumpEchoLogs({0x2870b33, 0x19}, {0x285ae52, 0x10}, {0x2c52aa0, 0xc00022eab0}, 0x0)
[1636](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1637)
	/home/runner/work/cilium/cilium/vendor/sigs.k8s.io/gateway-api/conformance/utils/kubernetes/logs.go:50 +0x399
[1637](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1638)
sigs.k8s.io/gateway-api/conformance/utils/http.ExpectMirroredRequest.func1()
[1638](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1639)
	/home/runner/work/cilium/cilium/vendor/sigs.k8s.io/gateway-api/conformance/utils/http/mirror.go:43 +0x1df
[1639](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1640)
github.com/stretchr/testify/assert.Eventually.func1()
[1640](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1641)
	/home/runner/work/cilium/cilium/vendor/github.com/stretchr/testify/assert/assertions.go:1852 +0x22
[1641](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1642)
created by github.com/stretchr/testify/assert.Eventually in goroutine 1747
[1642](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1643)
	/home/runner/work/cilium/cilium/vendor/github.com/stretchr/testify/assert/assertions.go:1852 +0x21e
[1643](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1644)
FAIL	github.com/cilium/cilium/operator/pkg/gateway-api	233.042s
[1644](https://github.com/cilium/cilium/actions/runs/5925798992/job/16065936819#step:14:1645)

[meyskens]

/test

[meyskens]

@weizhoublue this was indeed an issue we just fixed on the main branch, thanks for rebasing!