Dns proxy use original source address and port

[jrajahalme]

Make Cilium DNS proxy transparent by using the original source address (and port) in upstream connections. Since clients can issue multiple requests from the same port without waiting for the responses in between, we must support multiple requests in flight at the same time on the client side as well. To this end we share the DNS client between all requests that use the same upstream 5-tuple.

Support for shared clients is imported from a new version of cilium/dns.

Cilium DNS proxy now uses the original pod's address as the source address towards the DNS servers.

[jrajahalme]

In local testing cilium monitor now shows the original pod source address and port is also used on the overlay:

<- endpoint 1929 flow 0x0 , identity 3664->unknown state unknown ifindex 0 orig-ip 0.0.0.0: 10.244.1.203:43094 -> 10.96.0.10:53 udp
Policy verdict log: flow 0x0 local EP ID 1929, remote ID 36693, proto 17, egress, action redirect, auth: disabled, match L3-L4, 10.244.1.203:43094 -> 10.244.0.124:53 udp
-> proxy port 37757 flow 0x0 , identity 3664->unknown state new ifindex 0 orig-ip 0.0.0.0: 10.244.1.203:43094 -> 10.244.0.124:53 udp
<- endpoint 1929 flow 0x0 , identity 3664->unknown state unknown ifindex 0 orig-ip 0.0.0.0: 10.244.1.203:43094 -> 10.96.0.10:53 udp
-> proxy port 37757 flow 0x0 , identity 3664->unknown state established ifindex 0 orig-ip 0.0.0.0: 10.244.1.203:43094 -> 10.244.0.124:53 udp
<- proxy flow 0x5164dc81 , identity 3664->unknown state unknown ifindex 0 orig-ip 0.0.0.0: 10.244.1.203:43094 -> 10.244.0.124:53 udp
-> overlay flow 0x5164dc81 , identity 3664->36693 state unknown ifindex cilium_vxlan orig-ip 0.0.0.0: 10.244.1.203:43094 -> 10.244.0.124:53 udp
<- proxy flow 0xa0e9948a , identity 3664->unknown state unknown ifindex 0 orig-ip 0.0.0.0: 10.244.1.203:43094 -> 10.244.0.124:53 udp
-> overlay flow 0xa0e9948a , identity 3664->36693 state unknown ifindex cilium_vxlan orig-ip 0.0.0.0: 10.244.1.203:43094 -> 10.244.0.124:53 udp
<- overlay flow 0x52db30f3 , identity 36693->unknown state unknown ifindex cilium_vxlan orig-ip 0.0.0.0: 10.244.0.124:53 -> 10.244.1.203:43094 udp
-> proxy flow 0x52db30f3 , identity 36693->3664 state reply ifindex lxc53a2958fd80a orig-ip 10.244.0.124: 10.244.0.124:53 -> 10.244.1.203:43094 udp
<- overlay flow 0x52db30f3 , identity 36693->unknown state unknown ifindex cilium_vxlan orig-ip 0.0.0.0: 10.244.0.124:53 -> 10.244.1.203:43094 udp
-> proxy flow 0x52db30f3 , identity 36693->3664 state reply ifindex lxc53a2958fd80a orig-ip 10.244.0.124: 10.244.0.124:53 -> 10.244.1.203:43094 udp
<- proxy flow 0x0 , identity unknown->unknown state unknown ifindex 0 orig-ip 0.0.0.0: 10.244.0.124:53 -> 10.244.1.203:43094 udp
-> endpoint 1929 flow 0x0 , identity 36693->3664 state reply ifindex lxc53a2958fd80a orig-ip 10.244.0.124: 10.96.0.10:53 -> 10.244.1.203:43094 udp
<- endpoint 1929 flow 0x0 , identity 3664->unknown state unknown ifindex 0 orig-ip 0.0.0.0: 10.244.1.203:43094 -> 10.96.0.10:53 udp
-> proxy port 37757 flow 0x0 , identity 3664->unknown state established ifindex 0 orig-ip 0.0.0.0: 10.244.1.203:43094 -> 10.244.0.124:53 udp
<- proxy flow 0xe1185fe7 , identity 3664->unknown state unknown ifindex 0 orig-ip 0.0.0.0: 10.244.1.203:43094 -> 10.244.0.124:53 udp
-> overlay flow 0xe1185fe7 , identity 3664->36693 state unknown ifindex cilium_vxlan orig-ip 0.0.0.0: 10.244.1.203:43094 -> 10.244.0.124:53 udp
<- overlay flow 0x52db30f3 , identity 36693->unknown state unknown ifindex cilium_vxlan orig-ip 0.0.0.0: 10.244.0.124:53 -> 10.244.1.203:43094 udp
-> proxy flow 0x52db30f3 , identity 36693->3664 state reply ifindex lxc53a2958fd80a orig-ip 10.244.0.124: 10.244.0.124:53 -> 10.244.1.203:43094 udp
<- proxy flow 0x0 , identity unknown->unknown state unknown ifindex 0 orig-ip 0.0.0.0: 10.244.0.124:53 -> 10.244.1.203:43094 udp
-> endpoint 1929 flow 0x0 , identity 36693->3664 state reply ifindex lxc53a2958fd80a orig-ip 10.244.0.124: 10.96.0.10:53 -> 10.244.1.203:43094 udp

[jrajahalme]

dnsproxy gets timeouts waiting for response from the dns server when curl is issuing both IPv4 (A) and IPv6 (AAAA) queries at the same time. Works flawlessly if limiting to IPv4 only (curl -4 ...).

Functionally significant change is that now there are two upstream connections (one for A and other for AAAA records) at the same time, with the same 5-tuple. So the connections are fighting for the same packets. What I think happens on the timeout is that the "missing" response was delivered to the same socket as the successful one, but the connection does not read multiple answers from the same socket.

[jrajahalme]

Added commit to share a DNS client between requests that share the same upstream 5-tuple.

[jrajahalme]

One of the commits needs to be moved to cilium/dns.

[jrajahalme]

Moved the refactoring commit to cilium/dns, added TryLock support in sasha-s/go-deadlock/pull/30

[jrajahalme]

/test

[jrajahalme]

marked as blocked as we need to decide if we wait for sasha-s/go-deadlock#30 to merge or use my fork with it like in the 1st commit via go mod replace right now.

[jrajahalme]

Avoiding the use of original source address if it is known to be from the host networking namespace (host endpoint or localhost).

[jrajahalme]

/test

[bimmlerd]

I know it's still a draft, but I've briefly looked through and the code seems to have become more complex; thus questions around the approach arose

[jrajahalme]

Rebased to restart CI in hopes cloudflare is better today.

[jrajahalme]

/test