Cilium dropping UDP fragments of packets of certain sizes

[El-Begawy]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

We're running a cluster in an OpenStack environment with the config attached in the end. The nodes in the cluster have several network interfaces (eth0, eth1). The problem appears when UDP packets are sent to eth1 (a public network interface) that have sizes between (1473 -> 1475) (inclusive).

eth0 MTU: 1450
eth1 MTU: 1500

The problem started appearing on the ingress side after updating to version 1.14.4 from 1.12.15. Detaching cil_from_netdev-eth1 bpf part in ingress tc causes the packets to stop dropping.

Same problem appears for the same packet sizes in the egress side if enable-bpf-masquerade is enabled with the following logs:

xx drop (Invalid packet) flow 0x0 to endpoint 0, ifindex 3, file nodeport.h:2893, , identity unknown->unknown: <redacted> -> <redacted>

agent-not-ready-taint-key: node.cilium.io/agent-not-ready
arping-refresh-period: 30s
auto-direct-node-routes: 'true'
bpf-lb-external-clusterip: 'false'
bpf-lb-map-max: '65536'
bpf-lb-sock: 'false'
bpf-lb-sock-hostns-only: 'true'
bpf-map-dynamic-size-ratio: '0.0025'
bpf-policy-map-max: '16384'
bpf-root: /sys/fs/bpf
cgroup-root: /run/cilium/cgroupv2
cilium-endpoint-gc-interval: 5m0s
cluster-id: '0'
cluster-name: default
cni-exclusive: 'true'
cni-log-file: /var/run/cilium/cilium-cni.log
cnp-node-status-gc-interval: 0s
custom-cni-conf: 'false'
debug: 'false'
debug-verbose: ''
direct-routing-device: eth0
disable-cnp-status-updates: 'true'
egress-gateway-reconciliation-trigger-interval: 1s
enable-auto-protect-node-port-range: 'true'
enable-bgp-control-plane: 'false'
enable-bpf-clock-probe: 'false'
enable-endpoint-health-checking: 'true'
enable-health-check-nodeport: 'true'
enable-health-checking: 'true'
enable-host-legacy-routing: 'false'
enable-hubble: 'true'
enable-ipv4: 'true'
enable-ipv4-big-tcp: 'false'
enable-ipv4-masquerade: 'true'
enable-ipv6: 'false'
enable-ipv6-big-tcp: 'false'
enable-ipv6-masquerade: 'false'
enable-k8s-networkpolicy: 'true'
enable-k8s-terminating-endpoint: 'true'
enable-l2-neigh-discovery: 'true'
enable-l7-proxy: 'true'
enable-local-redirect-policy: 'true'
enable-metrics: 'true'
enable-policy: default
enable-remote-node-identity: 'true'
enable-sctp: 'false'
enable-svc-source-range-check: 'true'
enable-vtep: 'false'
enable-well-known-identities: 'false'
enable-xt-socket-fallback: 'true'
external-envoy-proxy: 'false'
hubble-disable-tls: 'false'
hubble-listen-address: ':4244'
hubble-socket-path: /var/run/cilium/hubble.sock
hubble-tls-cert-file: /var/lib/cilium/tls/hubble/server.crt
hubble-tls-client-ca-files: /var/lib/cilium/tls/hubble/client-ca.crt
hubble-tls-key-file: /var/lib/cilium/tls/hubble/server.key
identity-allocation-mode: crd
identity-gc-interval: 15m0s
identity-heartbeat-timeout: 30m0s
install-no-conntrack-iptables-rules: 'false'
ipam: kubernetes
ipam-cilium-node-update-rate: 15s
k8s-client-burst: '10'
k8s-client-qps: '5'
kube-proxy-replacement: strict
kube-proxy-replacement-healthz-bind-address: ''
mesh-auth-enabled: 'true'
mesh-auth-gc-interval: 5m0s
mesh-auth-queue-size: '1024'
mesh-auth-rotated-identities-queue-size: '1024'
monitor-aggregation: medium
monitor-aggregation-flags: all
monitor-aggregation-interval: 5s
node-port-bind-protection: 'true'
node-port-range: 3000,5000
nodes-gc-interval: 5m0s
operator-api-serve-addr: 127.0.0.1:9234
operator-prometheus-serve-addr: ':9963'
preallocate-bpf-maps: 'false'
procfs: /host/proc
prometheus-serve-addr: ':9962'
proxy-connect-timeout: '2'
proxy-max-connection-duration-seconds: '0'
proxy-max-requests-per-connection: '0'
proxy-prometheus-port: '9964'
remove-cilium-node-taints: 'true'
routing-mode: native
set-cilium-is-up-condition: 'true'
set-cilium-node-taints: 'true'
sidecar-istio-proxy-image: cilium/istio_proxy
skip-cnp-status-startup-clean: 'false'
synchronize-k8s-nodes: 'true'
tofqdns-dns-reject-response-code: refused
tofqdns-enable-dns-compression: 'true'
tofqdns-endpoint-max-ip-per-hostname: '50'
tofqdns-idle-connection-grace-period: 0s
tofqdns-max-deferred-connection-deletes: '10000'
tofqdns-proxy-response-max-delay: 100ms
unmanaged-pod-watcher-interval: '15'

Cilium monitor logs attached below

Cilium Version

1.14.4

Kernel Version

5.15.0-1051-aws #56~20.04.1-Ubuntu SMP Tue Nov 28 15:43:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

v1.23.5

Regression

No response

Sysdump

No response

Relevant log output

Logs when receiving UDP packets of increasing size from 1473 to 1475

------------------------------------------------------------------------------
Ethernet	{Contents=[..14..] Payload=[..26..] SrcMAC=<redacted> DstMAC=<redacted> EthernetType=IPv4 Length=0}
IPv4	{Contents=[..20..] Payload=[32] Version=4 IHL=5 TOS=0 Length=21 Id=63085 Flags= FragOffset=185 TTL=64 Protocol=UDP Checksum=27494 SrcIP=<redacted> DstIP=<redacted> Options=[] Padding=[]}
  Failed to decode layer: No decoder for layer type Fragment
CPU 01: MARK 0x0 FROM 1566 DROP: 35 bytes, reason Invalid packet
------------------------------------------------------------------------------
Ethernet	{Contents=[..14..] Payload=[..26..] SrcMAC=<redacted> DstMAC=<redacted> EthernetType=IPv4 Length=0}
IPv4	{Contents=[..20..] Payload=[32, 32] Version=4 IHL=5 TOS=0 Length=22 Id=63279 Flags= FragOffset=185 TTL=64 Protocol=UDP Checksum=27299 SrcIP=<redacted> DstIP=<redacted> Options=[] Padding=[]}
  Failed to decode layer: No decoder for layer type Fragment
CPU 01: MARK 0x0 FROM 1566 DROP: 36 bytes, reason Invalid packet
------------------------------------------------------------------------------
Ethernet	{Contents=[..14..] Payload=[..26..] SrcMAC=<redacted> DstMAC=<redacted> EthernetType=IPv4 Length=0}
IPv4	{Contents=[..20..] Payload=[32, 32, 32] Version=4 IHL=5 TOS=0 Length=23 Id=63439 Flags= FragOffset=185 TTL=64 Protocol=UDP Checksum=27138 SrcIP=<redacted> DstIP=<redacted> Options=[] Padding=[]}
  Failed to decode layer: No decoder for layer type Fragment
CPU 01: MARK 0x0 FROM 1566 DROP: 37 bytes, reason Invalid packet

Anything else?

No response

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct

[squeed]

Odd!
Would you be able to capture a sysdump on a node experiencing the issue?

Also, v1.14.4 is quite old; while I don't see any changes that should have an effect, it is almost 6 months old.

[El-Begawy]

I have updated to v1.15.4 and the issue disappeared.

I believe the issue was fixed in this PR: #25340

snat_v4_nat didn't account for fragments. In the packet size range where I was encountering the issue the second fragment would have a size less than 4. This would cause failure when reading port numbers causing the packet to drop.

[squeed]

@El-Begawy sounds good. I'll close this, then. Poke me if you feel it should be re-opened.