netpol - TCP Allow needed when switching from v1.9.3-eksbuild.7 to v1.10.1-eksbuild.[1-7]

[mvink-guida]

We were having a k8s cilium networkpolicy as shown below,running on k8s 1.27

  egress:
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
          rules:
            dns:
              - matchPattern: "*"
    - toEntities:
        - world

Using coredns v1.10.1-eksbuild.7 (or .1 or .4), when starting a busybox and doing a nslookup on registry.yarnpkg.com it seems to hang, no answer. coredns verbose logging shows only udp lines, but client does not get an answer.

When switching to coredns version v1.9.3-eksbuild.7 client gets an answer.

Using coredns v1.10.1-eksbuild.7 (or .1 or .4) with the networkpolicy on ANY for port 53 we were able to make it work and you can see the last two lines using tcp.

[INFO] 10.10.26.253:50072 - 15069 "AAAA IN registry.yarnpkg.com.customer.svc.cluster.local. udp 65 false 512" NXDOMAIN qr,aa,rd 158 0.000167929s
[INFO] 10.10.26.253:58656 - 39899 "A IN registry.yarnpkg.com.customer.svc.cluster.local. udp 65 false 512" NXDOMAIN qr,aa,rd 158 0.000174894s
[INFO] 10.10.26.253:60170 - 60612 "A IN registry.yarnpkg.com.svc.cluster.local. udp 56 false 512" NXDOMAIN qr,aa,rd 149 0.000201645s
[INFO] 10.10.26.253:60008 - 40306 "AAAA IN registry.yarnpkg.com.svc.cluster.local. udp 56 false 512" NXDOMAIN qr,aa,rd 149 0.000308035s
[INFO] 10.10.26.253:60497 - 46619 "A IN registry.yarnpkg.com.cluster.local. udp 52 false 512" NXDOMAIN qr,aa,rd 145 0.000506197s
[INFO] 10.10.26.253:45379 - 38426 "AAAA IN registry.yarnpkg.com.cluster.local. udp 52 false 512" NXDOMAIN qr,aa,rd 145 0.000222034s
[INFO] 10.10.26.253:40105 - 26999 "AAAA IN registry.yarnpkg.com.eu-central-1.compute.internal. udp 68 false 512" NXDOMAIN qr,rd,ra 187 0.006375552s
[INFO] 10.10.26.253:48719 - 9518 "A IN registry.yarnpkg.com.eu-central-1.compute.internal. udp 68 false 512" NXDOMAIN qr,rd,ra 187 0.006472747s
[INFO] 10.10.26.253:38674 - 3983 "A IN registry.yarnpkg.com. udp 38 false 512" NOERROR qr,rd,ra 446 0.001697531s
[INFO] 10.10.26.253:52101 - 61023 "AAAA IN registry.yarnpkg.com. udp 38 false 512" NOERROR qr,tc,rd,ra 86 0.002337759s
[INFO] 10.10.26.253:42740 - 24461 "A IN registry.yarnpkg.com. tcp 38 false 65535" NOERROR qr,aa,rd,ra 446 0.000117197s
[INFO] 10.10.26.253:42756 - 60749 "AAAA IN registry.yarnpkg.com. tcp 38 false 65535" NOERROR qr,rd,ra 590 0.005075246s

Questions:

Thoughts on the different lookup udp/tcp behaviour between v1.9.3-eksbuild.7 and v1.10.1-eksbuild.* would be appreciated.

added:
_Maybe something in buffersize behaviour, as one of my colegues noted
116ee6f

Is there something documented on this behaviour._

[mvink-guida]

probably the message is bigger on 1.10 then 1.9.

2.3.4. Size limits

Various objects and parameters in the DNS have size limits.  They are
listed below.  Some could be easily changed, others are more
fundamental.

labels          63 octets or less

names           255 octets or less

TTL             positive values of a signed 32 bit number.

UDP messages    512 octets or less

"To comply with DNS standards, responses sent over UDP are no more than 512 bytes in size. Responses exceeding 512 bytes are truncated and the resolver must re-issue the request over TCP."

https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/DNSBehavior.html

[chrisohaver]

[INFO] 10.10.26.253:52101 - 61023 "AAAA IN registry.yarnpkg.com. udp 38 false 512" NOERROR qr,tc,rd,ra 86 0.002337759s
[INFO] 10.10.26.253:42740 - 24461 "A IN registry.yarnpkg.com. tcp 38 false 65535" NOERROR qr,aa,rd,ra 446 0.000117197s
[INFO] 10.10.26.253:42756 - 60749 "AAAA IN registry.yarnpkg.com. tcp 38 false 65535" NOERROR qr,rd,ra 590 0.005075246s

Note the tc header flag set in the prior AAAA response. This means the response was truncated. This should prompt the client to retry over TCP, which it did as expected.

I recall there being a bug that was fixed at some point regarding stripping of TC bits from upstream responses. Possibly the bug was present in 1.9.3 and fixed some time before 1.10.1 release. I don't know the exact version.

Anyways, the current behavior appears correct.

[chrisohaver]

Also possibly related to recent fix that went in to fix malformed overflowed responses from upstream.