Pods unable to access each other, pod CIDR in cali-cidr-block

[rf152]

The ingress-nginx pod is not able to access the other pods in order to allow ingress to the applications running on the cluster. On closer inspection, all traffic inter-pod is not working.

Putting a manual rule at the top of the forward chain to allow inter-pod traffic works, but Calico quickly places its cali-FORWARD rule before it.

The iptables rule it is hitting is in the cali-cidr-block chain:

Chain cali-cidr-block (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       0    --  *      *       0.0.0.0/0            10.1.128.0/18        /* cali:MBFnMC9HO-P01G4D */
   18  1080 DROP       0    --  *      *       0.0.0.0/0            10.1.64.0/18         /* cali:9vWkRqRgEwhV2A-K */

10.1.64.0/18 is the pod network

Expected Behavior

ingress-nginx should be able to reach pods in order to serve applications

Current Behavior

ingress-nginx is not able to reach the pods to serve the applications. no pods are able to communicate with each other.

Steps to Reproduce (for bugs)

1. Install with kubespray enabling BGP, disabling nat_outgoing, and peering with a BGP router
2. Deploy ingress-nginx
3. Deploy a web application and create an ingress resource for it
4. browse to the newly created ingress

Your Environment

• Calico version: v3.26.4
• ClusterType: kubespray,bgp,kubeadm,kdd,k8s
• Orchestrator version kubernetes
• Operating System and version: debian 12
• Link to your project (optional):

[fasaxc]

IIUC, the cali-cidr-block chain drops traffic to advertised cluster IPs that didn't get resolved to Pod IPs by kube-proxy. Perhaps you have misconfigured the service CIDR to overlap with your Pod CIDR range?

[rf152]

Definitely don't have overlapping subnets - I have 10.1.0.0/18 for service, 10.1.128.0/18 for load balancers, and 10.1.64.0/18 for pods. It was stood up with kubespray, if that makes any difference?

[rf152]

I've got the same issue on a brand new cluster that I have set up. No pod is able to browse to any service. Browsing directly to the pod IP is fine.