Submariner doesn't work with Calico CNI

[zcz-1020]

Recently, I used calico CNI to build a k8s cluster on multiple virtual machines and encountered the same problem 【 #407 】. In cross-cluster scenarios, only pods on the gateway can communicate with each other. The data plane on non-gateway nodes cannot communicate with each other. I also followed the instructions in https://submariner.io/operations/deployment/calico/ to create IPPool, but it didn't work, everything is correct when using subctl diagnose command.Traffic can be transmitted from the vx-submariner of other nodes in the cluster to the vx-submariner of the gateway node, but the packet is not sent to the vxlan-tunnel of the gateway node. As a result, a tunnel cannot be established with the remote cluster.

The environment version is as follows:
k8s: 1.21.0
calico: 3.21
submariner: 0.13.0

[zcz-1020]

Recently, we spent some time trying to locate the problem. It is found that packet loss occurs on the FORWARD chain of the filter table. The default policy of the FORWARD chain is DROP. Because the request packet does not match any of the rules, the packet is discarded. After the default policy of the Forward chain is changed to ACCEPT or the rules that can be matched by data packets are added, the access is normal.

[sridhargaddam]

@zcz-1020 we recently validated Calico with Submariner and noticed that there are some issues when Calico is deployed with IPIP overlay tunnels. But when we changed the Calico configuration to VxLAN overlay tunnels and configured the required Submariner IPPools as documented [here](Calico supports different types of overlay networking. Currently, Submariner is validated only when Calico is deployed with VXLAN encapsulation.), we noticed that everything works fine. Did you deploy Calico with IPIP tunnels or VxLAN tunnels?

[ii2day]

@zcz-1020 we recently validated Calico with Submariner and noticed that there are some issues when Calico is deployed with IPIP overlay tunnels. But when we changed the Calico configuration to VxLAN overlay tunnels and configured the required Submariner IPPools as documented [here](Calico supports different types of overlay networking. Currently, Submariner is validated only when Calico is deployed with VXLAN encapsulation.), we noticed that everything works fine. Did you deploy Calico with IPIP tunnels or VxLAN tunnels?

Hi I have the same problem, I determined that my calico is in vxlan mode, my calico version 3.24.0
my submariner cableDriver is vxlan

[sridhargaddam]

@zcz-1020 we recently validated Calico with Submariner and noticed that there are some issues when Calico is deployed with IPIP overlay tunnels. But when we changed the Calico configuration to VxLAN overlay tunnels and configured the required Submariner IPPools as documented [here](Calico supports different types of overlay networking. Currently, Submariner is validated only when Calico is deployed with VXLAN encapsulation.), we noticed that everything works fine. Did you deploy Calico with IPIP tunnels or VxLAN tunnels?

Hi I have the same problem, I determined that my calico is in vxlan mode, my calico version 3.24.0 my submariner cableDriver is vxlan

We enhanced subctl recently to validate the Calico Overlay mode and also the IPPools that Submariner requires. Can you try running subctl diagnose cni using the latest devel which you can download as shown below.

curl -Ls https://get.submariner.io | VERSION=devel bash

[zcz-1020]

@sridhargaddam I'm using IPIP mode deployment, but I understand that this has nothing to do with IPIP or VXLAN mode. You can focus on the question I said above.
It is found that packet loss occurs on the FORWARD chain of the filter table. The default policy of the FORWARD chain is DROP. Because the request packet does not match any of the rules, the packet is discarded. After the default policy of the Forward chain is changed to ACCEPT or the rules that can be matched by data packets are added, the access is normal.

[zcz-1020]

@sridhargaddam I have another question. Recently, we are testing the calico BGP mode. We find that the data plane of pods in different clusters is also unreachable. Have you verified this before?

[sridhargaddam]

@sridhargaddam I'm using IPIP mode deployment, but I understand that this has nothing to do with IPIP or VXLAN mode. You can focus on the question I said above. It is found that packet loss occurs on the FORWARD chain of the filter table. The default policy of the FORWARD chain is DROP. Because the request packet does not match any of the rules, the packet is discarded. After the default policy of the Forward chain is changed to ACCEPT or the rules that can be matched by data packets are added, the access is normal.

Not sure if the above observation applies to Gateway node or non-Gateway node, but either way when we configure the IPPools for the remote clusters (as documented here), the remote cluster packets should not be dropped AFAICT. Anyways, currently we do not validate Calico in our CI pipeline and this needs some investigation.

CC: @yboaron JFYI

[sridhargaddam]

@sridhargaddam I have another question. Recently, we are testing the calico BGP mode. We find that the data plane of pods in different clusters is also unreachable. Have you verified this before?

We verified Calico with VxLAN overlay in which BGP is disabled.

[yboaron]

@zcz-1020 Can you share the content of Calico's default ippool from your environment?

[zcz-1020]

@yboaron Sorry, the environment is gone. My IPPool is configured according to this . The cidr in the spec in the IPPool is set to the pod_cidr and the service_cidr of the remote cluster.

[yboaron]

@yboaron Sorry, the environment is gone. My IPPool is configured according to this . The cidr in the spec in the IPPool is set to the pod_cidr and the service_cidr of the remote cluster.

I meant to the content of the default ippool (named 'default-ipv4-ippool') and not the ippools created to allow Submariner functionality with Calico CNI.

In case you have an environment please run (on both clusters):
A. Update default ippool spec to one of [1] or [2]
B. Restart all Submariner submariner-routeagent pods by running [3] command

And then run Submariner connectivity test.

Let us know if you still have issues.

[1]
ipipMode: Always
vxlanMode: Never
[2]
ipipMode: Never
vxlanMode: Always
[3]
kubectl delete pod -n submariner-operator -l app=submariner-routeagent

[leberknecht]

Maybe a little bit off-topic, but maybe also a little bit on-topic: i have an old gke cluster (i think we started it with 1.13) and im pretty sure we once installed calico manually back then to use NetworkPolicy resources. Now its been upgraded over time to 1.24, and the gcloud console UI shows Calico Kubernetes Network policy | Enabled

But subctl diagnose cni gives

 ⚠ Checking Submariner support for the CNI network plugin
 ⚠ Submariner could not detect the CNI network plugin and is using ("generic") plugin. It may or may not work.
 ✓ Trying to detect the Calico ConfigMap 

It seems odd that it does find a config-map but cant detect the plugin.. just out of curiosity, what config-map does it look for here?

[sridhargaddam]

@leberknecht the Network Plugin discovery code looks for configMap named "calico-config".

For more details, you can see the corresponding code here
https://github.com/submariner-io/submariner-operator/blob/0a98c9c95d5bfb10d3a7fcb0507f741397160c23/pkg/discovery/network/calico.go#L43

[stale]

This issue has been automatically marked as stale because it has not had activity for 60 days. It will be closed if no further activity occurs. Please make a comment if this issue/pr is still valid. Thank you for your contributions.

[sridhargaddam]

@leberknecht the Network Plugin discovery code looks for configMap named "calico-config".

For more details, you can see the corresponding code here https://github.com/submariner-io/submariner-operator/blob/0a98c9c95d5bfb10d3a7fcb0507f741397160c23/pkg/discovery/network/calico.go#L43

Also, we recently enhanced the discovery code for Calico CNI - submariner-io/submariner-operator#2769

[sridhargaddam]

@leberknecht did you get a chance to try out the suggestion from @yboaron here - #2121 (comment)

[cangyin]

I encountered the same problem, where cross cluster communication only works on gateway nodes, while non-gateway nodes not.

[cangyin]

Recently, we spent some time trying to locate the problem. It is found that packet loss occurs on the FORWARD chain of the filter table. The default policy of the FORWARD chain is DROP. Because the request packet does not match any of the rules, the packet is discarded. After the default policy of the Forward chain is changed to ACCEPT or the rules that can be matched by data packets are added, the access is normal.

How do you fixed the iptables rules? Could you detail the operations, thanks !

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further
activity occurs. Thank you for your contributions.

[yboaron]

Starting from release V0.16.0 Submariner automatically creates the necessary Calico IPPools for remote cluster connectivity when the Calico API Server is installed in the cluster and Calico CNI detection also includes searching for calico-node CNI pods when the calico-config map is not detected.

Please try to install Submariner with Calico [1] again using the latest version of Submariner and let us know if you encounter any problem.

I'm going to close this issue.

[1]
https://submariner.io/operations/deployment/calico/