Calico's performance is not as expected and seems unstable on my environment.

[chenlein]

Expected Behavior

Under normal circumstances, the network performance loss of Calico should be within 10%, right?

Current Behavior

Calico's performance is not as expected and seems unstable. How can I further analyze the cause of network degradation and solve this problem?

The network loss from testing on nodes to pods is over 50%！！！

Possible Solution

Have I missed any important system kernel parameters or Calico configurations?

Steps to Reproduce (for bugs)

Context

[root@master-1 ~]# kubectl get pods -o wide | grep iperf
iperf3-0        1/1     Running     0          63m    10.233.97.252    worker-3   <none>           <none>
iperf3-1        1/1     Running     0          63m    10.233.36.107    master-0   <none>           <none>
iperf3-2        1/1     Running     0          63m    10.233.226.112   worker-1   <none>           <none>
iperf3-3        1/1     Running     0          63m    10.233.36.108    master-0   <none>           <none>

[root@master-1 ~]# calicoctl get workloadEndpoint -o wide | grep iperf
worker--3-k8s-iperf3--0-eth0   iperf3-0    worker-3   10.233.97.252/32    caliee510e5ea76   kns.default,ksa.default.default                   
master--0-k8s-iperf3--1-eth0   iperf3-1    master-0   10.233.36.107/32    cali9d99c50de22   kns.default,ksa.default.default                   
worker--1-k8s-iperf3--2-eth0   iperf3-2    worker-1   10.233.226.112/32   cali9431011ac1f   kns.default,ksa.default.default                   
master--0-k8s-iperf3--3-eth0   iperf3-3    master-0   10.233.36.108/32    cali78a252f89c2   kns.default,ksa.default.default                   

1、node to node：

# master-0（10.83.3.48） --> worker-1（10.83.3.52）
[root@master-0 ~]# iperf3 -c worker-1 -i 1 -t 15
Connecting to host worker-1, port 5201
[  5] local 10.83.3.48 port 47406 connected to 10.83.3.52 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.10 GBytes  9.43 Gbits/sec   75   1.43 MBytes       
[  5]   1.00-2.00   sec  1.09 GBytes  9.41 Gbits/sec   23   1.43 MBytes       
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.44 MBytes       
[  5]   3.00-4.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.44 MBytes       
[  5]   4.00-5.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.45 MBytes       
[  5]   5.00-6.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.47 MBytes       
[  5]   6.00-7.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.49 MBytes       
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.53 MBytes       
[  5]   8.00-9.00   sec  1.09 GBytes  9.36 Gbits/sec  308   1.14 MBytes       
[  5]   9.00-10.00  sec  1.10 GBytes  9.42 Gbits/sec    0   1.37 MBytes       
[  5]  10.00-11.00  sec  1.09 GBytes  9.41 Gbits/sec    0   1.39 MBytes       
[  5]  11.00-12.00  sec  1.10 GBytes  9.42 Gbits/sec    0   1.40 MBytes       
[  5]  12.00-13.00  sec  1.09 GBytes  9.41 Gbits/sec    0   1.41 MBytes       
[  5]  13.00-14.00  sec  1.10 GBytes  9.42 Gbits/sec    0   1.41 MBytes       
[  5]  14.00-15.00  sec  1.10 GBytes  9.42 Gbits/sec    0   1.66 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-15.00  sec  16.4 GBytes  9.41 Gbits/sec  406             sender
[  5]   0.00-15.04  sec  16.4 GBytes  9.38 Gbits/sec                  receiver

iperf Done.

2、pod to node:

# iperf3-1 --> worker-1
[root@master-0 ~]# kubectl exec -ti iperf3-1 -- bash
root@iperf3-1:/# iperf3 -c 10.83.3.52 -i 1 -t 15
Connecting to host 10.83.3.52, port 5201
[  5] local 10.233.36.107 port 44216 connected to 10.83.3.52 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   632 MBytes  5.30 Gbits/sec  777    682 KBytes       
[  5]   1.00-2.00   sec   556 MBytes  4.67 Gbits/sec   32    724 KBytes       
[  5]   2.00-3.00   sec   544 MBytes  4.56 Gbits/sec   35   1.01 MBytes       
[  5]   3.00-4.00   sec   522 MBytes  4.38 Gbits/sec   44   1.02 MBytes       
[  5]   4.00-5.00   sec   465 MBytes  3.90 Gbits/sec    7   1.02 MBytes       
[  5]   5.00-6.00   sec   461 MBytes  3.87 Gbits/sec   63    984 KBytes       
[  5]   6.00-7.00   sec   496 MBytes  4.16 Gbits/sec   11    954 KBytes       
[  5]   7.00-8.00   sec   532 MBytes  4.47 Gbits/sec   10    974 KBytes       
[  5]   8.00-9.00   sec   491 MBytes  4.12 Gbits/sec    7   1010 KBytes       
[  5]   9.00-10.00  sec   522 MBytes  4.38 Gbits/sec   14   1.02 MBytes       
[  5]  10.00-11.00  sec   531 MBytes  4.46 Gbits/sec    8   1.08 MBytes       
[  5]  11.00-12.00  sec   415 MBytes  3.48 Gbits/sec    3   1.06 MBytes       
[  5]  12.00-13.00  sec   564 MBytes  4.73 Gbits/sec    4    830 KBytes       
[  5]  13.00-14.00  sec   492 MBytes  4.13 Gbits/sec   71    847 KBytes       
[  5]  14.00-15.00  sec   505 MBytes  4.24 Gbits/sec    1    930 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-15.00  sec  7.55 GBytes  4.32 Gbits/sec  1087             sender
[  5]   0.00-15.04  sec  7.55 GBytes  4.31 Gbits/sec                  receiver

iperf Done.
root@iperf3-1:/# iperf3 -c 10.83.3.52 -i 1 -t 15
Connecting to host 10.83.3.52, port 5201
[  5] local 10.233.36.107 port 46434 connected to 10.83.3.52 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   980 MBytes  8.22 Gbits/sec  1529    839 KBytes       
[  5]   1.00-2.00   sec   605 MBytes  5.08 Gbits/sec  261    888 KBytes       
[  5]   2.00-3.00   sec   572 MBytes  4.80 Gbits/sec   17   1.12 MBytes       
[  5]   3.00-4.00   sec   658 MBytes  5.52 Gbits/sec  183   1.02 MBytes       
[  5]   4.00-5.00   sec   609 MBytes  5.11 Gbits/sec   81    813 KBytes       
[  5]   5.00-6.00   sec   661 MBytes  5.55 Gbits/sec   28    990 KBytes       
[  5]   6.00-7.00   sec   569 MBytes  4.77 Gbits/sec   14   1.04 MBytes       
[  5]   7.00-8.00   sec   509 MBytes  4.27 Gbits/sec   12   1.06 MBytes       
[  5]   8.00-9.00   sec   991 MBytes  8.31 Gbits/sec    2   1.39 MBytes       
[  5]   9.00-10.00  sec   831 MBytes  6.97 Gbits/sec  196   1.09 MBytes       
[  5]  10.00-11.00  sec   846 MBytes  7.10 Gbits/sec   60   1022 KBytes       
[  5]  11.00-12.00  sec   586 MBytes  4.92 Gbits/sec   30   1.07 MBytes       
[  5]  12.00-13.00  sec   568 MBytes  4.76 Gbits/sec  711    945 KBytes       
[  5]  13.00-14.00  sec   726 MBytes  6.09 Gbits/sec    8   1.10 MBytes       
[  5]  14.00-15.00  sec   562 MBytes  4.72 Gbits/sec    4    858 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-15.00  sec  10.0 GBytes  5.75 Gbits/sec  3136             sender
[  5]   0.00-15.04  sec  10.0 GBytes  5.73 Gbits/sec                  receiver

iperf Done.
root@iperf3-1:/# iperf3 -c 10.83.3.52 -i 1 -t 15
Connecting to host 10.83.3.52, port 5201
[  5] local 10.233.36.107 port 47750 connected to 10.83.3.52 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   636 MBytes  5.33 Gbits/sec  958    796 KBytes       
[  5]   1.00-2.00   sec   585 MBytes  4.91 Gbits/sec   18   1.05 MBytes       
[  5]   2.00-3.00   sec   509 MBytes  4.27 Gbits/sec    7   1.04 MBytes       
[  5]   3.00-4.00   sec   544 MBytes  4.56 Gbits/sec   23   1.04 MBytes       
[  5]   4.00-5.00   sec   450 MBytes  3.77 Gbits/sec   16   1.03 MBytes       
[  5]   5.00-6.00   sec   521 MBytes  4.37 Gbits/sec    3   1.07 MBytes       
[  5]   6.00-7.00   sec   471 MBytes  3.95 Gbits/sec    7   1.03 MBytes       
[  5]   7.00-8.00   sec   586 MBytes  4.92 Gbits/sec   13    796 KBytes       
[  5]   8.00-9.00   sec   555 MBytes  4.66 Gbits/sec    6    888 KBytes       
[  5]   9.00-10.00  sec   601 MBytes  5.04 Gbits/sec    8   1010 KBytes       
[  5]  10.00-11.00  sec   506 MBytes  4.25 Gbits/sec   35   1024 KBytes       
[  5]  11.00-12.00  sec   571 MBytes  4.79 Gbits/sec   12   1.05 MBytes       
[  5]  12.00-13.00  sec   662 MBytes  5.56 Gbits/sec   19   1.10 MBytes       
[  5]  13.00-14.00  sec   480 MBytes  4.03 Gbits/sec    4    803 KBytes       
[  5]  14.00-15.00  sec   509 MBytes  4.27 Gbits/sec    2    871 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-15.00  sec  8.00 GBytes  4.58 Gbits/sec  1131             sender
[  5]   0.00-15.04  sec  7.99 GBytes  4.57 Gbits/sec                  receiver

iperf Done.
root@iperf3-1:/# iperf3 -c 10.83.3.52 -i 1 -t 15
Connecting to host 10.83.3.52, port 5201
[  5] local 10.233.36.107 port 47384 connected to 10.83.3.52 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   801 MBytes  6.72 Gbits/sec  1954    665 KBytes       
[  5]   1.00-2.00   sec   820 MBytes  6.88 Gbits/sec  153    836 KBytes       
[  5]   2.00-3.00   sec   888 MBytes  7.45 Gbits/sec  571   1.13 MBytes       
[  5]   3.00-4.00   sec   815 MBytes  6.84 Gbits/sec  309    911 KBytes       
[  5]   4.00-5.00   sec   581 MBytes  4.88 Gbits/sec    5    981 KBytes       
[  5]   5.00-6.00   sec   599 MBytes  5.02 Gbits/sec   13   1.06 MBytes       
[  5]   6.00-7.00   sec   605 MBytes  5.08 Gbits/sec   34    865 KBytes       
[  5]   7.00-8.00   sec   772 MBytes  6.48 Gbits/sec    7   1.10 MBytes       
[  5]   8.00-9.00   sec   978 MBytes  8.20 Gbits/sec  178   1.00 MBytes       
[  5]   9.00-10.00  sec   605 MBytes  5.08 Gbits/sec   25   1.11 MBytes       
[  5]  10.00-11.00  sec   542 MBytes  4.55 Gbits/sec   15    837 KBytes       
[  5]  11.00-12.00  sec   556 MBytes  4.67 Gbits/sec   17    926 KBytes       
[  5]  12.00-13.00  sec   562 MBytes  4.72 Gbits/sec   69    994 KBytes       
[  5]  13.00-14.00  sec   532 MBytes  4.47 Gbits/sec  420   1.07 MBytes       
[  5]  14.00-15.00  sec   519 MBytes  4.35 Gbits/sec    6   1.10 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-15.00  sec  9.94 GBytes  5.69 Gbits/sec  3776             sender
[  5]   0.00-15.04  sec  9.93 GBytes  5.67 Gbits/sec                  receiver

iperf Done.

3、node to pod：

# worker-1（10.83.3.52） --> iperf3-1（10.233.36.107）
[root@worker-1 ~]# iperf3 -c 10.233.36.107 -i 1 -t 15
Connecting to host 10.233.36.107, port 5201
[  5] local 10.83.3.52 port 60422 connected to 10.233.36.107 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.09 GBytes  9.37 Gbits/sec   14   1.24 MBytes       
[  5]   1.00-2.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.38 MBytes       
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec   12   1.40 MBytes       
[  5]   3.00-4.00   sec  1.04 GBytes  8.94 Gbits/sec  1438   1.02 MBytes       
[  5]   4.00-5.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.38 MBytes       
[  5]   5.00-6.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.39 MBytes       
[  5]   6.00-7.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.64 MBytes       
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.90 MBytes       
[  5]   8.00-9.00   sec  1.09 GBytes  9.33 Gbits/sec    0   1.90 MBytes       
[  5]   9.00-10.00  sec  1.10 GBytes  9.42 Gbits/sec    0   1.90 MBytes       
[  5]  10.00-11.00  sec  1.10 GBytes  9.42 Gbits/sec    0   1.90 MBytes       
[  5]  11.00-12.00  sec  1.09 GBytes  9.41 Gbits/sec   43   1.90 MBytes       
[  5]  12.00-13.00  sec  1.10 GBytes  9.42 Gbits/sec    0   1.90 MBytes       
[  5]  13.00-14.00  sec  1.10 GBytes  9.42 Gbits/sec    0   1.90 MBytes       
[  5]  14.00-15.00  sec  1.09 GBytes  9.41 Gbits/sec    0   1.90 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-15.00  sec  16.4 GBytes  9.37 Gbits/sec  1507             sender
[  5]   0.00-15.04  sec  16.4 GBytes  9.35 Gbits/sec                  receiver

iperf Done.

4、pod to pod：

# iperf3-1（10.233.36.107） --> iperf3-2（10.233.226.112）
[root@master-0 ~]# kubectl exec -ti iperf3-1 -- bash
root@iperf3-1:/# iperf3 -c 10.233.226.112 -i 1 -t 15
Connecting to host 10.233.226.112, port 5201
[  5] local 10.233.36.107 port 53670 connected to 10.233.226.112 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   867 MBytes  7.27 Gbits/sec  3080    587 KBytes       
[  5]   1.00-2.00   sec   825 MBytes  6.92 Gbits/sec  1864    551 KBytes       
[  5]   2.00-3.00   sec   654 MBytes  5.48 Gbits/sec  1865    448 KBytes       
[  5]   3.00-4.00   sec   680 MBytes  5.70 Gbits/sec  2474    416 KBytes       
[  5]   4.00-5.00   sec   684 MBytes  5.74 Gbits/sec  2649    431 KBytes       
[  5]   5.00-6.00   sec   556 MBytes  4.67 Gbits/sec  2480    238 KBytes       
[  5]   6.00-7.00   sec   685 MBytes  5.75 Gbits/sec  2410    229 KBytes       
[  5]   7.00-8.00   sec   740 MBytes  6.21 Gbits/sec  2563    580 KBytes       
[  5]   8.00-9.00   sec   638 MBytes  5.35 Gbits/sec  1373    211 KBytes       
[  5]   9.00-10.00  sec   652 MBytes  5.47 Gbits/sec  1347    547 KBytes       
[  5]  10.00-11.00  sec   761 MBytes  6.39 Gbits/sec  1816    631 KBytes       
[  5]  11.00-12.00  sec   761 MBytes  6.39 Gbits/sec  2100    472 KBytes       
[  5]  12.00-13.00  sec   706 MBytes  5.92 Gbits/sec  2034    609 KBytes       
[  5]  13.00-14.00  sec   710 MBytes  5.96 Gbits/sec  3791    329 KBytes       
[  5]  14.00-15.00  sec   749 MBytes  6.28 Gbits/sec  684    658 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-15.00  sec  10.4 GBytes  5.97 Gbits/sec  32530             sender
[  5]   0.00-15.04  sec  10.4 GBytes  5.95 Gbits/sec                  receiver

iperf Done.
root@iperf3-1:/# iperf3 -c 10.233.226.112 -i 1 -t 15
Connecting to host 10.233.226.112, port 5201
[  5] local 10.233.36.107 port 35270 connected to 10.233.226.112 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   709 MBytes  5.95 Gbits/sec  2567    509 KBytes       
[  5]   1.00-2.00   sec   639 MBytes  5.36 Gbits/sec   44    918 KBytes       
[  5]   2.00-3.00   sec   582 MBytes  4.89 Gbits/sec   54    602 KBytes       
[  5]   3.00-4.00   sec   620 MBytes  5.20 Gbits/sec   79    416 KBytes       
[  5]   4.00-5.00   sec   675 MBytes  5.66 Gbits/sec   46    776 KBytes       
[  5]   5.00-6.00   sec   591 MBytes  4.96 Gbits/sec   33    445 KBytes       
[  5]   6.00-7.00   sec   528 MBytes  4.42 Gbits/sec   24    619 KBytes       
[  5]   7.00-8.00   sec   570 MBytes  4.78 Gbits/sec   95    448 KBytes       
[  5]   8.00-9.00   sec   619 MBytes  5.19 Gbits/sec   14    612 KBytes       
[  5]   9.00-10.00  sec   639 MBytes  5.36 Gbits/sec   42    536 KBytes       
[  5]  10.00-11.00  sec   585 MBytes  4.91 Gbits/sec   66    560 KBytes       
[  5]  11.00-12.00  sec   569 MBytes  4.77 Gbits/sec   54    560 KBytes       
[  5]  12.00-13.00  sec   636 MBytes  5.34 Gbits/sec   23    611 KBytes       
[  5]  13.00-14.00  sec   538 MBytes  4.51 Gbits/sec   29    700 KBytes       
[  5]  14.00-15.00  sec   631 MBytes  5.30 Gbits/sec  143    491 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-15.00  sec  8.92 GBytes  5.11 Gbits/sec  3313             sender
[  5]   0.00-15.04  sec  8.91 GBytes  5.09 Gbits/sec                  receiver

iperf Done.

Your Environment

• Calico version

[root@master-0 ~]# calicoctl version
Client Version:    v3.27.2
Git commit:        402c0b381
Cluster Version:   v3.27.2
Cluster Type:      typha,kdd,k8s,operator,bgp,kubeadm

• Orchestrator version (e.g. kubernetes, mesos, rkt):

[root@master-0 ~]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.12", GitCommit:"12031002905c0410706974560cbdf2dad9278919", GitTreeState:"clean", BuildDate:"2024-03-15T02:15:31Z", GoVersion:"go1.21.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.12", GitCommit:"12031002905c0410706974560cbdf2dad9278919", GitTreeState:"clean", BuildDate:"2024-03-15T02:06:14Z", GoVersion:"go1.21.8", Compiler:"gc", Platform:"linux/amd64"}

• Operating System and version:

[root@master-0 ~]# uname -a
Linux master-0 4.19.90-52.33.v2207.ky10.x86_64 #1 SMP Fri Dec 22 17:04:59 CST 2023 x86_64 x86_64 x86_64 GNU/Linux

• Testing process and results:

[root@worker-0 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: p1p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master nm-bond state UP group default qlen 1000
    link/ether 9c:c2:c4:55:f6:4a brd ff:ff:ff:ff:ff:ff
3: em1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 9c:c2:c4:5f:0f:aa brd ff:ff:ff:ff:ff:ff
4: p1p2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master nm-bond state UP group default qlen 1000
    link/ether 9c:c2:c4:55:f6:4a brd ff:ff:ff:ff:ff:ff
5: p5p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master nm-bond state UP group default qlen 1000
    link/ether 9c:c2:c4:55:f6:4a brd ff:ff:ff:ff:ff:ff
6: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 9c:c2:c4:5f:0f:ab brd ff:ff:ff:ff:ff:ff
7: p5p2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master nm-bond state UP group default qlen 1000
    link/ether 9c:c2:c4:55:f6:4a brd ff:ff:ff:ff:ff:ff
8: nm-bond: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9c:c2:c4:55:f6:4a brd ff:ff:ff:ff:ff:ff
    inet 10.83.3.51/24 brd 10.83.3.255 scope global noprefixroute nm-bond
       valid_lft forever preferred_lft forever

[root@worker-0 ~]# ethtool nm-bond
Settings for nm-bond:
        Supported ports: [ ]
        Supported link modes:   Not reported
        Supported pause frame use: No
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 40000Mb/s
        Duplex: Full
        Port: Other
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Link detected: yes

[root@worker-0 ~]# ethtool -k nm-bond
Features for nm-bond:
rx-checksumming: off [fixed]
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [requested on]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: on
        tx-tcp-mangleid-segmentation: on
        tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: on [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]

[root@master-1 ~]# kubectl get felixconfigurations.projectcalico.org default -o yaml
apiVersion: projectcalico.org/v3
kind: FelixConfiguration
metadata:
  creationTimestamp: "2024-04-24T12:35:04Z"
  generation: 1
  name: default
  resourceVersion: "3943999"
  uid: cbd24723-21c4-4afd-81cd-5d8b4eb8a184
spec:
  bpfConnectTimeLoadBalancing: TCP
  bpfHostNetworkedNATWithoutCTLB: Enabled
  bpfLogLevel: ""
  floatingIPs: Disabled
  healthPort: 9099
  logSeverityScreen: Info
  mtuIfacePattern: nm-bond
  reportingInterval: 0s
  vxlanVNI: 4096


[root@master-1 ~]# kubectl get installations.operator.tigera.io default -o yaml
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  annotations:
    meta.helm.sh/release-name: calico
    meta.helm.sh/release-namespace: tigera-operator
  creationTimestamp: "2024-04-24T12:31:14Z"
  finalizers:
  - tigera.io/operator-cleanup
  generation: 7
  labels:
    app.kubernetes.io/managed-by: Helm
    operator.dameng.com/instance: calico
    operator.dameng.com/set: calico
  name: default
  resourceVersion: "4025413"
  uid: 44ed66f9-f993-4a42-a23e-b28564ecac4d
spec:
  calicoNetwork:
    bgp: Enabled
    hostPorts: Enabled
    ipPools:
    - blockSize: 26
      cidr: 10.233.0.0/16
      disableBGPExport: false
      encapsulation: None
      natOutgoing: Enabled
      nodeSelector: all()
    linuxDataplane: Iptables
    multiInterfaceMode: None
    nodeAddressAutodetectionV4:
      kubernetes: NodeInternalIP
    windowsDataplane: Disabled
  cni:
    ipam:
      type: Calico
    type: Calico
  controlPlaneReplicas: 2
  flexVolumePath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
  imagePullSecrets: []
  kubeletVolumePluginPath: /var/lib/kubelet
  kubernetesProvider: ""
  logging:
    cni:
      logFileMaxAgeDays: 30
      logFileMaxCount: 10
      logFileMaxSize: 100Mi
      logSeverity: Info
  nodeUpdateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
  nonPrivileged: Disabled
  variant: Calico
status:
  calicoVersion: v3.27.2
  computed:
    calicoNetwork:
      bgp: Enabled
      hostPorts: Enabled
      ipPools:
      - blockSize: 26
        cidr: 10.233.0.0/16
        disableBGPExport: false
        encapsulation: None
        natOutgoing: Enabled
        nodeSelector: all()
      linuxDataplane: Iptables
      multiInterfaceMode: None
      nodeAddressAutodetectionV4:
        kubernetes: NodeInternalIP
      windowsDataplane: Disabled
    cni:
      ipam:
        type: Calico
      type: Calico
    controlPlaneReplicas: 2
    flexVolumePath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
    kubeletVolumePluginPath: /var/lib/kubelet
    logging:
      cni:
        logFileMaxAgeDays: 30
        logFileMaxCount: 10
        logFileMaxSize: 100Mi
        logSeverity: Info
    nodeUpdateStrategy:
      rollingUpdate:
        maxUnavailable: 1
      type: RollingUpdate
    nonPrivileged: Disabled
    variant: Calico
  conditions:
  - lastTransitionTime: "2024-04-25T05:14:57Z"
    message: All Objects Available
    observedGeneration: 7
    reason: AllObjectsAvailable
    status: "False"
    type: Degraded
  - lastTransitionTime: "2024-04-25T05:14:57Z"
    message: All objects available
    observedGeneration: 7
    reason: AllObjectsAvailable
    status: "True"
    type: Ready
  - lastTransitionTime: "2024-04-25T05:14:57Z"
    message: All Objects Available
    observedGeneration: 7
    reason: AllObjectsAvailable
    status: "False"
    type: Progressing
  mtu: 1500
  variant: Calico

[root@master-1 ~]# cat /etc/cni/net.d/10-calico.conflist | jq
{
  "name": "k8s-pod-network",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "container_settings": {
        "allow_ip_forwarding": false
      },
      "datastore_type": "kubernetes",
      "ipam": {
        "assign_ipv4": "true",
        "assign_ipv6": "false",
        "type": "calico-ipam"
      },
      "kubernetes": {
        "k8s_api_root": "https://10.96.0.1:443",
        "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
      },
      "log_file_max_age": 30,
      "log_file_max_count": 10,
      "log_file_max_size": 100,
      "log_file_path": "/var/log/calico/cni/cni.log",
      "log_level": "Info",
      "mtu": 0,
      "nodename_file_optional": false,
      "policy": {
        "type": "k8s"
      },
      "type": "calico"
    },
    {
      "capabilities": {
        "bandwidth": true
      },
      "type": "bandwidth"
    },
    {
      "capabilities": {
        "portMappings": true
      },
      "snat": true,
      "type": "portmap"
    }
  ]
}

[root@master-0 ~]# ip link show cali9d99c50de22
76833: cali9d99c50de22@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-e677d036-a8cf-3323-ce92-c0682de0a022

[root@master-0 ~]# ethtool -k cali9d99c50de22
Features for cali9d99c50de22:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: on
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: on
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: on
        tx-tcp-mangleid-segmentation: on
        tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: off [fixed]
tx-sctp-segmentation: on
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: on
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]

[tomastigera]

Do you use vxlan? Are the nodes in different subnets? I suppose there is offloading turned off on vxlan.calico. It is turned off by default due to a kernel bug in older kernels, but we are turning it on in 3.28 again. You can set "ChecksumOffloadBroken=true" in the FelixConfiguration's featureDetectOverride field. You would need to restart the nodes. You can also manually turn it on using ethtool. Let us know if it helped.

[chenlein]

@tomastigera Yes, all nodes are connected to the same switch and VXLAN is not being used. I have set the encapsulation to None. I found in subsequent testing that the total bandwidth can reach the expected value when using concurrency parameters(-P 10), but it is still only about half of the physical bandwidth in single-threaded scenarios.

I also tried using ethtool to disable rx-checksumming and tx-checksumming, but I didn't see any significant change.

I also tried starting two iperf3 containers on the same node to test the same target simultaneously. I expected the results of the two iperf3 containers to add up to the physical bandwidth. However, in fact, the test results of each iperf3 were lower. I can't think of where the problem might be. I thought that in the case of "encapsulation: None", calico only needs to maintain the local routing table and veth pair, and there should not be such a large difference with the physical network.