Calico incompatible ipset protocol version

[arana198]

Expected Behavior

Calico's use of ipset should be as broadly compatible as possible.

Current Behavior

Calico v3.27.3 crashloops on Alpine and Ubuntu Server hosts now:

2024-04-30 12:40:40.998 [INFO][11817] felix/int_dataplane.go 1387: Linux interface state changed. ifIndex=924 ifaceName="calico_tmp_B" state=""
2024-04-30 12:40:40.998 [INFO][11817] felix/int_dataplane.go 1431: Linux interface addrs changed. addrs=<nil> ifaceName="calico_tmp_B"
2024-04-30 12:40:40.999 [ERROR][11817] felix/ipsets.go 656: Bad return code from 'ipset list cali40this-host'. error=exit status 1 family="inet" stderr="ipset v7.11: Kernel and userspace incompatible: settype hash:ip with revision 6 not supported by userspace.\n"
2024-04-30 12:40:41.000 [ERROR][11817] felix/ipsets.go 415: Failed to parse ipset cali40this-host error=exit status 1 family="inet"
2024-04-30 12:40:41.000 [WARNING][11817] felix/ipsets.go 346: Failed to resync with dataplane error=exit status 1 family="inet"
2024-04-30 12:40:41.016 [INFO][11817] felix/ipsets.go 337: Retrying after an ipsets update failure... family="inet"

This is similar to an issue that happened a few years ago (#5011) and few months ago (#8372)

Possible Solution

All k0s nodes have the following:

ipset --version
ipset v7.19, protocol version: 7

kube-proxy:

k -n kube-system exec -it kube-proxy-vdhv7 -- ipset --version                                                                 
ipset v7.19, protocol version: 7

Your Environment

• Calico version: v3.27.3
• Orchestrator version: k0s 1.29
• Operating System and version: RPI 5 running Alpine and Intel NUC running Ubuntu Server
• Kernel: 6.6.14-0-rpi on RPI and 6.5.0-28-generic on ubuntu server
• kube-proxy IPVS mode

[mazdakn]

Calico bundles ipset in its image, and only accesses ipsets that it owns. With the latest fix, this issue only happens when Calico ipsets are created with a newer ipset not bundled with calico node. I am not familiar with k0s. Is Calico deployed in a special way? Also do you see the same error only in IPVS mode?

[arana198]

I'm installing calico using the official helm chart. I haven't tried in any other mode yet

[fasaxc]

@mazdakn is the fix in v3.27.3? @arana198 what version of Alpine/Ubuntu are you using (i.e. OS version, not kernel version).

Is ipset shipped inside the kube-proxy pod or is it mounted in from the host? (You could check the volume mounts on the pod.)

[arana198]

Thanks for looking into this and for the prompt response.
Alpine version is 3.19
Ubuntu server is 22.04.4 LTS
Ipset is shipped inside kube-proxy

PS: I downgraded the calico version to V3.26.0 and it works.

[mazdakn]

@fasaxc the fix went into v3.27.2: https://github.com/projectcalico/calico/blob/v3.27.3/release-notes/v3.27.2-release-notes.md

@fasaxc I'm not much familiar with the implementation of IPVS mode. Is there any shared ipset between Calico and kube-proxy?

[fasaxc]

Is there any shared ipset between Calico and kube-proxy?

No, there shouldn't be