gha: cleanup stale AWS-CNI iptables rules in EKS tunnel workflow

[giorio94]

Extend the eks-tunnel script to additionally flush the stale AWS-CNI iptables rules left by the aws-node DaemonSet, which break host to pod connectivity. Using the cleanup commands from the documentation 1.

For #2035.

[giorio94]

Link to successful run: https://github.com/cilium/cilium-cli/actions/runs/6529848997/job/17728137356?pr=2045
Dropping the temporary commit.

[pchaigno]

I thought we already did that 🤔

[nebril]

This should be done by this post-start cilium Daemonset hook: https://github.com/cilium/cilium/blob/main/install/kubernetes/cilium/files/agent/poststart-eni.bash

Should we update this script to also remove AWS-CONNMARK-CHAIN rules?

@giorio94 ^^

[giorio94]

This should be done by this post-start cilium Daemonset hook: https://github.com/cilium/cilium/blob/main/install/kubernetes/cilium/files/agent/poststart-eni.bash

Should we update this script to also remove AWS-CONNMARK-CHAIN rules?

@giorio94 ^^

I don't know the details of the poststart-eni script, but makes definitely sense to me (it is very easy to miss the documentation and hit this issue). I edited this workflow because from a quick look at cilium/cilium#25804 it seemed that the script was not always executed.

Even if we extend the poststart-eni.bash script, I guess we'll still need this change for the moment, until a new release is cut.

[giorio94]

it seemed that the script was not always executed.

Just double-checked, and it is not run when running on EKS in tunnel mode, because eni.mode is not set.

/cc @nebril

[giorio94]

AKS failure is unrelated, marking ready to merge

[joestringer]

This seems like it's fixing CI, but isn't the same problem true in production systems?

[giorio94]

This seems like it's fixing CI, but isn't the same problem true in production systems?

Unfortunately yes. More info in cilium/cilium#25804

[joestringer]

Ah right. I was gonna comment that this should be fine as long as we document that this is temporary code to work around a bug in Cilium. But I see that the code comment already references that open item. Sure, let's merge this to keep things moving, then we can revisit how to approach that issue for a longer term fix.