Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to access SCTP Server using NodePort from host when running cilium without kube-proxy #32412

Open
2 of 3 tasks
rajeevsatya opened this issue May 8, 2024 · 1 comment
Labels
area/loadbalancing Impacts load-balancing and Kubernetes service implementations kind/community-report This was reported by a user in the Cilium community, eg via Slack. kind/question Frequently asked questions & answers. This issue will be linked from the documentation's FAQ. needs/triage This issue requires triaging to establish severity and next steps. sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages.

Comments

@rajeevsatya
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

In kubeadm based k8s cluster, I am running cilium as the primary CNI without kube-proxy. I have deployed the sample sctp-server as deployment and exposed the NodePort service just as mentioned in https://isovalent.com/labs/cilium-sctp/.
When I try to access the sctp server from host using ncat with NodeIP and NodePort, I get connection refused. It doesn't work with service IP too. It works when I initiate ncat with sctp service-name/podIP from inside another pod.

ubuntu@k8s-master1:~$ lsmod | grep sctp
sctp                  385024  87
libcrc32c              16384  5 nf_conntrack,nf_nat,btrfs,raid456,sctp
ubuntu@k8s-master1:~$

ubuntu@k8s-master1:~$ kubectl get nodes -o wide
NAME                    STATUS   ROLES           AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k8s-master1             Ready    control-plane   39m   v1.29.3   10.x.x.190       <none>        Ubuntu 20.04.6 LTS   5.4.0-174-generic   containerd://1.7.13
ubuntu@k8s-master1:~$
ubuntu@k8s-master1:~$ kubectl get svc | grep sctp
sctp-deployment   NodePort    10.109.168.181   <none>        9999:32010/SCTP   25m
ubuntu@k8s-master1:~$

# Inside cilium agent pod
root@k8s-master1:/home/cilium# cilium service list | grep 9999
45   10.109.168.181:9999    ClusterIP      1 => 50.0.0.210:9999 (active)
54   10.x.x.190:32010       NodePort       1 => 50.0.0.210:9999 (active)
55   0.0.0.0:32010          NodePort       1 => 50.0.0.210:9999 (active)
root@k8s-master1:/home/cilium#

ubuntu@k8s-master1:~$ ncat --sctp 10.x.x.190 32010
Ncat: Connection refused.
ubuntu@k8s-master1:~$

# Works with podIP from host
ubuntu@k8s-master1:~$ ncat --sctp 50.0.0.210 9999
Howdy! What's your name?
Test
Thanks for calling, Test. Bye, now.
ubuntu@k8s-master1:~$

I also tried capturing pcap and see INIT/ABORT messages for SCTP protocol, where Verification tag is non-zero in ABORT message.

When I try the same experiment with the same cilium, k8s, kernel versions but with kube-proxy enabled, I see that sctp connection is established from host with NodeIP and NodePort. Can you please let me know what am I missing here? Is SCTP access via NodePort without kube-proxy is not supported yet or I need to enable any other config?
cilium-config.txt

Cilium Version

cilium-cli: v0.16.6 compiled with go1.22.2 on linux/amd64
cilium image (default): v1.15.4

Kernel Version

5.4.0-174-generic

Kubernetes Version

Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.3

Regression

No response

Sysdump

No response

Relevant log output

No response

Anything else?

No response

Cilium Users Document

  • Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

  • I agree to follow this project's Code of Conduct
@rajeevsatya rajeevsatya added kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. needs/triage This issue requires triaging to establish severity and next steps. labels May 8, 2024
@julianwiedmann
Copy link
Member

👋 hi! Please have a look at the SCTP docs, in particular the part about:

SCTP support does not support rewriting ports for SCTP packets. This means that when defining services, the targetPort MUST equal the port, otherwise the packet will be dropped.

@julianwiedmann julianwiedmann added kind/question Frequently asked questions & answers. This issue will be linked from the documentation's FAQ. area/loadbalancing Impacts load-balancing and Kubernetes service implementations sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. and removed kind/bug This is a bug in the Cilium logic. labels May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/loadbalancing Impacts load-balancing and Kubernetes service implementations kind/community-report This was reported by a user in the Cilium community, eg via Slack. kind/question Frequently asked questions & answers. This issue will be linked from the documentation's FAQ. needs/triage This issue requires triaging to establish severity and next steps. sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages.
Projects
None yet
Development

No branches or pull requests

2 participants