AWS EKS 1.26 Flux 2.2.2 - failed to download archive - SYN_SENT

[aptslawomirs]

Hi,

EKS 1.26 (and on 1.24)
VPC CNI (1.11.1 and 1.15.1 and on 1.16.xx)
Flux v2.2.2 and 0.43.0
Connections between nodes are open for all traffic.

Kustomize-controller works by couple hours/days and after that it is crashed. Problem disappear when i did hard restart by scale down and scale up for kustomize-controller.

failed to download archive: GET http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/gitlab/123456890qwertyuiop.tar.gz giving up after 10 attempt(s): Get \"http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/gitlab/123456890qwertyuiop.tar.gz\": dial tcp yyy.yyy.yy.y:80: i/o timeout"

from kustomize controller it looks like that

/tmp/$ netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      1 kustomize-controller-aaaaa-c6ztr:53304 source-controller.flux-system.svc.cluster.local:http SYN_SENT
tcp        0      0 kustomize-controller-aaaaa-c6ztr:50208 kubernetes.default.svc.cluster.local:https ESTABLISHED
tcp        0      1 kustomize-controller-aaaaa-c6ztr:46764 notification-controller.flux-system.svc.cluster.local:http SYN_SENT

/tmp/ $ wget http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/gitlab/123456890qwertyuiop.tar.gz
Connecting to source-controller.flux-system.svc.cluster.local. (yyy.yyy.yy.y:80)
wget: can't connect to remote host (yyy.yyy.yy.y): Operation timed out

From node where this kustomize-controller works
xxx.xxx.xxx.xxx - source controler pod IP
yyy.yyy.yy.y - service source controler cluster IP
zz.zz.zz.zz - kustomize-controller IP

# conntrack -L | grep zz.zz.zz.zz | grep yyy
tcp      6 12 SYN_SENT src=zz.zz.zz.zz dst=yyy.yyy.yy.y sport=51288 dport=80 [UNREPLIED] src=xxx.xxx.xxx.xxx dst=zz.zz.zz.zz sport=9090 dport=51288 mark=0 use=1
tcp      6 110 SYN_SENT src=zz.zz.zz.zz dst=yyy.yyy.yy.y sport=54148 dport=80 [UNREPLIED] src=xxx.xxx.xxx.xxx dst=zz.zz.zz.zz sport=9090 dport=54148 mark=0 use=1
tcp      6 112 SYN_SENT src=zz.zz.zz.zz dst=yyy.yyy.yy.y sport=47178 dport=80 [UNREPLIED] src=xxx.xxx.xxx.xxx dst=zz.zz.zz.zz sport=9090 dport=47178 mark=0 use=1
tcp      6 52 SYN_SENT src=zz.zz.zz.zz dst=yyy.yyy.yy.y sport=60650 dport=80 [UNREPLIED] src=xxx.xxx.xxx.xxx dst=zz.zz.zz.zz sport=9090 dport=60650 mark=0 use=1
conntrack v1.4.4 (conntrack-tools): 371 flow entries have been shown.
tcp      6 82 SYN_SENT src=zz.zz.zz.zz dst=yyy.yyy.yy.y sport=44562 dport=80 [UNREPLIED] src=xxx.xxx.xxx.xxx dst=zz.zz.zz.zz sport=9090 dport=44562 mark=0 use=1
tcp      6 117 SYN_SENT src=zz.zz.zz.zz dst=yyy.yyy.yy.y sport=47194 dport=80 [UNREPLIED] src=xxx.xxx.xxx.xxx dst=zz.zz.zz.zz sport=9090 dport=47194 mark=0 use=1
tcp      6 87 SYN_SENT src=zz.zz.zz.zz dst=yyy.yyy.yy.y sport=44576 dport=80 [UNREPLIED] src=xxx.xxx.xxx.xxx dst=zz.zz.zz.zz sport=9090 dport=44576 mark=0 use=1

In the same time when problem exists on kustomize-controller , I can download exactly the same tar file from source-controller, so service is available.

I will be grateful for any suggestion

[aptslawomirs]

Problem solved - ACL blocked traffic on port 9090 beetwen subnets:)