envoy: Change socket option from 'STATE_LISTENING' to 'STATE_PREBIND'

[chaunceyjiang]

Refer to:
envoyproxy/envoy#18107 (comment)
solo-io/gloo#5842

envoy: Change socket option from 'STATE_LISTENING' to 'STATE_PREBIND'

[chaunceyjiang]

/test

[chaunceyjiang]

/cc @sayboras Please take a look.

[sayboras]

The changes seem reasonable for AWS NLB, however, I am not 100% sure if it will fix the mentioned issues. Can you explain a little bit more ?

Also, please help to update commit message why this change is required. Thank a lot.

[chaunceyjiang]

I am not 100% sure if it will fix the mentioned issues.

@acelinkio is testing with the image built from this PR, as he often encountered the issues mentioned in the issue in his environment. We can wait for his test results.

[ekarlso]

@chaunceyjiang I just tested the built image and it doesn't help any in my case. I am still seeing the issues mentioned in my issue.

[acelinkio]

I am still seeing the upstream connect error or disconnect/reset before headers. reset reason: connection timeout when using the gateway/routes. I am able to reliably reproduce this issue by cycling through the ~10 websites hosted. I noticed if I restart the Cilium daemonset, the error will typically jump to another website.

This was tested with:

• Version 1.15.0 helm

autoDirectNodeRoutes: true
bpf:
  masquerade: true
cluster:
  name: home-cluster
  id: 1
containerRuntime:
  integration: containerd
  socketPath: /var/run/k3s/containerd/containerd.sock
endpointRoutes:
  enabled: true
externalIPs:
  enabled: true
hubble:
  enabled: true
  metrics:
    enabled:
      - dns:query
      - drop
      - tcp
      - flow
      - port-distribution
      - icmp
      - http
  relay:
    enabled: true
    rollOutPods: true
  ui:
    enabled: true
    rollOutPods: true
    ingress:
      enabled: false
ipam:
  mode: kubernetes
ipv4NativeRoutingCIDR: "10.42.0.0/16"
k8sServiceHost: "192.168.1.195"
k8sServicePort: 6443
kubeProxyReplacement: true
kubeProxyReplacementHealthzBindAddr: 0.0.0.0:10256
l2announcements:
  enabled: true
  # https://github.com/cilium/cilium/issues/26586
  leaseDuration: 120s
  leaseRenewDeadline: 60s
  leaseRetryPeriod: 1s
loadBalancer:
  algorithm: maglev
  mode: dsr
localRedirectPolicy: true
operator:
  replicas: 1
  rollOutPods: true
rollOutCiliumPods: true
securityContext:
  privileged: true
routingMode: native
gatewayAPI:
  enabled: true
  secretsNamespace:
    create: false
    name: cilium-secrets
ingressController:
  enabled: false
  loadbalancerMode: shared
  enforceHttps: false
  secretsNamespace:
    create: false
    name: cilium-secrets
  service:
    allocateLoadBalancerNodePorts: false

• Version 1.15.0 helm with the following additional values to consume CI images from this pipeline

image:
  override: "quay.io/cilium/cilium-ci:ac42dc9c3899d5bcc21fda3e72f6687e5b5c128f"
operator:
  image:
    override: "quay.io/cilium/operator-generic-ci:ac42dc9c3899d5bcc21fda3e72f6687e5b5c128f"

I see the following error on the daemonset pod where the websites are failing.

level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=251
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=251
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=255
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=248
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=251
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=251
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=247
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=248
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=242
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=248
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=251
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=247
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=245
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=245
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=245
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=247
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=247
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=242
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=253
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=255
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=255
level=info msg="[cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success" subsys=envoy-filter threadID=242
level=info msg="Endpoint garbage collection is ineffective, ignoring endpoint" helpMessage="For more information, see the linked URL. Pass endpoint-gc-interval=\"0\" to disable" subsys=endpoint url="https://github.com/cilium/cilium/pull/14541"
level=info msg="Envoy: Version 13f6142b9c02268b10d547c8b093ef16724538e3/1.27.2/Distribution/RELEASE/BoringSSL" subsys=envoy-manager

I think #30581 seems related, not that I am not using any network policies tho.

[sayboras]

While this change didn't fix mentioned issue, it's still a good improvement.