ServiceEntry is causing SSL validation errors when acting as a TLS passthrough to an external IPv4 HTTPS service on an IPv6 cluster

[mmerickel]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

I'm somewhat convinced at this point that the behavior is related to running istio on IPv6. I think the HTTPS endpoints I am reproducing the problem on all do not support IPv6, so istio is allocating an IPv6 address within the pod and then failing to proxy the traffic properly to the IPv4 endpoints.

• Create a simple ServiceEntry to allow egress control / visibility of traffic from a pod:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: aws-external-https
spec:
  hosts:
    - sts.us-east-2.amazonaws.com
    - s3.us-east-2.amazonaws.com
    - s3.dualstack.us-east-2.amazonaws.com
  ports:
    - number: 443
      name: https
      protocol: TLS
  resolution: DNS
  location: MESH_EXTERNAL
  exportTo:
    - .

• Deploy a pod.
• Open a shell within the pod and confirm you can hit endpoints like curl https://sts.us-east-2.amazonaws.com successfully.
• Modify your istio config with:

meshConfig:
  defaultConfig:
    proxyMetadata:
      ISTIO_META_DNS_CAPTURE: "true"
      ISTIO_META_DNS_AUTO_ALLOCATE: "true"

• Create a new pod injected with the new config and open a new shell.
• Try the same curl https://sts.us-east-2.amazonaws.com:

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to sts.us-east-2.amazonaws.com:443

• If I uninstall the ServiceEntry then traffic works again, confirming it's related to how istio is handling the traffic.

Version

$ istioctl version
client version: 1.21.2
control plane version: 1.21.2
data plane version: 1.21.2 (6 proxies)

$ kubectl version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.1-eks-b9c9ed7

• Deployed on an EKS cluster in IPv6 mode.

Extra configs:

• ISTIO_ENABLE_IPV4_OUTBOUND_LISTENER_FOR_IPV6_CLUSTERS: "true"
• ENABLE_NATIVE_SIDECARS: "true"

Additional Information

attempts to view certificates

app@myapp-mq-main-757dd77ddc-bdn7v:~$ true | openssl s_client -connect sts.us-east-2.amazonaws.com:443 -servername sts.us-east-2.amazonaws.com 2>/dev/null |
openssl x509
unable to load certificate
139779969938560:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE

app@myapp-mq-main-757dd77ddc-bdn7v:~$ true | openssl s_client -connect sts.us-east-2.amazonaws.com:443 -servername sts.us-east-2.amazonaws.com 2>/dev/null
CONNECTED(00000003)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 319 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

istioctl proxy-config

Note the auto-allocated ip addresses used here via the ISTIO_META_DNS_AUTO_ALLOCATE setting.

❯ istioctl pc l myapp-mq-main-757dd77ddc-bdn7v | grep 443
2001:2::f0f0:6366    443   ALL                                                                 Cluster: outbound|443||s3.us-east-2.amazonaws.com
2001:2::f0f0:a18     443   ALL                                                                 Cluster: outbound|443||sts.us-east-2.amazonaws.com
2001:2::f0f0:e873    443   ALL                                                                 Cluster: outbound|443||s3.dualstack.us-east-2.amazonaws.com
::                   443   Trans: raw_buffer; App: http/1.1,h2c                                Route: 443
::                   443   ALL                                                                 PassthroughCluster
fdf5:2e6e:6925::1    443   ALL                                                                 Cluster: outbound|443||kubernetes.default.svc.cluster.local
fdf5:2e6e:6925::52e4 443   ALL                                                                 Cluster: outbound|443||metrics-server.kube-system.svc.cluster.local
fdf5:2e6e:6925::6372 443   ALL                                                                 Cluster: outbound|443||istiod.istio-system.svc.cluster.local
fdf5:2e6e:6925::88a5 443   ALL                                                                 Cluster: outbound|443||cert-manager-webhook.cert-manager.svc.cluster.local
fdf5:2e6e:6925::a50b 443   ALL                                                                 Cluster: outbound|443||istio-ingressgateway.istio-ingressgateway-private.svc.cluster.local
fdf5:2e6e:6925::eba5 443   Trans: raw_buffer; App: http/1.1,h2c                                Route: external-secrets-webhook.external-secrets.svc.cluster.local:443
fdf5:2e6e:6925::eba5 443   ALL                                                                 Cluster: outbound|443||external-secrets-webhook.external-secrets.svc.cluster.local
fdf5:2e6e:6925::f50b 443   Trans: raw_buffer; App: http/1.1,h2c                                Route: aws-load-balancer-webhook-service.kube-system.svc.cluster.local:443
fdf5:2e6e:6925::f50b 443   ALL                                                                 Cluster: outbound|443||aws-load-balancer-webhook-service.kube-system.svc.cluster.local

[hzxuzhonghu]

I think pure ipv6 node can not access ipv4 service, maybe i am wrong?

If your cluster support dual stack, should configure istio with dual stack mode

[mmerickel]

This is in EKS in which there is a link local interface allowing only egress ipv4. Hence a special flag was added ISTIO_ENABLE_IPV4_OUTBOUND_LISTENER_FOR_IPV6_CLUSTERS and it does not require enabling dual stack according to previous comments in other threads.

[mmerickel]

@hzxuzhonghu please see #46719 (comment) and the followup comment by @day0ops indicating dual stack mode shouldn't be required. If it is, I'd like to know because istio hasn't been exactly clear on this issue.