Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sidecar CRD is ignored when pod rebalance from one Istiod to another and is endpoint in a ServiceEnrty #50968

Closed
2 tasks done
liorfranko opened this issue May 9, 2024 · 11 comments · Fixed by #51003
Closed
2 tasks done

Sidecar CRD is ignored when pod rebalance from one Istiod to another and is endpoint in a ServiceEnrty #50968

liorfranko opened this issue May 9, 2024 · 11 comments · Fixed by #51003
Labels
area/perf and scalability

Comments

@liorfranko
Copy link

liorfranko commented May 9, 2024 •
edited

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

We have a Sidecar CRD with configurations of egress.hosts that match pods with sidecar injected by workloadSelector.
We also create a ServiceEntry and add those pods as endpoints in a ServiceEntry by IP address.

Once we do it, we see that 30 minutes after the sidecars boot (when the sidecar connection to Istiod is recreated), Istiod ignores the Sidecar CRD and the egress configurations and starts to push all the mesh configs to the sidecars.

{"level":"info","time":"2024-05-09T18:54:04.373168Z","scope":"ads","msg":"CDS: PUSH for node:sleep-ilo-2-9ff6468bc-5bl8n.devops-apps resources:8 size:3.1kB cached:4/4"}
{"level":"info","time":"2024-05-09T18:54:04.373897Z","scope":"ads","msg":"LDS: PUSH for node:sleep-ilo-2-9ff6468bc-5bl8n.devops-apps resources:6 size:9.1kB"}
{"level":"info","time":"2024-05-09T18:54:04.373947Z","scope":"ads","msg":"RDS: PUSH for node:sleep-ilo-2-9ff6468bc-5bl8n.devops-apps resources:2 size:1.2kB cached:2/2"}
{"level":"info","time":"2024-05-09T18:54:04.374017Z","scope":"ads","msg":"NDS: PUSH for node:sleep-ilo-2-9ff6468bc-5bl8n.devops-apps resources:1 size:92B"}
{"level":"info","time":"2024-05-09T19:13:59.430100Z","scope":"ads","msg":"ADS: \"10.120.19.17:36808\" sleep-ilo-2-9ff6468bc-5bl8n.devops-apps-348 terminated"}
{"level":"info","time":"2024-05-09T19:13:59.699261Z","scope":"ads","msg":"ADS: new connection for node:sleep-ilo-2-9ff6468bc-5bl8n.devops-apps-357"}
{"level":"info","time":"2024-05-09T19:13:59.700163Z","scope":"ads","msg":"CDS: PUSH request for node:sleep-ilo-2-9ff6468bc-5bl8n.devops-apps resources:154 size:91.9kB cached:139/151"}
{"level":"info","time":"2024-05-09T19:13:59.700397Z","scope":"ads","msg":"EDS: PUSH request for node:sleep-ilo-2-9ff6468bc-5bl8n.devops-apps resources:4 size:1.3kB empty:0 cached:4/4"}
{"level":"info","time":"2024-05-09T19:13:59.702351Z","scope":"ads","msg":"LDS: PUSH request for node:sleep-ilo-2-9ff6468bc-5bl8n.devops-apps resources:152 size:169.7kB"}
{"level":"info","time":"2024-05-09T19:13:59.702907Z","scope":"ads","msg":"NDS: PUSH request for node:sleep-ilo-2-9ff6468bc-5bl8n.devops-apps resources:1 size:19.8kB"}
{"level":"info","time":"2024-05-09T19:13:59.702960Z","scope":"ads","msg":"RDS: PUSH request for node:sleep-ilo-2-9ff6468bc-5bl8n.devops-apps resources:2 size:1.2kB cached:2/2"}
{"level":"info","time":"2024-05-09T19:13:59.742769Z","scope":"ads","msg":"EDS: PUSH request for node:sleep-ilo-2-9ff6468bc-5bl8n.devops-apps resources:122 size:53.4kB empty:0 cached:114/122"}
{"level":"info","time":"2024-05-09T19:13:59.866555Z","scope":"ads","msg":"RDS: PUSH request for node:sleep-ilo-2-9ff6468bc-5bl8n.devops-apps resources:68 size:74.1kB cached:48/68"}

It happens even if the connection is recreated to the same Istiod.
We tried setting the ServiceEntry.spec.location to MESH_EXTERNAL and MESH_INTERNAL, but it didn’t change.

This can also be reproduced by deleting the Istiod that the sidecar is connected to:

 ~  istioctl pc endpoints -n devops-apps sleep-ilo-2-9ff6468bc-m2jl2 | wc -l
      13
 ~  istioctl ps | grep sleep-ilo-2-9ff6468bc-m2jl2
sleep-ilo-2-9ff6468bc-m2jl2.devops-apps         general-dev-02     SYNCED     SYNCED     SYNCED     SYNCED     istiod-77b697dd85-m929j     1.21.1
 ~  kubectl -n istio-system delete pods istiod-77b697dd85-m929j
pod "istiod-77b697dd85-m929j" deleted
 ~  istioctl pc endpoints -n devops-apps sleep-ilo-2-9ff6468bc-m2jl2 | wc -l
     321
 ~

Version

istioctl version
client version: 1.13.5
control plane version: 1.21.1
data plane version: 1.21.1 (15 proxies)

kubectl version
Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.3-eks-adc7111

Additional Information

No response
@howardjohn
Copy link
Member

Can you maybe post the SE and Sidecar or an example?

Does it happen every time?

If you want to trigger it more (or less)often for testing, the --keepaliveMaxServerConnectionAge flag on istiod controls the 30min time btw

@liorfranko
Copy link
Author

Yes, it happens every time.

Here is the SE:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: sleep-ilo-2-80
  namespace: devops-apps
spec:
  endpoints:
  - address: 10.120.17.59
    weight: 1
  - address: 10.120.17.180
    weight: 1
  - address: 10.120.17.226
    weight: 1
  hosts:
  - sleep-ilo-2.devops-apps.svc.cluster.local
  ports:
  - name: http
    number: 80
    protocol: http
  resolution: STATIC

Here is the Sidecar:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  labels:
    app: sleep-ilo-2
    argocd.argoproj.io/instance: sleep-ilo-2
    clusterName: general-dev-02
    commit: latest
    environment: dev
    name: sleep-ilo-2
    owner: general
    project: support
    sidecar.istio.io/inject: "true"
    team: devops
  name: sleep-ilo-2
  namespace: devops-apps
spec:
  egress:
  - hosts:
    - istio-system/istiod.istio-system.svc.cluster.local
  workloadSelector:
    labels:
      app: sleep-ilo-2

@hzxuzhonghu
Copy link
Member

curl 127.0.0.1:8080/debug/sidecarz?proxyID=sleep-ilo-2 in istiod. it will show you the inner object

@liorfranko
Copy link
Author

Without SE:

{
    "destinationRules": {},
    "name": "sleep-ilo-2",
    "namespace": "devops-apps",
    "outboundTrafficPolicy": {
        "mode": "ALLOW_ANY"
    },
    "rootNamespace": "istio-system",
    "services": [
        <Small list of services>
    ],
    "sidecar": {
        "workloadSelector": {
            "labels": {
                "app": "sleep-ilo-2"
            }
        },
        "egress": [
            {
                "hosts": [
                    "istio-system/istiod.istio-system.svc.cluster.local"
                ]
            }
        ]
    },
    "version": "2024-05-10T10:55:19Z/1"
}

With SE:

{
    "destinationRules": {
        "abir-test.devops-apps.svc.cluster.local": [
            {}
        ],
        "sleep-ilo-2.devops-apps.svc.cluster.local": [
            {}
        ]
    },
    "name": "default-sidecar",
    "namespace": "devops-apps",
    "outboundTrafficPolicy": {
        "mode": "ALLOW_ANY"
    },
    "rootNamespace": "istio-system",
    "services": [
        <Very long list of services>
   ],
    "sidecar": null,
    "version": "2024-05-10T10:54:00Z/23"
}

@hzxuzhonghu
Copy link
Member

This looks weird, with SE. The sleep-ilo-2 cannot matches the sidecar declared, rather it use the default sidecar, actually no sidecar.

Let me think about hoiw that can happen

@hzxuzhonghu
Copy link
Member

I doubt in Proxy.SetWorkloadLabels, we overwrite the node labels with nil, because of the SE/Endppoints has not labels.

So PushContext.getSidecarScope can not match the sidecar.

Want you to help confirm by curl /debug/adsz

@liorfranko
Copy link
Author

Sure

{
    "totalClients": 7,
    "clients": [
        {
            "connectionId": "abir-test-659456cddf-wt6kn.devops-apps-409",
            "connectedAt": "2024-05-11T14:14:49.560212062Z",
            "address": "10.120.12.25:41934",
            "labels": {
                "app": "abir-test",
                "clusterName": "general-dev-02",
                "commit": "1.16.1",
                "environment": "dev",
                "kubernetes.io/hostname": "ip-10-120-15-151.ec2.internal",
                "kyverno-protected": "false",
                "name": "abir-test",
                "owner": "general",
                "project": "support",
                "rollouts-pod-template-hash": "659456cddf",
                "security.istio.io/tlsMode": "istio",
                "service.istio.io/canonical-name": "abir-test",
                "service.istio.io/canonical-revision": "latest",
                "sidecar.istio.io/inject": "true",
                "team": "devops",
                "test": "dfddfw",
                "topology.istio.io/cluster": "general-dev-02",
                "topology.kubernetes.io/region": "us-east-1",
                "topology.kubernetes.io/zone": "us-east-1b"
            },
            "metadata": {
                "PROXY_CONFIG": {
                    "configPath": "./etc/istio/proxy",
                    "binaryPath": "/usr/local/bin/envoy",
                    "serviceCluster": "istio-proxy",
                    "drainDuration": "45s",
                    "discoveryAddress": "istiod.istio-system.svc:15012",
                    "proxyAdminPort": 15000,
                    "controlPlaneAuthPolicy": "MUTUAL_TLS",
                    "statNameLength": 189,
                    "concurrency": 2,
                    "tracing": {},
                    "proxyMetadata": {
                        "EXIT_ON_ZERO_ACTIVE_CONNECTIONS": "true",
                        "ISTIO_META_DNS_CAPTURE": "true"
                    },
                    "statusPort": 15020,
                    "terminationDrainDuration": "5s",
                    "meshId": "general-dev-02",
                    "proxyHeaders": {
                        "forwardedClientCert": "SANITIZE",
                        "requestId": {
                            "disabled": true
                        },
                        "server": {
                            "disabled": true
                        },
                        "attemptCount": {
                            "disabled": true
                        },
                        "envoyDebugHeaders": {
                            "disabled": true
                        },
                        "metadataExchangeHeaders": {
                            "mode": "IN_MESH"
                        }
                    }
                },
                "ISTIO_VERSION": "1.21.1",
                "LABELS": {
                    "app": "abir-test",
                    "clusterName": "general-dev-02",
                    "commit": "1.16.1",
                    "environment": "dev",
                    "kyverno-protected": "false",
                    "name": "abir-test",
                    "owner": "general",
                    "project": "support",
                    "rollouts-pod-template-hash": "659456cddf",
                    "security.istio.io/tlsMode": "istio",
                    "service.istio.io/canonical-name": "abir-test",
                    "service.istio.io/canonical-revision": "latest",
                    "sidecar.istio.io/inject": "true",
                    "team": "devops",
                    "test": "dfddfw"
                },
                "ANNOTATIONS": {
                    "chart_version": "2.2.10",
                    "instance_type": "c6i.large",
                    "istio.io/rev": "default",
                    "kubectl.kubernetes.io/default-container": "telegraf",
                    "kubectl.kubernetes.io/default-logs-container": "telegraf",
                    "kubernetes.io/config.seen": "2024-05-08T21:40:18.804418462Z",
                    "kubernetes.io/config.source": "api",
                    "node_name": "ip-10-120-15-151.ec2.internal",
                    "nodegroup": "abir-test",
                    "prometheus.io/path": "/stats/prometheus",
                    "prometheus.io/port": "15020",
                    "prometheus.io/scrape": "true",
                    "sidecar.istio.io/interceptionMode": "REDIRECT",
                    "sidecar.istio.io/proxyCPU": "500m",
                    "sidecar.istio.io/proxyMemory": "512Mi",
                    "sidecar.istio.io/proxyMemoryLimit": "512Mi",
                    "sidecar.istio.io/status": "{\"initContainers\":[\"istio-validation\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"workload-socket\",\"credential-socket\",\"workload-certs\",\"istio-envoy\",\"istio-data\",\"istio-podinfo\",\"istio-token\",\"istiod-ca-cert\"],\"imagePullSecrets\":null,\"revision\":\"default\"}",
                    "traffic.sidecar.istio.io/excludeInboundPorts": "15020",
                    "traffic.sidecar.istio.io/includeInboundPorts": "80, 8080",
                    "traffic.sidecar.istio.io/includeOutboundIPRanges": "*",
                    "traffic.sidecar.istio.io/includeOutboundPorts": "80, 8080",
                    "zone": "us-east-1b"
                },
                "INSTANCE_IPS": "10.120.12.25",
                "NAMESPACE": "devops-apps",
                "NODE_NAME": "ip-10-120-15-151.ec2.internal",
                "WORKLOAD_NAME": "abir-test-659456cddf",
                "INTERCEPTION_MODE": "REDIRECT",
                "SERVICE_ACCOUNT": "default",
                "MESH_ID": "general-dev-02",
                "CLUSTER_ID": "general-dev-02",
                "POD_PORTS": "[{\"name\":\"udp-statsd\",\"containerPort\":8125,\"protocol\":\"TCP\"},{\"name\":\"prometheus\",\"containerPort\":9273,\"protocol\":\"TCP\"},{\"name\":\"default-port\",\"containerPort\":80,\"protocol\":\"TCP\"},{\"name\":\"metrics-port\",\"containerPort\":8080,\"protocol\":\"TCP\"},{\"name\":\"exporter\",\"containerPort\":19000,\"protocol\":\"TCP\"}]",
                "DNS_CAPTURE": "true",
                "PLATFORM_METADATA": {
                    "aws_availability_zone": "us-east-1b",
                    "aws_instance_id": "i-09a7729d86ea52c2e",
                    "aws_region": "us-east-1"
                },
                "ENVOY_STATUS_PORT": 15021,
                "ENVOY_PROMETHEUS_PORT": 15090,
                "EXIT_ON_ZERO_ACTIVE_CONNECTIONS": "true"
            },
            "locality": {
                "region": "us-east-1",
                "zone": "us-east-1b"
            },
            "watches": {
                "type.googleapis.com/envoy.config.cluster.v3.Cluster": [],
                "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment": [
                    "outbound|15014||istiod.istio-system.svc.cluster.local",
                    "outbound|15012||istiod.istio-system.svc.cluster.local",
                    "outbound|15010||istiod.istio-system.svc.cluster.local",
                    "outbound|443||istiod.istio-system.svc.cluster.local"
                ],
                "type.googleapis.com/envoy.config.listener.v3.Listener": [],
                "type.googleapis.com/envoy.config.route.v3.RouteConfiguration": [
                    "15014",
                    "15010"
                ],
                "type.googleapis.com/istio.networking.nds.v1.NameTable": []
            }
        },
        {
            "connectionId": "infra-lb-59fd56b796-55j27.istio-ingress-405",
            "connectedAt": "2024-05-11T13:54:41.879582482Z",
            "address": "10.120.18.21:34168",
            "labels": {
                "app": "infra-lb",
                "install.operator.istio.io/owning-resource": "unknown",
                "istio": "infra-lb",
                "istio.io/rev": "default",
                "kubernetes.io/hostname": "ip-10-120-16-83.ec2.internal",
                "operator.istio.io/component": "IngressGateways",
                "pod-template-hash": "59fd56b796",
                "service.istio.io/canonical-name": "infra-lb",
                "service.istio.io/canonical-revision": "latest",
                "sidecar.istio.io/inject": "false",
                "topology.istio.io/cluster": "general-dev-02",
                "topology.kubernetes.io/region": "us-east-1",
                "topology.kubernetes.io/zone": "us-east-1d",
                "type": "ingressgateway"
            },
            "metadata": {
                "PROXY_CONFIG": {
                    "configPath": "./etc/istio/proxy",
                    "binaryPath": "/usr/local/bin/envoy",
                    "serviceCluster": "istio-proxy",
                    "drainDuration": "300s",
                    "discoveryAddress": "istiod.istio-system.svc:15012",
                    "proxyAdminPort": 15000,
                    "controlPlaneAuthPolicy": "MUTUAL_TLS",
                    "statNameLength": 189,
                    "concurrency": 2,
                    "tracing": {
                        "zipkin": {
                            "address": "zipkin.istio-system:9411"
                        }
                    },
                    "statusPort": 15020,
                    "terminationDrainDuration": "5s"
                },
                "ISTIO_VERSION": "1.21.1",
                "LABELS": {
                    "app": "infra-lb",
                    "install.operator.istio.io/owning-resource": "unknown",
                    "istio": "infra-lb",
                    "istio.io/rev": "default",
                    "operator.istio.io/component": "IngressGateways",
                    "service.istio.io/canonical-name": "infra-lb",
                    "service.istio.io/canonical-revision": "latest",
                    "sidecar.istio.io/inject": "false",
                    "type": "ingressgateway"
                },
                "ANNOTATIONS": {
                    "instance_type": "c5d.xlarge",
                    "istio.io/rev": "default",
                    "kubernetes.io/config.seen": "2024-05-09T08:49:45.280464846Z",
                    "kubernetes.io/config.source": "api",
                    "node_name": "ip-10-120-16-83.ec2.internal",
                    "nodegroup": "control-plane-ingress",
                    "prometheus.io/path": "/stats/prometheus",
                    "prometheus.io/port": "15020",
                    "prometheus.io/scrape": "true",
                    "sidecar.istio.io/inject": "false",
                    "zone": "us-east-1d"
                },
                "INSTANCE_IPS": "10.120.18.21",
                "NAMESPACE": "istio-ingress",
                "NODE_NAME": "ip-10-120-16-83.ec2.internal",
                "WORKLOAD_NAME": "infra-lb",
                "SERVICE_ACCOUNT": "infra-lb-service-account",
                "MESH_ID": "general-dev-02",
                "CLUSTER_ID": "general-dev-02",
                "PLATFORM_METADATA": {
                    "aws_availability_zone": "us-east-1d",
                    "aws_instance_id": "i-05370d463211a084b",
                    "aws_region": "us-east-1"
                },
                "ENVOY_STATUS_PORT": 15021,
                "ENVOY_PROMETHEUS_PORT": 15090
            },
            "locality": {
                "region": "us-east-1",
                "zone": "us-east-1d"
            },
            "watches": {
                "type.googleapis.com/envoy.config.cluster.v3.Cluster": [],
                "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment": [
                    "outbound|8080||sleep-ilo-2.devops-apps.svc.cluster.local",
                    "outbound|80||sleep-ilo-2.devops-apps.svc.cluster.local",
                    "outbound|4318||otelcol-sidecar-collector.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-sidecar-collector.monitoring.svc.cluster.local",
                    "outbound|8888||otelcol-sidecar-collector-monitoring.monitoring.svc.cluster.local",
                    "outbound|4318||otelcol-sidecar-collector-headless.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-sidecar-collector-headless.monitoring.svc.cluster.local",
                    "outbound|4318||otelcol-dev-collector.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-dev-collector.monitoring.svc.cluster.local",
                    "outbound|8888||otelcol-dev-collector-monitoring.monitoring.svc.cluster.local",
                    "outbound|4318||otelcol-dev-collector-headless.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-dev-collector-headless.monitoring.svc.cluster.local",
                    "outbound|9094||vmalertmanager-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|9093||vmalertmanager-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8401||vmstorage-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8400||vmstorage-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8482||vmstorage-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8481||vmselect-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8480||vminsert-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8429||vmagent-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8080||vmalert-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|443||dev-query-lb.monitoring.svc.cluster.local",
                    "outbound|80||dev-query-lb.monitoring.svc.cluster.local",
                    "outbound|61678||k8s-resources-aws-node.kube-system.svc.cluster.local",
                    "outbound|8080||keda-operator-metrics-apiserver.keda.svc.cluster.local",
                    "outbound|9091||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|3888||elk-zookeeper-nodes.devops-kafka-elk.svc.cluster.local",
                    "outbound|4318||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|5558||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|14250||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|8084||argocd-apps-repo-server-metrics.argocd-apps.svc.cluster.local",
                    "outbound|8084||argo-cd-repo-server-metrics.argocd.svc.cluster.local",
                    "outbound|8081||argocd-apps-repo-server.argocd-apps.svc.cluster.local",
                    "outbound|9001||argocd-apps-notifications-controller-metrics.argocd-apps.svc.cluster.local",
                    "outbound|443||observability-k8s-stack-victoria-metrics-operator.monitoring.svc.cluster.local",
                    "outbound|2020||fluent-bit.logging.svc.cluster.local",
                    "outbound|80||argo-cd-server.argocd.svc.cluster.local",
                    "outbound|9100||observability-k8s-stack-prometheus-node-exporter.monitoring.svc.cluster.local",
                    "outbound|8000||kyverno-background-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|2181||elk-zookeeper-nodes.devops-kafka-elk.svc.cluster.local",
                    "outbound|443||argo-cd-server.argocd.svc.cluster.local",
                    "outbound|5557||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|8443||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|5557||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|9115||prometheus-blackbox-exporter.devops.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1d-0.devops-kafka-elk.svc.cluster.local",
                    "outbound|443||cert-manager-webhook.cert-manager.svc.cluster.local",
                    "outbound|8090||argo-rollouts-metrics.devops.svc.cluster.local",
                    "outbound|8443||actions-runner-controller-metrics-service.github-actions.svc.cluster.local",
                    "outbound|443||elastic-operator-webhook.kube-system.svc.cluster.local",
                    "outbound|80||strimzi-drain-cleaner.devops.svc.cluster.local",
                    "outbound|80||argocd-apps-server.argocd-apps.svc.cluster.local",
                    "outbound|443||kubernetes.default.svc.cluster.local",
                    "outbound|443||dashboard-kubernetes-dashboard.kube-system.svc.cluster.local",
                    "outbound|14268||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|443||private-lb.istio-ingress.svc.cluster.local",
                    "outbound|4317||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|8082||argocd-apps-application-controller-metrics.argocd-apps.svc.cluster.local",
                    "outbound|3100||argo-rollouts-dashboard.devops.svc.cluster.local",
                    "outbound|2181||elk-zookeeper-client.devops-kafka-elk.svc.cluster.local",
                    "outbound|9200||devops-elk-es-internal-http.devops-apps.svc.cluster.local",
                    "outbound|8443||observability-k8s-stack-opentelemetry-operator.monitoring.svc.cluster.local",
                    "outbound|9153||kube-dns.kube-system.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-kube-state-metrics.monitoring.svc.cluster.local",
                    "outbound|443||kyverno-svc.kyverno.svc.cluster.local",
                    "outbound|8000||kyverno-cleanup-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|9106||prometheus-cloudwatch-exporter.devops.svc.cluster.local",
                    "outbound|80||cruise-control-ui-oauth-proxy.devops-apps.svc.cluster.local",
                    "outbound|53||kube-dns.kube-system.svc.cluster.local",
                    "outbound|9153||observability-k8s-stack-coredns.kube-system.svc.cluster.local",
                    "outbound|9090||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|8081||argo-cd-repo-server.argocd.svc.cluster.local",
                    "outbound|443||keda-admission-webhooks.keda.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-opentelemetry-operator.monitoring.svc.cluster.local",
                    "outbound|443||argocd-apps-server.argocd-apps.svc.cluster.local",
                    "outbound|8001||k8s-pvc-tagger.kube-system.svc.cluster.local",
                    "outbound|7000||argocd-apps-applicationset-controller.argocd-apps.svc.cluster.local",
                    "outbound|443||infra-lb.istio-ingress.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1b-101.devops-kafka-elk.svc.cluster.local",
                    "outbound|80||elk-kafka-ui.devops-kafka-elk.svc.cluster.local",
                    "outbound|8443||druid-operator-metrics-service.druid.svc.cluster.local",
                    "outbound|9090||elk-cruise-control.devops-kafka-elk.svc.cluster.local",
                    "outbound|44180||cruise-control-ui-oauth-proxy.devops-apps.svc.cluster.local",
                    "outbound|9666||keda-operator.keda.svc.cluster.local",
                    "outbound|8000||k8s-pvc-tagger.kube-system.svc.cluster.local",
                    "outbound|80||telepresence-prometheus.ambassador.svc.cluster.local",
                    "outbound|8085||velero.velero.svc.cluster.local",
                    "outbound|9273||telegraf.telegraf.svc.cluster.local",
                    "outbound|443||metrics-server.kube-system.svc.cluster.local",
                    "outbound|9092||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|9091||elk-kafka-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|9200||devops-elk-es-hot.devops-apps.svc.cluster.local",
                    "outbound|8383||istio-operator.istio-system.svc.cluster.local",
                    "outbound|8083||argocd-apps-server-metrics.argocd-apps.svc.cluster.local",
                    "outbound|6379||argo-cd-redis.argocd.svc.cluster.local",
                    "outbound|10249||observability-k8s-stack-kube-proxy.kube-system.svc.cluster.local",
                    "outbound|9092||elk-kafka-bootstrap-lb.devops-kafka-elk.svc.cluster.local",
                    "outbound|8082||argo-cd-application-controller-metrics.argocd.svc.cluster.local",
                    "outbound|80||cruise-control-ui.devops-apps.svc.cluster.local",
                    "outbound|15766||traffic-manager.ambassador.svc.cluster.local",
                    "outbound|443||protected-lb.istio-ingress.svc.cluster.local",
                    "outbound|9121||argocd-apps-redis-metrics.argocd-apps.svc.cluster.local",
                    "outbound|80||infra-lb.istio-ingress.svc.cluster.local",
                    "outbound|80||cerebro-devops-elk.devops-apps.svc.cluster.local",
                    "outbound|8000||kyverno-svc-metrics.kyverno.svc.cluster.local",
                    "outbound|443||keda-operator-metrics-apiserver.keda.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1d-1.devops-kafka-elk.svc.cluster.local",
                    "outbound|443||agent-injector.ambassador.svc.cluster.local",
                    "outbound|8083||argo-cd-server-metrics.argocd.svc.cluster.local",
                    "outbound|9600||logging-stack-logstash-headless.devops-apps.svc.cluster.local",
                    "outbound|8888||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|7979||external-dns.kube-system.svc.cluster.local",
                    "outbound|80||protected-lb.istio-ingress.svc.cluster.local",
                    "outbound|5601||devops-elk-kb-http.devops-apps.svc.cluster.local",
                    "outbound|9411||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|443||istiod.istio-system.svc.cluster.local",
                    "outbound|9200||devops-elk-es-warm.devops-apps.svc.cluster.local",
                    "outbound|15014||istiod.istio-system.svc.cluster.local",
                    "outbound|9092||elk-kafka-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1b-100.devops-kafka-elk.svc.cluster.local",
                    "outbound|9200||devops-elk-es-http.devops-apps.svc.cluster.local",
                    "outbound|443||observability-k8s-stack-opentelemetry-operator-webhook.monitoring.svc.cluster.local",
                    "outbound|5556||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|6379||argocd-apps-redis.argocd-apps.svc.cluster.local",
                    "outbound|80||traffic-manager-ambassador-agent.ambassador.svc.cluster.local",
                    "outbound|5558||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|8081||traffic-manager.ambassador.svc.cluster.local",
                    "outbound|80||private-lb.istio-ingress.svc.cluster.local",
                    "outbound|80||abir-test.devops-apps.svc.cluster.local",
                    "outbound|9402||cert-manager.cert-manager.svc.cluster.local",
                    "outbound|9200||devops-elk-es-masters.devops-apps.svc.cluster.local",
                    "outbound|8200||devops-elk-apm-http.devops-apps.svc.cluster.local",
                    "outbound|2888||elk-zookeeper-nodes.devops-kafka-elk.svc.cluster.local",
                    "outbound|443||aws-load-balancer-webhook-service.kube-system.svc.cluster.local",
                    "outbound|9001||argo-cd-notifications-controller-metrics.argocd.svc.cluster.local",
                    "outbound|8000||karpenter.karpenter.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-victoria-metrics-operator.monitoring.svc.cluster.local",
                    "outbound|8000||kyverno-reports-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|443||strimzi-drain-cleaner.devops.svc.cluster.local",
                    "outbound|443||actions-runner-controller-webhook.github-actions.svc.cluster.local",
                    "outbound|15010||istiod.istio-system.svc.cluster.local",
                    "outbound|5556||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|80||node-cleanup-controller.devops.svc.cluster.local",
                    "outbound|9108||logging-stack-prometheus-elasticsearch-exporter.devops-apps.svc.cluster.local",
                    "outbound|9200||devops-elk-es-cold.devops-apps.svc.cluster.local",
                    "outbound|443||kyverno-cleanup-controller.kyverno.svc.cluster.local",
                    "outbound|9300||devops-elk-es-transport.devops-apps.svc.cluster.local",
                    "outbound|9094||elk-kafka-external-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|15012||istiod.istio-system.svc.cluster.local",
                    "outbound|8080||keda-operator.keda.svc.cluster.local"
                ],
                "type.googleapis.com/envoy.config.listener.v3.Listener": [],
                "type.googleapis.com/envoy.config.route.v3.RouteConfiguration": [
                    "https.443.https.infra-lb-gw.istio-ingress",
                    "http.80"
                ],
                "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret": [
                    "kubernetes://certificate-infra-lb-tls"
                ]
            }
        },
        {
            "connectionId": "sleep-ilo-2-9ff6468bc-vzfcq.devops-apps-407",
            "connectedAt": "2024-05-11T14:05:37.048560386Z",
            "address": "10.120.18.245:58002",
            "labels": {
                "kubernetes.io/hostname": "ip-10-120-19-57.ec2.internal",
                "topology.istio.io/cluster": "general-dev-02",
                "topology.kubernetes.io/region": "us-east-1",
                "topology.kubernetes.io/zone": "us-east-1d"
            },
            "metadata": {
                "PROXY_CONFIG": {
                    "configPath": "./etc/istio/proxy",
                    "binaryPath": "/usr/local/bin/envoy",
                    "serviceCluster": "istio-proxy",
                    "drainDuration": "45s",
                    "discoveryAddress": "istiod.istio-system.svc:15012",
                    "proxyAdminPort": 15000,
                    "controlPlaneAuthPolicy": "MUTUAL_TLS",
                    "statNameLength": 189,
                    "concurrency": 4,
                    "tracing": {},
                    "proxyMetadata": {
                        "EXIT_ON_ZERO_ACTIVE_CONNECTIONS": "true",
                        "ISTIO_META_DNS_CAPTURE": "true"
                    },
                    "statusPort": 15020,
                    "terminationDrainDuration": "5s",
                    "meshId": "general-dev-02",
                    "proxyHeaders": {
                        "forwardedClientCert": "SANITIZE",
                        "requestId": {
                            "disabled": true
                        },
                        "server": {
                            "disabled": true
                        },
                        "attemptCount": {
                            "disabled": true
                        },
                        "envoyDebugHeaders": {
                            "disabled": true
                        },
                        "metadataExchangeHeaders": {
                            "mode": "IN_MESH"
                        }
                    }
                },
                "ISTIO_VERSION": "1.21.1",
                "LABELS": {
                    "app": "sleep-ilo-2",
                    "clusterName": "general-dev-02",
                    "commit": "latest",
                    "environment": "dev",
                    "name": "sleep-ilo-2",
                    "owner": "general",
                    "project": "support",
                    "rollouts-pod-template-hash": "9ff6468bc",
                    "security.istio.io/tlsMode": "istio",
                    "service.istio.io/canonical-name": "sleep-ilo-2",
                    "service.istio.io/canonical-revision": "latest",
                    "sidecar.istio.io/inject": "true",
                    "team": "devops"
                },
                "ANNOTATIONS": {
                    "chart_version": "2.2.2",
                    "checksum/config": "659f4c48bb3f9f73c7a610cf1babf4ae633bb00d955ad7a904b06504c5132ece",
                    "instance_type": "c5d.xlarge",
                    "istio.io/rev": "default",
                    "kubectl.kubernetes.io/default-container": "sleep-ilo-2",
                    "kubectl.kubernetes.io/default-logs-container": "sleep-ilo-2",
                    "kubernetes.io/config.seen": "2024-05-10T06:49:56.117159859Z",
                    "kubernetes.io/config.source": "api",
                    "node_name": "ip-10-120-19-57.ec2.internal",
                    "nodegroup": "mobile-devops-dev-spot",
                    "prometheus.io/path": "/stats/prometheus",
                    "prometheus.io/port": "15020",
                    "prometheus.io/scrape": "true",
                    "sidecar.istio.io/interceptionMode": "REDIRECT",
                    "sidecar.istio.io/proxyCPU": "500m",
                    "sidecar.istio.io/proxyMemory": "512Mi",
                    "sidecar.istio.io/proxyMemoryLimit": "512Mi",
                    "sidecar.istio.io/status": "{\"initContainers\":[\"istio-validation\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"workload-socket\",\"credential-socket\",\"workload-certs\",\"istio-envoy\",\"istio-data\",\"istio-podinfo\",\"istio-token\",\"istiod-ca-cert\"],\"imagePullSecrets\":null,\"revision\":\"default\"}",
                    "traffic.sidecar.istio.io/excludeInboundPorts": "15020",
                    "traffic.sidecar.istio.io/includeInboundPorts": "80, 8080",
                    "traffic.sidecar.istio.io/includeOutboundIPRanges": "*",
                    "traffic.sidecar.istio.io/includeOutboundPorts": "80, 8080",
                    "zone": "us-east-1d"
                },
                "INSTANCE_IPS": "10.120.18.245",
                "NAMESPACE": "devops-apps",
                "NODE_NAME": "ip-10-120-19-57.ec2.internal",
                "WORKLOAD_NAME": "sleep-ilo-2-9ff6468bc",
                "INTERCEPTION_MODE": "REDIRECT",
                "SERVICE_ACCOUNT": "default",
                "MESH_ID": "general-dev-02",
                "CLUSTER_ID": "general-dev-02",
                "DNS_CAPTURE": "true",
                "PLATFORM_METADATA": {
                    "aws_availability_zone": "us-east-1d",
                    "aws_instance_id": "i-0f721401be01d6bf4",
                    "aws_region": "us-east-1"
                },
                "ENVOY_STATUS_PORT": 15021,
                "ENVOY_PROMETHEUS_PORT": 15090,
                "EXIT_ON_ZERO_ACTIVE_CONNECTIONS": "true"
            },
            "locality": {
                "region": "us-east-1",
                "zone": "us-east-1d"
            },
            "watches": {
                "type.googleapis.com/envoy.config.cluster.v3.Cluster": [],
                "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment": [
                    "outbound|8080||sleep-ilo-2.devops-apps.svc.cluster.local",
                    "outbound|80||sleep-ilo-2.devops-apps.svc.cluster.local",
                    "outbound|4318||otelcol-sidecar-collector.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-sidecar-collector.monitoring.svc.cluster.local",
                    "outbound|8888||otelcol-sidecar-collector-monitoring.monitoring.svc.cluster.local",
                    "outbound|4318||otelcol-dev-collector.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-dev-collector.monitoring.svc.cluster.local",
                    "outbound|8888||otelcol-dev-collector-monitoring.monitoring.svc.cluster.local",
                    "outbound|8480||vminsert-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8429||vmagent-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8080||vmalert-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|443||dev-query-lb.monitoring.svc.cluster.local",
                    "outbound|80||dev-query-lb.monitoring.svc.cluster.local",
                    "outbound|443||observability-k8s-stack-victoria-metrics-operator.monitoring.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-victoria-metrics-operator.monitoring.svc.cluster.local",
                    "outbound|9100||observability-k8s-stack-prometheus-node-exporter.monitoring.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-opentelemetry-operator.monitoring.svc.cluster.local",
                    "outbound|8443||observability-k8s-stack-opentelemetry-operator.monitoring.svc.cluster.local",
                    "outbound|443||observability-k8s-stack-opentelemetry-operator-webhook.monitoring.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-kube-state-metrics.monitoring.svc.cluster.local",
                    "outbound|443||protected-lb.istio-ingress.svc.cluster.local",
                    "outbound|80||protected-lb.istio-ingress.svc.cluster.local",
                    "outbound|443||private-lb.istio-ingress.svc.cluster.local",
                    "outbound|80||private-lb.istio-ingress.svc.cluster.local",
                    "outbound|443||infra-lb.istio-ingress.svc.cluster.local",
                    "outbound|80||infra-lb.istio-ingress.svc.cluster.local",
                    "outbound|80||abir-test.devops-apps.svc.cluster.local",
                    "outbound|9090||elk-cruise-control.devops-kafka-elk.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1d-1.devops-kafka-elk.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1d-0.devops-kafka-elk.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1b-101.devops-kafka-elk.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1b-100.devops-kafka-elk.svc.cluster.local",
                    "outbound|9094||elk-kafka-external-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|9092||elk-kafka-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|9091||elk-kafka-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|8200||devops-elk-apm-http.devops-apps.svc.cluster.local",
                    "outbound|5601||devops-elk-kb-http.devops-apps.svc.cluster.local",
                    "outbound|2181||elk-zookeeper-client.devops-kafka-elk.svc.cluster.local",
                    "outbound|9108||logging-stack-prometheus-elasticsearch-exporter.devops-apps.svc.cluster.local",
                    "outbound|80||elk-kafka-ui.devops-kafka-elk.svc.cluster.local",
                    "outbound|9092||elk-kafka-bootstrap-lb.devops-kafka-elk.svc.cluster.local",
                    "outbound|9200||devops-elk-es-internal-http.devops-apps.svc.cluster.local",
                    "outbound|9200||devops-elk-es-http.devops-apps.svc.cluster.local",
                    "outbound|80||cerebro-devops-elk.devops-apps.svc.cluster.local",
                    "outbound|80||traffic-manager-ambassador-agent.ambassador.svc.cluster.local",
                    "outbound|443||aws-load-balancer-webhook-service.kube-system.svc.cluster.local",
                    "outbound|443||kyverno-svc.kyverno.svc.cluster.local",
                    "outbound|8000||kyverno-svc-metrics.kyverno.svc.cluster.local",
                    "outbound|8000||kyverno-reports-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|443||kyverno-cleanup-controller.kyverno.svc.cluster.local",
                    "outbound|8000||kyverno-cleanup-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|8000||kyverno-background-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|2020||fluent-bit.logging.svc.cluster.local",
                    "outbound|9411||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|4318||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|4317||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|8888||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|14268||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|14250||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|80||cruise-control-ui.devops-apps.svc.cluster.local",
                    "outbound|44180||cruise-control-ui-oauth-proxy.devops-apps.svc.cluster.local",
                    "outbound|80||cruise-control-ui-oauth-proxy.devops-apps.svc.cluster.local",
                    "outbound|443||actions-runner-controller-webhook.github-actions.svc.cluster.local",
                    "outbound|8443||actions-runner-controller-metrics-service.github-actions.svc.cluster.local",
                    "outbound|80||telepresence-prometheus.ambassador.svc.cluster.local",
                    "outbound|443||agent-injector.ambassador.svc.cluster.local",
                    "outbound|443||strimzi-drain-cleaner.devops.svc.cluster.local",
                    "outbound|80||strimzi-drain-cleaner.devops.svc.cluster.local",
                    "outbound|9402||cert-manager.cert-manager.svc.cluster.local",
                    "outbound|443||cert-manager-webhook.cert-manager.svc.cluster.local",
                    "outbound|8090||argo-rollouts-metrics.devops.svc.cluster.local",
                    "outbound|3100||argo-rollouts-dashboard.devops.svc.cluster.local",
                    "outbound|7979||external-dns.kube-system.svc.cluster.local",
                    "outbound|8085||velero.velero.svc.cluster.local",
                    "outbound|443||elastic-operator-webhook.kube-system.svc.cluster.local",
                    "outbound|443||argocd-apps-server.argocd-apps.svc.cluster.local",
                    "outbound|80||argocd-apps-server.argocd-apps.svc.cluster.local",
                    "outbound|8083||argocd-apps-server-metrics.argocd-apps.svc.cluster.local",
                    "outbound|8081||argocd-apps-repo-server.argocd-apps.svc.cluster.local",
                    "outbound|8084||argocd-apps-repo-server-metrics.argocd-apps.svc.cluster.local",
                    "outbound|6379||argocd-apps-redis.argocd-apps.svc.cluster.local",
                    "outbound|9001||argocd-apps-notifications-controller-metrics.argocd-apps.svc.cluster.local",
                    "outbound|5558||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|5557||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|5556||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|7000||argocd-apps-applicationset-controller.argocd-apps.svc.cluster.local",
                    "outbound|8082||argocd-apps-application-controller-metrics.argocd-apps.svc.cluster.local",
                    "outbound|8000||karpenter.karpenter.svc.cluster.local",
                    "outbound|80||node-cleanup-controller.devops.svc.cluster.local",
                    "outbound|8080||keda-operator.keda.svc.cluster.local",
                    "outbound|9666||keda-operator.keda.svc.cluster.local",
                    "outbound|8080||keda-operator-metrics-apiserver.keda.svc.cluster.local",
                    "outbound|443||keda-operator-metrics-apiserver.keda.svc.cluster.local",
                    "outbound|443||keda-admission-webhooks.keda.svc.cluster.local",
                    "outbound|9273||telegraf.telegraf.svc.cluster.local",
                    "outbound|9106||prometheus-cloudwatch-exporter.devops.svc.cluster.local",
                    "outbound|443||metrics-server.kube-system.svc.cluster.local",
                    "outbound|8443||druid-operator-metrics-service.druid.svc.cluster.local",
                    "outbound|8001||k8s-pvc-tagger.kube-system.svc.cluster.local",
                    "outbound|8000||k8s-pvc-tagger.kube-system.svc.cluster.local",
                    "outbound|9115||prometheus-blackbox-exporter.devops.svc.cluster.local",
                    "outbound|443||dashboard-kubernetes-dashboard.kube-system.svc.cluster.local",
                    "outbound|61678||k8s-resources-aws-node.kube-system.svc.cluster.local",
                    "outbound|8383||istio-operator.istio-system.svc.cluster.local",
                    "outbound|443||argo-cd-server.argocd.svc.cluster.local",
                    "outbound|80||argo-cd-server.argocd.svc.cluster.local",
                    "outbound|8083||argo-cd-server-metrics.argocd.svc.cluster.local",
                    "outbound|8081||argo-cd-repo-server.argocd.svc.cluster.local",
                    "outbound|8084||argo-cd-repo-server-metrics.argocd.svc.cluster.local",
                    "outbound|6379||argo-cd-redis.argocd.svc.cluster.local",
                    "outbound|9001||argo-cd-notifications-controller-metrics.argocd.svc.cluster.local",
                    "outbound|5558||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|5557||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|5556||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|8082||argo-cd-application-controller-metrics.argocd.svc.cluster.local",
                    "outbound|9153||kube-dns.kube-system.svc.cluster.local",
                    "outbound|53||kube-dns.kube-system.svc.cluster.local",
                    "outbound|443||kubernetes.default.svc.cluster.local",
                    "outbound|443||istiod.istio-system.svc.cluster.local",
                    "outbound|15010||istiod.istio-system.svc.cluster.local",
                    "outbound|15014||istiod.istio-system.svc.cluster.local",
                    "outbound|15012||istiod.istio-system.svc.cluster.local"
                ],
                "type.googleapis.com/envoy.config.listener.v3.Listener": [],
                "type.googleapis.com/envoy.config.route.v3.RouteConfiguration": [
                    "observability-k8s-stack-victoria-metrics-operator.monitoring.svc.cluster.local:443",
                    "10249",
                    "8429",
                    "5601",
                    "otelcol-sidecar-collector-monitoring.monitoring.svc.cluster.local:8888",
                    "observability-k8s-stack-opentelemetry-operator.monitoring.svc.cluster.local:8080",
                    "9600",
                    "k8s-pvc-tagger.kube-system.svc.cluster.local:8001",
                    "9093",
                    "kyverno-reports-controller-metrics.kyverno.svc.cluster.local:8000",
                    "keda-operator.keda.svc.cluster.local:8080",
                    "5557",
                    "keda-operator-metrics-apiserver.keda.svc.cluster.local:8080",
                    "argo-rollouts-dashboard.devops.svc.cluster.local:3100",
                    "kyverno-cleanup-controller-metrics.kyverno.svc.cluster.local:8000",
                    "5556",
                    "actions-runner-controller-metrics-service.github-actions.svc.cluster.local:8443",
                    "telegraf.telegraf.svc.cluster.local:9273",
                    "7979",
                    "traffic-manager.ambassador.svc.cluster.local:8081",
                    "9115",
                    "15766",
                    "9121",
                    "keda-operator.keda.svc.cluster.local:9666",
                    "4317",
                    "telepresence-prometheus.ambassador.svc.cluster.local:80",
                    "9153",
                    "9108",
                    "8083",
                    "80",
                    "kube-dns.kube-system.svc.cluster.local:9153",
                    "4318",
                    "9200",
                    "argo-rollouts-metrics.devops.svc.cluster.local:8090",
                    "8000",
                    "8383",
                    "5558",
                    "9001",
                    "443",
                    "9106",
                    "vmstorage-observability-k8s-stack.monitoring.svc.cluster.local:8401",
                    "7000",
                    "opentelemetry-collector.logging.svc.cluster.local:14268",
                    "8481",
                    "kyverno-svc-metrics.kyverno.svc.cluster.local:8000",
                    "opentelemetry-collector.logging.svc.cluster.local:8888",
                    "observability-k8s-stack-prometheus-node-exporter.monitoring.svc.cluster.local:9100",
                    "8480",
                    "8084",
                    "44180",
                    "opentelemetry-collector.logging.svc.cluster.local:14250",
                    "elk-cruise-control.devops-kafka-elk.svc.cluster.local:9090",
                    "elk-kafka-bootstrap-lb.devops-kafka-elk.svc.cluster.local:9092",
                    "8085",
                    "8080",
                    "8082",
                    "otelcol-dev-collector-monitoring.monitoring.svc.cluster.local:8888",
                    "aws-load-balancer-webhook-service.kube-system.svc.cluster.local:443",
                    "2020",
                    "8482",
                    "kyverno-background-controller-metrics.kyverno.svc.cluster.local:8000",
                    "opentelemetry-collector.logging.svc.cluster.local:9411",
                    "k8s-resources-aws-node.kube-system.svc.cluster.local:61678",
                    "vmstorage-observability-k8s-stack.monitoring.svc.cluster.local:8400",
                    "observability-k8s-stack-opentelemetry-operator-webhook.monitoring.svc.cluster.local:443",
                    "opentelemetry-collector.logging.svc.cluster.local:4318",
                    "15014",
                    "15010"
                ],
                "type.googleapis.com/istio.networking.nds.v1.NameTable": []
            }
        },
        {
            "connectionId": "infra-lb-59fd56b796-f9xtx.istio-ingress-404",
            "connectedAt": "2024-05-11T13:49:10.855253648Z",
            "address": "10.120.18.71:34174",
            "labels": {
                "app": "infra-lb",
                "install.operator.istio.io/owning-resource": "unknown",
                "istio": "infra-lb",
                "istio.io/rev": "default",
                "kubernetes.io/hostname": "ip-10-120-19-125.ec2.internal",
                "operator.istio.io/component": "IngressGateways",
                "pod-template-hash": "59fd56b796",
                "service.istio.io/canonical-name": "infra-lb",
                "service.istio.io/canonical-revision": "latest",
                "sidecar.istio.io/inject": "false",
                "topology.istio.io/cluster": "general-dev-02",
                "topology.kubernetes.io/region": "us-east-1",
                "topology.kubernetes.io/zone": "us-east-1d",
                "type": "ingressgateway"
            },
            "metadata": {
                "PROXY_CONFIG": {
                    "configPath": "./etc/istio/proxy",
                    "binaryPath": "/usr/local/bin/envoy",
                    "serviceCluster": "istio-proxy",
                    "drainDuration": "300s",
                    "discoveryAddress": "istiod.istio-system.svc:15012",
                    "proxyAdminPort": 15000,
                    "controlPlaneAuthPolicy": "MUTUAL_TLS",
                    "statNameLength": 189,
                    "concurrency": 2,
                    "tracing": {
                        "zipkin": {
                            "address": "zipkin.istio-system:9411"
                        }
                    },
                    "statusPort": 15020,
                    "terminationDrainDuration": "5s"
                },
                "ISTIO_VERSION": "1.21.1",
                "LABELS": {
                    "app": "infra-lb",
                    "install.operator.istio.io/owning-resource": "unknown",
                    "istio": "infra-lb",
                    "istio.io/rev": "default",
                    "operator.istio.io/component": "IngressGateways",
                    "service.istio.io/canonical-name": "infra-lb",
                    "service.istio.io/canonical-revision": "latest",
                    "sidecar.istio.io/inject": "false",
                    "type": "ingressgateway"
                },
                "ANNOTATIONS": {
                    "instance_type": "c5d.xlarge",
                    "istio.io/rev": "default",
                    "kubernetes.io/config.seen": "2024-05-09T08:50:19.606143057Z",
                    "kubernetes.io/config.source": "api",
                    "node_name": "ip-10-120-19-125.ec2.internal",
                    "nodegroup": "control-plane-ingress",
                    "prometheus.io/path": "/stats/prometheus",
                    "prometheus.io/port": "15020",
                    "prometheus.io/scrape": "true",
                    "sidecar.istio.io/inject": "false",
                    "zone": "us-east-1d"
                },
                "INSTANCE_IPS": "10.120.18.71",
                "NAMESPACE": "istio-ingress",
                "NODE_NAME": "ip-10-120-19-125.ec2.internal",
                "WORKLOAD_NAME": "infra-lb",
                "SERVICE_ACCOUNT": "infra-lb-service-account",
                "MESH_ID": "general-dev-02",
                "CLUSTER_ID": "general-dev-02",
                "PLATFORM_METADATA": {
                    "aws_availability_zone": "us-east-1d",
                    "aws_instance_id": "i-08c1bca5f475df5d5",
                    "aws_region": "us-east-1"
                },
                "ENVOY_STATUS_PORT": 15021,
                "ENVOY_PROMETHEUS_PORT": 15090
            },
            "locality": {
                "region": "us-east-1",
                "zone": "us-east-1d"
            },
            "watches": {
                "type.googleapis.com/envoy.config.cluster.v3.Cluster": [],
                "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment": [
                    "outbound|8080||sleep-ilo-2.devops-apps.svc.cluster.local",
                    "outbound|80||sleep-ilo-2.devops-apps.svc.cluster.local",
                    "outbound|4318||otelcol-sidecar-collector.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-sidecar-collector.monitoring.svc.cluster.local",
                    "outbound|8888||otelcol-sidecar-collector-monitoring.monitoring.svc.cluster.local",
                    "outbound|4318||otelcol-sidecar-collector-headless.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-sidecar-collector-headless.monitoring.svc.cluster.local",
                    "outbound|4318||otelcol-dev-collector.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-dev-collector.monitoring.svc.cluster.local",
                    "outbound|8888||otelcol-dev-collector-monitoring.monitoring.svc.cluster.local",
                    "outbound|4318||otelcol-dev-collector-headless.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-dev-collector-headless.monitoring.svc.cluster.local",
                    "outbound|9094||vmalertmanager-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|9093||vmalertmanager-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8401||vmstorage-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8400||vmstorage-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8482||vmstorage-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8481||vmselect-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8480||vminsert-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8429||vmagent-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8080||vmalert-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|443||dev-query-lb.monitoring.svc.cluster.local",
                    "outbound|80||dev-query-lb.monitoring.svc.cluster.local",
                    "outbound|8084||argocd-apps-repo-server-metrics.argocd-apps.svc.cluster.local",
                    "outbound|8000||karpenter.karpenter.svc.cluster.local",
                    "outbound|443||keda-admission-webhooks.keda.svc.cluster.local",
                    "outbound|5558||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|8001||k8s-pvc-tagger.kube-system.svc.cluster.local",
                    "outbound|8080||keda-operator-metrics-apiserver.keda.svc.cluster.local",
                    "outbound|9092||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1d-1.devops-kafka-elk.svc.cluster.local",
                    "outbound|9091||elk-kafka-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|80||abir-test.devops-apps.svc.cluster.local",
                    "outbound|3100||argo-rollouts-dashboard.devops.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-opentelemetry-operator.monitoring.svc.cluster.local",
                    "outbound|8085||velero.velero.svc.cluster.local",
                    "outbound|443||protected-lb.istio-ingress.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1d-0.devops-kafka-elk.svc.cluster.local",
                    "outbound|8443||actions-runner-controller-metrics-service.github-actions.svc.cluster.local",
                    "outbound|80||argocd-apps-server.argocd-apps.svc.cluster.local",
                    "outbound|4318||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|8200||devops-elk-apm-http.devops-apps.svc.cluster.local",
                    "outbound|9300||devops-elk-es-transport.devops-apps.svc.cluster.local",
                    "outbound|9411||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|443||kubernetes.default.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-victoria-metrics-operator.monitoring.svc.cluster.local",
                    "outbound|9402||cert-manager.cert-manager.svc.cluster.local",
                    "outbound|8081||argo-cd-repo-server.argocd.svc.cluster.local",
                    "outbound|8082||argo-cd-application-controller-metrics.argocd.svc.cluster.local",
                    "outbound|9200||devops-elk-es-cold.devops-apps.svc.cluster.local",
                    "outbound|9153||observability-k8s-stack-coredns.kube-system.svc.cluster.local",
                    "outbound|14250||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|9666||keda-operator.keda.svc.cluster.local",
                    "outbound|8000||kyverno-svc-metrics.kyverno.svc.cluster.local",
                    "outbound|80||cruise-control-ui.devops-apps.svc.cluster.local",
                    "outbound|9200||devops-elk-es-internal-http.devops-apps.svc.cluster.local",
                    "outbound|8081||argocd-apps-repo-server.argocd-apps.svc.cluster.local",
                    "outbound|443||observability-k8s-stack-victoria-metrics-operator.monitoring.svc.cluster.local",
                    "outbound|8080||keda-operator.keda.svc.cluster.local",
                    "outbound|443||kyverno-svc.kyverno.svc.cluster.local",
                    "outbound|80||argo-cd-server.argocd.svc.cluster.local",
                    "outbound|8081||traffic-manager.ambassador.svc.cluster.local",
                    "outbound|443||dashboard-kubernetes-dashboard.kube-system.svc.cluster.local",
                    "outbound|15012||istiod.istio-system.svc.cluster.local",
                    "outbound|80||cruise-control-ui-oauth-proxy.devops-apps.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1b-100.devops-kafka-elk.svc.cluster.local",
                    "outbound|9108||logging-stack-prometheus-elasticsearch-exporter.devops-apps.svc.cluster.local",
                    "outbound|443||istiod.istio-system.svc.cluster.local",
                    "outbound|443||cert-manager-webhook.cert-manager.svc.cluster.local",
                    "outbound|80||private-lb.istio-ingress.svc.cluster.local",
                    "outbound|80||infra-lb.istio-ingress.svc.cluster.local",
                    "outbound|9092||elk-kafka-bootstrap-lb.devops-kafka-elk.svc.cluster.local",
                    "outbound|443||keda-operator-metrics-apiserver.keda.svc.cluster.local",
                    "outbound|9200||devops-elk-es-warm.devops-apps.svc.cluster.local",
                    "outbound|8000||kyverno-reports-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|443||aws-load-balancer-webhook-service.kube-system.svc.cluster.local",
                    "outbound|8090||argo-rollouts-metrics.devops.svc.cluster.local",
                    "outbound|9115||prometheus-blackbox-exporter.devops.svc.cluster.local",
                    "outbound|44180||cruise-control-ui-oauth-proxy.devops-apps.svc.cluster.local",
                    "outbound|9200||devops-elk-es-hot.devops-apps.svc.cluster.local",
                    "outbound|9600||logging-stack-logstash-headless.devops-apps.svc.cluster.local",
                    "outbound|7000||argocd-apps-applicationset-controller.argocd-apps.svc.cluster.local",
                    "outbound|8000||kyverno-cleanup-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|80||telepresence-prometheus.ambassador.svc.cluster.local",
                    "outbound|443||actions-runner-controller-webhook.github-actions.svc.cluster.local",
                    "outbound|443||agent-injector.ambassador.svc.cluster.local",
                    "outbound|443||observability-k8s-stack-opentelemetry-operator-webhook.monitoring.svc.cluster.local",
                    "outbound|443||elastic-operator-webhook.kube-system.svc.cluster.local",
                    "outbound|61678||k8s-resources-aws-node.kube-system.svc.cluster.local",
                    "outbound|8083||argo-cd-server-metrics.argocd.svc.cluster.local",
                    "outbound|9001||argocd-apps-notifications-controller-metrics.argocd-apps.svc.cluster.local",
                    "outbound|9121||argocd-apps-redis-metrics.argocd-apps.svc.cluster.local",
                    "outbound|80||cerebro-devops-elk.devops-apps.svc.cluster.local",
                    "outbound|8000||k8s-pvc-tagger.kube-system.svc.cluster.local",
                    "outbound|53||kube-dns.kube-system.svc.cluster.local",
                    "outbound|9153||kube-dns.kube-system.svc.cluster.local",
                    "outbound|9092||elk-kafka-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|80||node-cleanup-controller.devops.svc.cluster.local",
                    "outbound|8383||istio-operator.istio-system.svc.cluster.local",
                    "outbound|2888||elk-zookeeper-nodes.devops-kafka-elk.svc.cluster.local",
                    "outbound|8000||kyverno-background-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|80||traffic-manager-ambassador-agent.ambassador.svc.cluster.local",
                    "outbound|5556||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|8443||observability-k8s-stack-opentelemetry-operator.monitoring.svc.cluster.local",
                    "outbound|443||private-lb.istio-ingress.svc.cluster.local",
                    "outbound|80||strimzi-drain-cleaner.devops.svc.cluster.local",
                    "outbound|4317||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|443||argocd-apps-server.argocd-apps.svc.cluster.local",
                    "outbound|8083||argocd-apps-server-metrics.argocd-apps.svc.cluster.local",
                    "outbound|5556||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|443||argo-cd-server.argocd.svc.cluster.local",
                    "outbound|10249||observability-k8s-stack-kube-proxy.kube-system.svc.cluster.local",
                    "outbound|9100||observability-k8s-stack-prometheus-node-exporter.monitoring.svc.cluster.local",
                    "outbound|8443||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|6379||argocd-apps-redis.argocd-apps.svc.cluster.local",
                    "outbound|9273||telegraf.telegraf.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1b-101.devops-kafka-elk.svc.cluster.local",
                    "outbound|2181||elk-zookeeper-client.devops-kafka-elk.svc.cluster.local",
                    "outbound|5557||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|9090||elk-cruise-control.devops-kafka-elk.svc.cluster.local",
                    "outbound|5601||devops-elk-kb-http.devops-apps.svc.cluster.local",
                    "outbound|443||kyverno-cleanup-controller.kyverno.svc.cluster.local",
                    "outbound|8888||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|2020||fluent-bit.logging.svc.cluster.local",
                    "outbound|9001||argo-cd-notifications-controller-metrics.argocd.svc.cluster.local",
                    "outbound|9106||prometheus-cloudwatch-exporter.devops.svc.cluster.local",
                    "outbound|8082||argocd-apps-application-controller-metrics.argocd-apps.svc.cluster.local",
                    "outbound|5558||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|80||protected-lb.istio-ingress.svc.cluster.local",
                    "outbound|443||infra-lb.istio-ingress.svc.cluster.local",
                    "outbound|9090||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|9200||devops-elk-es-masters.devops-apps.svc.cluster.local",
                    "outbound|443||strimzi-drain-cleaner.devops.svc.cluster.local",
                    "outbound|7979||external-dns.kube-system.svc.cluster.local",
                    "outbound|2181||elk-zookeeper-nodes.devops-kafka-elk.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-kube-state-metrics.monitoring.svc.cluster.local",
                    "outbound|8084||argo-cd-repo-server-metrics.argocd.svc.cluster.local",
                    "outbound|15014||istiod.istio-system.svc.cluster.local",
                    "outbound|14268||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|15766||traffic-manager.ambassador.svc.cluster.local",
                    "outbound|6379||argo-cd-redis.argocd.svc.cluster.local",
                    "outbound|443||metrics-server.kube-system.svc.cluster.local",
                    "outbound|8443||druid-operator-metrics-service.druid.svc.cluster.local",
                    "outbound|9094||elk-kafka-external-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|80||elk-kafka-ui.devops-kafka-elk.svc.cluster.local",
                    "outbound|9200||devops-elk-es-http.devops-apps.svc.cluster.local",
                    "outbound|9091||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|3888||elk-zookeeper-nodes.devops-kafka-elk.svc.cluster.local",
                    "outbound|5557||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|15010||istiod.istio-system.svc.cluster.local"
                ],
                "type.googleapis.com/envoy.config.listener.v3.Listener": [],
                "type.googleapis.com/envoy.config.route.v3.RouteConfiguration": [
                    "http.80",
                    "https.443.https.infra-lb-gw.istio-ingress"
                ],
                "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret": [
                    "kubernetes://certificate-infra-lb-tls"
                ]
            }
        },
        {
            "connectionId": "private-lb-57bcdb9546-4xzbw.istio-ingress-406",
            "connectedAt": "2024-05-11T14:00:51.984760966Z",
            "address": "10.120.16.139:56152",
            "labels": {
                "app": "private-lb",
                "install.operator.istio.io/owning-resource": "unknown",
                "istio": "private-lb",
                "istio.io/rev": "default",
                "kubernetes.io/hostname": "ip-10-120-17-76.ec2.internal",
                "operator.istio.io/component": "IngressGateways",
                "pod-template-hash": "57bcdb9546",
                "service.istio.io/canonical-name": "private-lb",
                "service.istio.io/canonical-revision": "latest",
                "sidecar.istio.io/inject": "false",
                "topology.istio.io/cluster": "general-dev-02",
                "topology.kubernetes.io/region": "us-east-1",
                "topology.kubernetes.io/zone": "us-east-1d",
                "type": "ingressgateway"
            },
            "metadata": {
                "PROXY_CONFIG": {
                    "configPath": "./etc/istio/proxy",
                    "binaryPath": "/usr/local/bin/envoy",
                    "serviceCluster": "istio-proxy",
                    "drainDuration": "300s",
                    "discoveryAddress": "istiod.istio-system.svc:15012",
                    "proxyAdminPort": 15000,
                    "controlPlaneAuthPolicy": "MUTUAL_TLS",
                    "statNameLength": 189,
                    "concurrency": 2,
                    "tracing": {
                        "zipkin": {
                            "address": "zipkin.istio-system:9411"
                        }
                    },
                    "statusPort": 15020,
                    "terminationDrainDuration": "5s"
                },
                "ISTIO_VERSION": "1.21.1",
                "LABELS": {
                    "app": "private-lb",
                    "install.operator.istio.io/owning-resource": "unknown",
                    "istio": "private-lb",
                    "istio.io/rev": "default",
                    "operator.istio.io/component": "IngressGateways",
                    "service.istio.io/canonical-name": "private-lb",
                    "service.istio.io/canonical-revision": "latest",
                    "sidecar.istio.io/inject": "false",
                    "type": "ingressgateway"
                },
                "ANNOTATIONS": {
                    "instance_type": "c5d.xlarge",
                    "istio.io/rev": "default",
                    "kubernetes.io/config.seen": "2024-05-09T08:50:20.610849884Z",
                    "kubernetes.io/config.source": "api",
                    "node_name": "ip-10-120-17-76.ec2.internal",
                    "nodegroup": "control-plane-ingress",
                    "prometheus.io/path": "/stats/prometheus",
                    "prometheus.io/port": "15020",
                    "prometheus.io/scrape": "true",
                    "sidecar.istio.io/inject": "false",
                    "zone": "us-east-1d"
                },
                "INSTANCE_IPS": "10.120.16.139",
                "NAMESPACE": "istio-ingress",
                "NODE_NAME": "ip-10-120-17-76.ec2.internal",
                "WORKLOAD_NAME": "private-lb",
                "SERVICE_ACCOUNT": "private-lb-service-account",
                "MESH_ID": "general-dev-02",
                "CLUSTER_ID": "general-dev-02",
                "PLATFORM_METADATA": {
                    "aws_availability_zone": "us-east-1d",
                    "aws_instance_id": "i-03481c6c4a2638c53",
                    "aws_region": "us-east-1"
                },
                "ENVOY_STATUS_PORT": 15021,
                "ENVOY_PROMETHEUS_PORT": 15090
            },
            "locality": {
                "region": "us-east-1",
                "zone": "us-east-1d"
            },
            "watches": {
                "type.googleapis.com/envoy.config.cluster.v3.Cluster": [],
                "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment": [
                    "outbound|8080||sleep-ilo-2.devops-apps.svc.cluster.local",
                    "outbound|80||sleep-ilo-2.devops-apps.svc.cluster.local",
                    "outbound|4318||otelcol-sidecar-collector.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-sidecar-collector.monitoring.svc.cluster.local",
                    "outbound|8888||otelcol-sidecar-collector-monitoring.monitoring.svc.cluster.local",
                    "outbound|4318||otelcol-sidecar-collector-headless.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-sidecar-collector-headless.monitoring.svc.cluster.local",
                    "outbound|4318||otelcol-dev-collector.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-dev-collector.monitoring.svc.cluster.local",
                    "outbound|8888||otelcol-dev-collector-monitoring.monitoring.svc.cluster.local",
                    "outbound|4318||otelcol-dev-collector-headless.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-dev-collector-headless.monitoring.svc.cluster.local",
                    "outbound|9094||vmalertmanager-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|9093||vmalertmanager-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8401||vmstorage-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8400||vmstorage-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8482||vmstorage-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8481||vmselect-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8480||vminsert-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8429||vmagent-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8080||vmalert-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|443||dev-query-lb.monitoring.svc.cluster.local",
                    "outbound|80||dev-query-lb.monitoring.svc.cluster.local",
                    "outbound|8000||kyverno-background-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|9600||logging-stack-logstash-headless.devops-apps.svc.cluster.local",
                    "outbound|8383||istio-operator.istio-system.svc.cluster.local",
                    "outbound|9300||devops-elk-es-transport.devops-apps.svc.cluster.local",
                    "outbound|2020||fluent-bit.logging.svc.cluster.local",
                    "outbound|9001||argocd-apps-notifications-controller-metrics.argocd-apps.svc.cluster.local",
                    "outbound|8084||argo-cd-repo-server-metrics.argocd.svc.cluster.local",
                    "outbound|9402||cert-manager.cert-manager.svc.cluster.local",
                    "outbound|9090||elk-cruise-control.devops-kafka-elk.svc.cluster.local",
                    "outbound|2181||elk-zookeeper-nodes.devops-kafka-elk.svc.cluster.local",
                    "outbound|443||keda-operator-metrics-apiserver.keda.svc.cluster.local",
                    "outbound|15010||istiod.istio-system.svc.cluster.local",
                    "outbound|5558||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-kube-state-metrics.monitoring.svc.cluster.local",
                    "outbound|9153||observability-k8s-stack-coredns.kube-system.svc.cluster.local",
                    "outbound|80||argo-cd-server.argocd.svc.cluster.local",
                    "outbound|44180||cruise-control-ui-oauth-proxy.devops-apps.svc.cluster.local",
                    "outbound|80||cruise-control-ui.devops-apps.svc.cluster.local",
                    "outbound|80||node-cleanup-controller.devops.svc.cluster.local",
                    "outbound|9666||keda-operator.keda.svc.cluster.local",
                    "outbound|9092||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|2181||elk-zookeeper-client.devops-kafka-elk.svc.cluster.local",
                    "outbound|8084||argocd-apps-repo-server-metrics.argocd-apps.svc.cluster.local",
                    "outbound|80||strimzi-drain-cleaner.devops.svc.cluster.local",
                    "outbound|443||metrics-server.kube-system.svc.cluster.local",
                    "outbound|80||private-lb.istio-ingress.svc.cluster.local",
                    "outbound|9090||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|80||infra-lb.istio-ingress.svc.cluster.local",
                    "outbound|443||observability-k8s-stack-opentelemetry-operator-webhook.monitoring.svc.cluster.local",
                    "outbound|8081||traffic-manager.ambassador.svc.cluster.local",
                    "outbound|5558||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|443||cert-manager-webhook.cert-manager.svc.cluster.local",
                    "outbound|8200||devops-elk-apm-http.devops-apps.svc.cluster.local",
                    "outbound|9153||kube-dns.kube-system.svc.cluster.local",
                    "outbound|443||kyverno-svc.kyverno.svc.cluster.local",
                    "outbound|443||infra-lb.istio-ingress.svc.cluster.local",
                    "outbound|9200||devops-elk-es-masters.devops-apps.svc.cluster.local",
                    "outbound|9200||devops-elk-es-hot.devops-apps.svc.cluster.local",
                    "outbound|8000||kyverno-reports-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|443||protected-lb.istio-ingress.svc.cluster.local",
                    "outbound|9092||elk-kafka-bootstrap-lb.devops-kafka-elk.svc.cluster.local",
                    "outbound|80||traffic-manager-ambassador-agent.ambassador.svc.cluster.local",
                    "outbound|5557||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|8443||druid-operator-metrics-service.druid.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1b-100.devops-kafka-elk.svc.cluster.local",
                    "outbound|8000||k8s-pvc-tagger.kube-system.svc.cluster.local",
                    "outbound|80||elk-kafka-ui.devops-kafka-elk.svc.cluster.local",
                    "outbound|9115||prometheus-blackbox-exporter.devops.svc.cluster.local",
                    "outbound|10249||observability-k8s-stack-kube-proxy.kube-system.svc.cluster.local",
                    "outbound|443||aws-load-balancer-webhook-service.kube-system.svc.cluster.local",
                    "outbound|8081||argocd-apps-repo-server.argocd-apps.svc.cluster.local",
                    "outbound|6379||argocd-apps-redis.argocd-apps.svc.cluster.local",
                    "outbound|7000||argocd-apps-applicationset-controller.argocd-apps.svc.cluster.local",
                    "outbound|8083||argocd-apps-server-metrics.argocd-apps.svc.cluster.local",
                    "outbound|80||argocd-apps-server.argocd-apps.svc.cluster.local",
                    "outbound|443||argocd-apps-server.argocd-apps.svc.cluster.local",
                    "outbound|6379||argo-cd-redis.argocd.svc.cluster.local",
                    "outbound|443||strimzi-drain-cleaner.devops.svc.cluster.local",
                    "outbound|5601||devops-elk-kb-http.devops-apps.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1d-0.devops-kafka-elk.svc.cluster.local",
                    "outbound|9200||devops-elk-es-internal-http.devops-apps.svc.cluster.local",
                    "outbound|80||abir-test.devops-apps.svc.cluster.local",
                    "outbound|8082||argo-cd-application-controller-metrics.argocd.svc.cluster.local",
                    "outbound|443||kubernetes.default.svc.cluster.local",
                    "outbound|14250||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|9091||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|8080||keda-operator-metrics-apiserver.keda.svc.cluster.local",
                    "outbound|8085||velero.velero.svc.cluster.local",
                    "outbound|3100||argo-rollouts-dashboard.devops.svc.cluster.local",
                    "outbound|80||cruise-control-ui-oauth-proxy.devops-apps.svc.cluster.local",
                    "outbound|8888||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|443||agent-injector.ambassador.svc.cluster.local",
                    "outbound|80||protected-lb.istio-ingress.svc.cluster.local",
                    "outbound|5556||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|14268||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|80||cerebro-devops-elk.devops-apps.svc.cluster.local",
                    "outbound|8443||observability-k8s-stack-opentelemetry-operator.monitoring.svc.cluster.local",
                    "outbound|8090||argo-rollouts-metrics.devops.svc.cluster.local",
                    "outbound|5557||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|61678||k8s-resources-aws-node.kube-system.svc.cluster.local",
                    "outbound|9200||devops-elk-es-warm.devops-apps.svc.cluster.local",
                    "outbound|15014||istiod.istio-system.svc.cluster.local",
                    "outbound|443||observability-k8s-stack-victoria-metrics-operator.monitoring.svc.cluster.local",
                    "outbound|9094||elk-kafka-external-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|9121||argocd-apps-redis-metrics.argocd-apps.svc.cluster.local",
                    "outbound|9108||logging-stack-prometheus-elasticsearch-exporter.devops-apps.svc.cluster.local",
                    "outbound|8000||kyverno-svc-metrics.kyverno.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1d-1.devops-kafka-elk.svc.cluster.local",
                    "outbound|9091||elk-kafka-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|443||dashboard-kubernetes-dashboard.kube-system.svc.cluster.local",
                    "outbound|8000||karpenter.karpenter.svc.cluster.local",
                    "outbound|443||istiod.istio-system.svc.cluster.local",
                    "outbound|9100||observability-k8s-stack-prometheus-node-exporter.monitoring.svc.cluster.local",
                    "outbound|443||elastic-operator-webhook.kube-system.svc.cluster.local",
                    "outbound|15766||traffic-manager.ambassador.svc.cluster.local",
                    "outbound|8083||argo-cd-server-metrics.argocd.svc.cluster.local",
                    "outbound|443||argo-cd-server.argocd.svc.cluster.local",
                    "outbound|9106||prometheus-cloudwatch-exporter.devops.svc.cluster.local",
                    "outbound|443||actions-runner-controller-webhook.github-actions.svc.cluster.local",
                    "outbound|9411||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|4318||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|8001||k8s-pvc-tagger.kube-system.svc.cluster.local",
                    "outbound|8082||argocd-apps-application-controller-metrics.argocd-apps.svc.cluster.local",
                    "outbound|8081||argo-cd-repo-server.argocd.svc.cluster.local",
                    "outbound|9001||argo-cd-notifications-controller-metrics.argocd.svc.cluster.local",
                    "outbound|5556||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|15012||istiod.istio-system.svc.cluster.local",
                    "outbound|80||telepresence-prometheus.ambassador.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-opentelemetry-operator.monitoring.svc.cluster.local",
                    "outbound|443||private-lb.istio-ingress.svc.cluster.local",
                    "outbound|9092||elk-kafka-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|9200||devops-elk-es-http.devops-apps.svc.cluster.local",
                    "outbound|7979||external-dns.kube-system.svc.cluster.local",
                    "outbound|2888||elk-zookeeper-nodes.devops-kafka-elk.svc.cluster.local",
                    "outbound|443||kyverno-cleanup-controller.kyverno.svc.cluster.local",
                    "outbound|443||keda-admission-webhooks.keda.svc.cluster.local",
                    "outbound|4317||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1b-101.devops-kafka-elk.svc.cluster.local",
                    "outbound|53||kube-dns.kube-system.svc.cluster.local",
                    "outbound|3888||elk-zookeeper-nodes.devops-kafka-elk.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-victoria-metrics-operator.monitoring.svc.cluster.local",
                    "outbound|9200||devops-elk-es-cold.devops-apps.svc.cluster.local",
                    "outbound|8443||actions-runner-controller-metrics-service.github-actions.svc.cluster.local",
                    "outbound|8080||keda-operator.keda.svc.cluster.local",
                    "outbound|8443||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|9273||telegraf.telegraf.svc.cluster.local",
                    "outbound|8000||kyverno-cleanup-controller-metrics.kyverno.svc.cluster.local"
                ],
                "type.googleapis.com/envoy.config.listener.v3.Listener": [],
                "type.googleapis.com/envoy.config.route.v3.RouteConfiguration": [
                    "https.443.https.private-lb-gw.istio-ingress",
                    "http.80"
                ],
                "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret": [
                    "kubernetes://certificate-private-lb-tls"
                ]
            }
        },
        {
            "connectionId": "dev-query-lb-74cd8649fd-fwpnd.monitoring-410",
            "connectedAt": "2024-05-11T14:14:52.671994784Z",
            "address": "10.120.19.156:51904",
            "labels": {
                "app": "dev-query-lb",
                "environment": "dev",
                "istio": "dev-query-lb",
                "istio.io/rev": "default",
                "kubernetes.io/hostname": "ip-10-120-16-37.ec2.internal",
                "name": "dev-query-lb",
                "operator.istio.io/component": "IngressGateways",
                "owner": "general",
                "pod-template-hash": "74cd8649fd",
                "project": "mobile-devops-monitoring",
                "release": "istio",
                "service.istio.io/canonical-name": "dev-query-lb",
                "service.istio.io/canonical-revision": "latest",
                "sidecar.istio.io/inject": "false",
                "team": "devops",
                "topology.istio.io/cluster": "general-dev-02",
                "topology.kubernetes.io/region": "us-east-1",
                "topology.kubernetes.io/zone": "us-east-1d",
                "type": "ingressgateway"
            },
            "metadata": {
                "PROXY_CONFIG": {
                    "configPath": "./etc/istio/proxy",
                    "binaryPath": "/usr/local/bin/envoy",
                    "serviceCluster": "istio-proxy",
                    "drainDuration": "240s",
                    "discoveryAddress": "istiod.istio-system.svc:15012",
                    "proxyAdminPort": 15000,
                    "controlPlaneAuthPolicy": "MUTUAL_TLS",
                    "statNameLength": 189,
                    "tracing": {
                        "zipkin": {
                            "address": "zipkin.istio-system:9411"
                        }
                    },
                    "statusPort": 15020,
                    "terminationDrainDuration": "5s"
                },
                "ISTIO_VERSION": "1.21.1",
                "LABELS": {
                    "app": "dev-query-lb",
                    "environment": "dev",
                    "istio": "dev-query-lb",
                    "istio.io/rev": "default",
                    "name": "dev-query-lb",
                    "operator.istio.io/component": "IngressGateways",
                    "owner": "general",
                    "project": "mobile-devops-monitoring",
                    "release": "istio",
                    "service.istio.io/canonical-name": "dev-query-lb",
                    "service.istio.io/canonical-revision": "latest",
                    "sidecar.istio.io/inject": "false",
                    "team": "devops",
                    "type": "ingressgateway"
                },
                "ANNOTATIONS": {
                    "instance_type": "c5d.xlarge",
                    "kubernetes.io/config.seen": "2024-05-09T09:09:18.375022802Z",
                    "kubernetes.io/config.source": "api",
                    "node_name": "ip-10-120-16-37.ec2.internal",
                    "nodegroup": "mobile-devops-monitoring",
                    "prometheus.io/path": "/stats/prometheus",
                    "prometheus.io/port": "15020",
                    "prometheus.io/scrape": "true",
                    "sidecar.istio.io/inject": "false",
                    "zone": "us-east-1d"
                },
                "INSTANCE_IPS": "10.120.19.156",
                "NAMESPACE": "monitoring",
                "WORKLOAD_NAME": "dev-query-lb",
                "SERVICE_ACCOUNT": "dev-query-lb-sa",
                "MESH_ID": "general-dev-02",
                "CLUSTER_ID": "general-dev-02",
                "PLATFORM_METADATA": {
                    "aws_availability_zone": "us-east-1d",
                    "aws_instance_id": "i-0cb7667d42d7da53f",
                    "aws_region": "us-east-1"
                },
                "ENVOY_STATUS_PORT": 15021,
                "ENVOY_PROMETHEUS_PORT": 15090
            },
            "locality": {
                "region": "us-east-1",
                "zone": "us-east-1d"
            },
            "watches": {
                "type.googleapis.com/envoy.config.cluster.v3.Cluster": [],
                "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment": [
                    "outbound|8080||sleep-ilo-2.devops-apps.svc.cluster.local",
                    "outbound|80||sleep-ilo-2.devops-apps.svc.cluster.local",
                    "outbound|4318||otelcol-sidecar-collector.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-sidecar-collector.monitoring.svc.cluster.local",
                    "outbound|8888||otelcol-sidecar-collector-monitoring.monitoring.svc.cluster.local",
                    "outbound|4318||otelcol-sidecar-collector-headless.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-sidecar-collector-headless.monitoring.svc.cluster.local",
                    "outbound|4318||otelcol-dev-collector.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-dev-collector.monitoring.svc.cluster.local",
                    "outbound|8888||otelcol-dev-collector-monitoring.monitoring.svc.cluster.local",
                    "outbound|4318||otelcol-dev-collector-headless.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-dev-collector-headless.monitoring.svc.cluster.local",
                    "outbound|9094||vmalertmanager-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|9093||vmalertmanager-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8401||vmstorage-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8400||vmstorage-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8482||vmstorage-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8481||vmselect-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8480||vminsert-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8429||vmagent-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8080||vmalert-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|443||agent-injector.ambassador.svc.cluster.local",
                    "outbound|8082||argocd-apps-application-controller-metrics.argocd-apps.svc.cluster.local",
                    "outbound|5556||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|3888||elk-zookeeper-nodes.devops-kafka-elk.svc.cluster.local",
                    "outbound|15010||istiod.istio-system.svc.cluster.local",
                    "outbound|443||keda-operator-metrics-apiserver.keda.svc.cluster.local",
                    "outbound|443||kubernetes.default.svc.cluster.local",
                    "outbound|8443||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|80||infra-lb.istio-ingress.svc.cluster.local",
                    "outbound|8888||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|44180||cruise-control-ui-oauth-proxy.devops-apps.svc.cluster.local",
                    "outbound|80||cruise-control-ui.devops-apps.svc.cluster.local",
                    "outbound|80||strimzi-drain-cleaner.devops.svc.cluster.local",
                    "outbound|8000||kyverno-cleanup-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|443||observability-k8s-stack-victoria-metrics-operator.monitoring.svc.cluster.local",
                    "outbound|9153||observability-k8s-stack-coredns.kube-system.svc.cluster.local",
                    "outbound|9090||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|443||keda-admission-webhooks.keda.svc.cluster.local",
                    "outbound|8443||druid-operator-metrics-service.druid.svc.cluster.local",
                    "outbound|8000||kyverno-background-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|8080||keda-operator.keda.svc.cluster.local",
                    "outbound|9001||argocd-apps-notifications-controller-metrics.argocd-apps.svc.cluster.local",
                    "outbound|5558||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|3100||argo-rollouts-dashboard.devops.svc.cluster.local",
                    "outbound|80||dev-query-lb.monitoring.svc.cluster.local",
                    "outbound|9108||logging-stack-prometheus-elasticsearch-exporter.devops-apps.svc.cluster.local",
                    "outbound|443||aws-load-balancer-webhook-service.kube-system.svc.cluster.local",
                    "outbound|5558||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|9273||telegraf.telegraf.svc.cluster.local",
                    "outbound|5557||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|443||kyverno-svc.kyverno.svc.cluster.local",
                    "outbound|8000||kyverno-reports-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|8090||argo-rollouts-metrics.devops.svc.cluster.local",
                    "outbound|9402||cert-manager.cert-manager.svc.cluster.local",
                    "outbound|8081||argocd-apps-repo-server.argocd-apps.svc.cluster.local",
                    "outbound|80||protected-lb.istio-ingress.svc.cluster.local",
                    "outbound|9200||devops-elk-es-warm.devops-apps.svc.cluster.local",
                    "outbound|9300||devops-elk-es-transport.devops-apps.svc.cluster.local",
                    "outbound|9092||elk-kafka-bootstrap-lb.devops-kafka-elk.svc.cluster.local",
                    "outbound|5557||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|2181||elk-zookeeper-client.devops-kafka-elk.svc.cluster.local",
                    "outbound|8083||argo-cd-server-metrics.argocd.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1d-0.devops-kafka-elk.svc.cluster.local",
                    "outbound|4317||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|14268||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|8080||keda-operator-metrics-apiserver.keda.svc.cluster.local",
                    "outbound|8081||argo-cd-repo-server.argocd.svc.cluster.local",
                    "outbound|15012||istiod.istio-system.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1b-100.devops-kafka-elk.svc.cluster.local",
                    "outbound|443||cert-manager-webhook.cert-manager.svc.cluster.local",
                    "outbound|80||node-cleanup-controller.devops.svc.cluster.local",
                    "outbound|4318||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|8082||argo-cd-application-controller-metrics.argocd.svc.cluster.local",
                    "outbound|443||metrics-server.kube-system.svc.cluster.local",
                    "outbound|2888||elk-zookeeper-nodes.devops-kafka-elk.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-opentelemetry-operator.monitoring.svc.cluster.local",
                    "outbound|9115||prometheus-blackbox-exporter.devops.svc.cluster.local",
                    "outbound|443||protected-lb.istio-ingress.svc.cluster.local",
                    "outbound|7979||external-dns.kube-system.svc.cluster.local",
                    "outbound|61678||k8s-resources-aws-node.kube-system.svc.cluster.local",
                    "outbound|80||telepresence-prometheus.ambassador.svc.cluster.local",
                    "outbound|80||traffic-manager-ambassador-agent.ambassador.svc.cluster.local",
                    "outbound|15014||istiod.istio-system.svc.cluster.local",
                    "outbound|9092||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|6379||argocd-apps-redis.argocd-apps.svc.cluster.local",
                    "outbound|9106||prometheus-cloudwatch-exporter.devops.svc.cluster.local",
                    "outbound|8383||istio-operator.istio-system.svc.cluster.local",
                    "outbound|8000||karpenter.karpenter.svc.cluster.local",
                    "outbound|9411||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|5556||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|9100||observability-k8s-stack-prometheus-node-exporter.monitoring.svc.cluster.local",
                    "outbound|8443||observability-k8s-stack-opentelemetry-operator.monitoring.svc.cluster.local",
                    "outbound|443||actions-runner-controller-webhook.github-actions.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-victoria-metrics-operator.monitoring.svc.cluster.local",
                    "outbound|9091||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|443||kyverno-cleanup-controller.kyverno.svc.cluster.local",
                    "outbound|8085||velero.velero.svc.cluster.local",
                    "outbound|9121||argocd-apps-redis-metrics.argocd-apps.svc.cluster.local",
                    "outbound|9090||elk-cruise-control.devops-kafka-elk.svc.cluster.local",
                    "outbound|15766||traffic-manager.ambassador.svc.cluster.local",
                    "outbound|8001||k8s-pvc-tagger.kube-system.svc.cluster.local",
                    "outbound|80||abir-test.devops-apps.svc.cluster.local",
                    "outbound|6379||argo-cd-redis.argocd.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-kube-state-metrics.monitoring.svc.cluster.local",
                    "outbound|80||cerebro-devops-elk.devops-apps.svc.cluster.local",
                    "outbound|53||kube-dns.kube-system.svc.cluster.local",
                    "outbound|9200||devops-elk-es-cold.devops-apps.svc.cluster.local",
                    "outbound|9153||kube-dns.kube-system.svc.cluster.local",
                    "outbound|9091||elk-kafka-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|80||argo-cd-server.argocd.svc.cluster.local",
                    "outbound|443||dashboard-kubernetes-dashboard.kube-system.svc.cluster.local",
                    "outbound|80||argocd-apps-server.argocd-apps.svc.cluster.local",
                    "outbound|9666||keda-operator.keda.svc.cluster.local",
                    "outbound|2020||fluent-bit.logging.svc.cluster.local",
                    "outbound|10249||observability-k8s-stack-kube-proxy.kube-system.svc.cluster.local",
                    "outbound|9200||devops-elk-es-hot.devops-apps.svc.cluster.local",
                    "outbound|8443||actions-runner-controller-metrics-service.github-actions.svc.cluster.local",
                    "outbound|443||private-lb.istio-ingress.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1d-1.devops-kafka-elk.svc.cluster.local",
                    "outbound|9200||devops-elk-es-internal-http.devops-apps.svc.cluster.local",
                    "outbound|443||istiod.istio-system.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1b-101.devops-kafka-elk.svc.cluster.local",
                    "outbound|14250||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|8083||argocd-apps-server-metrics.argocd-apps.svc.cluster.local",
                    "outbound|80||cruise-control-ui-oauth-proxy.devops-apps.svc.cluster.local",
                    "outbound|8000||kyverno-svc-metrics.kyverno.svc.cluster.local",
                    "outbound|2181||elk-zookeeper-nodes.devops-kafka-elk.svc.cluster.local",
                    "outbound|80||elk-kafka-ui.devops-kafka-elk.svc.cluster.local",
                    "outbound|8084||argo-cd-repo-server-metrics.argocd.svc.cluster.local",
                    "outbound|8000||k8s-pvc-tagger.kube-system.svc.cluster.local",
                    "outbound|443||dev-query-lb.monitoring.svc.cluster.local",
                    "outbound|8200||devops-elk-apm-http.devops-apps.svc.cluster.local",
                    "outbound|7000||argocd-apps-applicationset-controller.argocd-apps.svc.cluster.local",
                    "outbound|9092||elk-kafka-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|9200||devops-elk-es-http.devops-apps.svc.cluster.local",
                    "outbound|443||argo-cd-server.argocd.svc.cluster.local",
                    "outbound|9200||devops-elk-es-masters.devops-apps.svc.cluster.local",
                    "outbound|9001||argo-cd-notifications-controller-metrics.argocd.svc.cluster.local",
                    "outbound|80||private-lb.istio-ingress.svc.cluster.local",
                    "outbound|8084||argocd-apps-repo-server-metrics.argocd-apps.svc.cluster.local",
                    "outbound|443||strimzi-drain-cleaner.devops.svc.cluster.local",
                    "outbound|9094||elk-kafka-external-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|5601||devops-elk-kb-http.devops-apps.svc.cluster.local",
                    "outbound|9600||logging-stack-logstash-headless.devops-apps.svc.cluster.local",
                    "outbound|443||elastic-operator-webhook.kube-system.svc.cluster.local",
                    "outbound|443||observability-k8s-stack-opentelemetry-operator-webhook.monitoring.svc.cluster.local",
                    "outbound|443||infra-lb.istio-ingress.svc.cluster.local",
                    "outbound|443||argocd-apps-server.argocd-apps.svc.cluster.local",
                    "outbound|8081||traffic-manager.ambassador.svc.cluster.local"
                ],
                "type.googleapis.com/envoy.config.listener.v3.Listener": [],
                "type.googleapis.com/envoy.config.route.v3.RouteConfiguration": [
                    "http.80",
                    "https.443.https.dev-query-lb.monitoring"
                ],
                "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret": [
                    "kubernetes://certificate-dev-query-lb"
                ]
            }
        },
        {
            "connectionId": "private-lb-57bcdb9546-zhphr.istio-ingress-408",
            "connectedAt": "2024-05-11T14:13:48.151205622Z",
            "address": "10.120.17.248:60268",
            "labels": {
                "app": "private-lb",
                "install.operator.istio.io/owning-resource": "unknown",
                "istio": "private-lb",
                "istio.io/rev": "default",
                "kubernetes.io/hostname": "ip-10-120-16-83.ec2.internal",
                "operator.istio.io/component": "IngressGateways",
                "pod-template-hash": "57bcdb9546",
                "service.istio.io/canonical-name": "private-lb",
                "service.istio.io/canonical-revision": "latest",
                "sidecar.istio.io/inject": "false",
                "topology.istio.io/cluster": "general-dev-02",
                "topology.kubernetes.io/region": "us-east-1",
                "topology.kubernetes.io/zone": "us-east-1d",
                "type": "ingressgateway"
            },
            "metadata": {
                "PROXY_CONFIG": {
                    "configPath": "./etc/istio/proxy",
                    "binaryPath": "/usr/local/bin/envoy",
                    "serviceCluster": "istio-proxy",
                    "drainDuration": "300s",
                    "discoveryAddress": "istiod.istio-system.svc:15012",
                    "proxyAdminPort": 15000,
                    "controlPlaneAuthPolicy": "MUTUAL_TLS",
                    "statNameLength": 189,
                    "concurrency": 2,
                    "tracing": {
                        "zipkin": {
                            "address": "zipkin.istio-system:9411"
                        }
                    },
                    "statusPort": 15020,
                    "terminationDrainDuration": "5s"
                },
                "ISTIO_VERSION": "1.21.1",
                "LABELS": {
                    "app": "private-lb",
                    "install.operator.istio.io/owning-resource": "unknown",
                    "istio": "private-lb",
                    "istio.io/rev": "default",
                    "operator.istio.io/component": "IngressGateways",
                    "service.istio.io/canonical-name": "private-lb",
                    "service.istio.io/canonical-revision": "latest",
                    "sidecar.istio.io/inject": "false",
                    "type": "ingressgateway"
                },
                "ANNOTATIONS": {
                    "instance_type": "c5d.xlarge",
                    "istio.io/rev": "default",
                    "kubernetes.io/config.seen": "2024-05-09T08:49:45.266921841Z",
                    "kubernetes.io/config.source": "api",
                    "node_name": "ip-10-120-16-83.ec2.internal",
                    "nodegroup": "control-plane-ingress",
                    "prometheus.io/path": "/stats/prometheus",
                    "prometheus.io/port": "15020",
                    "prometheus.io/scrape": "true",
                    "sidecar.istio.io/inject": "false",
                    "zone": "us-east-1d"
                },
                "INSTANCE_IPS": "10.120.17.248",
                "NAMESPACE": "istio-ingress",
                "NODE_NAME": "ip-10-120-16-83.ec2.internal",
                "WORKLOAD_NAME": "private-lb",
                "SERVICE_ACCOUNT": "private-lb-service-account",
                "MESH_ID": "general-dev-02",
                "CLUSTER_ID": "general-dev-02",
                "PLATFORM_METADATA": {
                    "aws_availability_zone": "us-east-1d",
                    "aws_instance_id": "i-05370d463211a084b",
                    "aws_region": "us-east-1"
                },
                "ENVOY_STATUS_PORT": 15021,
                "ENVOY_PROMETHEUS_PORT": 15090
            },
            "locality": {
                "region": "us-east-1",
                "zone": "us-east-1d"
            },
            "watches": {
                "type.googleapis.com/envoy.config.cluster.v3.Cluster": [],
                "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment": [
                    "outbound|8080||sleep-ilo-2.devops-apps.svc.cluster.local",
                    "outbound|80||sleep-ilo-2.devops-apps.svc.cluster.local",
                    "outbound|4318||otelcol-sidecar-collector.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-sidecar-collector.monitoring.svc.cluster.local",
                    "outbound|8888||otelcol-sidecar-collector-monitoring.monitoring.svc.cluster.local",
                    "outbound|4318||otelcol-sidecar-collector-headless.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-sidecar-collector-headless.monitoring.svc.cluster.local",
                    "outbound|4318||otelcol-dev-collector.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-dev-collector.monitoring.svc.cluster.local",
                    "outbound|8888||otelcol-dev-collector-monitoring.monitoring.svc.cluster.local",
                    "outbound|4318||otelcol-dev-collector-headless.monitoring.svc.cluster.local",
                    "outbound|4317||otelcol-dev-collector-headless.monitoring.svc.cluster.local",
                    "outbound|9094||vmalertmanager-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|9093||vmalertmanager-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8401||vmstorage-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8400||vmstorage-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8482||vmstorage-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8481||vmselect-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8480||vminsert-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8429||vmagent-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|8080||vmalert-observability-k8s-stack.monitoring.svc.cluster.local",
                    "outbound|443||dev-query-lb.monitoring.svc.cluster.local",
                    "outbound|80||dev-query-lb.monitoring.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1b-101.devops-kafka-elk.svc.cluster.local",
                    "outbound|9200||devops-elk-es-internal-http.devops-apps.svc.cluster.local",
                    "outbound|443||actions-runner-controller-webhook.github-actions.svc.cluster.local",
                    "outbound|6379||argo-cd-redis.argocd.svc.cluster.local",
                    "outbound|8085||velero.velero.svc.cluster.local",
                    "outbound|2888||elk-zookeeper-nodes.devops-kafka-elk.svc.cluster.local",
                    "outbound|9100||observability-k8s-stack-prometheus-node-exporter.monitoring.svc.cluster.local",
                    "outbound|8082||argo-cd-application-controller-metrics.argocd.svc.cluster.local",
                    "outbound|9200||devops-elk-es-http.devops-apps.svc.cluster.local",
                    "outbound|15012||istiod.istio-system.svc.cluster.local",
                    "outbound|8000||kyverno-svc-metrics.kyverno.svc.cluster.local",
                    "outbound|9092||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|80||elk-kafka-ui.devops-kafka-elk.svc.cluster.local",
                    "outbound|443||strimzi-drain-cleaner.devops.svc.cluster.local",
                    "outbound|8000||k8s-pvc-tagger.kube-system.svc.cluster.local",
                    "outbound|443||argo-cd-server.argocd.svc.cluster.local",
                    "outbound|5557||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|10249||observability-k8s-stack-kube-proxy.kube-system.svc.cluster.local",
                    "outbound|9091||elk-kafka-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|5556||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|8082||argocd-apps-application-controller-metrics.argocd-apps.svc.cluster.local",
                    "outbound|9300||devops-elk-es-transport.devops-apps.svc.cluster.local",
                    "outbound|443||observability-k8s-stack-opentelemetry-operator-webhook.monitoring.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-victoria-metrics-operator.monitoring.svc.cluster.local",
                    "outbound|5601||devops-elk-kb-http.devops-apps.svc.cluster.local",
                    "outbound|9090||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|8443||druid-operator-metrics-service.druid.svc.cluster.local",
                    "outbound|8443||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|8080||keda-operator.keda.svc.cluster.local",
                    "outbound|6379||argocd-apps-redis.argocd-apps.svc.cluster.local",
                    "outbound|14250||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|9200||devops-elk-es-warm.devops-apps.svc.cluster.local",
                    "outbound|9091||elk-kafka-brokers.devops-kafka-elk.svc.cluster.local",
                    "outbound|9108||logging-stack-prometheus-elasticsearch-exporter.devops-apps.svc.cluster.local",
                    "outbound|443||elastic-operator-webhook.kube-system.svc.cluster.local",
                    "outbound|443||observability-k8s-stack-victoria-metrics-operator.monitoring.svc.cluster.local",
                    "outbound|8081||traffic-manager.ambassador.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1d-0.devops-kafka-elk.svc.cluster.local",
                    "outbound|9153||observability-k8s-stack-coredns.kube-system.svc.cluster.local",
                    "outbound|443||protected-lb.istio-ingress.svc.cluster.local",
                    "outbound|15010||istiod.istio-system.svc.cluster.local",
                    "outbound|80||telepresence-prometheus.ambassador.svc.cluster.local",
                    "outbound|443||keda-operator-metrics-apiserver.keda.svc.cluster.local",
                    "outbound|8200||devops-elk-apm-http.devops-apps.svc.cluster.local",
                    "outbound|3888||elk-zookeeper-nodes.devops-kafka-elk.svc.cluster.local",
                    "outbound|8443||observability-k8s-stack-opentelemetry-operator.monitoring.svc.cluster.local",
                    "outbound|443||kyverno-svc.kyverno.svc.cluster.local",
                    "outbound|443||kyverno-cleanup-controller.kyverno.svc.cluster.local",
                    "outbound|9106||prometheus-cloudwatch-exporter.devops.svc.cluster.local",
                    "outbound|9001||argo-cd-notifications-controller-metrics.argocd.svc.cluster.local",
                    "outbound|4318||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|9200||devops-elk-es-cold.devops-apps.svc.cluster.local",
                    "outbound|8081||argo-cd-repo-server.argocd.svc.cluster.local",
                    "outbound|8443||actions-runner-controller-metrics-service.github-actions.svc.cluster.local",
                    "outbound|8383||istio-operator.istio-system.svc.cluster.local",
                    "outbound|8083||argo-cd-server-metrics.argocd.svc.cluster.local",
                    "outbound|9092||elk-kafka-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|5558||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|80||cerebro-devops-elk.devops-apps.svc.cluster.local",
                    "outbound|2020||fluent-bit.logging.svc.cluster.local",
                    "outbound|3100||argo-rollouts-dashboard.devops.svc.cluster.local",
                    "outbound|2181||elk-zookeeper-client.devops-kafka-elk.svc.cluster.local",
                    "outbound|9090||elk-cruise-control.devops-kafka-elk.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1d-1.devops-kafka-elk.svc.cluster.local",
                    "outbound|443||aws-load-balancer-webhook-service.kube-system.svc.cluster.local",
                    "outbound|443||agent-injector.ambassador.svc.cluster.local",
                    "outbound|7000||argocd-apps-applicationset-controller.argocd-apps.svc.cluster.local",
                    "outbound|443||argocd-apps-server.argocd-apps.svc.cluster.local",
                    "outbound|53||kube-dns.kube-system.svc.cluster.local",
                    "outbound|9273||telegraf.telegraf.svc.cluster.local",
                    "outbound|80||protected-lb.istio-ingress.svc.cluster.local",
                    "outbound|5556||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|80||cruise-control-ui-oauth-proxy.devops-apps.svc.cluster.local",
                    "outbound|80||infra-lb.istio-ingress.svc.cluster.local",
                    "outbound|9094||elk-kafka-us-east-1b-100.devops-kafka-elk.svc.cluster.local",
                    "outbound|8084||argo-cd-repo-server-metrics.argocd.svc.cluster.local",
                    "outbound|8090||argo-rollouts-metrics.devops.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-opentelemetry-operator.monitoring.svc.cluster.local",
                    "outbound|8000||karpenter.karpenter.svc.cluster.local",
                    "outbound|9094||elk-kafka-external-bootstrap.devops-kafka-elk.svc.cluster.local",
                    "outbound|15014||istiod.istio-system.svc.cluster.local",
                    "outbound|443||kubernetes.default.svc.cluster.local",
                    "outbound|8083||argocd-apps-server-metrics.argocd-apps.svc.cluster.local",
                    "outbound|9402||cert-manager.cert-manager.svc.cluster.local",
                    "outbound|443||private-lb.istio-ingress.svc.cluster.local",
                    "outbound|9092||elk-kafka-bootstrap-lb.devops-kafka-elk.svc.cluster.local",
                    "outbound|8080||observability-k8s-stack-kube-state-metrics.monitoring.svc.cluster.local",
                    "outbound|80||abir-test.devops-apps.svc.cluster.local",
                    "outbound|44180||cruise-control-ui-oauth-proxy.devops-apps.svc.cluster.local",
                    "outbound|14268||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|80||node-cleanup-controller.devops.svc.cluster.local",
                    "outbound|7979||external-dns.kube-system.svc.cluster.local",
                    "outbound|9200||devops-elk-es-masters.devops-apps.svc.cluster.local",
                    "outbound|4317||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|2181||elk-zookeeper-nodes.devops-kafka-elk.svc.cluster.local",
                    "outbound|9411||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|443||keda-admission-webhooks.keda.svc.cluster.local",
                    "outbound|61678||k8s-resources-aws-node.kube-system.svc.cluster.local",
                    "outbound|9200||devops-elk-es-hot.devops-apps.svc.cluster.local",
                    "outbound|80||private-lb.istio-ingress.svc.cluster.local",
                    "outbound|80||argocd-apps-server.argocd-apps.svc.cluster.local",
                    "outbound|443||cert-manager-webhook.cert-manager.svc.cluster.local",
                    "outbound|9001||argocd-apps-notifications-controller-metrics.argocd-apps.svc.cluster.local",
                    "outbound|9666||keda-operator.keda.svc.cluster.local",
                    "outbound|5557||argocd-apps-dex-server.argocd-apps.svc.cluster.local",
                    "outbound|443||infra-lb.istio-ingress.svc.cluster.local",
                    "outbound|8081||argocd-apps-repo-server.argocd-apps.svc.cluster.local",
                    "outbound|8084||argocd-apps-repo-server-metrics.argocd-apps.svc.cluster.local",
                    "outbound|80||argo-cd-server.argocd.svc.cluster.local",
                    "outbound|80||cruise-control-ui.devops-apps.svc.cluster.local",
                    "outbound|15766||traffic-manager.ambassador.svc.cluster.local",
                    "outbound|8888||opentelemetry-collector.logging.svc.cluster.local",
                    "outbound|5558||argo-cd-dex-server.argocd.svc.cluster.local",
                    "outbound|443||metrics-server.kube-system.svc.cluster.local",
                    "outbound|8000||kyverno-reports-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|9121||argocd-apps-redis-metrics.argocd-apps.svc.cluster.local",
                    "outbound|9600||logging-stack-logstash-headless.devops-apps.svc.cluster.local",
                    "outbound|8000||kyverno-cleanup-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|443||istiod.istio-system.svc.cluster.local",
                    "outbound|80||traffic-manager-ambassador-agent.ambassador.svc.cluster.local",
                    "outbound|80||strimzi-drain-cleaner.devops.svc.cluster.local",
                    "outbound|9115||prometheus-blackbox-exporter.devops.svc.cluster.local",
                    "outbound|9153||kube-dns.kube-system.svc.cluster.local",
                    "outbound|443||dashboard-kubernetes-dashboard.kube-system.svc.cluster.local",
                    "outbound|8000||kyverno-background-controller-metrics.kyverno.svc.cluster.local",
                    "outbound|8080||keda-operator-metrics-apiserver.keda.svc.cluster.local",
                    "outbound|8001||k8s-pvc-tagger.kube-system.svc.cluster.local"
                ],
                "type.googleapis.com/envoy.config.listener.v3.Listener": [],
                "type.googleapis.com/envoy.config.route.v3.RouteConfiguration": [
                    "https.443.https.private-lb-gw.istio-ingress",
                    "http.80"
                ],
                "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret": [
                    "kubernetes://certificate-private-lb-tls"
                ]
            }
        }
    ]
}

@hzxuzhonghu
Copy link
Member

Is the above get without SE, can you get the result with SE

BTW, i cannot get the ip from the result. As you said the ips are actually pods ip

  endpoints:
  - address: 10.120.17.59
    weight: 1
  - address: 10.120.17.180
    weight: 1
  - address: 10.120.17.226
    weight: 1

@hzxuzhonghu
Copy link
Member

I can see the labels do not include any "app": "sleep-ilo-2", so definitely cannot match the sidecar with selector

        {
            "connectionId": "sleep-ilo-2-9ff6468bc-vzfcq.devops-apps-407",
            "connectedAt": "2024-05-11T14:05:37.048560386Z",
            "address": "10.120.18.245:58002",
            "labels": {
                "kubernetes.io/hostname": "ip-10-120-19-57.ec2.internal",
                "topology.istio.io/cluster": "general-dev-02",
                "topology.kubernetes.io/region": "us-east-1",
                "topology.kubernetes.io/zone": "us-east-1d"
            },

@hzxuzhonghu
Copy link
Member

I found the cause, now we add registry serviceEntry in front of kube. The order should be reversed.

// initServiceControllers creates and initializes the service controllers
func (s *Server) initServiceControllers(args *PilotArgs) error {
	serviceControllers := s.ServiceController()

	s.serviceEntryController = serviceentry.NewController(
		s.configController, s.XDSServer,
		serviceentry.WithClusterID(s.clusterID),
	)
	serviceControllers.AddRegistry(s.serviceEntryController)  // 1111. first added serviceentry registry

	registered := sets.New[provider.ID]()
	for _, r := range args.RegistryOptions.Registries {
		serviceRegistry := provider.ID(r)
		if registered.Contains(serviceRegistry) {
			log.Warnf("%s registry specified multiple times.", r)
			continue
		}
		registered.Insert(serviceRegistry)
		log.Infof("Adding %s registry adapter", serviceRegistry)
		switch serviceRegistry {
		case provider.Kubernetes:
			if err := s.initKubeRegistry(args); err != nil {   // 2222. add kube registry 
				return err
			}
		default:
			return fmt.Errorf("service registry %s is not supported", r)
		}
	}

@hzxuzhonghu
Copy link
Member

This is a regression from long before, since we refactored multicluster

@hzxuzhonghu hzxuzhonghu mentioned this issue May 13, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/perf and scalability
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add kubernetes registry in front of ServiceEntry hzxuzhonghu/istio
4 participants
@howardjohn @hzxuzhonghu @liorfranko @istio-policy-bot