Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OpenYurt Knative Setup Issue #950

Open
yulinzou opened this issue Mar 5, 2024 · 2 comments
Open

OpenYurt Knative Setup Issue #950

yulinzou opened this issue Mar 5, 2024 · 2 comments
Assignees
@yulinzou @lrq619 @jchua99
Labels
bug Something isn't working

Comments

@yulinzou
Copy link
Contributor

yulinzou commented Mar 5, 2024

Describe the bug
Failed to setup Knative for OpenYurt, the cluster-local-gateway and istio-ingressgateway pod are not in READY status

- Processing resources for Istio core.
✔ Istio core installed
- Processing resources for Istiod.
- Processing resources for Istiod. Waiting for Deployment/istio-system/istiod
✔ Istiod installed
- Processing resources for Ingress gateways.
- Processing resources for Ingress gateways. Waiting for Deployment/istio-system/cluster-local-ga...
✘ Ingress gateways encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition
  Deployment/istio-system/cluster-local-gateway (containers with unready status: [istio-proxy])
  Deployment/istio-system/istio-ingressgateway (containers with unready status: [istio-proxy])
- Pruning removed resourcesError: failed to install manifests: errors occurred during operation

To Reproduce
After setting up Kubernetes cluster include cloud and edge nodes, run ./openyurt_deployer knative, following the OpenYurt setup manual.

Expected behavior
Knative should be successfully setup for OpenYurt.

Logs
kubectl get pods -n istio-system

NAME                                     READY   STATUS    RESTARTS   AGE
cluster-local-gateway-76bbc4bf78-lnjm2   0/1     Running   0          15m
istio-ingressgateway-dbcbdd6d5-lvw2q     0/1     Running   0          15m
istiod-657b54846b-5sxrq                  1/1     Running   0          15m

kubectl get svc -n istio-system

NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                      AGE
cluster-local-gateway   ClusterIP      10.105.191.168   <none>        15020/TCP,80/TCP,443/TCP                     20m
istio-ingressgateway    LoadBalancer   10.97.13.94      <pending>     15021:31926/TCP,80:30778/TCP,443:31594/TCP   20m
istiod                  ClusterIP      10.99.107.159    <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP        20m
knative-local-gateway   ClusterIP      10.100.181.234   <none>        80/TCP                                       14m

kubectl logs cluster-local-gateway-76bbc4bf78-lnjm2 -n istio-system --tail=20

2024-03-05T08:39:30.972101Z	warn	ca	ca request failed, starting attempt 4 in 797.764779ms
2024-03-05T08:39:31.770157Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.99.107.159:15012: i/o timeout"
2024-03-05T08:39:31.912041Z	warn	ca	ca request failed, starting attempt 1 in 102.654655ms
2024-03-05T08:39:32.015484Z	warn	ca	ca request failed, starting attempt 2 in 203.884035ms
2024-03-05T08:39:32.220005Z	warn	ca	ca request failed, starting attempt 3 in 367.516646ms
2024-03-05T08:39:32.588638Z	warn	ca	ca request failed, starting attempt 4 in 846.114595ms
2024-03-05T08:39:33.434912Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.99.107.159:15012: i/o timeout"
2024-03-05T08:39:33.642198Z	warn	ca	ca request failed, starting attempt 1 in 93.76391ms
2024-03-05T08:39:33.736643Z	warn	ca	ca request failed, starting attempt 2 in 188.254413ms
2024-03-05T08:39:33.925052Z	warn	ca	ca request failed, starting attempt 3 in 430.149563ms
2024-03-05T08:39:34.355720Z	warn	ca	ca request failed, starting attempt 4 in 852.894099ms
2024-03-05T08:39:35.209024Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.99.107.159:15012: i/o timeout"
2024-03-05T08:39:35.608001Z	warn	ca	ca request failed, starting attempt 1 in 94.065982ms
2024-03-05T08:39:35.702460Z	warn	ca	ca request failed, starting attempt 2 in 193.463351ms
2024-03-05T08:39:35.896931Z	warn	ca	ca request failed, starting attempt 3 in 392.137813ms
2024-03-05T08:39:36.289533Z	warn	ca	ca request failed, starting attempt 4 in 871.494812ms
2024-03-05T08:39:37.161684Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.99.107.159:15012: i/o timeout"
2024-03-05T08:39:37.563544Z	warn	ca	ca request failed, starting attempt 1 in 91.317602ms
2024-03-05T08:39:45.811213Z	warning	envoy config	StreamAggregatedResources gRPC config stream to xds-grpc closed since 3220s ago: 14, connection error: desc = "transport: Error while dialing dial tcp 10.99.107.159:15012: i/o timeout"
2024-03-05T08:40:14.606232Z	warning	envoy config	StreamAggregatedResources gRPC config stream to xds-grpc closed since 3249s ago: 14, connection error: desc = "transport: Error while dialing dial tcp 10.99.107.159:15012: i/o timeout"

kubectl logs istio-ingressgateway-dbcbdd6d5-lvw2q -n istio-system --tail=20

2024-03-05T08:39:18.676212Z	warn	ca	ca request failed, starting attempt 1 in 109.786691ms
2024-03-05T08:39:18.786619Z	warn	ca	ca request failed, starting attempt 2 in 215.392111ms
2024-03-05T08:39:19.003146Z	warn	ca	ca request failed, starting attempt 3 in 399.242009ms
2024-03-05T08:39:19.267186Z	warning	envoy config	StreamAggregatedResources gRPC config stream to xds-grpc closed since 3194s ago: 14, connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2024-03-05T08:39:19.402866Z	warn	ca	ca request failed, starting attempt 4 in 799.453932ms
2024-03-05T08:39:20.203095Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 192.168.32.2:34056->10.96.0.10:53: i/o timeout"
2024-03-05T08:39:20.340578Z	warn	ca	ca request failed, starting attempt 1 in 94.646846ms
2024-03-05T08:39:20.436013Z	warn	ca	ca request failed, starting attempt 2 in 219.345007ms
2024-03-05T08:39:20.656524Z	warn	ca	ca request failed, starting attempt 3 in 399.759668ms
2024-03-05T08:39:21.057245Z	warn	ca	ca request failed, starting attempt 4 in 863.372867ms
2024-03-05T08:39:21.921368Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 192.168.32.2:34056->10.96.0.10:53: i/o timeout"
2024-03-05T08:39:22.142503Z	warn	ca	ca request failed, starting attempt 1 in 94.927632ms
2024-03-05T08:39:22.237921Z	warn	ca	ca request failed, starting attempt 2 in 206.659364ms
2024-03-05T08:39:22.445396Z	warn	ca	ca request failed, starting attempt 3 in 381.600133ms
2024-03-05T08:39:22.828096Z	warn	ca	ca request failed, starting attempt 4 in 830.636367ms
2024-03-05T08:39:23.659210Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 192.168.32.2:34056->10.96.0.10:53: i/o timeout"
2024-03-05T08:39:23.966994Z	warn	ca	ca request failed, starting attempt 1 in 101.581298ms
2024-03-05T08:39:24.069457Z	warn	ca	ca request failed, starting attempt 2 in 189.259981ms
2024-03-05T08:39:24.258852Z	warn	ca	ca request failed, starting attempt 3 in 362.50633ms
2024-03-05T08:39:24.622570Z	warn	ca	ca request failed, starting attempt 4 in 873.762276ms

Notes
The configuration of nodes is as following, two nodes on cloud site, two nodes on edge sides, using image emulab-ops/UBUNTU20-64-STD

{
    "master": "yulin001@hp101.utah.cloudlab.us",
    "workers": {
        "cloud": [
            "yulin001@hp118.utah.cloudlab.us"
        ],
        "edge": [
            "yulin001@hp111.utah.cloudlab.us",
	    "yulin001@hp086.utah.cloudlab.us"
        ]
    }
}
@yulinzou yulinzou added the bug Something isn't working label Mar 5, 2024
@yulinzou
Copy link
Contributor Author

yulinzou commented Mar 5, 2024

kubectl describe pod istio-ingressgateway-dbcbdd6d5-gd9jf -n istio-system

Name:             istio-ingressgateway-dbcbdd6d5-gd9jf
Namespace:        istio-system
Priority:         0
Service Account:  istio-ingressgateway-service-account
Node:             edge-0.yulin001-194981.ntu-cloud-pg0.utah.cloudlab.us/128.110.218.150
Start Time:       Tue, 05 Mar 2024 02:16:53 -0700
Labels:           app=istio-ingressgateway
                  chart=gateways
                  heritage=Tiller
                  install.operator.istio.io/owning-resource=unknown
                  istio=ingressgateway
                  istio.io/rev=default
                  operator.istio.io/component=IngressGateways
                  pod-template-hash=dbcbdd6d5
                  release=istio
                  service.istio.io/canonical-name=istio-ingressgateway
                  service.istio.io/canonical-revision=latest
                  sidecar.istio.io/inject=false
Annotations:      cni.projectcalico.org/containerID: ad4a6fe1c266d4264a17c4c037581461abceec9904bc5253f6cbd562400aac79
                  cni.projectcalico.org/podIP: 192.168.157.194/32
                  cni.projectcalico.org/podIPs: 192.168.157.194/32
                  prometheus.io/path: /stats/prometheus
                  prometheus.io/port: 15020
                  prometheus.io/scrape: true
                  sidecar.istio.io/inject: false
Status:           Running
IP:               192.168.157.194
IPs:
  IP:           192.168.157.194
Controlled By:  ReplicaSet/istio-ingressgateway-dbcbdd6d5
Containers:
  istio-proxy:
    Container ID:  containerd://694a4a8c920e222c07990a2762cb14669a7c05ae1507f02e108e56e8f59c456e
    Image:         docker.io/istio/proxyv2:1.16.3
    Image ID:      docker.io/istio/proxyv2@sha256:35ecc61d241242e8d68746fcccb253c4abc7d3b7671702ddb9e20b532cc514f2
    Ports:         15021/TCP, 8080/TCP, 8443/TCP, 15090/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      proxy
      router
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --log_output_level=default:info
    State:          Running
      Started:      Tue, 05 Mar 2024 02:17:00 -0700
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   128Mi
    Readiness:  http-get http://:15021/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
    Environment:
      JWT_POLICY:                   first-party-jwt
      PILOT_CERT_PROVIDER:          istiod
      CA_ADDR:                      istiod.istio-system.svc:15012
      NODE_NAME:                     (v1:spec.nodeName)
      POD_NAME:                     istio-ingressgateway-dbcbdd6d5-gd9jf (v1:metadata.name)
      POD_NAMESPACE:                istio-system (v1:metadata.namespace)
      INSTANCE_IP:                   (v1:status.podIP)
      HOST_IP:                       (v1:status.hostIP)
      SERVICE_ACCOUNT:               (v1:spec.serviceAccountName)
      ISTIO_META_WORKLOAD_NAME:     istio-ingressgateway
      ISTIO_META_OWNER:             kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-ingressgateway
      ISTIO_META_MESH_ID:           cluster.local
      TRUST_DOMAIN:                 cluster.local
      ISTIO_META_UNPRIVILEGED_POD:  true
      ISTIO_META_CLUSTER_ID:        Kubernetes
    Mounts:
      /etc/istio/config from config-volume (rw)
      /etc/istio/ingressgateway-ca-certs from ingressgateway-ca-certs (ro)
      /etc/istio/ingressgateway-certs from ingressgateway-certs (ro)
      /etc/istio/pod from podinfo (rw)
      /etc/istio/proxy from istio-envoy (rw)
      /var/lib/istio/data from istio-data (rw)
      /var/run/secrets/credential-uds from credential-socket (rw)
      /var/run/secrets/istio from istiod-ca-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9sggs (ro)
      /var/run/secrets/workload-spiffe-credentials from workload-certs (rw)
      /var/run/secrets/workload-spiffe-uds from workload-socket (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  workload-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  credential-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  workload-certs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  istiod-ca-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-ca-root-cert
    Optional:  false
  podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
  istio-envoy:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  istio-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio
    Optional:  true
  ingressgateway-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-certs
    Optional:    true
  ingressgateway-ca-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-ca-certs
    Optional:    true
  kube-api-access-9sggs:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  9m2s                  default-scheduler  Successfully assigned istio-system/istio-ingressgateway-dbcbdd6d5-gd9jf to edge-0.yulin001-194981.ntu-cloud-pg0.utah.cloudlab.us
  Normal   Pulling    9m2s                  kubelet            Pulling image "docker.io/istio/proxyv2:1.16.3"
  Normal   Pulled     8m55s                 kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.16.3" in 443.023989ms (6.245443949s including waiting)
  Normal   Created    8m55s                 kubelet            Created container istio-proxy
  Normal   Started    8m55s                 kubelet            Started container istio-proxy
  Warning  Unhealthy  4m (x152 over 8m54s)  kubelet            Readiness probe failed: Get "http://192.168.157.194:15021/healthz/ready": dial tcp 192.168.157.194:15021: connect: connection refused

kubectl describe pod cluster-local-gateway-76bbc4bf78-xmjpv -n istio-system

Name:             cluster-local-gateway-76bbc4bf78-xmjpv
Namespace:        istio-system
Priority:         0
Service Account:  cluster-local-gateway-service-account
Node:             edge-0.yulin001-194981.ntu-cloud-pg0.utah.cloudlab.us/128.110.218.150
Start Time:       Tue, 05 Mar 2024 02:16:53 -0700
Labels:           app=cluster-local-gateway
                  chart=gateways
                  heritage=Tiller
                  install.operator.istio.io/owning-resource=unknown
                  istio=cluster-local-gateway
                  istio.io/rev=default
                  operator.istio.io/component=IngressGateways
                  pod-template-hash=76bbc4bf78
                  release=istio
                  service.istio.io/canonical-name=cluster-local-gateway
                  service.istio.io/canonical-revision=latest
                  sidecar.istio.io/inject=false
Annotations:      cni.projectcalico.org/containerID: 1b4b8a4991baa26046ca537fb6a765198453cb4e7848cc35cbec4d3a28044394
                  cni.projectcalico.org/podIP: 192.168.157.193/32
                  cni.projectcalico.org/podIPs: 192.168.157.193/32
                  prometheus.io/path: /stats/prometheus
                  prometheus.io/port: 15020
                  prometheus.io/scrape: true
                  sidecar.istio.io/inject: false
Status:           Running
IP:               192.168.157.193
IPs:
  IP:           192.168.157.193
Controlled By:  ReplicaSet/cluster-local-gateway-76bbc4bf78
Containers:
  istio-proxy:
    Container ID:  containerd://343cad521d50edf23bfa3d741a08034082ab1962663f354f1fdd7da84b2633a7
    Image:         docker.io/istio/proxyv2:1.16.3
    Image ID:      docker.io/istio/proxyv2@sha256:35ecc61d241242e8d68746fcccb253c4abc7d3b7671702ddb9e20b532cc514f2
    Ports:         15020/TCP, 8080/TCP, 8443/TCP, 15090/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      proxy
      router
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --log_output_level=default:info
    State:          Running
      Started:      Tue, 05 Mar 2024 02:16:59 -0700
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   128Mi
    Readiness:  http-get http://:15021/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
    Environment:
      JWT_POLICY:                   first-party-jwt
      PILOT_CERT_PROVIDER:          istiod
      CA_ADDR:                      istiod.istio-system.svc:15012
      NODE_NAME:                     (v1:spec.nodeName)
      POD_NAME:                     cluster-local-gateway-76bbc4bf78-xmjpv (v1:metadata.name)
      POD_NAMESPACE:                istio-system (v1:metadata.namespace)
      INSTANCE_IP:                   (v1:status.podIP)
      HOST_IP:                       (v1:status.hostIP)
      SERVICE_ACCOUNT:               (v1:spec.serviceAccountName)
      ISTIO_META_WORKLOAD_NAME:     cluster-local-gateway
      ISTIO_META_OWNER:             kubernetes://apis/apps/v1/namespaces/istio-system/deployments/cluster-local-gateway
      ISTIO_META_MESH_ID:           cluster.local
      TRUST_DOMAIN:                 cluster.local
      ISTIO_META_UNPRIVILEGED_POD:  true
      ISTIO_META_CLUSTER_ID:        Kubernetes
    Mounts:
      /etc/istio/config from config-volume (rw)
      /etc/istio/ingressgateway-ca-certs from ingressgateway-ca-certs (ro)
      /etc/istio/ingressgateway-certs from ingressgateway-certs (ro)
      /etc/istio/pod from podinfo (rw)
      /etc/istio/proxy from istio-envoy (rw)
      /var/lib/istio/data from istio-data (rw)
      /var/run/secrets/credential-uds from credential-socket (rw)
      /var/run/secrets/istio from istiod-ca-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dzjnl (ro)
      /var/run/secrets/workload-spiffe-credentials from workload-certs (rw)
      /var/run/secrets/workload-spiffe-uds from workload-socket (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  workload-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  credential-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  workload-certs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  istiod-ca-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-ca-root-cert
    Optional:  false
  podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
  istio-envoy:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  istio-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio
    Optional:  true
  ingressgateway-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-certs
    Optional:    true
  ingressgateway-ca-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-ca-certs
    Optional:    true
  kube-api-access-dzjnl:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                      From               Message
  ----     ------     ----                     ----               -------
  Normal   Scheduled  9m21s                    default-scheduler  Successfully assigned istio-system/cluster-local-gateway-76bbc4bf78-xmjpv to edge-0.yulin001-194981.ntu-cloud-pg0.utah.cloudlab.us
  Normal   Pulling    9m21s                    kubelet            Pulling image "docker.io/istio/proxyv2:1.16.3"
  Normal   Pulled     9m15s                    kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.16.3" in 5.809399086s (5.809488756s including waiting)
  Normal   Created    9m15s                    kubelet            Created container istio-proxy
  Normal   Started    9m15s                    kubelet            Started container istio-proxy
  Warning  Unhealthy  4m19s (x154 over 9m14s)  kubelet            Readiness probe failed: Get "http://192.168.157.193:15021/healthz/ready": dial tcp 192.168.157.193:15021: connect: connection refused

@yulinzou
Copy link
Contributor Author

yulinzou commented Mar 5, 2024

kubectl describe pod istiod-657b54846b-2ncl8 -n istio-system

Name:             istiod-657b54846b-2ncl8
Namespace:        istio-system
Priority:         0
Service Account:  istiod
Node:             edge-1.yulin001-194981.ntu-cloud-pg0.utah.cloudlab.us/128.110.218.125
Start Time:       Tue, 05 Mar 2024 02:16:43 -0700
Labels:           app=istiod
                  install.operator.istio.io/owning-resource=unknown
                  istio=pilot
                  istio.io/rev=default
                  operator.istio.io/component=Pilot
                  pod-template-hash=657b54846b
                  sidecar.istio.io/inject=false
Annotations:      cni.projectcalico.org/containerID: 5cdc8c69e6be7075b384931191d7f08a742cc61928743a598e886165e56d6642
                  cni.projectcalico.org/podIP: 192.168.32.1/32
                  cni.projectcalico.org/podIPs: 192.168.32.1/32
                  prometheus.io/port: 15014
                  prometheus.io/scrape: true
                  sidecar.istio.io/inject: false
Status:           Running
IP:               192.168.32.1
IPs:
  IP:           192.168.32.1
Controlled By:  ReplicaSet/istiod-657b54846b
Containers:
  discovery:
    Container ID:  containerd://f1789e0fd9199390262c119cfca3a118ab2c7bb127298e07b9860a87fd958174
    Image:         docker.io/istio/pilot:1.16.3
    Image ID:      docker.io/istio/pilot@sha256:91a8907fee81051fe22d767cbca2584d1b07b475c686403395a7207d82e8f36e
    Ports:         8080/TCP, 15010/TCP, 15017/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      discovery
      --monitoringAddr=:15014
      --log_output_level=default:info
      --domain
      cluster.local
      --keepaliveMaxServerConnectionAge
      30m
    State:          Running
      Started:      Tue, 05 Mar 2024 02:16:48 -0700
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      500m
      memory:   2Gi
    Readiness:  http-get http://:8080/ready delay=1s timeout=5s period=3s #success=1 #failure=3
    Environment:
      REVISION:                                     default
      JWT_POLICY:                                   first-party-jwt
      PILOT_CERT_PROVIDER:                          istiod
      POD_NAME:                                     istiod-657b54846b-2ncl8 (v1:metadata.name)
      POD_NAMESPACE:                                istio-system (v1:metadata.namespace)
      SERVICE_ACCOUNT:                               (v1:spec.serviceAccountName)
      KUBECONFIG:                                   /var/run/secrets/remote/config
      PILOT_TRACE_SAMPLING:                         1
      PILOT_ENABLE_PROTOCOL_SNIFFING_FOR_OUTBOUND:  true
      PILOT_ENABLE_PROTOCOL_SNIFFING_FOR_INBOUND:   true
      ISTIOD_ADDR:                                  istiod.istio-system.svc:15012
      PILOT_ENABLE_ANALYSIS:                        false
      CLUSTER_ID:                                   Kubernetes
    Mounts:
      /etc/cacerts from cacerts (ro)
      /var/run/secrets/istio-dns from local-certs (rw)
      /var/run/secrets/istiod/ca from istio-csr-ca-configmap (ro)
      /var/run/secrets/istiod/tls from istio-csr-dns-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rvxxs (ro)
      /var/run/secrets/remote from istio-kubeconfig (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  local-certs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  cacerts:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cacerts
    Optional:    true
  istio-kubeconfig:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-kubeconfig
    Optional:    true
  istio-csr-dns-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istiod-tls
    Optional:    true
  istio-csr-ca-configmap:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-ca-root-cert
    Optional:  true
  kube-api-access-rvxxs:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  34m                default-scheduler  Successfully assigned istio-system/istiod-657b54846b-2ncl8 to edge-1.yulin001-194981.ntu-cloud-pg0.utah.cloudlab.us
  Normal   Pulling    34m                kubelet            Pulling image "docker.io/istio/pilot:1.16.3"
  Normal   Pulled     34m                kubelet            Successfully pulled image "docker.io/istio/pilot:1.16.3" in 4.771692006s (4.771701671s including waiting)
  Normal   Created    34m                kubelet            Created container discovery
  Normal   Started    34m                kubelet            Started container discovery
  Warning  Unhealthy  34m (x2 over 34m)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yulinzou yulinzou

@lrq619 lrq619

@jchua99 jchua99

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@yulinzou @lrq619 @jchua99