istio side car proxy Version istio/proxyv2:1.15.3 is crashing

[sakulka]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

we are using istio sidecar for all our workloads, we havent faces any problem but since yesterday, the envoy proxy has crashed and hence the app is unavailable. however, the proxy version on other clusters are working fine.

Version

kubernetes version 1.23.16
istio version:v2:1.15.3

Additional Information

2024-05-09T05:28:36.469821Z warning envoy config gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) 0.0.0.0_0: cannot bind '0.0.0.0': Read-only file system
99
98
2024-05-09T05:28:36.497922Z warn Envoy proxy is NOT ready: config received from XDS server, but was rejected: cds updates: 15 successful, 0 rejected; lds updates: 0 successful, 15 rejected
97
2024-05-09T05:28:37.012659Z warn Envoy proxy is NOT ready: config received from XDS server, but was rejected: cds updates: 16 successful, 0 rejected; lds updates: 0 successful, 15 rejected
96
2024-05-09T05:28:37.115585Z error envoy config listener '0.0.0.0_0' failed to bind or apply socket options: cannot bind '0.0.0.0': Read-only file system
95
2024-05-09T05:28:37.185936Z warning envoy config gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) 0.0.0.0_0: cannot bind '0.0.0.0': Read-only file system
94
93
2024-05-09T05:28:37.296195Z error envoy config listener '0.0.0.0_0' failed to bind or apply socket options: cannot bind '0.0.0.0': Read-only file system

[sakulka]

i have pasted the error logs above, failing to understand what could be the reason, i tried rolling restart of the istiod deployment, didnt help. seeing the same error logs as above in istiod pod logs as well

[zirain]

1.15 was EOL for a long long time, please upgrade and retest.

[sakulka]

@zirain thanks for responding, i do agree, we have to upgrade, do we have any solution for this problem, as we are currently using the same version in prod, it would take us a week to upgrade to new version until we push it prod

[zirain]

0.0.0.0_0 this's wired, possible cause by some incorrect configuration.

[sakulka]

can you please help me, which configuration to verify, i'm currently blocked, istio is only being used as a proxy and other features are currenlty not in use on these clusters

[sakulka]

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: >
{"apiVersion":"install.istio.io/v1alpha1","kind":"IstioOperator","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"b-ae1-1-istio-controlplane"},"name":"istio-controlplane","namespace":"istio-system"},"spec":{"components":{"base":{"enabled":true},"cni":{"enabled":false},"egressGateways":[{"enabled":false,"name":"istio-egressgateway"}],"ingressGateways":[{"enabled":false,"name":"istio-ingressgateway"}],"istiodRemote":{"enabled":false},"pilot":{"enabled":true}},"hub":"docker.io/istio","meshConfig":{"accessLogFile":"/dev/stdout","connectTimeout":"6s","defaultConfig":{"proxyMetadata":{"ISTIO_META_DNS_CAPTURE":"true"}},"enableAutoMtls":false,"enablePrometheusMerge":true,"outboundTrafficPolicy":{"mode":"ALLOW_ANY"}},"profile":"empty","tag":"1.15.3","values":{"base":{"enableCRDTemplates":false,"validationURL":""},"gateways":{"istio-egressgateway":{"autoscaleEnabled":true,"env":{},"name":"istio-egressgateway","secretVolumes":[{"mountPath":"/etc/istio/egressgateway-certs","name":"egressgateway-certs","secretName":"istio-egressgateway-certs"},{"mountPath":"/etc/istio/egressgateway-ca-certs","name":"egressgateway-ca-certs","secretName":"istio-egressgateway-ca-certs"}],"type":"ClusterIP","zvpn":{}},"istio-ingressgateway":{"autoscaleEnabled":true,"env":{},"name":"istio-ingressgateway","secretVolumes":[{"mountPath":"/etc/istio/ingressgateway-certs","name":"ingressgateway-certs","secretName":"istio-ingressgateway-certs"},{"mountPath":"/etc/istio/ingressgateway-ca-certs","name":"ingressgateway-ca-certs","secretName":"istio-ingressgateway-ca-certs"}],"type":"LoadBalancer","zvpn":{}}},"global":{"arch":{"amd64":2,"ppc64le":2,"s390x":2},"configValidation":true,"defaultNodeSelector":{},"defaultPodDisruptionBudget":{"enabled":true},"defaultResources":{"requests":{"cpu":"10m"}},"imagePullPolicy":"IfNotPresent","imagePullSecrets":[],"istioNamespace":"istio-system","istiod":{"enableAnalysis":false},"jwtPolicy":"third-party-jwt","logAsJson":false,"logging":{"level":"default:info"},"meshNetworks":{},"mountMtlsCerts":false,"multiCluster":{"clusterName":"","enabled":false},"network":"","omitSidecarInjectorConfigMap":false,"oneNamespace":false,"operatorManageWebhooks":false,"pilotCertProvider":"istiod","priorityClassName":"","proxy":{"autoInject":"enabled","clusterDomain":"cluster.local","componentLogLevel":"misc:error","enableCoreDump":false,"excludeIPRanges":"","excludeInboundPorts":"","excludeOutboundPorts":"","holdApplicationUntilProxyStarts":true,"image":"proxyv2","includeIPRanges":"","logLevel":"warning","privileged":false,"readinessFailureThreshold":30,"readinessInitialDelaySeconds":10,"readinessPeriodSeconds":2,"resources":{"limits":{"cpu":"2000m","memory":"1024Mi"},"requests":{"cpu":"100m","memory":"128Mi"}},"statusPort":15020,"tracer":"zipkin"},"proxy_init":{"image":"proxyv2","resources":{"limits":{"cpu":"2000m","memory":"1024Mi"},"requests":{"cpu":"10m","memory":"10Mi"}}},"sds":{"token":{"aud":"istio-ca"}},"sts":{"servicePort":0},"tracer":{"datadog":{},"lightstep":{},"stackdriver":{},"zipkin":{}},"useMCP":false},"istiodRemote":{"injectionURL":""},"pilot":{"autoscaleEnabled":true,"autoscaleMax":200,"autoscaleMin":50,"configMap":true,"cpu":{"targetAverageUtilization":80},"deploymentLabels":null,"enableProtocolSniffingForInbound":true,"enableProtocolSniffingForOutbound":true,"env":{},"image":"pilot","keepaliveMaxServerConnectionAge":"30m","nodeSelector":{},"replicaCount":1,"traceSampling":1},"sidecarInjectorWebhook":{"rewriteAppHTTPProbe":false},"telemetry":{"enabled":true,"v2":{"enabled":true,"metadataExchange":{"wasmEnabled":false},"prometheus":{"enabled":true,"wasmEnabled":false},"stackdriver":{"configOverride":{},"enabled":false,"logging":false,"monitoring":false,"topology":false}}}}}}
creationTimestamp: '2023-04-12T14:51:27Z'
finalizers:
- istio-finalizer.install.istio.io
generation: 2
labels:
app.kubernetes.io/instance: beta-1-istio-controlplane
managedFields:
- apiVersion: install.istio.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:finalizers':
.: {}
'v:"istio-finalizer.install.istio.io"': {}
manager: operator
operation: Update
time: '2023-04-12T14:51:28Z'
- apiVersion: install.istio.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
'f:status':
.: {}
'f:componentStatus':
.: {}
'f:Base':
.: {}
'f:status': {}
'f:Pilot':
.: {}
'f:status': {}
'f:status': {}
manager: operator
operation: Update
subresource: status
time: '2023-04-12T14:52:25Z'
- apiVersion: install.istio.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:annotations':
.: {}
'f:kubectl.kubernetes.io/last-applied-configuration': {}
'f:labels':
.: {}
'f:app.kubernetes.io/instance': {}
'f:spec':
.: {}
'f:components':
.: {}
'f:base':
.: {}
'f:enabled': {}
'f:cni':
.: {}
'f:enabled': {}
'f:egressGateways': {}
'f:ingressGateways': {}
'f:istiodRemote':
.: {}
'f:enabled': {}
'f:pilot':
.: {}
'f:enabled': {}
'f:hub': {}
'f:meshConfig':
.: {}
'f:accessLogFile': {}
'f:connectTimeout': {}
'f:defaultConfig':
.: {}
'f:proxyMetadata':
.: {}
'f:ISTIO_META_DNS_CAPTURE': {}
'f:enableAutoMtls': {}
'f:enablePrometheusMerge': {}
'f:outboundTrafficPolicy':
.: {}
'f:mode': {}
'f:profile': {}
'f:tag': {}
'f:values':
.: {}
'f:base':
.: {}
'f:enableCRDTemplates': {}
'f:validationURL': {}
'f:gateways':
.: {}
'f:istio-egressgateway':
.: {}
'f:autoscaleEnabled': {}
'f:env': {}
'f:name': {}
'f:secretVolumes': {}
'f:type': {}
'f:zvpn': {}
'f:istio-ingressgateway':
.: {}
'f:autoscaleEnabled': {}
'f:env': {}
'f:name': {}
'f:secretVolumes': {}
'f:type': {}
'f:zvpn': {}
'f:global':
.: {}
'f:arch':
.: {}
'f:amd64': {}
'f:ppc64le': {}
'f:s390x': {}
'f:configValidation': {}
'f:defaultNodeSelector': {}
'f:defaultPodDisruptionBudget':
.: {}
'f:enabled': {}
'f:defaultResources':
.: {}
'f:requests':
.: {}
'f:cpu': {}
'f:imagePullPolicy': {}
'f:imagePullSecrets': {}
'f:istioNamespace': {}
'f:istiod':
.: {}
'f:enableAnalysis': {}
'f:jwtPolicy': {}
'f:logAsJson': {}
'f:logging':
.: {}
'f:level': {}
'f:meshNetworks': {}
'f:mountMtlsCerts': {}
'f:multiCluster':
.: {}
'f:clusterName': {}
'f:enabled': {}
'f:network': {}
'f:omitSidecarInjectorConfigMap': {}
'f:oneNamespace': {}
'f:operatorManageWebhooks': {}
'f:pilotCertProvider': {}
'f:priorityClassName': {}
'f:proxy':
.: {}
'f:autoInject': {}
'f:clusterDomain': {}
'f:componentLogLevel': {}
'f:enableCoreDump': {}
'f:excludeIPRanges': {}
'f:excludeInboundPorts': {}
'f:excludeOutboundPorts': {}
'f:holdApplicationUntilProxyStarts': {}
'f:image': {}
'f:includeIPRanges': {}
'f:logLevel': {}
'f:privileged': {}
'f:readinessFailureThreshold': {}
'f:readinessInitialDelaySeconds': {}
'f:readinessPeriodSeconds': {}
'f:resources':
.: {}
'f:limits':
.: {}
'f:cpu': {}
'f:memory': {}
'f:requests':
.: {}
'f:cpu': {}
'f:memory': {}
'f:statusPort': {}
'f:tracer': {}
'f:proxy_init':
.: {}
'f:image': {}
'f:resources':
.: {}
'f:limits':
.: {}
'f:cpu': {}
'f:memory': {}
'f:requests':
.: {}
'f:cpu': {}
'f:memory': {}
'f:sds':
.: {}
'f:token':
.: {}
'f:aud': {}
'f:sts':
.: {}
'f:servicePort': {}
'f:tracer':
.: {}
'f:datadog': {}
'f:lightstep': {}
'f:stackdriver': {}
'f:zipkin': {}
'f:useMCP': {}
'f:istiodRemote':
.: {}
'f:injectionURL': {}
'f:pilot':
.: {}
'f:autoscaleEnabled': {}
'f:autoscaleMax': {}
'f:autoscaleMin': {}
'f:configMap': {}
'f:cpu':
.: {}
'f:targetAverageUtilization': {}
'f:deploymentLabels': {}
'f:enableProtocolSniffingForInbound': {}
'f:enableProtocolSniffingForOutbound': {}
'f:env': {}
'f:image': {}
'f:keepaliveMaxServerConnectionAge': {}
'f:nodeSelector': {}
'f:replicaCount': {}
'f:traceSampling': {}
'f:sidecarInjectorWebhook':
.: {}
'f:rewriteAppHTTPProbe': {}
'f:telemetry':
.: {}
'f:enabled': {}
'f:v2':
.: {}
'f:enabled': {}
'f:metadataExchange':
.: {}
'f:wasmEnabled': {}
'f:prometheus':
.: {}
'f:enabled': {}
'f:wasmEnabled': {}
'f:stackdriver':
.: {}
'f:configOverride': {}
'f:enabled': {}
'f:logging': {}
'f:monitoring': {}
'f:topology': {}
manager: argocd-application-controller
operation: Update
time: '2023-04-25T14:17:17Z'
name: istio-controlplane
namespace: istio-system
resourceVersion: '97708879'
uid: 90e94bed-a8e5-4a0d-9b03-4c5d9f9181e3
spec:
components:
base:
enabled: true
cni:
enabled: false
egressGateways:
- enabled: false
name: istio-egressgateway
ingressGateways:
- enabled: false
name: istio-ingressgateway
istiodRemote:
enabled: false
pilot:
enabled: true
hub: docker.io/istio
meshConfig:
accessLogFile: /dev/stdout
connectTimeout: 6s
defaultConfig:
proxyMetadata:
ISTIO_META_DNS_CAPTURE: 'true'
enableAutoMtls: false
enablePrometheusMerge: true
outboundTrafficPolicy:
mode: ALLOW_ANY
profile: empty
tag: 1.15.3
values:
base:
enableCRDTemplates: false
validationURL: ''
gateways:
istio-egressgateway:
autoscaleEnabled: true
env: {}
name: istio-egressgateway
secretVolumes:
- mountPath: /etc/istio/egressgateway-certs
name: egressgateway-certs
secretName: istio-egressgateway-certs
- mountPath: /etc/istio/egressgateway-ca-certs
name: egressgateway-ca-certs
secretName: istio-egressgateway-ca-certs
type: ClusterIP
zvpn: {}
istio-ingressgateway:
autoscaleEnabled: true
env: {}
name: istio-ingressgateway
secretVolumes:
- mountPath: /etc/istio/ingressgateway-certs
name: ingressgateway-certs
secretName: istio-ingressgateway-certs
- mountPath: /etc/istio/ingressgateway-ca-certs
name: ingressgateway-ca-certs
secretName: istio-ingressgateway-ca-certs
type: LoadBalancer
zvpn: {}
global:
arch:
amd64: 2
ppc64le: 2
s390x: 2
configValidation: true
defaultNodeSelector: {}
defaultPodDisruptionBudget:
enabled: true
defaultResources:
requests:
cpu: 10m
imagePullPolicy: IfNotPresent
imagePullSecrets: []
istioNamespace: istio-system
istiod:
enableAnalysis: false
jwtPolicy: third-party-jwt
logAsJson: false
logging:
level: 'default:info'
meshNetworks: {}
mountMtlsCerts: false
multiCluster:
clusterName: ''
enabled: false
network: ''
omitSidecarInjectorConfigMap: false
oneNamespace: false
operatorManageWebhooks: false
pilotCertProvider: istiod
priorityClassName: ''
proxy:
autoInject: enabled
clusterDomain: cluster.local
componentLogLevel: 'misc:error'
enableCoreDump: false
excludeIPRanges: ''
excludeInboundPorts: ''
excludeOutboundPorts: ''
holdApplicationUntilProxyStarts: true
image: proxyv2
includeIPRanges: ''
logLevel: warning
privileged: false
readinessFailureThreshold: 30
readinessInitialDelaySeconds: 10
readinessPeriodSeconds: 2
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 100m
memory: 128Mi
statusPort: 15020
tracer: zipkin
proxy_init:
image: proxyv2
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 10m
memory: 10Mi
sds:
token:
aud: istio-ca
sts:
servicePort: 0
tracer:
datadog: {}
lightstep: {}
stackdriver: {}
zipkin: {}
useMCP: false
istiodRemote:
injectionURL: ''
pilot:
autoscaleEnabled: true
autoscaleMax: 200
autoscaleMin: 50
configMap: true
cpu:
targetAverageUtilization: 80
deploymentLabels: null
enableProtocolSniffingForInbound: true
enableProtocolSniffingForOutbound: true
env: {}
image: pilot
keepaliveMaxServerConnectionAge: 30m
nodeSelector: {}
replicaCount: 1
traceSampling: 1
sidecarInjectorWebhook:
rewriteAppHTTPProbe: false
telemetry:
enabled: true
v2:
enabled: true
metadataExchange:
wasmEnabled: false
prometheus:
enabled: true
wasmEnabled: false
stackdriver:
configOverride: {}
enabled: false
logging: false
monitoring: false
topology: false
status:
componentStatus:
Base:
status: HEALTHY
Pilot:
status: HEALTHY
status: HEALTHY
this is the current istio-control plane manifest file

[sakulka]

Exec lifecycle hook ([pilot-agent wait]) for Container "istio-proxy" in Pod "cache-7b6b-cb57t_next(852fb1c9-93db-78169afce051)" failed - error: command 'pilot-agent wait' exited with 255: Error: timeout waiting for Envoy proxy to become ready. Last error: HTTP status code 503...
this is the error seen in the worload events

[howardjohn]

I think I saw this in the past, I recall it was from some invalid config that had port unset and missed validation.
As mentioned though 1.15 is super EOL. It's best to stay on top of upgrades so you don't have to scramble to upgrade when you hit issues... especially if you are relying on community support

[sakulka]

thanks for responding @howardjohn , we are in process to migrate, since this is currently in prod, we are trying to mitigate this issue, can you please share more info when you say invalid config that had port unset. which config are you referring here, can you please elaborate, thanks

[howardjohn]

I think it was a Gateway but not sure, and that wouldn't apply here. It was at a previous company so I don't have access to more info. I recall it was for using an old version though TBH and the issue has been fixed already