Didn't find a registered implementation for 'envoy.filters.http.router' when upgrading to Kourier 1.12 #1162
Comments
I don't think that is something that is supported. To do updates, you have to go through all the versions to make sure all the migrations are done properly. For the specific issue, I haven't seen that one before. Can you start by comparing your current config to the one we define for 1.12: Please make sure to restart all the components, to be sure that there is no old config applied to envoy anymore |
|
Thank you for the reply @ReToCode 🙏 The contents of
I guess this refers to all of Knative serving not just Kourier, right? The Knative serving upgrade actually worked just fine, but I'll make sure we go through all the versions in our staging and production environments. Out of desperation, I reverted Kourier to 1.7 and left Knative serving at 1.12, and things seem to work just fine from my tests so far. It's probably still best if I have Kourier 1.12 installed instead, right? |
Hm interesting, that should define the initial config for envoy, having the correct image and Any idea @nak3?
Yeah, that is not something we test, so it can work, but it could also break for specific features. You'll definitely want to be on 1.12 with kourier as well. |
|
In case it's helpful, I did have to add this to the deployment because of my cluster version: env:
- name: KUBERNETES_MIN_VERSION
value: v1.25.15 |
|
Hm not sure if this could even have an impact, but also we do not support 1.12 with K8s 1.25, see https://github.com/knative/community/blob/main/mechanics/RELEASE-SCHEDULE.md#releases-supported-by-community. |
|
@ReToCode So sorry to comment again on this, but even though I upgraded k8s to 1.26, kourier 1.12.3 is still not working for us. I cleanup the namespace and remove any configmaps before applying, but this is what I get: $ kubectl -n kourier-system logs -f 3scale-kourier-gateway-6f84654dc4-q47hc
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:404] initializing epoch 0 (base id=1, hot restart version=11.104)
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:406] statically linked extensions:
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.common.key_value: envoy.key_value.file_based
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.request_id: envoy.request_id.uuid
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.formatter: envoy.formatter.metadata, envoy.formatter.req_without_query
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.quic.proof_source: envoy.quic.proof_source.filter_chain
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.path.match: envoy.path.match.uri_template.uri_template_matcher
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.listener_manager_impl: envoy.listener_manager_impl.default
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.regex_engines: envoy.regex_engines.google_re2
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.load_balancing_policies: envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.random, envoy.load_balancing_policies.round_robin
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.dubbo_proxy.serializers: dubbo.hessian2
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.health_checkers: envoy.health_checkers.redis, envoy.health_checkers.thrift
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.matching.input_matchers: envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.matching.action: envoy.matching.actions.format_string, filter-chain-name
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_system_buffer, envoy.filters.http.gcp_authn, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2024-03-11 08:55:14.719][1][info][main] [source/server/server.cc:408] envoy.dubbo_proxy.protocols: dubbo
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] network.connection.client: default, envoy_internal
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.upstreams: envoy.filters.connection_pools.tcp.generic
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.matching.http.input: envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.connection_handler: envoy.connection_handler.default
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.http.header_validators: envoy.http.header_validators.envoy_default
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.thrift_proxy.transports: auto, framed, header, unframed
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.resolvers: envoy.ip
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.filters.http.upstream: envoy.buffer, envoy.filters.http.admission_control, envoy.filters.http.buffer, envoy.filters.http.upstream_codec
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] quic.http_server_connection: quic.http_server_connection.default
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.header
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.quic.connection_id_generator: envoy.quic.deterministic_connection_id_generator
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.route.early_data_policy: envoy.route.early_data_policy.default
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.tracers: envoy.dynamic.ot, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2024-03-11 08:55:14.720][1][info][main] [source/server/server.cc:408] envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
[2024-03-11 08:55:14.726][1][info][main] [source/server/server.cc:456] HTTP header map info:
[2024-03-11 08:55:14.728][1][info][main] [source/server/server.cc:459] request header map: 672 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-ot-span-context,x-request-id
[2024-03-11 08:55:14.728][1][info][main] [source/server/server.cc:459] request trailer map: 120 bytes:
[2024-03-11 08:55:14.728][1][info][main] [source/server/server.cc:459] response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
[2024-03-11 08:55:14.728][1][info][main] [source/server/server.cc:459] response trailer map: 144 bytes: grpc-message,grpc-status
[2024-03-11 08:55:14.736][1][info][main] [source/server/server.cc:819] runtime: layers:
- name: static-layer
static_layer:
envoy.reloadable_features.override_request_timeout_by_gateway_timeout: false
[2024-03-11 08:55:14.736][1][error][envoy_bug] [source/common/runtime/runtime_impl.cc:385] envoy bug failure: false. Details: Using a removed guard envoy.reloadable_features.override_request_timeout_by_gateway_timeout. In future version of Envoy this will be treated as invalid configuration
[2024-03-11 08:55:14.736][1][error][envoy_bug] [./source/common/common/assert.h:38] stacktrace for envoy bug
[2024-03-11 08:55:14.736][1][error][envoy_bug] [./source/common/common/assert.h:45] #0 UNKNOWN [0x55ff89f32fdc]
[2024-03-11 08:55:14.737][1][error][envoy_bug] [./source/common/common/assert.h:45] #1 UNKNOWN [0x55ff89f36a53]
[2024-03-11 08:55:14.737][1][error][envoy_bug] [./source/common/common/assert.h:45] #2 UNKNOWN [0x55ff89f34ba3]
[2024-03-11 08:55:14.737][1][error][envoy_bug] [./source/common/common/assert.h:45] #3 UNKNOWN [0x55ff89f33dcf]
[2024-03-11 08:55:14.737][1][error][envoy_bug] [./source/common/common/assert.h:45] #4 UNKNOWN [0x55ff8987aa70]
[2024-03-11 08:55:14.737][1][error][envoy_bug] [./source/common/common/assert.h:45] #5 UNKNOWN [0x55ff8822b8f4]
[2024-03-11 08:55:14.737][1][error][envoy_bug] [./source/common/common/assert.h:45] #6 UNKNOWN [0x55ff8987330b]
[2024-03-11 08:55:14.737][1][error][envoy_bug] [./source/common/common/assert.h:45] #7 UNKNOWN [0x55ff8986ecf5]
[2024-03-11 08:55:14.737][1][error][envoy_bug] [./source/common/common/assert.h:45] #8 UNKNOWN [0x55ff8822cbd1]
[2024-03-11 08:55:14.737][1][error][envoy_bug] [./source/common/common/assert.h:45] #9 UNKNOWN [0x55ff8822c07d]
[2024-03-11 08:55:14.737][1][error][envoy_bug] [./source/common/common/assert.h:45] #10 UNKNOWN [0x55ff8820769c]
[2024-03-11 08:55:14.737][1][error][envoy_bug] [./source/common/common/assert.h:45] #11 UNKNOWN [0x55ff8820781c]
[2024-03-11 08:55:14.737][1][error][envoy_bug] [./source/common/common/assert.h:45] #12 UNKNOWN [0x55ff8820511c]
[2024-03-11 08:55:14.738][1][error][envoy_bug] [./source/common/common/assert.h:43] #13 __libc_start_main [0x7fcc8821b083]
[2024-03-11 08:55:14.738][1][info][admin] [source/server/admin/admin.cc:67] admin address: /tmp/envoy.admin
[2024-03-11 08:55:14.738][1][info][config] [source/server/configuration_impl.cc:131] loading tracing configuration
[2024-03-11 08:55:14.738][1][info][config] [source/server/configuration_impl.cc:91] loading 0 static secret(s)
[2024-03-11 08:55:14.738][1][info][config] [source/server/configuration_impl.cc:97] loading 2 cluster(s)
[2024-03-11 08:55:14.741][1][info][config] [source/server/configuration_impl.cc:101] loading 1 listener(s)
[2024-03-11 08:55:14.791][1][info][config] [source/server/configuration_impl.cc:113] loading stats configuration
[2024-03-11 08:55:14.791][1][info][main] [source/server/server.cc:915] starting main dispatch loop
[2024-03-11 08:55:14.803][1][info][runtime] [source/common/runtime/runtime_impl.cc:463] RTDS has finished initialization
[2024-03-11 08:55:14.803][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:222] cm init: initializing cds
[2024-03-11 08:55:14.804][1][warning][main] [source/server/server.cc:794] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
[2024-03-11 08:55:29.802][1][warning][config] [source/common/config/grpc_subscription_impl.cc:120] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.cluster.v3.Cluster
[2024-03-11 08:55:29.802][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:226] cm init: all clusters initialized
[2024-03-11 08:55:29.802][1][info][main] [source/server/server.cc:896] all clusters initialized. initializing init manager
[2024-03-11 08:55:44.802][1][warning][config] [source/common/config/grpc_subscription_impl.cc:120] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.listener.v3.Listener
[2024-03-11 08:55:44.802][1][info][config] [source/extensions/listener_managers/listener_manager/listener_manager_impl.cc:852] all dependencies initialized. starting workers
[2024-03-11 08:55:56.063][1][warning][config] [./source/common/config/grpc_stream.h:201] StreamAggregatedResources gRPC config stream to xds_cluster closed since 41s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111
[2024-03-11 08:56:08.438][1][warning][main] [source/server/server.cc:854] caught ENVOY_SIGTERM
[2024-03-11 08:56:08.438][1][info][main] [source/server/server.cc:985] shutting down server instance
[2024-03-11 08:56:08.438][1][info][main] [source/server/server.cc:920] main dispatch loop exited
[2024-03-11 08:56:08.438][1][info][main] [source/server/server.cc:972] exitingI tried removing this section from the config but no good: layered_runtime:
layers:
- name: static-layer
static_layer:
envoy.reloadable_features.override_request_timeout_by_gateway_timeout: falseIs there anything I could do here? |
|
It seems that your error (envoy is terminated, SIGTERM is caught) comes from the fact that ingress pod (3scale-kourier-gateway) cannot connect to the net-kourier-controller (acts as the xDs server) in knative-serving ns. Could you paste the status of your installation eg. |
|
/reopen |
|
@ReToCode: Reopened this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Thank you for your reply! Looking at the The deployment is there, just unhealthy: $ kubectl -n knative-serving get deployments.apps net-kourier-controller
NAME READY UP-TO-DATE AVAILABLE AGE
net-kourier-controller 0/1 0 0 23mHere's the end part of the deployment's Conditions:
Type Status Reason
---- ------ ------
Available False MinimumReplicasUnavailable
ReplicaFailure True FailedCreate
Progressing False ProgressDeadlineExceeded
OldReplicaSets: <none>
NewReplicaSet: net-kourier-controller-95f56644f (0/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 22m deployment-controller Scaled up replica set net-kourier-controller-95f56644f to 1From describing the corresponding replicaset: Warning FailedCreate 27m replicaset-controller Error creating: Pod "net-kourier-controller-95f56644f-2zj5p" is invalid: [spec.containers[0].livenessProbe: Required value: must specify a handler type, spec.containers[0].readinessProbe: Required value: must specify a handler type] |
|
We're on kubernetes 1.26 but grpc health probes are a feature available at version 1.27, it seems... This (perhaps?) doesn't match this info? https://github.com/knative/community/blob/main/mechanics/RELEASE-SCHEDULE.md#releases-supported-by-community (cc @ReToCode) Is it still possible to do some |
|
I tried to replace the grpc probe with |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem summary:
As part of my knative serving upgrade attempt from 1.7 to 1.12, I'm also upgrading kourier from 1.7 to 1.12, but I'm seeing an error in the
3scale-kourier-gatewaypods that's preventing the pods from becoming READY (and allowing my services to communicate via the internal knative service URLs).Environment:
v1.25.15-eks-4f4795d1.12.11.12.1Error logs:
Any tips on how to resolve this would be appreciated.
The text was updated successfully, but these errors were encountered: