Active health checks

[jan-machacek-kosik]

I am testing the active health check in Envoy Gateway, but it looks like it doesn’t work as I expected.
I have a BackendTrafficPolicy with a healthCheck that points to an endpoint that always returns response code 500.

I can see that Envoy calls the healthcheck endpoint, and in the cluster stats, I can see that the cluster is unhealthy (health_flags::/failed_active_hc).
So I expect that the entire cluster will be removed from traffic. But when I call the endpoint, the traffic is routed to the cluster regardless of whether the health checks return false or not.

Should Envoy really stop traffic to an unhealthy cluster?

[arkodg]

@jan-machacek-kosik can you share a sample BTP and HTTPRoute sample config, so we can better understand what your backend and policy looks like

[arkodg]

stats might be the easiest way to understand what is happening
can you run watch -n 5 egctl x stats envoy-proxy and keep sending traffic, we'll be able to see what changed and why
after installing egctl https://gateway.envoyproxy.io/v1.0.1/install/install-egctl/

[jan-machacek-kosik]

Here is otput from your command

stats.txt

[jan-machacek-kosik]

@arkodg According to the Envoy Proxy documentation, if health checks for all endpoints in a cluster fail, the cluster enters panic mode, causing traffic to be sent to all endpoints.
This behavior is influenced by the panic threshold, which defaults to 0.5 (50%). If I want to stop all traffic when healthchecks of all endpoints fail, I need to set the panic threshold to 0.0.

Is there a way to do this in Envoy Proxy? I tried to find a solution in the Envoy Gateway API but found nothing.

[arkodg]

thanks for root causing this one @jan-machacek-kosik, you can set that xds field using the EnvoyPatchPolicy feature
https://gateway.envoyproxy.io/v1.0.1/tasks/extensibility/envoy-patch-policy/ and also raise a GH issue around making the panic threshold configurable