You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing envoy-security@googlegroups.com where the issue will be triaged appropriately.
One of the cortex ingester instance was killed by the OOM killer because of the memory limit, then the ingester start to startup. but the ingester is still receive request at the same time, so it generate a lot error message:
"Apr 27 17:16:43 cortex-1.16.0[3069602]: ts=2024-04-27T15:16:43.536890158Z caller=grpc_logging.go:64 level=warn method=/cortex.Ingester/QueryStream duration=55.725µs err="rpc error: code = Unavailable desc = Starting" msg=gRPC"
It seems that the distributor still route request to this ingester instance even it have not startup successfully.
and the client that send request to cortex distributor get error message at the same time:
"Request failed. Status: 500 Body: maxFailure (quorum) on a given error family, rpc error: code = Code(500) desc = addr=xxxxxx:port state=ACTIVE zone=www, rpc error: code = Unavailable desc = Starting
"
expect:
the cortex ingester receive request only when it startup totally.
The text was updated successfully, but these errors were encountered:
Is this an issue that should be reported to https://github.com/cortexproject/cortex ?
If it is Envoy related, can you provide a more detailed explanation related to Envoy?
If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing
envoy-security@googlegroups.com where the issue will be triaged appropriately.
Title: rpc error: code = Unavailable desc = Starting
Description:
It seems that the distributor still route request to this ingester instance even it have not startup successfully.
and the client that send request to cortex distributor get error message at the same time:
"Request failed. Status: 500 Body: maxFailure (quorum) on a given error family, rpc error: code = Code(500) desc = addr=xxxxxx:port state=ACTIVE zone=www, rpc error: code = Unavailable desc = Starting
"
expect:
the cortex ingester receive request only when it startup totally.
The text was updated successfully, but these errors were encountered: