Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rpc error: code = Unavailable desc = Starting #33831

Open
wolf666666 opened this issue Apr 27, 2024 · 1 comment
Open

rpc error: code = Unavailable desc = Starting #33831

wolf666666 opened this issue Apr 27, 2024 · 1 comment
Labels
bug question Questions that are neither investigations, bugs, nor enhancements

Comments

@wolf666666
Copy link

If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing
envoy-security@googlegroups.com where the issue will be triaged appropriately.

Title: rpc error: code = Unavailable desc = Starting

Description:

One of the cortex ingester instance was killed by the OOM killer because of the memory limit, then the ingester start to startup. but the ingester is still receive request at the same time, so it generate a lot error message:
"Apr 27 17:16:43 cortex-1.16.0[3069602]: ts=2024-04-27T15:16:43.536890158Z caller=grpc_logging.go:64 level=warn method=/cortex.Ingester/QueryStream duration=55.725µs err="rpc error: code = Unavailable desc = Starting" msg=gRPC"

It seems that the distributor still route request to this ingester instance even it have not startup successfully.
and the client that send request to cortex distributor get error message at the same time:
"Request failed. Status: 500 Body: maxFailure (quorum) on a given error family, rpc error: code = Code(500) desc = addr=xxxxxx:port state=ACTIVE zone=www, rpc error: code = Unavailable desc = Starting
"
expect:
the cortex ingester receive request only when it startup totally.

@wolf666666 wolf666666 added bug triage Issue requires triage labels Apr 27, 2024
@adisuissa
Copy link
Contributor

Is this an issue that should be reported to https://github.com/cortexproject/cortex ?
If it is Envoy related, can you provide a more detailed explanation related to Envoy?

@adisuissa adisuissa added question Questions that are neither investigations, bugs, nor enhancements and removed triage Issue requires triage labels Apr 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug question Questions that are neither investigations, bugs, nor enhancements
Projects
None yet
Development

No branches or pull requests

2 participants