Failed pushing to ingester: context deadline exceeded error observed in 2.11.0

[sivesh1989]

Failed pushing to ingester: context deadline exceeded error observed in 2.11.0

We have upgraded Mimir from 2.10.1 to 2.11.0 and started observing below 500 code error in distributor logs with a frequency of 2 times in a day for approx. 5 mins( only less than 1% failure observed),

ts=2024-04-09T08:00:34.598762452Z caller=logging.go:126 level=warn traceID=22cbeb4b92a3fa6c msg="POST /api/v1/push (500) 10.008223844s Response: "failed pushing to ingester: context deadline exceeded

On analyzing further, we observe p99 latency in distributor and ingester has degraded post upgrade and observe a spike in latency of ingester during the error occurrence.

Before upgrade ( v 2.10.1):

After upgrade (v 2.11.0)

Spike in ingester latency when error is observed,

Infrastructure: Azure Kubernetes service

• Deployment tool: Helm

[aknuds1]

Have you seen this Grafana Community Slack thread? They report a similar problem, and could work around it by downgrading only the distributor back to v2.10. Have you tried this, to see if you can isolate the latency issue to the distributor?

[aknuds1]

Also, could you produce traces for failing requests, so we could see where the time is spent?