Large external label set in store gateway resulted in empty announced label set and data rage in Querier

[v1jayr]

Hi All,

Recently we encountered an issue were store gateway instances were discovered by Thanos Querier but the label set was empty and time range announced was Min & Max Integer.

store Addr: 100.89.119.151:10901 LabelSets:  Mint: 9223372036854775807 Maxt: -9223372036854775808 filtered out: does not have data within this time period: [1683753524997,1683757124997]

This happened after we introduced a new value of an external label, this doubled the size of label set. Queries for blocks under the scope of the gateway instances came back with empty result. During this time queriers produced enormous log lines (suspecting it had all labelsets from every discovered store) so we suspected the issue might be related to the total size of labelset. We updated the relabelconfig for gateway to limit the scope to a smaller set of external labels and the gateways were immediately discovered with expected external labels and time range.

I have looked at the Info grpc_method performance emitted by grpc_client (from querier) and grpc_server (from gateway) and they are look normal relative to post fix. Logs do not indicate any issue and I do not see unusual context deadline exceeded error in logs either. I am having trouble figuring out when this started happening as I could not find any metric or log showing unusual errors.

Has anyone else seen similar issue? Lack of smoking gun in metric is scary as we wont be able to detect this issue when it happens the next time.

[v1jayr]

The issue turned out to be related to relabel config and not really about the size of the label set.

[theodrim]

@v1jayr how did you solve your problem?
We intermediately see same issue, although it's non persistent and only occurs for us in lower environments - but still annoying.
Any way to reliably fix this?