Huge amount of logs on the tempo querier after a single query #3517
-
Hello! I am not sure if this is a bug or simply a misconfiguration issue, but we have been using Tempo for some time, and our cloud costs for it are outrageously high for the very little load we push at it (it receives less than a trace per second). The reason is difficult to pinpoint, but while looking at the tempo querier logs, I found it was spamming with info logs for every single query I do. Let take a example: And I get thousands of logs in return:
If I get it right, the query is itself querying thousands of blocks to find For context: we're using tempo-distributed helm chart v1.8.5. We have been using tempo since v1.0.0. Our data is stored in GCS. Compactor is running and is configured with the helm chart default values. Thanks in advance! |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 4 replies
-
When searching for traces by id Tempo does in fact check every block. First it checks a bloom filter, if that passes then it moves on to checking the actual block for data. That is a fairly intense log and should probably be dropped to "debug" level. You can check Also, you can pass start/end parameters to that endpoint. This will force Tempo to search fewer blocks. |
Beta Was this translation helpful? Give feedback.
When searching for traces by id Tempo does in fact check every block. First it checks a bloom filter, if that passes then it moves on to checking the actual block for data. That is a fairly intense log and should probably be dropped to "debug" level.
You can check
tempodb_blocklist_length
to determine the length of the current blocklist. There is no magic number for the perfect number of blocks. You can scale up compactors, reduce theircompaction_window
, or just reduce retention to help drive them down if desired.Also, you can pass start/end parameters to that endpoint. This will force Tempo to search fewer blocks.