offset table size exceeds 4 bytes & symbol table size exceeds 4 bytes || Thanos Compactor error

[abhishekgoel1410]

Can anybody suggest the root cause & ways to handle these issues. Facing these 2 highlighted errors in thanos compactor component. I did remove the corrupted blocks to some backup s3 bucket and restarted the compactor process but it re-occurred after a few hours only. Due to this memory & storage factors are getting exhausted for storegateway component.

level=error ts=2023-06-06T06:27:00.33013803Z caller=compact.go:495 msg="critical error detected; halting" err="compaction: group 0@12094533570599070506: compact blocks
[/data/compact/0@12094533570599070506/01H13xxxxx8XP
/data/compact/0@12094533570599070506/01H16xxxxPXHD
/data/compact/0@12094533570599070506/01H16xxxxxBT4F
/data/compact/0@12094533570599070506/01H162xxxxx33A0N
/data/compact/0@12094533570599070506/01H1xxxxxx3HAFT
/data/compact/0@12094533570599070506/01H1xxxxxWPJBM]: postings offset table size exceeds 4 bytes: 4355208691"

level=error ts=2023-06-06T03:39:06.1064242Z caller=compact.go:495 msg="critical error detected; halting" err="compaction: group 0@15540937320163341612: compact blocks [/data/compact/0@15540937320163341612/01H19Hxxxxxx49T3 /data/compact/0@15540937320163341612/01H19JxxxxxG9RB
/data/compact/0@15540937320163341612/01H19xxxx0WPB
/data/compact/0@15540937320163341612/01H1xxxxADF03
/data/compact/0@15540937320163341612/01H1xxxxAXCN3
/data/compact/0@15540937320163341612/01H19xxxxx83VG8]: 2 errors: populate block: add series: symbol table size exceeds 4 bytes: 4634597500; symbol table size exceeds 4 bytes: 4634597500"

[sm0nt]

We're seeing the same issue with the symbol table size exceeds 4 bytes error that causes halting. This is keeping the compactor from never being able to get through all of the compactions and always having compactions to do. We recently upgraded to Thanos 0.33 in hopes that maybe it would solve the issue but it has not and we still see it. We have tried deleting all of the compaction data and letting it start from scratch again to see if that helps but we still see the problems.

[ggabijaa]

Hello, maybe you have found better way to solve it? Problem seems to be occurring every few hours. marking block is very inefficient.

[MichaHoffmann]

Do the blocks have all chunk files and index file?

[yeya24]

I don't think there is a good way to deal with this in Thanos without a compactor that could vertically shard your block.
You probably have some very high cardinality labels in your metric and they take a large amount space on your symbol table. I suggest revisiting the high cardinality metrics and use relabel configurations to either modify or drop those metrics

[ggabijaa]

@MichaHoffmann Yes, all blocks have all the chunks and index files

[GiedriusS]

Yeah, this is bad experience :( this needs to be fixed on the Prometheus side: prometheus/prometheus#9709. Thanos has some functionality in place to drop some blocks from compaction if the sum of indices becomes >64GiB but unfortunately it's not a silver bullet :/ I started working a bit on 64bit postings but the len fields also need to be bumped to 8 bytes

My recommendation would be to work on splitting scraping/data sources into multiple ones. For example, have smaller Prometheus+Sidecar pairs and so on.