Is it possible that BBolt cannot delete a key when the storage has no space left?

[andrzej-stencel]

Hi,

I'm dealing with an issue in the OpenTelemetry Collector that uses BBolt for the exporter queues (using the File Storage extension). It seems that I've hit a scenario when the exporter wasn't able to remove an item from the queue after the underlying storage device had no space left. The actual operation was bucket.Delete in a transaction: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.93.0/extension/storage/filestorage/client.go#L118. It was called from here: https://github.com/open-telemetry/opentelemetry-collector/blob/v0.92.0/exporter/exporterhelper/internal/persistent_queue.go#L489.

I admit I know next to nothing about BBolt. My thinking is that due to the transactional/versioning nature of BBolt, it might need to reserve a new page to create a new version of the node(s) that don't contain the deleted data. Does this make sense?

[ahrtr]

I do not get what's the exact issue you are running into. Do you mean that bucket.Delete returned an error or the tx.Commit somehow failed?

[andrzej-stencel]

Please provide the detailed error you received.

The error was write /var/lib/storage/otc/exporter_otlphttp__logs: no space left on device.

[ahrtr]

The error was write /var/lib/storage/otc/exporter_otlphttp__logs: no space left on device.

The error was returned by OS (syscall), and it's clear that there is no space left. Each time when committing a bolt transaction, it will try to reuse the existing free pages, and allocate more disk space (pages) only when the existing free pages can't meet the requirement.

So this isn't an issue. Can we close this session?

[andrzej-stencel]

I didn't say it was an issue 😄 I asked a specific question: Is it possible that BBolt fails to remove a key if there's not enough storage available?

I'm assuming from your replies that the answer to the question is "Yes". That was also my initial assumption, but I wanted to confirm this. Thank you!

[ahrtr]

I didn't say it was an issue 😄 I asked a specific question: Is it possible that BBolt fails to remove a key if there's not enough storage available?

Oh, sorry for the confusion. The answer is indeed "YES". The same reason as I mentioned in #685 (reply in thread). Each time when updating (no matter it's writing or deleting), bbolt will always write the updated data into new pages, the previous pages will keep unchanged to ensure that the db file isn't corrupted if the transaction runs into any issue.

Please refer to https://github.com/ahrtr/etcd-issues/blob/master/docs/cncf_storage_tag_etcd.md#storage-boltdb-feature

[andrzej-stencel]

Awesome, thank you so much. The linked information is very useful too. I believe it would be great to add these sections to the BBolt documentation (the README file):

• Storage: BoltDB
• Storage: BoltDB feature
• Perhaps this one too: Storage: tools to analyze BoltDB file