Temporary files are not getting deleted

[isavitsky]

Bug description

Temporary files networks*.csv.gz are not getting cleaned from the temporary directory, causing disk space to fill up with gigabytes of unused files.

Akvorado version:

akvorado v1.10.2-73-gcf6726c1-dirty
  Built with: go1.21.9 X:loopvar

Steps to reproduce the problem

docker compose up -d

Expected outcome

Temporary files are getting deleted from corresponding docker volumes when not in use.

Current outcome

In a few days' time after starting up docker containers:

sudo find /var/lib/docker -name "networks*.csv.gz" | wc -l
908

The possible cause could be the Akvorado container(s) restarting.

Preliminary patch to solve the problem:

diff --git a/orchestrator/clickhouse/networks.go b/orchestrator/clickhouse/networks.go
index 0fec4892..675a6a6d 100644
--- a/orchestrator/clickhouse/networks.go
+++ b/orchestrator/clickhouse/networks.go
@@ -126,6 +126,7 @@ func (c *Component) networksCSVRefresher() {
                        c.r.Err(err).Msg("cannot create temporary file for networks.csv")
                        return
                }
+               defer os.Remove(tmpfile.Name())

                // Write a gzip dump to the disk
                gzipWriter := gzip.NewWriter(tmpfile)

[vincentbernat]

The file should not be removed as it is the one that is used at the end. It should be automatically removed when rotated and also on shutdown. I don't have the issue on my own installation.

[vincentbernat]

On another install, I have leftover files. I don't know exactly what may trigger that, maybe a slow shutdown. I'll think of a better way to cleanup.

[isavitsky]

I've conducted a few tests on two akvorado VMs (patched and unpatched) and can now confirm that my patch is ineffective against OOM events when containers get killed due to lack of memory.
I think there ought to be another way of dealing with this problem…

[vincentbernat]

I'll add some cleanup on start.