cacache cleanup doesn't return diskspace

[gg-kialo]

What would you like help with?

I think I found a bug

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

GitHub, renovate-ce:6.11.0 (renovate 37.170.0)

Please tell us more about your question or problem

Renovate's cacache has ever-increasing disk usage.
The cleanup (https://github.com/renovatebot/renovate/blob/main/lib/util/cache/package/file.ts#L77) is calling cacache.rm.entry(cacheFileName, cachedItem.key), which (from cacache docs):

Removes the index entry for key. Content will still be accessible if requested directly by content address (get.stream.byDigest).
...
To remove the content itself (which might still be used by other entries), use rm.content. Or, to safely vacuum any unused content, use verify.

Indeed, if from the renovate container I run await cacache.verify("RENOVATE_BASE_DIR/cache/renovate/renovate-cache-v1/") then disk space used by expired cache contents is reclaimed.

Logs (if relevant)

Logs

> await cacache.verify("/var/cache/renovate/cache/renovate/renovate-cache-v1/")
{
  startTime: 2024-03-28T14:07:28.214Z,
  runTime: {
    markStartTime: 1,
    fixPerms: 0,
    garbageCollect: 642216,
    rebuildIndex: 29559,
    cleanTmp: 16,
    writeVerifile: 59,
    markEndTime: 0,
    total: 671851
  },
  verifiedContent: 649,
  reclaimedCount: 926253,
  reclaimedSize: 6215064669,
  badContentCount: 0,
  keptSize: 6488665,
  missingContent: 0,
  rejectedEntries: 0,
  totalEntries: 649,
  endTime: 2024-03-28T14:18:40.065Z
}

[viceice]

@secustor @rarkins it seems we've a bug 🐛

[gg-kialo]

@viceice Would you like me to open a PR for this? It's a simple call to cacache.verify, which is usually fast, but the first time it's called on an exiting cache can take a while (11 minutes in my case). If aborted then it should pick up where it was after the next run.

[viceice]

sure

[gg-kialo]

Cool, I've opened #28275

[rarkins]

@viceice @secustor @gg-kialo should we be switching to rm.content in addition to instead of rm.entry?

[viceice]

sounds good to me

[rarkins]

@gg-kialo did you do any research into this? I'm guessing that verify is inefficient because it has to check that no entry is referencing the digest. So longer term we're better to not need verification

[gg-kialo]

Observing my renovate instance, a call to verify on a recently verified cache takes about 20-30s and the cleanup using just rm.entry takes 10-15s. I don't expect adding rm.content to that would increase the time by much; very few items are actually removed each iteration so it shouldn't add much more work over rm.entry.

Even though it's slower, I wonder if running an occasional verify would still be useful to catch things that are missed somehow (bugs, containers being killed, users doing manual cleanup) to keep the caching from continuing to grow.