New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Garbage collection does not delete data from cache, preventing reupload of deleted image #4269
Comments
I also encountered this problem, how to solve it? |
roblabla
added a commit
to roblabla/harbor-helm
that referenced
this issue
Feb 15, 2024
There are bugs in the registry with regards to cache invalidation when deleting an image that can lead to some really problematic issues. See distribution/distribution#4269. To work around that problem, add a new field to allow disabling redis in the registry.
roblabla
added a commit
to roblabla/harbor-helm
that referenced
this issue
Feb 15, 2024
There are bugs in the registry with regards to cache invalidation when deleting an image that can lead to some really problematic issues. See distribution/distribution#4269. To work around that problem, add a new field to allow disabling redis in the registry. Signed-off-by: roblabla <unfiltered@roblab.la>
I think this might be related to PR #3323 |
roblabla
added a commit
to roblabla/harbor-helm
that referenced
this issue
Feb 22, 2024
There are bugs in the registry with regards to cache invalidation when deleting an image that can lead to some really problematic issues. See distribution/distribution#4269. To work around that problem, add a new field to allow disabling redis in the registry. Signed-off-by: roblabla <unfiltered@roblab.la>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Currently, GC does not delete data from the redis cache. This can result in some very weird scenario where trying to reupload an image that was deleted will appear to succeed (the registry thinks it already has the files we're uploading, and so returns 201), but then trying to use it fails with a MANIFEST_UNKNOWN error as the file is not actually present.
Reproduce
We'll need a registry configured to use a redis cache. I used a filesystem storage with redis configuration for simplicity, but any backend storage will do, the problem stems from the cache.
Here's my configuration
here's the reproducer:
Expected behavior
I expect pushing the image to actually work. Either pushing should peer through the cache and check whether it is up-to-date, or garbage-collection should remove the keys from the cache.
registry version
Tested both on 2.8.3 and main branch (9b3eac8)
./bin/registry github.com/distribution/distribution/v3 v3.0.0-alpha.1.m+unknown
Additional Info
This was discovered via harbor's retention policy. Harbor uses distribution as the underlying backend for its storage. When it applies its retention policy, it deletes the manifests, and automatically runs garbage-collection to free up space, triggering this bug.
The text was updated successfully, but these errors were encountered: