Issue with pulling an image from the microk8s registry

[nimi13]

Summary

Image cannot pulled from the local registry

Reproduction Steps

When I disable the registry, destroy-storage and enable the registry it will work but after few upgrades it will failed again, can be after one upgrade and can be much more

Introspection Report

inspection-report-20240410_115206.tar.gz

Are you interested in contributing with a fix?

Yes if I can

The issue in detailed

I am using few microk8s 1.28 and 1.29 on Ubuntu 20.04 and 22.04, I got strange behavior from one of the pods, it is always the same pod (the admin) and we got 20 pods, sometime after system upgrade when I change the pod container version/tag it failed to start with status ImagePullBackOff.

When I run describe on that pod I see kubelet Back-off pulling image "localhost:32000/admin:8.3.1989"

Try to pull the image from the local registry will failed
The image exist on the server
When pushed it to the local registry it seems that all the layer are already exist in the registry
But it will not pull even after the push
Checking with curl and I can see the image and the tag in the registry

microk8s registry is set to 40G and the default-storage directory size is 24G

Create the registry using the command: microk8s enable registry:size=40Gi

Any help will be much appreciated

In detailed commands and output:

Try to pull the image from the local registry will failed

docker pull localhost:32000/admin:8.3.1989
Error response from daemon: manifest for localhost:32000/admin:8.3.1989 not found: manifest unknown: manifest unknown

It exist on the server
docker images | grep admin
localhost:32000/admin 8.3.1989 2890fcb56766 7 weeks ago 500MB

When push it it seems that all the layer are already exist

docker push localhost:32000/admin:8.3.1989
The push refers to repository [localhost:32000/admin]
94a14ce94eef: Layer already exists 
b0aed591e1f6: Layer already exists 
06855bd23f0b: Layer already exists 
c2fc9fe8dc06: Layer already exists 
6c804dddc7ac: Layer already exists 
5bc86e618a6c: Layer already exists 
425d004af7ba: Layer already exists 
d101c9453715: Layer already exists 
8.3.1989: digest: sha256:6df1160a2880c5c69af2d949a9c06df8fb6c7de4779427bef528a1ed9a825987 size: 1995

But it will not pull even after the push

docker pull localhost:32000/admin:8.3.1989
Error response from daemon: manifest for localhost:32000/admin:8.3.1989 not found: manifest unknown: manifest unknown

looking for the container tag it exist on the registry

curl -sSL "localhost:32000/v2/admin/tags/list"
{"name":"admin","tags":["8.3.1989","8.2.1617","8.4.5138","8.4.5160","8.4.5169","8.4.5178","8.4.5173","8.4.5144","8.4.5168"]}