Docker containers with hash-only image cannot be updated

[piksel]

Recently, containers without an image tag has been encountered by watchtower users:

• Containers created with docker compose cannot be updated #1019
• Watchtower causes a cron crash on Ubuntu when checking for updates #1026
• Watchtower crashes when containers with hash-only images are encountered #1037
• Non-standard image refs does not map to expected repositories in config.json #796 (comment)

Since the containers' configurations does not contain the necessary information for updating, watchtower cannot check for updates or recreate them.
Since #1027 (available in watchtower:latest-dev) it should at least no longer crash, but just skip said containers.

One source of these containers seem to be docker compose (with no dash), but it's unclear whether they can be created in some other fashion (you could of course just manually specify the sha256:xxx hash in docker run, but then, what would you expect?)

To verify that containers does not have any named image, you can run docker ps:

$ docker ps
CONTAINER ID   IMAGE             COMMAND                  CREATED          STATUS          PORTS                                       NAMES
fb0c2a735c38   4cdc5dd7eaad      "/docker-entrypoint.…"   18 minutes ago   Up 20 seconds   80/tcp         

In the second column (IMAGE), instead of the image tag, it shows just a short image id hash.

I made a PoC attempting to retrieve the image tag based on the ID, and it is possible to do so under some conditions, but it's highly volatile.
If this is a permanent change in future Docker versions, it may render watchtower unusable... 😩

Add a comment if you know anything more about the source of these containers!

[piksel]

Thanks to @amarcu5 filing docker/compose-cli#1983, this should be solved (merged in docker/compose-cli#1997) in upcoming releases.

Until then, use the "vanilla" docker-compose command to enable watchtower support.
Note: If you have enabled Docker Compose V2 in Docker Desktop, use docker-compose disable-v2 to disable it.

[tiritibambix]

I'm sorry, I'm a real noob.

How can I solve this with my RPi4 ? What's the "vanilla" docker compose ?

I have the exact same stack running on an amd64 in portainer and it works great

version: "3"
services:
    watchtower:
        image: containrrr/watchtower
        container_name: watchtower
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock
        environment:
            - TZ=Europe/Paris
            - WATCHTOWER_MONITOR_ONLY=true
            - WATCHTOWER_SCHEDULE=0 0 10 ? * SAT
            - WATCHTOWER_CLEANUP=true
            - WATCHTOWER_NOTIFICATIONS=email
            - WATCHTOWER_NOTIFICATION_EMAIL_FROM=xxx
            - WATCHTOWER_NOTIFICATION_EMAIL_TO=xxx
            - WATCHTOWER_NOTIFICATION_EMAIL_SERVER=xxx
            - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD=xxx
            - WATCHTOWER_NOTIFICATION_EMAIL_SUBJECTTAG=xxx
            - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_USER=xxx
            - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT=xxx
        restart: unless-stopped

[piksel]

It should only be a matter of running docker-compose disable-v2, and you can verify that it's using v1 with docker-compose -v:

$ docker-compose -v
Docker Compose version v2.0.0

$ docker-compose disable-v2

$ docker-compose -v
docker-compose version 1.29.2, build 5becea4c

[tiritibambix]

It should only be a matter of running docker-compose disable-v2, and you can verify that it's using v1 with docker-compose

Thank you for your answer.
This is my docker compose version : docker-compose version 1.28.4, build cabd5cfb

Shouldn't this be working ?

[RoggerFabri]

It should only be a matter of running docker-compose disable-v2, and you can verify that it's using v1 with docker-compose

Thank you for your answer. This is my docker compose version : docker-compose version 1.28.4, build cabd5cfb

Shouldn't this be working ?

I guess if you're using Portainer on the latest version, it's using docker-compose v2 internally. Can someone confirm it?

[niawag]

Yes, I can confirm it: portainer/portainer#5799 (comment)

[tiritibambix]

So is there a workaround for people using portainer 2.9.1 like me ?

[niawag]

AFAIK not using portainer to deploy your stack(s) is the only workaround ATM. That is what I'm doing, I deploy the stack with "docker-compose up -d" on the CLI and then use Portainer to manage the created stack(s). It gives you almost all Portainer functions except stack creation.

[tiritibambix]

AFAIK not using portainer to deploy your stack(s) is the only workaround ATM. That is what I'm doing, I deploy the stack with "docker-compose up -d" on the CLI and then use Portainer to manage the created stack(s). It gives you almost all Portainer functions except stack creation.

Thanks for the suggestion. I did that, but I face the exact same problem o_O

[niawag]

If you're having the same problem from the CLI it must mean that you're using docker compose (as portainer is) and not docker-compose (mind the space and the -). You should try with docker-compose from https://github.com/linuxserver/docker-docker-compose/releases and install it like that :

sudo curl -L --fail https://github.com/linuxserver/docker-docker-compose/releases/download/1.27.4-ls26/docker-compose-armhf -o /usr/local/bin/docker-compose

Mind the release number and the plateform amd64/arm64 or armhf, depending on your hardware

sudo chmod +x /usr/local/bin/docker-compose

It may also be a problem with the docker-compose version, here is the one working for me: docker-compose version 1.27.4, build 40524192

[tiritibambix]

Thanks for your answer. I do use docker-compose (version 1.28.4, build cabd5cfb).

Would a more recent version break things up ?

[niawag]

I'm not sure but I think it should be ok, the problem is with docker compose from docker CLI, not docker-compose...

[mastan30]

Is there a solution for this yet, can someone please help me with this, I am facing the same issue.

[Smiggel]

A work-around is editing the container. Set the image container url in the config, and save it. That will solve it temporarily.

I personally use portainer to deploy images, etc. and update the container image.

Just replace the hash with image name. For example: linuxserver/portainer

[mastan30]

@Smiggel Thanks for the response. Can you please let me know where can we find the config for container?

[Smiggel]

@Smiggel Thanks for the response. Can you please let me know where can we find the config for container?

I use Portainer. In Portainer I go to "Containers" in the left menu. Then I click on an image. You see several buttons on top of the page and press "Duplicate/Edit". Under "Image" you can set the "Image configuration". Behind "Docker.io" just type for example "containrrr/watchtower" or what ever you need there.

Just make sure you replace the sha256 hash there. When you do that and update the image, Watchtower will be able to find new versions.

[mastan30]

@Smiggel, thank you so much this makes sense a lot.

[NeoTrace82]

@Smiggel Thanks for the response. Can you please let me know where can we find the config for container?

I use Portainer. In Portainer I go to "Containers" in the left menu. Then I click on an image. You see several buttons on top of the page and press "Duplicate/Edit". Under "Image" you can set the "Image configuration". Behind "Docker.io" just type for example "containrrr/watchtower" or what ever you need there.

Just make sure you replace the sha256 hash there. When you do that and update the image, Watchtower will be able to find new versions.

this works but as far as I understand this is only a workaround as soon as watchtower will update the container or you update the container it will be set back to the hash?

[Smiggel]

@Smiggel Thanks for the response. Can you please let me know where can we find the config for container?

I use Portainer. In Portainer I go to "Containers" in the left menu. Then I click on an image. You see several buttons on top of the page and press "Duplicate/Edit". Under "Image" you can set the "Image configuration". Behind "Docker.io" just type for example "containrrr/watchtower" or what ever you need there.
Just make sure you replace the sha256 hash there. When you do that and update the image, Watchtower will be able to find new versions.

this works but as far as I understand this is only a workaround as soon as watchtower will update the container or you update the container it will be set back to the hash?

I only let watchtower send me a message and not update the container. So, I update it when I have time and just rebuild the container with pulling the last image.

[NeoTrace82]

guess that's the best solution for now, thanks

[Trony71]

This doesn't happen to me when deploying a stack in Portainer with version "3.3". It keeps the chosen image names and watchtower is able to update the containers

[mastan30]

Looks like the latest portainer is using docker-compose instead of docker compose @Trony71 . That will be very helpful actually.

[Guido83]

This doesn't happen to me when deploying a stack in Portainer with version "3.3". It keeps the chosen image names and watchtower is able to update the containers

It does not work for every stack I have tested it with

[MelleD]

I use version: '3.3' and when I update the watchtower stack initially, the image does not get a hash. As soon as I change the stack, e.g. update env variable, the image gets the sha hash as a name, although the image is not downloaded again. Anyone have an idea why?

Unfortunately, no scan is performed even with the correct image name, I'm not sure yet whether it is related to this problem

[piksel]

@mbourd25 I don't think this is related to hash-only images. Please create a new Discussion for this.

That being said, the current hash is sha256:a66d4579ef01e017d7c7af3da9d462388fc824ade4c6be1efe7521d34d314349 and the hash for the image in dockerhub is sha256:a66d4579ef01e017d7c7af3da9d462388fc824ade4c6be1efe7521d34d314349, so there is no update. Also, when looking at dockerhub, it seems like the latest tag hasn't been updated for a month

[davtur19]

As a workaround, could you put a label with the container image?
something like that:

labels:
    "com.centurylinklabs.watchtower.image": "containrrr/watchtower:latest"

[Stdocdure]

time="2022-02-01T18:00:08Z" level=info msg="Watchtower 1.4.0"
time="2022-02-01T18:00:08Z" level=info msg="Using no notifications"
time="2022-02-01T18:00:08Z" level=info msg="Only checking containers with name "qing-dragon" or "aliyun--web" or "qbittorrent" or "epic--game" or "watchtower""
time="2022-02-01T18:00:08Z" level=info msg="Running a one time update."
time="2022-02-01T18:00:09Z" level=warning msg="Failed to retrieve container image info: Error: No such image: sha256:6fe21a5ad3ab585034e1dab71e099313098b9acdfd7e7988811e91bc99ad8448"
time="2022-02-01T18:00:09Z" level=warning msg="Failed to retrieve container image info: Error: No such image: sha256:dd78a816fb764ac6f44bd30d599e3605ebce50bc987e627573ab495092561a69"
time="2022-02-01T18:00:24Z" level=info msg="Session done" Failed=0 Scanned=5 Updated=0 notify=no
time="2022-02-01T18:00:24Z" level=info msg="Waiting for the notification goroutine to finish" notify=no

[devedse]

Any updates on fixing this? I'm also running into this issue with a stack running on portainer which refers to some ghcr.io docker images.

[davtur19]

For anyone who uses portainers, this solved my problem:
portainer/portainer#5799 (comment)

Now the images have the correct names again and no longer hashes

[piksel]

@devedse This is not something that can be fixed by watchtower. The information necessary for watchtower to function (what the name of the image is) has been omitted, so we have no idea what to check for updates for.

[devedse]

@piksel , understandable that if the image name is missing watchtower can't pull it.

As a workaround though, could we think of any of the solutions mentioned above?
For example, adding a custom label with the actual image name to the docker-compose file?

Or do you have any idea why this fix is supposed to work?:

For anyone who uses portainers, this solved my problem: portainer/portainer#5799 (comment)

Now the images have the correct names again and no longer hashes

Edit: I tried this, but for me it doesn't seem to work:

[davtur19]

Or do you have any idea why this fix is supposed to work?

I have no idea why the fix works, I suppose there was a problem in docker-compose for ARM, and deleting the file leads to the rebuild/copy/download of the new docker-compose without the bug.

For me the problem is solved and I'm on ARM64.
You could try to switch to the 64bit version of your OS, if you are using a 32bit version.

[devedse]

@davtur19 , I'm using 64bit. Could you show me your docker inspect output?

Edit, I actually just found out that docker inspect always shows a hash for the "Image" field. However it seems that in docker ps it now shows an image name correctly.

[davtur19]

@davtur19 , I'm using 64bit. Could you show me your docker I spect output?

Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:44:28 2021
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:05 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[devedse]

Apparently I was looking at the wrong image field. The one containing the Image name seems to be in .Config.Image:

docker inspect CoolImage | jq '.[0].Config.Image'

This now shows: devedse/coolimage:latest

Edit:
You can use this command to show all image names:

docker inspect $(docker ps -aq) | jq '.[].Config.Image'

This does display image names now correctly for stacks that I created.

[piksel]

Yeah, Config.Image is indeed the field that watchtower uses to find out the image name. If that's populated with a name, it should work correctly.

[Fizcko]

In my case, i use watchtower with portainer (2.11.1).

The image won't update with watchtower if you don't specify a tag in the docker-compose definition.

Just add :latest tag in the image name and this will fix the problem.

[piksel]

@Fizcko, it might be the case that it pins the image to a version if you omit the tag. Or it was using a version of docker-compose v2 before that used the image hashes, and now it doesn't?

In any case, I added an additional check for image hashes to give a better error message for pinned images in #1277. Now it will instead inform that the image is pinned and as such cannot be updated by watchtower. That is a lot more helpful than errors about library/sha256 not existing.

[Fizcko]

@piksel I don't really know. Now it works with the tag.
Thanks for add a better error message.