Watchtower is unable to update itself. No access to repository.

[shanduur]

Describe the bug

Watchtower is unable to update itself. No access to repository.

To Reproduce

Expected behavior

Screenshots

time="2022-04-21T15:42:04Z" level=warning msg="Reason: registry responded to head request with \"401 Unauthorized\", auth: \"Bearer realm=\\\"https://auth.docker.io/token\\\",service=\\\"registry.docker.io\\\",scope=\\\"repository:library/sha256:pull\\\",error=\\\"insufficient_scope\\\"\"" container=/watchtower_watchtower_1 image="sha256:ac708a0cd96706c6ba4bc0bed52e9113b21dd789322ea69561c85e3d9838abc7"
time="2022-04-21T15:42:05Z" level=info msg="Unable to update container \"/watchtower_watchtower_1\": Error response from daemon: pull access denied for sha256, repository does not exist or may require 'docker login': denied: requested access to the resource is denied. Proceeding to next."

Environment

Client: Docker Engine - Community
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:47:02 2022
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:45:51 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Logs from running watchtower with the --debug option

Additional context

[github-actions]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[simskij]

repository:library/sha256 is very suspicious. How did you deploy watchtower?

[shanduur]

Through Portainer/Docker Compose:

version: "3"

services:
  watchtower:
    image: containrrr/watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

This update was failed on all my containers, and rendered all of them down, with almost the same error message. Since then, it works perfectly fine, and I was not able to reproduce this bug once again.

I think it is worth noting, that the Docker is deployed on Oracle free tier machine (4 ARM cores, 24GB of RAM), and running under Oracle Linux:

[opc@oracle ~]$ uname -a
Linux oracle 5.4.17-2136.302.6.1.el8uek.aarch64 #2 SMP Tue Dec 14 12:00:21 PST 2021 aarch64 aarch64 aarch64 GNU/Linux
[opc@oracle ~]$ neofetch --off
opc@oracle
----------
OS: Oracle Linux Server 8.5 aarch64
Host: KVM Virtual Machine virt-4.2
Kernel: 5.4.17-2136.302.6.1.el8uek.aarch64
Uptime: 99 days, 13 hours, 32 mins
Packages: 847 (rpm)
Shell: bash 4.4.20
Resolution: 1024x768
Terminal: /dev/pts/0
CPU: (4)
GPU: 00:01.0 Red Hat, Inc. Virtio GPU
Memory: 1828MiB / 23239MiB

[shanduur]

Here is full log after running it with debug flag, through compose:

version: "3"

services:
  watchtower:
    image: containrrr/watchtower
    command: --debug --cleanup --run-once
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

_watchtower_watchtower_1_logs.txt

[piksel]

/drone_server_1 is using a pinned version and hence cannot be updated. This is a duplicate of #1050, which contains some solutions for Portainer. They have fixed it in a development version, but I don't think it has been released to stable yet.
As per #1277, the error message will be much clearer, but it still cannot update containers that don't specify what image they are using.

[shanduur]

@piksel okay, the pinned images - I understand. But What do you mean by:

it still cannot update containers that don't specify what image they are using

From what I understood from the initial issue description, it untagged image of itself, and failed to relaunch - as it was stopped after this failure message.

[piksel]

The two lines in the initial log only said that it was skipping it:

Unable to update container "/watchtower_watchtower_1". [...] Proceeding to next.

If it actually did an update, but failed to start it, the error would look very different. So that is probably somewhere else in the logs.