Version checker seems to pick the wrong latest version on multi-arch images when using SHA

[tfadeyi]

What happened?
When I was running version-checker on KIND (amd64), the latest image version reported for the following container docker.io/ealen/echo-server:latest was 3f390e3cb38a1bb7a5169b27ae56285335ffede3cf334156e4d826173ce54ae5, for arm/v6, instead of the tag for amd64 or the tag used for the manifests collection sha256:3419f7cf35d10c9096ce359df762283ccfa7af12095f9be51542566bef8e9034 which is the actual ID assigned to the container when using :latest.

    Image:          docker.io/ealen/echo-server:latest
    Image ID:       docker.io/ealen/echo-server@sha256:3419f7cf35d10c9096ce359df762283ccfa7af12095f9be51542566bef8e9034

Logs
Echo Image Manifests and digests:

docker buildx imagetools inspect docker.io/ealen/echo-server:latest
Name:      docker.io/ealen/echo-server:latest
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:3419f7cf35d10c9096ce359df762283ccfa7af12095f9be51542566bef8e9034
           
Manifests: 
  Name:      docker.io/ealen/echo-server:latest@sha256:397996a29090fde2eeb8866d972b683fd64afadbe7161d0e64c53350b4a83fd7
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64
             
  Name:      docker.io/ealen/echo-server:latest@sha256:3f390e3cb38a1bb7a5169b27ae56285335ffede3cf334156e4d826173ce54ae5
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v6
             
  Name:      docker.io/ealen/echo-server:latest@sha256:80e5ed2796aa8242b965c80678bc512ae2396c58f13614be204e7e70ce4529b6
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v7
             
  Name:      docker.io/ealen/echo-server:latest@sha256:1865bce901f1565d6262a730f0ae6f562a86e4cca04f65fea6cfe2c30896ec7f
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64
             
  Name:      docker.io/ealen/echo-server:latest@sha256:1f183d751287c846a8d1c5c21e8c635b175141b23af81df067631d8cd9906787
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/386

vc metrics:

 curl http://localhost:8083/metrics
# HELP version_checker_is_latest_version Where the container in use is using the latest upstream registry version
# TYPE version_checker_is_latest_version gauge
version_checker_is_latest_version{container="echo-2",current_version="sha256:3419f7cf35d10c9096ce359df762283ccfa7af12095f9be51542566bef8e9034",image="docker.io/ealen/echo-server",latest_version="latest@sha256:3f390e3cb38a1bb7a5169b27ae56285335ffede3cf334156e4d826173ce54ae5",namespace="default",pod="echo-2-79fbc9bc76-dvggs"} 0
version_checker_is_latest_version{container="version-checker",current_version="v0.2.1",image="quay.io/jetstack/version-checker",latest_version="v0.2.1",namespace="version-checker",pod="version-checker-5677485967-b7czz"} 1

The targeted pod:

Name:                      echo-2-6dfbb54cfb-fxkrq
Namespace:                 default
Priority:                  0
Node:                      kind-control-plane/172.17.0.2
Start Time:                Tue, 10 Nov 2020 11:20:23 +0000
Labels:                    app=echo
                           pod-template-hash=6dfbb54cfb
Annotations:               enable.version-checker.io/echo-2: true
Status:                    Terminating (lasts <invalid>)
Termination Grace Period:  30s
IP:                        
IPs:
  IP:           
Controlled By:  ReplicaSet/echo-2-6dfbb54cfb
Containers:
  echo-2:
    Container ID:   containerd://45bc033cb1b6fef9a6d46633a7c99f5e22541e218a410e71f083541dc8a37804
    Image:          docker.io/ealen/echo-server:latest
    Image ID:       docker.io/ealen/echo-server@sha256:3419f7cf35d10c9096ce359df762283ccfa7af12095f9be51542566bef8e9034
    Port:           8081/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 10 Nov 2020 11:20:25 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>

[JoshVanL]

This is a tricky one. There is a couple things going on here.

First thing I noticed was that we are not handling paging properly in most/all our clients. I will create another issue for that.

Next, version-checker is doing a bad job of being architecture aware, leading to the issue above where it picks the wrong architecture as the latest when they both have the same timestamp. To fix this correctly, we should create a node controller. This will expose a map[string]struct{OS, Arch} which should be passed down into our searcher. The searcher can then be aware of which architecture and OS to be filtering for, when getting the latest. An override option may be appropriate here, but let's leave that for later. We should update the metrics to include the os and arch as labels.

The next thing to fix is the clients which do not expose the arch and OS of image tags available. In these cases, we will likely need to swap out the native API for our first class supported clients, in favour of using the fallback generic docker API, which does support getting this information. We will likely need to do a bit of a dance in terms of permissions, in that we get a valid access token/username/password using the native client first, then passing them on correctly to the generic docker client. We shouldn't expect any config API changes.

Lastly, we need to fix the issue of the "composite hashes". When using an image in Kubernetes which has the tag of "", or "latest", and the selected tag that docker pulls is a multi-architecture image, the reported image SHA as part of the "imageID", will be the "composite hash". This means that, the reported image hash will be the hash of the full json payload (denoted in the HTTP header as DOCKER-IMAGE-DIGEST), rather than the particular image digest. To fix this, we should include a "CompositeHash" field as part of the "ImageTag" struct which includes this value, along with the image SHA. The searcher then need to be aware of whether it should compare against the raw SHA value, or the CompositeHash value. Depending on the client, it may never report a composite hash in these cases, and should be tested. If this is the case, we can save some KB and make the CompositeHash field a *string, making the searcher fall back to SHA when nil.