feat(datasource/docker): add support for architecture-specific image digests

[Churro]

Changes

• Ensures that platform-specific image digests are not updated to manifest digests but to digests of the according platform (amd64, arm64, etc.).

Test repositories:

• https://github.com/Churro/renovate-archdigests
• https://github.com/Churro/renovate-bazel-docker
  • Reproduction repo from Bazel docker digest update uses manifest digest #9786
  • Digest in update corresponds to amd64 as found here

Note: I'm not sure if the changes to the caching key are correct, necessary and effective. Any insights will be appreciated.

Context

• Closes Docker datasource getDigest should return image digest, if current digest is an image digest. #11409
• Closes Bazel docker digest update uses manifest digest #9786

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

How I've tested my work (please tick one)

I have verified these changes via:

• Code inspection only, or
• Newly added/modified unit tests, or
• No unit tests but ran on a real repository, or
• Both unit tests + ran on a real repository

[Churro]

Thanks for the hint on the content-type check, quite a neat trick!

I've now added a HEAD request prior to requesting a full body manifest for the image with currentValue. So if getDigest() gets called with a currentDigest that relates to a manifest list, architecture remains null and the behavior is exactly as it was until now.

[Churro]

@viceice, I think we have a different understanding of how to get the architecture. Based on your comments, I believe you think one could get the architecture also from the digest, which is not the case.

Example

Let's take FROM hashicorp/vault-k8s:0.12.0@sha256:81c09f6d42c2db8121bcd759565ea244cedc759f36a0f090ec7da9de4f7f8fe4:

getManifestResponse(..., 'sha256:81c09f6d42c2db8121bcd759565ea244cedc759f36a0f090ec7da9de4f7f8fe4') returns:

{
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "digest": "sha256:a5d44c0fc90734573afaa286e5138f14d3a9a067493052fa8f7cba950301b09b",
    "size": 4912
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:29291e31a76a7e560b9b7ad3cada56e8c18d50a96cca8a2573e4f4689d7aca77",
      "size": 2813006
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:07afe598755c46e2334a7e7c9378b8b08802604f9975e7293723ec777de2cdd9",
      "size": 5426
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:a1fa93ca544e33240a8a124c0fdf6b429caa27b77244bd124d7aff1894a29dd5",
      "size": 1256
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:f04c54d8e54c64fe12dfd51a02e1168772d8a0cc995b8f73694855546509ee31",
      "size": 19444232
    }
  ]
}

The mapping of digests to architecture is only retrievable by querying the currentValue:

getManifestResponse(..., '0.12.0') returns:

{
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:81c09f6d42c2db8121bcd759565ea244cedc759f36a0f090ec7da9de4f7f8fe4",
      "size": 1156,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:2ffd023131d0184923d237f18f4ad5009b8afe71d83ded7b4c272db204ab7b8a",
      "size": 1156,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:295b5db1283633daf8d82f0b794bd543118508a5568c90b055d7a48ad6afcfd1",
      "size": 1156,
      "platform": {
        "architecture": "386",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:197c58a9c30faf97f16412d63e333a0adffed856b06994759899d070f28db51e",
      "size": 1156,
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v6"
      }
    }
  ]
}

Architecture-Lookup Flow

1. Check if the current digest relates to a concrete architecture or a manifest list. If manifest list, skip the architecture lookup since the digest doesn't map to an architecture anyway (and proceed with regular manifest list updates):
   

   renovate/lib/modules/datasource/docker/index.ts

   Lines 845 to 855 in 1e67faf

   	let manifestResponse = await this.getManifestResponse(
   	registryHost,
   	dockerRepository,
   	currentDigest,
   	'head'
   	);
   	if (
   	manifestResponse?.headers['content-type'] === MediaType.manifestV2 ||
   	manifestResponse?.headers['content-type'] === MediaType.ociManifestV1
   	) {

2. Fetch the full manifest list based on currentValue, since currentDigest is supposed to be included there:
   

   renovate/lib/modules/datasource/docker/index.ts

   Lines 856 to 860 in 1e67faf

   	manifestResponse = await this.getManifestResponse(
   	registryHost,
   	dockerRepository,
   	currentValue
   	);

3. If the full manifest list includes currentDigest, it gets mapped to the architecture, otherwise architecture remains null (and manifest list updates will be done as until now):
   

   renovate/lib/modules/datasource/docker/index.ts

   Lines 874 to 876 in 1e67faf

   	if (manifest.digest === currentDigest) {
   	architecture =
   	(manifest.platform['architecture'] as string) ?? null;

I hope this helps to clarify things a bit 😊

[viceice]

@viceice, I think we have a different understanding of how to get the architecture. Based on your comments, I believe you think one could get the architecture also from the digest, which is not the case.

Example

Let's take FROM hashicorp/vault-k8s:0.12.0@sha256:81c09f6d42c2db8121bcd759565ea244cedc759f36a0f090ec7da9de4f7f8fe4:

getManifestResponse(..., 'sha256:81c09f6d42c2db8121bcd759565ea244cedc759f36a0f090ec7da9de4f7f8fe4') returns:

{
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "digest": "sha256:a5d44c0fc90734573afaa286e5138f14d3a9a067493052fa8f7cba950301b09b",
    "size": 4912
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:29291e31a76a7e560b9b7ad3cada56e8c18d50a96cca8a2573e4f4689d7aca77",
      "size": 2813006
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:07afe598755c46e2334a7e7c9378b8b08802604f9975e7293723ec777de2cdd9",
      "size": 5426
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:a1fa93ca544e33240a8a124c0fdf6b429caa27b77244bd124d7aff1894a29dd5",
      "size": 1256
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:f04c54d8e54c64fe12dfd51a02e1168772d8a0cc995b8f73694855546509ee31",
      "size": 19444232
    }
  ]
}

The mapping of digests to architecture is only retrievable by querying the currentValue:

getManifestResponse(..., '0.12.0') returns:

{
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:81c09f6d42c2db8121bcd759565ea244cedc759f36a0f090ec7da9de4f7f8fe4",
      "size": 1156,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:2ffd023131d0184923d237f18f4ad5009b8afe71d83ded7b4c272db204ab7b8a",
      "size": 1156,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:295b5db1283633daf8d82f0b794bd543118508a5568c90b055d7a48ad6afcfd1",
      "size": 1156,
      "platform": {
        "architecture": "386",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:197c58a9c30faf97f16412d63e333a0adffed856b06994759899d070f28db51e",
      "size": 1156,
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v6"
      }
    }
  ]
}

Architecture-Lookup Flow

1. Check if the current digest relates to a concrete architecture or a manifest list. If manifest list, skip the architecture lookup since the digest doesn't map to an architecture anyway (and proceed with regular manifest list updates):
   

   renovate/lib/modules/datasource/docker/index.ts

   Lines 845 to 855 in 1e67faf

   	let manifestResponse = await this.getManifestResponse(
   	registryHost,
   	dockerRepository,
   	currentDigest,
   	'head'
   	);
   	if (
   	manifestResponse?.headers['content-type'] === MediaType.manifestV2 ||
   	manifestResponse?.headers['content-type'] === MediaType.ociManifestV1
   	) {

2. Fetch the full manifest list based on currentValue, since currentDigest is supposed to be included there:
   

   renovate/lib/modules/datasource/docker/index.ts

   Lines 856 to 860 in 1e67faf

   	manifestResponse = await this.getManifestResponse(
   	registryHost,
   	dockerRepository,
   	currentValue
   	);

3. If the full manifest list includes currentDigest, it gets mapped to the architecture, otherwise architecture remains null (and manifest list updates will be done as until now):
   

   renovate/lib/modules/datasource/docker/index.ts

   Lines 874 to 876 in 1e67faf

   	if (manifest.digest === currentDigest) {
   	architecture =
   	(manifest.platform['architecture'] as string) ?? null;

I hope this helps to clarify things a bit 😊

as i sad in comment, current value probably no longer point to the original manifest list, docker tags are floating.

so algorithm should be:

• fetch head of current digest
• if it's manifest list do existing flow
• if it's a image, fetch image manifest, then config manifest to get architecture

[Churro]

Thanks for the info regarding image config. I've now adapted the implementation to that. Moreover:

• moved architecture fetching to its own method to enable caching
• replicated the flow of of getLabels() with try / catch, getConfigDigest() etc. to extract the architecture
• added getImageConfig() to fetch and cache image config blobs

In a follow-up PR (or this one, depending on your preference), getLabels() could be refactored to use getImageConfig().

[viceice]

please validate latest changes against real repo, should test all scenarios

[Churro]

I have now re-run the latest version on the two test repositories: working as expected.

• https://github.com/Churro/renovate-archdigests

  • Tested even more combinations than in the initial PR:
    • some random digest (expecting answer 404 from registry) -> fetch manifest digest ✅
    • tag + amd64 digest provided -> fetch amd64 digest ✅
    • amd64 digest provided only -> fetch amd64 digest ✅
    • tag + manifest digest provided -> fetch manifest digest ✅
    • tag only -> no digest ✅
  • The expected amd64 digest was correctly retrieved for vault-k8s:0.17.0 and distroless/java17

• https://github.com/Churro/renovate-bazel-docker

  • Digest in update corresponds to amd64 as found here ✅

[viceice]

i expect a pin PR would also use the manifest list digest, it's same case as no current digest?

[Churro]

My bad, I was missing the docker:pinDigests preset. With that enabled, the manifest list digest is added: https://github.com/Churro/renovate-archdigests/pull/5/files

[renovate-release]

🎉 This PR is included in version 32.163.0 🎉

The release is available on:

• GitHub release
• 32.163.0

Your semantic-release bot 📦🚀