docker: Replacement causes existing dependencies to use wrong digest #20304

uhthomas · 2023-02-08T22:49:02Z

How are you running Renovate?

Mend Renovate hosted app on github.com

If you're self-hosting Renovate, tell us what version of Renovate you run.

34.125.1

If you're self-hosting Renovate, select which platform you are using.

Bitbucket Server

If you're self-hosting Renovate, tell us what version of the platform you run.

N/A

Was this something which used to work for you, and then stopped?

I never saw this working

Describe the bug

We're replacing an old image with a new one, but the digest for the new image is different.

This is fine and the replacement PRs look fine too, but Renovate seems to assume the replacements are also aliases.

This means that Renovate is now opening PRs for the old image with the digest for the new image...

To help illustrate this, here are our two images:

legacy, digest: 44dea30fe85df1f3d554cee602fa69d553da7ab746c8c9f20defb1f1fc3e03b8
current, digest: e742813303ad00bbe1409ea01eb909cb67818a8fbaae8c07df79036a7dea3e1d

Renovate is opening PRs which look like:

-FROM legacy@sha256:44dea30fe85df1f3d554cee602fa69d553da7ab746c8c9f20defb1f1fc3e03b8
+FROM legacy@sha256:e742813303ad00bbe1409ea01eb909cb67818a8fbaae8c07df79036a7dea3e1d

This is wrong and unexpected. I've read these issues, but they're not completely the same:

Relevant debug logs

Logs

Copy/paste the relevant log(s) here, between the starting and ending backticks

Have you created a minimal reproduction repository?

No reproduction repository

The text was updated successfully, but these errors were encountered:

uhthomas · 2023-02-08T22:59:00Z

Please see this snippet from the logs, with relevant modifications for privacy.

"deps": [
        {
                "autoReplaceStringTemplate": "example.com/legacy:{{#if newValue}}{{newValue}}{{/if}}@{{#if newDigest}}{{newDigest}}{{/if}}",
                "currentDigest": "sha256:46cea30fe85df1f3d554cee602fa69d553da7ab746c8c9f20defb1f1fc3e03b8",
                "currentValue": "some-tag",
                "currentVersion": "some-tag",
                "datasource": "docker",
                "depIndex": 0,
                "depName": "example.com/legacy",
                "depType": "final",
                "fixedVersion": "some-tag",
                "registryUrl": "https://example.com",
                "replaceString": "example.com/legacy:some-tag@sha256:46cea30fe85df1f3d554cee602fa69d553da7ab746c8c9f20defb1f1fc3e03b8",
                "versioning": "docker",
                "warnings": [],
                "updates": [
                        {
                                "updateType": "replacement",
                                "newName": "example.com/current",
                                "newValue": "some-tag",
                                "newDigest": "sha256:174f813303ad00bbe1409ea01eb909cb67818a8fbaae8c07df79036a7dea3e1d",
                                "branchName": "renovate/example-com-legacy-replacement"
                        },
                        {
                                "updateType": "digest",
                                "newValue": "some-tag",
                                "newDigest": "sha256:174f813303ad00bbe1409ea01eb909cb67818a8fbaae8c07df79036a7dea3e1d",
                                "branchName": "renovate/example-com-legacy-some-tag"
                        }
                ]
        }
],

github-actions · 2023-02-09T04:06:33Z

Hi there,

Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

uhthomas · 2023-02-09T18:02:03Z

I tried to create a reproduction repository but Renovate isn't able to create any PRs at all.

https://github.com/uhthomas/renovate20304

https://app.renovatebot.com/dashboard#github/uhthomas/renovate20304/1001308789

WARN: Error updating branch: update failure(branch="renovate/alpine-replacement")

viceice · 2023-02-09T20:36:56Z

I tried to create a reproduction repository but Renovate isn't able to create any PRs at all.

https://github.com/uhthomas/renovate20304

https://app.renovatebot.com/dashboard#github/uhthomas/renovate20304/1001308789
WARN: Error updating branch: update failure(branch="renovate/alpine-replacement")

provide more context debug logs

uhthomas · 2023-02-09T20:40:34Z

provide more context debug logs

I really am not sure what more to provide. This is the hosted version of Renovate, so all the debug logs are:

https://app.renovatebot.com/dashboard#github/uhthomas/renovate20304/1001308789

The repository only contains a renovate.json and a Dockerfile which reproduce the issue(s).

https://github.com/uhthomas/renovate20304

The dependency dashboard also shows that it can't make a PR for some reason.

uhthomas/renovate20304#2

uhthomas · 2023-02-09T20:43:57Z

If it helps, I've copied the output from the linked debug log and put it in this gist.

viceice · 2023-02-09T20:46:29Z

the logs doesn't contain the warning, so wrong run

uhthomas · 2023-02-09T20:54:56Z

the logs doesn't contain the warning, so wrong run

Yes they do.

https://gist.github.com/uhthomas/da50bee0a31a3e0755bc5e6644780f34#file-renovate-debug-log-L254

viceice · 2023-02-10T13:49:00Z

looks like a duplicate of

Docker Replacement Rules: Not working as expected #20128

uhthomas · 2023-02-10T14:29:00Z

@viceice I did see that issue, but it's not quite the same. The replacement rules work as expected in our case, but it just seems that for some reason Renovate will now open PRs with the replacement digest for the original image.

I am struggling to demonstrate this as the hosted version of Renovate is running into a different(?) issue in the debug log.

uhthomas · 2023-02-10T15:41:05Z

Okay @viceice, I have a full working reproduction now.

Repository: https://github.com/uhthomas/renovate20304/tree/445bbce6bf13dc2da5f4350e10c74d8f9d946601

Logs: https://app.renovatebot.com/dashboard#github/uhthomas/renovate20304/1001783011 (gist).

Here's what Renovate is trying to do:

See: https://github.com/uhthomas/renovate20304/pull/3/files

It's trying to update the original image with a digest from the replacement image...

I can confirm it's incorrect as docker pull does not work, whereas it does work if I use the digest with the replacement image and version.

❯ docker run gcr.io/distroless/base-debian10:debug@sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d
Unable to find image 'gcr.io/distroless/base-debian10:debug@sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d' locally
docker: Error response from daemon: manifest for gcr.io/distroless/base-debian10@sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d not found: manifest unknown: Failed to fetch "sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d" from request "/v2/distroless/base-debian10/manifests/sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d".
See 'docker run --help'.
❯ docker run gcr.io/distroless/base-debian11:nonroot@sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d
Unable to find image 'gcr.io/distroless/base-debian11:nonroot@sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d' locally
gcr.io/distroless/base-debian11@sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d: Pulling from distroless/base-debian11
5f80a38cb015: Pull complete 
9fb3436fe506: Pull complete 
4f70717a8bb3: Pull complete 
Digest: sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d
Status: Downloaded newer image for gcr.io/distroless/base-debian11@sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d

Is what I've written clear?

viceice · 2023-02-10T15:44:09Z

please write expected and current behavior to the repo readme, then we'll fork it

uhthomas · 2023-02-10T15:44:47Z

Will do.

uhthomas · 2023-02-10T15:47:48Z

@viceice Done!

viceice · 2023-02-10T16:06:29Z

ok, i see a pin PR but no replacement PR. is this only happen on self hosted renovate or also on the GitHub app?

@rarkins @secustor @JamieMagee it seems the pin PR uses the replacement data for pin and blocks any other PR.

i think the pin update type shouldn't use the replacement rule data.

uhthomas · 2023-02-10T17:29:25Z

I think this is the same behaviour as self hosted. We saw correct replacement in some repos, but just a pin PR with the incorrect digest in others.

uhthomas · 2023-02-17T19:18:15Z

Is there anything I can do to help fix this?

viceice · 2023-02-17T22:22:24Z

disable pin for those does or pin before enable the replacement

uhthomas · 2023-02-18T01:54:19Z

I believe this issue occurs even if the original dependency was pinned. I can add that to the reproduction repository to prove it if that would be helpful.

uhthomas · 2023-02-21T16:09:43Z

I tried disabling digest pinning for those deps, but the same thing happened. Have I misconfigured it?

{
  "packageRules": {
    "matchDatasources": [
      "docker"
    ],
    "matchPackageNames": [
      "example.com/old"
    ],
    "pinDigests": false,
    "replacementName": "example.com/new",
    "replacementVersion": "some-tag"
  }
}

As noted in my previous comment, this happens for dependencies which already have digests. So I imagine this probably prevents this for dependencies which don't already have a digest, but this is still a problem for dependencies which do already have a digest.

So, it's not possible to use this feature for what we need unfortunately. Is the demotion from p2 to p3 correct given the potential severity of using bad digests?

uhthomas · 2023-03-20T16:52:06Z

I was really hoping some of the recent changes in v35 around replacement would have inadvertently fixed this issue, but that does not appear to be the case.

@rarkins @viceice Any ideas on how we can resolve this? We really want to use Renovate to replace older versions of internal debian images (jessie, stretch, buster) with bullseye or bookworm but just can't because Renovate creates PRs to update those old images to bad digests.

lixdavid94 · 2024-02-08T19:31:19Z

We are also seeing this issue. We want to use replacementName to replace our java11 images with java17 ones. While renovate does create a PR to replace the image name correctly, it also creates a PR related to the original java11 image using the java17 image digest. This is an issue for our developers since we're using renovate to open these PRs ahead of time to give time for developers to migrate over. But using replacementName is breaking updates to the original java11 image.

In other words, we're expecting renovate to open java11 image PR updates normally, and also a java11 image replacement PR with java17

ssegato81 · 2024-02-09T19:11:16Z

I'd like to add more information.

I have the following renovate.json file

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "configWarningReuseIssue": false,
  "extends": [
    "config:recommended"
  ],
  "packageRules": [
    {
      "matchDatasources": [
        "docker"
      ],
      "matchPackagePatterns": [
        "^registry.ms.io/company/node-16.*"
      ],
      "replacementName": "registry.ms.io/company/openjdk-17",
      "replacementVersion": "5.2.148"
    }
  ]
}

The following Dockerfile

FROM registry.ms.io/company/node-16:4.7.100@sha256:5a5fe176b9fa2f706a5b1b09f26137bf69de27bc40627c80ad212d0c6ea9de8b as base

SHELL ["/bin/bash", "-o", "pipefail", "-c"]
USER root

Last node-16 docker image version: 4.7.102
The openjdk docker image only has versions above 5.0.0, so it doesn't have 4.7.102 version

In this case, Renovate should create two PR

updates the node-16 image to the latest version (4.1.102)
change the docker image to openjdk version 5.2.148 (this is that latest one)

PR 2 is created correctly
PR 1 throughs the following error

Renovate failed to look up the following dependencies: Could not determine new digest for update (docker package registry.ms.io/company/node-16).

Files affected: Dockerfile

Looking at the logs I see the following

DEBUG: getManifestResponse(https://registry.ms.io, comany/openjdk-17, 4.7.102, head) (repository=seba/testRenovate)
DEBUG: HEAD https://registry.ms.io/v2/company/openjdk-17/manifests/4.7.102 = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=734) (repository=seba/testRenovate)
DEBUG: Docker Manifest is unknown (repository=seba/testRenovate)
       "err": {
         "name": "HTTPError",
         "code": "ERR_NON_2XX_3XX_RESPONSE",
         "timings": {
           "start": 1707503296111,
           "socket": 1707503296112,
           "lookup": 1707503296113,
           "connect": 1707503296114,
           "secureConnect": 1707503296568,
           "upload": 1707503296568,
           "response": 1707503296845,
           "end": 1707503296845,
           "phases": {
             "wait": 1,
             "dns": 1,
             "tcp": 1,
             "tls": 454,
             "request": 0,
             "firstByte": 277,
             "download": 0,
             "total": 734
           }
         },
         "message": "Response code 404 (Not Found)",
         "stack": "HTTPError: Response code 404 (Not Found)\n    at Request.<anonymous> (/usr/src/app/node_modules/got/dist/source/as-promise/index.js:118:42)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)",
         "options": {
           "headers": {
             "user-agent": "RenovateBot/37.105.1 (https://github.com/renovatebot/renovate)",
             "authorization": "***********",
             "accept": "application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json",
             "accept-encoding": "gzip, deflate, br"
           },
           "url": "https://registry.ms.io/v2/company/openjdk-17/manifests/4.7.102",
           "hostType": "docker",
           "username": "",
           "password": "",
           "method": "HEAD",
           "http2": false
         },
         "response": {
           "statusCode": 404,
           "statusMessage": "Not Found",
           "body": "",
           "headers": {
             "content-length": "114",
             "content-type": "application/json; charset=utf-8",
             "date": "Fri, 09 Feb 2024 18:28:16 GMT",
             "server": "nginx",
             "set-cookie": ["sid=53cdc4189fb79a4091b3f745675d0d06; Path=/; HttpOnly"],
             "x-request-id": "a55db2a4-7760-4dad-8135-9e6a1a192f0c",
             "connection": "Close"
           },
           "httpVersion": "1.1",
           "retryCount": 0
         }
       },
       "registryHost": "https://registry.ms.io",
       "dockerRepository": "company/openjdk-17",
       "tag": "4.7.102"
DEBUG: Could not determine new digest for update. (repository=seba/testRenovate)
       "packageName": "registry.ms.io/company/node-16",
       "currentValue": "4.7.100",
       "datasource": "docker",
       "newValue": "4.7.102",
       "bucket": "non-major"

It seems that Renovate is replacing only the docker image name in the PR (1)

uhthomas added priority-5-triage status:requirements Full requirements are not yet known, so implementation should not be started type:bug Bug fix of existing functionality labels Feb 8, 2023

rarkins added the auto:reproduction A minimal reproduction is necessary to proceed label Feb 9, 2023

HonkingGoose added reproduction:provided and removed auto:reproduction A minimal reproduction is necessary to proceed labels Feb 9, 2023

viceice added priority-3-medium Default priority, "should be done" but isn't prioritised ahead of others manager:dockerfile Dockerfile files and removed priority-5-triage labels Feb 10, 2023

rarkins added priority-3-medium Default priority, "should be done" but isn't prioritised ahead of others and removed priority-2-high Bugs impacting wide number of users or very important features labels Feb 21, 2023

rarkins removed the reproduction:provided label Sep 26, 2023

rarkins removed the status:ready label Oct 8, 2023

ssegato81 mentioned this issue Mar 7, 2024

fix: Correct digest resolution when the replacementName and replacementVersion options are defined #27773

Closed

6 tasks

This was referenced Apr 25, 2024

fix: Correct digest resolution when the replacementName and replacementVersion options are defined #27773 #28654

Closed

fix(autoReplace): Correct digest resolution when the replacementName and replacementVersion options are defined #28655

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker: Replacement causes existing dependencies to use wrong digest #20304

docker: Replacement causes existing dependencies to use wrong digest #20304

uhthomas commented Feb 8, 2023

uhthomas commented Feb 8, 2023 •

edited

github-actions bot commented Feb 9, 2023

uhthomas commented Feb 9, 2023

viceice commented Feb 9, 2023

uhthomas commented Feb 9, 2023 •

edited

uhthomas commented Feb 9, 2023

viceice commented Feb 9, 2023

uhthomas commented Feb 9, 2023

viceice commented Feb 10, 2023

uhthomas commented Feb 10, 2023

uhthomas commented Feb 10, 2023

viceice commented Feb 10, 2023

uhthomas commented Feb 10, 2023

uhthomas commented Feb 10, 2023

viceice commented Feb 10, 2023

uhthomas commented Feb 10, 2023 •

edited

uhthomas commented Feb 17, 2023

viceice commented Feb 17, 2023

uhthomas commented Feb 18, 2023

uhthomas commented Feb 21, 2023 •

edited

uhthomas commented Mar 20, 2023 •

edited

lixdavid94 commented Feb 8, 2024

ssegato81 commented Feb 9, 2024 •

edited

docker: Replacement causes existing dependencies to use wrong digest #20304

docker: Replacement causes existing dependencies to use wrong digest #20304

Comments

uhthomas commented Feb 8, 2023

How are you running Renovate?

If you're self-hosting Renovate, tell us what version of Renovate you run.

If you're self-hosting Renovate, select which platform you are using.

If you're self-hosting Renovate, tell us what version of the platform you run.

Was this something which used to work for you, and then stopped?

Describe the bug

Relevant debug logs

Have you created a minimal reproduction repository?

uhthomas commented Feb 8, 2023 • edited

github-actions bot commented Feb 9, 2023

uhthomas commented Feb 9, 2023

viceice commented Feb 9, 2023

uhthomas commented Feb 9, 2023 • edited

uhthomas commented Feb 9, 2023

viceice commented Feb 9, 2023

uhthomas commented Feb 9, 2023

viceice commented Feb 10, 2023

uhthomas commented Feb 10, 2023

uhthomas commented Feb 10, 2023

viceice commented Feb 10, 2023

uhthomas commented Feb 10, 2023

uhthomas commented Feb 10, 2023

viceice commented Feb 10, 2023

uhthomas commented Feb 10, 2023 • edited

uhthomas commented Feb 17, 2023

viceice commented Feb 17, 2023

uhthomas commented Feb 18, 2023

uhthomas commented Feb 21, 2023 • edited

uhthomas commented Mar 20, 2023 • edited

lixdavid94 commented Feb 8, 2024

ssegato81 commented Feb 9, 2024 • edited

uhthomas commented Feb 8, 2023 •

edited

uhthomas commented Feb 9, 2023 •

edited

uhthomas commented Feb 10, 2023 •

edited

uhthomas commented Feb 21, 2023 •

edited

uhthomas commented Mar 20, 2023 •

edited

ssegato81 commented Feb 9, 2024 •

edited