404 while listing Docker images with Renovate 32 and Sonatype NXRM 3 #15016

ssube · 2022-04-07T18:58:50Z

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

32.14.1

Please select which platform you are using if self-hosting.

GitLab self-hosted

If you're self-hosting Renovate, tell us what version of the platform you run.

Gitlab EE

Was this something which used to work for you, and then stopped?

It used to work, and then stopped

Describe the bug

After upgrading from Renovate 27.31.10 to 32.14.1 (we were a little behind 🙈 ), I am seeing a 404 response from our private Nexus Repository Manager 3.38 server that I cannot reproduce locally.

From the job logs:

 WARN: Invalid registry response (repository=devops/cluster/gitlab)
       "apiCheckUrl": "https://nexus-docker-out.build.company.com/v2/library/docker/tags/list?n=10000",
       "res": {
         "statusCode": 404,
         "body": "",
         "headers": {
           "server": "nginx/1.13.12",
           "date": "Thu, 07 Apr 2022 17:31:41 GMT",
           "content-type": "text/html",
           "connection": "close",
           "vary": "Accept-Encoding",
           "x-content-type-options": "nosniff",
           "content-security-policy": "sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation",
           "x-xss-protection": "1; mode=block",
           "cache-control": "no-cache, no-store, max-age=0, must-revalidate, post-check=0, pre-check=0",
           "pragma": "no-cache",
           "expires": "0",
           "x-frame-options": "DENY",
           "content-encoding": "gzip"
         },
         "authorization": false
       }

Run locally using the same renovate user/pass:

> curl -H 'Authorization: Basic ...'  https://nexus-docker-out.build.company.com/v2/library/docker/tags/list?n=10000
{"name":"library/docker","tags":["1","1-dind","1-experimental","1-experimental-dind","1-experimental-git","1-git","1.0","1.0-dind","1.0-git","1.0.1",...

My config is:

module.exports = {                                                            
  endpoint: 'https://gitlab.build.company.com/api/v4',                            
  token: GITLAB_TOKEN,                                              
  platform: 'gitlab',                                                         
  onboarding: true,                                                           
  onboardingConfig: {                                                         
    extends: ['config:base'],                                                 
  },                                                                          
  repositories: RENOVATE_PROJECTS,                                                                 
  hostRules: [{                                                               
    matchHost: 'https://gitlab.build.company.com/',                                       
    username: 'renovate',                                                 
    token: GITLAB_TOKEN,                                            
  }, {                                                                        
    hostType: 'docker',                                                       
    matchHost: 'https://nexus-docker-out.build.company.com/',      
    username: 'renovate',                                                     
    password: NEXUS_PASS,                         
  }, {                                                                        
    hostType: 'npm',                                                       
    matchHost: 'https://nexus.build.company.com/',      
    username: 'renovate',                                                     
    password: NEXUS_PASS,
    npmrc: NEXUS_NPMRC,             # this part is weird, but npm still works fine        
  }],                                                                         
  prConcurrentLimit: 2,
  branchPrefix: 'deps/',
  branchTopic: '{{{depNameSanitized}}}',
}

I tried matchHost with and without the protocol and trailing slash, that did not seem to make a difference. I'm not sure if the npm host needs both rc and pass, but that one still works with 32.x, this only seems to impact docker. I looked at #5263 and #8200, as well as https://issues.sonatype.org/browse/NEXUS-26313, but the www-authenticate header is present on the initial 401 response, and the URL would suggest the error is happening later in the process.

This depends on my company's private Gitlab and Nexus, so I do not have a repro repo. https://nexus.build.company.com is running Sonatype Nexus Repository Manager PRO 3.38.0-01.

Any idea what might have changed?

Relevant debug logs

Logs

{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":20,"logContext":"E1lWoD36u2ccZ3qd4ra23","msg":"Setting global hostRules","time":"2022-04-07T17:31:37.557Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":20,"logContext":"E1lWoD36u2ccZ3qd4ra23","msg":"Adding token authentication for gitlab.build.company.com to hostRules","time":"2022-04-07T17:31:37.557Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":20,"logContext":"E1lWoD36u2ccZ3qd4ra23","msg":"Adding password authentication for nexus-docker-out.build.company.com to hostRules","time":"2022-04-07T17:31:37.558Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":20,"logContext":"E1lWoD36u2ccZ3qd4ra23","msg":"Adding password authentication for nexus.build.company.com to hostRules","time":"2022-04-07T17:31:37.559Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":20,"logContext":"E1lWoD36u2ccZ3qd4ra23","msg":"Adding token authentication for github.com to hostRules","time":"2022-04-07T17:31:37.560Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":20,"logContext":"E1lWoD36u2ccZ3qd4ra23","msg":"Adding token authentication for gitlab.build.company.com to hostRules","time":"2022-04-07T17:31:37.560Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":20,"logContext":"E1lWoD36u2ccZ3qd4ra23","msg":"validatePresets()","time":"2022-04-07T17:31:37.560Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":20,"logContext":"E1lWoD36u2ccZ3qd4ra23","msg":"Reinitializing hostRules for repo","time":"2022-04-07T17:31:37.595Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":20,"logContext":"E1lWoD36u2ccZ3qd4ra23","msg":"Clearing hostRules","time":"2022-04-07T17:31:37.595Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":20,"logContext":"E1lWoD36u2ccZ3qd4ra23","msg":"Adding token authentication for gitlab.build.company.com to hostRules","time":"2022-04-07T17:31:37.596Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":20,"logContext":"E1lWoD36u2ccZ3qd4ra23","msg":"Adding password authentication for nexus-docker-out.build.company.com to hostRules","time":"2022-04-07T17:31:37.596Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":20,"logContext":"E1lWoD36u2ccZ3qd4ra23","msg":"Adding password authentication for nexus.build.company.com to hostRules","time":"2022-04-07T17:31:37.597Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":20,"logContext":"E1lWoD36u2ccZ3qd4ra23","msg":"Adding token authentication for github.com to hostRules","time":"2022-04-07T17:31:37.598Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":20,"logContext":"E1lWoD36u2ccZ3qd4ra23","msg":"Adding token authentication for gitlab.build.company.com to hostRules","time":"2022-04-07T17:31:37.598Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":30,"logContext":"E1lWoD36u2ccZ3qd4ra23","repository":"devops/cluster/gitlab","renovateVersion":"32.14.1","msg":"Repository started","time":"2022-04-07T17:31:37.604Z","v":0}

# some logs omitted

{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":20,"logContext":"E1lWoD36u2ccZ3qd4ra23","repository":"devops/cluster/gitlab","msg":"Failed to get authHeaders for getTags lookup","time":"2022-04-07T17:31:41.075Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":40,"logContext":"E1lWoD36u2ccZ3qd4ra23","repository":"devops/cluster/gitlab","apiCheckUrl":"https://nexus-docker-out.build.company.com/v2/library/docker/tags/list?n=10000","res":{"statusCode":404,"body":"","headers":{"server":"nginx/1.13.12","date":"Thu, 07 Apr 2022 17:31:41 GMT","content-type":"text/html","connection":"close","vary":"Accept-Encoding","x-content-type-options":"nosniff","content-security-policy":"sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation","x-xss-protection":"1; mode=block","cache-control":"no-cache, no-store, max-age=0, must-revalidate, post-check=0, pre-check=0","pragma":"no-cache","expires":"0","x-frame-options":"DENY","content-encoding":"gzip"},"authorization":false},"msg":"Invalid registry response","time":"2022-04-07T17:31:41.079Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":20,"logContext":"E1lWoD36u2ccZ3qd4ra23","repository":"devops/cluster/gitlab","msg":"Failed to get authHeaders for getTags lookup","time":"2022-04-07T17:31:41.080Z","v":0}
{"name":"renovate","hostname":"runner-sksmtss1-project-1064-concurrent-3rzjs2","pid":22,"level":40,"logContext":"E1lWoD36u2ccZ3qd4ra23","repository":"devops/cluster/gitlab","apiCheckUrl":"https://nexus-docker-out.build.company.com/v2/ssube/salty-dog/tags/list?n=10000","res":{"statusCode":404,"body":"","headers":{"server":"nginx/1.13.12","date":"Thu, 07 Apr 2022 17:31:41 GMT","content-type":"text/html","connection":"close","vary":"Accept-Encoding","x-content-type-options":"nosniff","content-security-policy":"sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation","x-xss-protection":"1; mode=block","cache-control":"no-cache, no-store, max-age=0, must-revalidate, post-check=0, pre-check=0","pragma":"no-cache","expires":"0","x-frame-options":"DENY","content-encoding":"gzip"},"authorization":false},"msg":"Invalid registry response","time":"2022-04-07T17:31:41.081Z","v":0}

Have you created a minimal reproduction repository?

No reproduction repository

The text was updated successfully, but these errors were encountered:

rarkins · 2022-04-08T06:25:34Z

Can you try release 32.12.0 to check that it wasn't the changes in #14744?

rslinckx · 2022-04-08T08:16:08Z

I commented on the other issue, but we started seeing 405 using gcr.io since 2 days, so this might be related as well

vlagorce · 2022-04-08T10:03:59Z

@ssube
what does $curl --location --head 'https://nexus-docker-out.build.company.com/v2/library/docker/tags/list?n=10000 returns
@rslinckx
Can you performe the same for your gcr.io repo/image ?

rslinckx · 2022-04-08T10:09:20Z

curl --location --head 'https://gcr.io/v2/datadoghq/cluster-agent/tags/list?n=10000'

HTTP/2 405
docker-distribution-api-version: registry/2.0
content-type: application/json
date: Fri, 08 Apr 2022 10:08:28 GMT
server: Docker Registry
cache-control: private
x-xss-protection: 0
x-frame-options: SAMEORIGIN
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
accept-ranges: none
vary: Accept-Encoding

vlagorce · 2022-04-08T10:32:40Z

On a private gcr.io registry we get the following

curl -iL --get 'https://gcr.io/v2/datadog/cluster-agent/tags/list?n=10000' 
HTTP/2 401 
docker-distribution-api-version: registry/2.0
content-type: application/json
www-authenticate: Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:datadog/cluster-agent:pull"
date: Fri, 08 Apr 2022 10:29:39 GMT
server: Docker Registry
cache-control: private
x-xss-protection: 0
x-frame-options: SAMEORIGIN
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
accept-ranges: none
vary: Accept-Encoding

curl -iL --head 'https://gcr.io/v2/datadog/cluster-agent/tags/list?n=10000'
HTTP/2 401 
docker-distribution-api-version: registry/2.0
content-type: application/json
www-authenticate: Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:datadog/cluster-agent:pull"
date: Fri, 08 Apr 2022 10:30:57 GMT
server: Docker Registry
cache-control: private
x-xss-protection: 0
x-frame-options: SAMEORIGIN
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
accept-ranges: none
vary: Accept-Encoding

HEAD seems not supported only for public gcr registry.

rslinckx · 2022-04-08T10:35:25Z

I also have 405 on our private registries, presumably because the call is properly authentified, which the curl above is not ?

vlagorce · 2022-04-08T10:48:26Z

I also have 405 on our private registries, presumably because the call is properly authentified, which the curl above is not ?

If you don't explicitly give a token with the above curl there is no reason. May be something more related to the repo I checked.

vlagorce · 2022-04-08T10:49:51Z

Looking at docker-cli; It seems they are using GET https://github.com/docker/cli/blob/master/cli/trust/trust.go#L139

viceice · 2022-04-08T11:28:22Z

Looking at docker-cli; It seems they are using GET https://github.com/docker/cli/blob/master/cli/trust/trust.go#L139

That is the wrong function, it for Notary.

viceice · 2022-04-08T11:32:45Z

It seems to do a POST to /auth endpoint 🤔

https://github.com/docker/cli/blob/429d716fbc11423f174490d448a78047928301c3/vendor/github.com/docker/docker/client/login.go#L15

viceice · 2022-04-08T11:41:42Z

OK, the daemon is doing the pull and is doing a get to /v2 as we've done before

viceice · 2022-04-08T11:43:07Z

I think this is now fixed by the revert PR:

fix(docker): Revert "fix(docker): use a HEAD request to the real resource auth." #15018

@vlagorce So we should continue on #14708

ssube · 2022-04-08T14:51:01Z

With --location --head (or --head alone), I can repro the 404 from my logs as well:

HTTP/1.1 404 Not Found
Server: nginx/1.13.12
Date: Fri, 08 Apr 2022 14:35:07 GMT
Content-Type: text/html
Content-Length: 1850
Connection: keep-alive
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
Content-Security-Policy: sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY

Can you try release 32.12.0 to check that it wasn't the changes in #14744?

It does not occur under 32.12.0, but I do see it with 32.12.1, so this change/fix seems to be the one.

ssube added priority-5-triage status:requirements Full requirements are not yet known, so implementation should not be started type:bug Bug fix of existing functionality labels Apr 7, 2022

viceice closed this as completed Apr 8, 2022

github-actions bot locked as resolved and limited conversation to collaborators May 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

404 while listing Docker images with Renovate 32 and Sonatype NXRM 3 #15016

404 while listing Docker images with Renovate 32 and Sonatype NXRM 3 #15016

ssube commented Apr 7, 2022

rarkins commented Apr 8, 2022

rslinckx commented Apr 8, 2022

vlagorce commented Apr 8, 2022 •

edited

rslinckx commented Apr 8, 2022

vlagorce commented Apr 8, 2022

rslinckx commented Apr 8, 2022

vlagorce commented Apr 8, 2022 •

edited

vlagorce commented Apr 8, 2022

viceice commented Apr 8, 2022

viceice commented Apr 8, 2022

viceice commented Apr 8, 2022

viceice commented Apr 8, 2022

ssube commented Apr 8, 2022

404 while listing Docker images with Renovate 32 and Sonatype NXRM 3 #15016

404 while listing Docker images with Renovate 32 and Sonatype NXRM 3 #15016

Comments

ssube commented Apr 7, 2022

How are you running Renovate?

If you're self-hosting Renovate, tell us what version of Renovate you run.

Please select which platform you are using if self-hosting.

If you're self-hosting Renovate, tell us what version of the platform you run.

Was this something which used to work for you, and then stopped?

Describe the bug

Relevant debug logs

Have you created a minimal reproduction repository?

rarkins commented Apr 8, 2022

rslinckx commented Apr 8, 2022

vlagorce commented Apr 8, 2022 • edited

rslinckx commented Apr 8, 2022

vlagorce commented Apr 8, 2022

rslinckx commented Apr 8, 2022

vlagorce commented Apr 8, 2022 • edited

vlagorce commented Apr 8, 2022

viceice commented Apr 8, 2022

viceice commented Apr 8, 2022

viceice commented Apr 8, 2022

viceice commented Apr 8, 2022

ssube commented Apr 8, 2022

vlagorce commented Apr 8, 2022 •

edited

vlagorce commented Apr 8, 2022 •

edited