Artifactory 406 Error when fetching tags

[mmkonrad]

This is a:

• Bug report (non-security related)
• Feature request
• I'm not sure which of those it is

Please describe the issue:

Fun Fact upfront: renovate works like a charm as described below when the dockerfiles already contain a sha256-Digest.

1. We run renovate as a cronjob on OpenShift.
2. It finds the configured repository (in fact tested of different ones with same result)
3. Handles only dockerfiles.
4. Is configured to pinDigests=true
5. Finds out that there are dockerfiles in the repository (2x node:8.9.4-alpine Image )
6. Tries to get a list of tags from Artifactory (self-hosted, mirroring public docker registries)
7. Fails with HTTP 406 - Not Acceptable
8. Onboarding PR states:

⚠ Dependency Lookup Warnings ⚠

Please correct - or verify that you can safely ignore - these lookup failures before you merge this MR.

Failed to look up dependency node

Log of Renovate states:

 INFO: Renovating repository (repository=mygroup/myrepo)
(node:1) Warning: Ignoring extra certs from `/var/run/secrets/kubernetes.io/serviceaccount/ca.crt`, load failed: error:0B07C065:x509 certificate routines:X509_STORE_add_cert:cert already in hash table

 INFO: Repo is not onboarded (repository=mygroup/myrepo)
 INFO: Onboarding branch is up to date (repository=mygroup/myrepo)
 INFO: Repository config (repository=mygroup/myrepo)
       "configFile": "renovate.json",
       "config": {"enabledManagers": ["dockerfile"], "pinDigests": true}
 INFO: Found dockerfile package files (repository=mygroup/myrepo)
 WARN: Error getting docker image tags (repository=mygroup/myrepo)
       "registry": "https://my-self-hosted-artifactory-url.com ",
       "dockerRepository": "node",
       "err": {
         "name": "HTTPError",
         "host": "my-self-hosted-artifactory-url.com",
         "hostname": "my-self-hosted-artifactory-url.com",
         "method": "GET",
         "path": "/v2/node/tags/list?n=10000",
         "protocol": "https:",
         "url": "https://my-self-hosted-artifactory-url.com/v2/node/tags/list?n=10000 ",
         "gotOptions": {
           "path": "/v2/node/tags/list?n=10000",
           "protocol": "https:",
           "slashes": true,
           "auth": null,
           "host": "my-self-hosted-artifactory-url.com",
           "port": null,
           "hostname": "my-self-hosted-artifactory-url.com",
           "hash": null,
           "search": "?n=10000",
           "pathname": "/v2/node/tags/list",
           "href": "https://my-self-hosted-artifactory-url.com/v2/node/tags/list?n=10000 ",
           "retry": {
             "methods": {},
             "statusCodes": {},
             "errorCodes": {},
             "maxRetryAfter": 10000
           },
           "headers": {
             "user-agent": "got/9.5.0 (https://github.com/sindresorhus/got )",
             "authorization": "Bearer AKCp2VpZDvgzzRrnxdqQ3nfeLAwXPsv1HcP19Jret7nhY5J4KYZLpd2S4DTabJKX2DaULEY9o",
             "accept": "application/vnd.docker.distribution.manifest.v2+json",
             "accept-encoding": "gzip, deflate"
           },
           "hooks": {
             "beforeRequest": [],
             "beforeRedirect": [],
             "beforeRetry": [],
             "afterResponse": []
           },
           "decompress": true,
           "throwHttpErrors": true,
           "followRedirect": true,
           "stream": false,
           "form": false,
           "json": true,
           "cache": false,
           "useElectronNet": false,
           "gotTimeout": {"request": 10000},
           "method": "GET"
         },
         "statusCode": 406,
         "statusMessage": "Not Acceptable",
         "headers": {
           "date": "Fri, 11 Jan 2019 13:32:15 GMT",
           "content-type": "application/json",
           "transfer-encoding": "chunked",
           "connection": "close",
           "server": "Artifactory/6.0.3",
           "x-artifactory-id": "b110e4a1a4ada9b6e19f4fa25488988fc2bdd8fd",
           "x-artifactory-node-id": "ip-10-104-193-164",
           "x-frame-options": "DENY",
           "x-content-type-options": "nosniff",
           "x-xss-protection": "1; mode=block"
         },
         "body": {"errors": [{"status": 406, "message": "Not Acceptable"}]},
         "message": "Response code 406 (Not Acceptable)",
         "stack": "HTTPError: Response code 406 (Not Acceptable)\n    at EventEmitter.emitter.on (/usr/src/app/node_modules/got/source/as-promise.js:74:19)\n    at <anonymous>\n    at process._tickCallback (internal/process/next_tick.js:189:7)"
       }
 INFO: Failed to look up dependency node (repository=mygroup/myrepo, packageFile=docker/delivery/my-app/Dockerfile, dependency=node)
 INFO: Failed to look up dependency node (repository=mygroup/myrepo, packageFile=docker/delivery/my-workers/Dockerfile, dependency=node)
 INFO: Found package lookup warnings in onboarding (repository=mygroup/myrepo)
       "warnings": ["Failed to look up dependency node"],
       "warningFiles": [
         "docker/delivery/my-app/Dockerfile",
         "docker/delivery/my-workers/Dockerfile"
       ]
 INFO: Found open onboarding PR (repository=mygroup/myrepo)
 INFO: Merge Request #60 does not need updating (repository=mygroup/myrepo)
 INFO: Finished repository (repository=mygroup/myrepo)
INFO: Renovate finished

I saw a similar issue with the same error code that got it solved, but we use renovate not with that command but defined everything in an openshift template like:

  stringData:
    config.js: |
      module.exports = {
        endpoint: 'https://my.private.gitlab.com/api/v4/',
        platform: 'gitlab',
        token: '${TOKEN}',
        onboarding: true,
        onboardingConfig: {
          enabledManagers: [
            'dockerfile'
          ],
          pinDigests: true
        },
        repositories: [
          {
            repository: '${REPOSITORY}',

          }
        ]
      };

What additional information do you need?

Where did I go wrong?

[rarkins]

I think we may have a compatibility problem with Artifactory. To start with, does your Artifactory Docker registry require any authentication?

[mmkonrad]

The mirror repository of that artifactory do not need authentication (afaik).

As stated before it works like a a charm when the base image in the dockerfile already contains a digest. In such case we already received a merge request to update the digest.
Since there are no config differences of renovate between both cases and the URL to the mirror repository did not change as well, I conclude that there is no authentication needed.

[mmkonrad]

The only thing that changed between the success case and the failed case is:
Success case:
Dockerfile contains: FROM my-self-hosted-artifactory-url.com/solr@sha256:dd81436461637eac88c4a2ebfb59af3cffd3d5b928fd68315d8a1a175243242c

Failure case:
Dockerfile contains FROM my-self-hosted-artifactory-url.com/node:8.9.2-alpine

[rarkins]

I agree with your assessment. Seems like this URL may be the culprit:

         "url": "https://my-self-hosted-artifactory-url.com/v2/node/tags/list?n=10000 ",

Can you try accessing it through a browser or perhaps Postman to see what the response is?

Strange thing is the space at the end there unless you added it by accident when search/replacing?

[rarkins]

Also can you confirm what exactly the tags/hashes look like when it works and doesn't work?

e.g. FROM node:latest and FROM node@sha256aaaaaaaa..........?

[mmkonrad]

I replayed the URL with Restlet Client and received the same result.
What I noticed: 406 with:

• "accept" header "application/vnd.docker.distribution.manifest.v2+json"
• "accept encoding" header "gzip, deflate"
• "authorization" header "Bearer "

If I remove the "accept" header I get ERROR 404 with:

{
    "errors":[
        {
            "code": "NAME_UNKNOWN",
            "message": "Repository name not known to registry.",
            "detail":{
                "name": "node"
            }
        }
    ]
}

[mmkonrad]

The tags in the success and failure case look exactly as stated before:

Success case:
FROM my-self-hosted-artifactory-url.com/solr@sha256:dd81436461637eac88c4a2ebfb59af3cffd3d5b928fd68315d8a1a175243242c

Failure case:
FROM my-self-hosted-artifactory-url.com/node:8.9.2-alpine

In the success case with the hash digest renovate also made an update-merge-request:

FROM smy-self-hosted-artifactory-url.com/solr@sha256:dd81436461637eac88c4a2ebfb59af3cffd3d5b928fd68315d8a1a175243242c

TO

FROM my-self-hosted-artifactory-url.com/solr@sha256:eb3196e4e75478d336741c7cacfbf422dd1adf62b8cd98d580a3124aa37f7646

[rarkins]

What if you add an sha256 to the node image and test again? I wonder if this is related to node and not necessarily to digest pinning

[mmkonrad]

I am going to check it out.
But I do not think it is related to node.

The solr image hash digest is in its own branch. The master still contains:
FROM my-self-hosted-artifactory-url.com/solr:7.4-alpine

When I let run renovate against the master branch it fails in the very same way as the node variant.

NOTE: the solr and the node dockerfiles are in different GitLab repositories, but the URL to the docker registry is always the same.

[mmkonrad]

I double checked that the registry url is always the same.
I am also able to pull the images (with hash digest and without) from that registry without further authentication (meaning that I am not logged in into that registry, but can pull the images).

[rarkins]

Do you get the same error message if you manually pull the tags for the “solr” image rather than “node”?

[mmkonrad]

you mean docker pull my-self-hosted-artifactory-url.com/solr:7.4-alpine ?

[mmkonrad]

I was able to pull all images manually without any errors:

my-self-hosted-artifactory-url.com/solr:7.4-alpine
my-self-hosted-artifactory-url.com/solr@sha256:dd81436461637eac88c4a2ebfb59af3cffd3d5b928fd68315d8a1a175243242c
my-self-hosted-artifactory-url.com/solr@sha256:eb3196e4e75478d336741c7cacfbf422dd1adf62b8cd98d580a3124aa37f7646
my-self-hosted-artifactory-url.com/node:8.9.2-alpine
my-self-hosted-artifactory-url.com/node:8.9.4-alpine

Before I pulled them I deleted all images and after each pull I deleted the recently pulled one.
Also I am still not logged in into the registry.

[mmkonrad]

Further Information!
After toying around with the tags of an image we come to the conclusion:
If the tag does not equal "latest" (or blank ->resolves to latest) -> renovate fails.

[rarkins]

@mmkonrad can you explain it to me with example URLs that work/fail? i.e. not "docker" commands but something you'd GET using Restlet

[rarkins]

Or maybe I understand already. Would this explanation be right?

• If an image's existing tag is blank, latest, or a non-version (e.g. "alpine") then Renovate knows it's not a version and doesn't look up tags because there's no existing version to compare it to. This works.

• If an image has a version-like tag (e.g. node:10.0.0 or node:10.0.0-alpine) then Renovate attempts to query the packages tags for more versions. This fails

Conclusion: Artifactory doesn't like our tags query.

[mmkonrad]

correct

[mmkonrad]

Alright I just tested the assumption again and it looks like that it is correct.

1. I created a repository.
2. created 8 directories with each a Dockerfile in it (node-a until node-d, solr-a until solr-d)

• a) node / solr
• b) node:latest / solr:latest
• c) node:8.9.4 / solr:7.4
• d) node:8.9.4-alpine / solr:7.4-alpine

3. let renovate run against it
4. renovate makes onboarding mr

• Pin Node.js:
  Pin node to sha256:5ea9b2d04e1da959087d93289bdcfa5011c3a173bddf3eaf98553676d28b8f63

• Pin solr Docker tag:
  Pin solr to sha256:eb3196e4e75478d336741c7cacfbf422dd1adf62b8cd98d580a3124aa37f7646

5. Failes for c/d directories:

⚠ Dependency Lookup Warnings ⚠

Please correct - or verify that you can safely ignore - these lookup failures before you merge this MR.

Failed to look up dependency node
Failed to look up dependency solr

Files affected: node-c/Dockerfile, node-d/Dockerfile, solr-c/Dockerfile, solr-d/Dockerfile

6. All Failures are: 406 Not Acceptable

[rarkins]

Thank you again for help in troubleshooting this. I think the right way to probably to fix this is to use Joyent's docker registry library for retrieving tags. If you don't mind, I'll first send you a test file you can run that will double check that this new approach will successfully retrieve tags from Artifactory?

[mmkonrad]

Can you tell me more about that library and the script you want to send me?
What is it and what does it?

[rarkins]

Currently we query Docker registries using "hand written" API queries. But it sounds like we're probably not fully compliant with all the way to request tags, because Artifactory is rejecting us.

There's an open source client for Docker written by Joyent we could switch to, if it solves the problem. There was a PR by @mikebryant that got most of the work done already that we could resuscitate: https://github.com/renovatebot/renovate/pull/2908/files

[mmkonrad]

Nice, I guess the easiest way to test this is to provide a docker image available by the official docker registry.
So our (to-be) cronjob in gitlab can fetch it through the mirror repo and run it.

Currently we use renovate/renovate

That would require me to only change a line -> the renovate image in the gitlab-ci

[rarkins]

Actually no, I wanted to test it first before patching Renovate - otherwise it's 50 lines of refactoring that might not even work.

I would provide you a repo with:

• package.json including the registry as only dependency
• index.js that does nothing but query the list of tags of "node" in your registry. You'd manually patch in the registry name into this file

You'd fun "npm install" and then "node index.js" and see if it errors or prints a list of tags.

So if that query worked and returned the list of tags, we'd know it's worth refactoring renovate. If not, then Artifactory has a bigger problem than we thought.

Sound ok?

[mmkonrad]

Okay, I will setup a Node container....so ia have a clean node version and installation.
What Node Version is required?

[rarkins]

node 8 or 10 should be fine

[renovate-bot]

🎉 This issue has been resolved in version 13.175.13 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[rarkins]

@mmkonrad please try testing with this new release and see if it resolves the tags problem. If not, please reopen

[mmkonrad]

It still nor working for the c/d-repositories:

c) node:8.9.4 / solr:7.4
d) node:8.9.4-alpine / solr:7.4-alpine

As I said I have to look into the issue of us depending so much on fixed versions and what registries are actually included in the mirror repository.

[rarkins]

@mmkonrad if you update your Dockerfiles to use the library/ prefix or amd64/ prefix, does everything work? And with the new release, is the error log exactly the same, or is it different? I think it should be different.

The problem is this:

1. The official Docker Hub registry is special and has special library packages like "node". Clients know to prefix "library/" to them for API calls, if they don't have a prefix already
2. All other registries including your mirror are not special and Docker clients have no way certainty that it's required to add the "library/" prefix
3. Artifactory seem to have worked around this for all the APIs necessary for pulling images and digests, but not for the "list tags" API, probably because it's never used by the docker CLI

[mmkonrad]

The Dockerfiles are currently updated to "library/" . (amd64 resolves to the same image id)
Since we use the "renovate/renovate" Docker Image I suppose we also use the latest version of renovate.

[rarkins]

Unless you have an exact version of Renovate's image, you may be running a cached version. Your logs will print which version is being run anyway, right near the top of each repository.

Anyway, please attach the new logs

[mmkonrad]

package.json and "node lib/renovate.js" state version 13.180.0

If it is not the latest version: which is it, so that I can add that to the gitlab-ci.yml image descriptor

[rarkins]

Latest release can change every 10 minutes, but that one you ran should have had the changes in it necessary. Can you take a look at the latest logs/failures?

[mmkonrad]

Logfile: https://github.com/mmkonrad/renovate-log/blob/master/renovate.log

[rarkins]

According to that log, the URL that fails is https://my-self-hosted-artifactory-url.com/v2/library/node/tags/list?n=10000.

But earlier you wrote:

Actually, we discovered that we receive the tag list with inserting "/library/" before node:
https://my-self-hosted-artifactory-url.com/v2/library/node/tags/list?n=10000

And I can't see any difference in URLs. In which case, is it something in our headers? It says:

      "headers": {
        "user-agent": "got/9.5.1 (https://github.com/sindresorhus/got)",
        "authorization": "Bearer ************************************************",
        "accept": "application/vnd.docker.distribution.manifest.v2+json",
        "accept-encoding": "gzip, deflate"
      },

Are you able to try again using Restlet?

By the way, do you know if the Docker Compose instructions for Artifactory Docker setup work as simply as they describe? Are you running Artifactory as Docker-only or for other types of packages on the same server?

[mmkonrad]

Weird. Now I get 406, too (with restlet)
...with library/, amd64/ and without

Afaik serves our Artifactory as mirror for Docker registries, npm and maven

[rarkins]

Now I’m at a bit of a loss and wondering if I have the root cause wrong!

BTW I knew Artifactory can do many types of packages but wasn’t sure if the same server does everything or you end up running separate ones for each protocol.

[mmkonrad]

Shall we summarize what we know and what we have to find out?
I still haven't got into contact with our artifactory guys to get more information on what we have there...

[TKappatsch]

#3124

we were inspecting this behavior and found out there are two overlaying causes

1. using accept = 'application/vnd.docker.distribution.manifest.v2+json' is only needed when going for the manifest. If used when reading the Tag-List. Artifactory quits with http-status 406, while dockerhub.io seems to ignore the header.

2. when reading Tag-Lists for images without repo-name from artifactory things are getting weird. Depending on reading from a local or reading from remote repo, using no tag or lastest-tag or special-tag the the library-prefix is needed or not in different cases . We opened a ticket at JFrogs on this issue.
   So there is no way to be sure witch url has to be set. The lazy fix then is to try again when a 404 appears in getTags.

My Pull-Request fixes both obove and is testet against dockerhub und artifactory.

[rarkins]

@TKappatsch thanks. Both of these make logical sense. The implicit "library/" prefix is a historical convention for Docker Hub only so when Artifactory proxies it, you can never be sure.

[renovate-bot]

🎉 This issue has been resolved in version 14.3.20 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀