Custom datasource GCP authentication

[zzzpoint]

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

37.68.4

If you're self-hosting Renovate, select which platform you are using.

GitLab self-hosted

What is your question?

Hello,
I need to use custom datasource to pull a list of VM images from GCP. It works if I use curl:
curl -H "Authorization: Bearer $GCP_TOKEN" https://compute.googleapis.com/compute/v1/projects/[VALUE REDACTED]/global/images

So I put this in Revonate config:

    {
      "hostType": "docker",
      "matchHost": "northamerica-northeast1-docker.pkg.dev",
      "token": process.env.GCP_TOKEN,
    },
    {
      "hostType": "docker",
      "matchHost": "gcr.io",
      "token": process.env.GCP_TOKEN,
    },
    {
      "matchHost": "googleapis.com",
      "token": process.env.GCP_TOKEN
    },
  ],
  "customDatasources": {
    "vm-image": {
      "defaultRegistryUrlTemplate": "https://compute.googleapis.com/compute/v1/projects/[VALUE REDACTED]/global/images",
      "format": "json",
      "transformTemplates": [
        "{\"items\":[{\"labels\": $string(gitlab_tag)}]}"
      ]
    }
  }

I know my token is good, because it works for gcr.io and artifact registry, but for some reason I got 401 when checking images. Also I know permissions are good, because curl command above works. I tried to change host rules to exactly match url, partially match url - it didn't help.
Any advise, please? I'm hitting the wall for several days.

Thank you,
Viacheslav

Logs (if relevant)

Logs

DEBUG: GET https://compute.googleapis.com/compute/v1/projects/[VALUE REDACTED]/global/images = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=401 retryCount=0, duration=70) (repository=[VALUE REDACTED])
DEBUG: Failed to look up custom.vm-image package VM Image (repository=[VALUE REDACTED], packageFile=test/terragrunt.hcl, dependency=VM Image)

[github-actions[bot]]

Hi, please format any copy/pasted code or logs so they're readable.

You can find a Markdown code formatting guide here as well as some GitHub-specific information formatting code blocks here.

Thanks, the Renovate team

[zzzpoint]

Sorry for that, I updated code formatting in the initial post.

[rarkins]

The hostRules for googleapis.com looks correct. Once #25967 is part of a release, please upgrade as you will get a little more debug logging around hostRules and maybe an indication of whether the hostRules are input correctly.

BTW you wrote vaguely "Revonate config" which is not really descriptive enough. e.g. is it in a config.js? Are you sure it's being read?

[zzzpoint]

Thank you for the reply. Yes, it's config.js. I think it's been read, I see it in debug output:

DEBUG: File config
       "config": {
         "repositories": [
           "[VALUE REDACTED]"
         ],
         "hostRules": [
           {
             "hostType": "docker",
             "matchHost": "northamerica-northeast1-docker.pkg.dev",
             "token": "***********"
           },
           {"hostType": "docker", "matchHost": "gcr.io", "token": "***********"},
           {"matchHost": "googleapis.com", "token": "***********"}
         ],
   ...

[zzzpoint]

I updated Renovate to v37.74, I see DEBUG: hostRules: ... for different hosts like github, artifact registry, etc, but there is nothing for googleapis.com. The only relevant entry in the log is:
DEBUG: GET https://compute.googleapis.com/compute/v1/projects/prj-netw-npe-npda-common-01/global/images = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=401 retryCount=0, duration=66) (repository=nbd/dataplatform/gcp/datapltf-modules/datapltf-dataproc-module)
Looks like it doesn't even try to authenticate to googleapis.com for some reason.

[rarkins]

Yes, that implies that the token is probably not applied.

Can you grep your logs for every occurrence of googleapis.com or hostRules and then paste here what you find?

[zzzpoint]

Here is the log filtered by grep -E 'googleapis.com|hostRules' :

  "selfLink": "https://www.googleapis.com/compute/v1/projects/[VALUE REDACTED]/global/images"
         "hostRules": [
           {"matchHost": "googleapis.com", "token": "***********"}
             "defaultRegistryUrlTemplate": "https://compute.googleapis.com/compute/v1/projects/[VALUE REDACTED]/global/images",
         "hostRules": [
         "hostRules": [
           {"matchHost": "googleapis.com", "token": "***********"},
             "defaultRegistryUrlTemplate": "https://compute.googleapis.com/compute/v1/projects/[VALUE REDACTED]/global/images",
DEBUG: Setting global hostRules
DEBUG: Adding token authentication for northamerica-northeast1-docker.pkg.dev (hostType=docker) to hostRules
DEBUG: Adding token authentication for gcr.io (hostType=docker) to hostRules
DEBUG: Adding token authentication for googleapis.com (hostType=undefined) to hostRules
DEBUG: Adding token authentication for github.com (hostType=github) to hostRules
DEBUG: hostRules: authentication already set for gitlab.int.[VALUE REDACTED]
DEBUG: Adding token authentication for gitlab.int.[VALUE REDACTED] (hostType=gitlab) to hostRules
DEBUG: Setting global hostRules
DEBUG: Adding token authentication for northamerica-northeast1-docker.pkg.dev (hostType=docker) to hostRules
DEBUG: Adding token authentication for gcr.io (hostType=docker) to hostRules
DEBUG: Adding token authentication for googleapis.com (hostType=undefined) to hostRules
DEBUG: Adding token authentication for github.com (hostType=github) to hostRules
DEBUG: Adding token authentication for gitlab.int.[VALUE REDACTED] (hostType=gitlab) to hostRules
DEBUG: Reinitializing hostRules for repo
DEBUG: Clearing hostRules
DEBUG: Adding token authentication for northamerica-northeast1-docker.pkg.dev (hostType=docker) to hostRules
DEBUG: Adding token authentication for gcr.io (hostType=docker) to hostRules
DEBUG: Adding token authentication for googleapis.com (hostType=undefined) to hostRules
DEBUG: Adding token authentication for github.com (hostType=github) to hostRules
DEBUG: Adding token authentication for gitlab.int.[VALUE REDACTED] (hostType=gitlab) to hostRules
DEBUG: hostRules: applying Bearer authentication for gitlab.int.[VALUE REDACTED] (repository=[VALUE REDACTED])
DEBUG: Setting hostRules from config (repository=[VALUE REDACTED])
DEBUG: hostRules: authentication already set for api.github.com (repository=[VALUE REDACTED])
DEBUG: hostRules: no authentication for pypi.org (repository=[VALUE REDACTED])
DEBUG: GET https://compute.googleapis.com/compute/v1/projects/[VALUE REDACTED]/global/images = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=401 retryCount=0, duration=66) (repository=[VALUE REDACTED])
DEBUG: hostRules: Bearer token auth for https://northamerica-northeast1-docker.pkg.dev (repository=[VALUE REDACTED])
         "https://compute.googleapis.com/compute/v1/projects/[VALUE REDACTED]/global/images (GET,401)": 1,
         "compute.googleapis.com": {

Thank you.