GITHUB_COM_TOKEN will not be used to download presets

[Tanemahuta]

How are you running Renovate?

Self-hosted

Please select which platform you are using if self-hosting.

GitLab self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

26.13.8

Describe the bug

In the documentation, it is mentioned to set GITHUB_COM_TOKEN to a private access token to get around the problems with rate limits. This does not work.
I suspect the following line:
https://github.com/renovatebot/renovate/blob/main/lib/config/presets/github/index.ts#L13

No options are set here.

Relevant debug logs

Logs

{"name":"renovate","hostname":"<redacted>","pid":39,"level":20,"logContext":"4O8RqOsYX","err":{"name":"HTTPError","timings":{"start":1630400096112,"socket":1630400096180,"lookup":1630400096180,"connect":1630400096180,"secureConnect":1630400096183,"upload":1630400096184,"response":1630400096189,"end":1630400096190,"phases":{"wait":68,"dns":0,"tcp":0,"tls":3,"request":1,"firstByte":5,"download":1,"total":78}},"message":"Response code 403 (rate limit exceeded)","stack":"HTTPError: Response code 403 (rate limit exceeded)\n    at Request.<anonymous> (/usr/src/app/node_modules/got/dist/source/as-promise/index.js:117:42)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)","options":{"headers":{"user-agent":"RenovateBot/26.10.0 (https://github.com/renovatebot/renovate)","accept":"application/vnd.github.v3+json","accept-encoding":"gzip, deflate, br"},"url":"https://api.github.com/repos/whitesource/merge-confidence/contents/beta.json","username":"","password":"","method":"GET","http2":false},"response":{"statusCode":403,"statusMessage":"rate limit exceeded","body":{"message":"API rate limit exceeded for <redacted>. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"},"headers":{"date":"Tue, 31 Aug 2021 08:54:56 GMT","server":"Varnish","strict-transport-security":"max-age=31536000; includeSubdomains; preload","x-content-type-options":"nosniff","x-frame-options":"deny","x-xss-protection":"1; mode=block","content-security-policy":"default-src 'none'; style-src 'unsafe-inline'","access-control-allow-origin":"*","access-control-expose-headers":"ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-RateLimit-Used, X-RateLimit-Resource, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset","content-type":"application/json; charset=utf-8","referrer-policy":"origin-when-cross-origin, strict-origin-when-cross-origin","x-github-media-type":"github.v3; format=json","x-ratelimit-limit":"60","x-ratelimit-remaining":"0","x-ratelimit-reset":"1630400264","x-ratelimit-resource":"core","x-ratelimit-used":"60","content-length":"278","x-github-request-id":"2761:32CC:413FE9A:4297457:612DEE60","connection":"close"},"httpVersion":"1.1"}},"msg":"GitHub failure: rate limit","time":"2021-08-31T08:54:56.193Z","v":0}

Have you created a minimal reproduction repository?

No reproduction repository

[rarkins]

Please provide the full (redacted) logs from startup to error

[Tanemahuta]

{"config":{"autodiscover":true,"autodiscoverFilter":"**/*","dockerImagePrefix":"<redacted>/renovate","force":{"gitLabAutomerge":true,"automerge":true},"onboardingConfig":{"extends":["config:base","group:unitTest","group:definitelyTyped","group:linters","group:monorepos",":prConcurrentLimit20",":prHourlyLimitNone",":automergeMinor","docker:enableMajor"]},"customEnvVariables":{"HTTP_PROXY":"http://<redacted>:8080","HTTPS_PROXY":"http://<redacted>:8080","NO_PROXY":"<redacted>","http_proxy":"http://<redacted>:8080","https_proxy":"http://<redacted>:8080","no_proxy":"<redacted>","GITHUB_COM_TOKEN":"<redacted>"},"gitLabAutomerge":true,"automerge":true},"msg":"File config"}
{"config":{},"msg":"CLI config"}
{"config":{"hostRules":[{"hostType":"github","matchHost":"github.com","token":"***********"}],"extends":["github>whitesource/merge-confidence:beta"],"repositoryCache":"true","baseDir":"<redacted>/renovate-bot/renovate","logFile":"renovate-log.ndjson","logFileLevel":"debug","onboarding":true,"onboardingConfig":{"$schema":"https://docs.renovatebot.com/renovate-schema.json","extends":["config:base"]},"requireConfig":false,"optimizeForDisabled":true,"platform":"gitlab","endpoint":"https://<redacted>/api/v4","token":"***********","gitAuthor":"Renovate Bot <no-reply@<redacted>>","ignorePrAuthor":true},"msg":"Env config"}
{"config":{"autodiscover":true,"autodiscoverFilter":"**/*","dockerImagePrefix":"<redacted>/renovate","force":{"gitLabAutomerge":true,"automerge":true},"onboardingConfig":{"extends":["config:base"],"$schema":"https://docs.renovatebot.com/renovate-schema.json"},"customEnvVariables":{"HTTP_PROXY":"http://<redacted>:8080","HTTPS_PROXY":"http://<redacted>:8080","NO_PROXY":"<redacted>","http_proxy":"http://<redacted>:8080","https_proxy":"http://<redacted>:8080","no_proxy":"<redacted>","GITHUB_COM_TOKEN":"<redacted>"},"gitLabAutomerge":true,"automerge":true,"hostRules":[{"hostType":"github","matchHost":"github.com","token":"***********"}],"extends":["github>whitesource/merge-confidence:beta"],"repositoryCache":"true","baseDir":"<redacted>/renovate-bot/renovate","logFile":"renovate-log.ndjson","logFileLevel":"debug","onboarding":true,"requireConfig":false,"optimizeForDisabled":true,"platform":"gitlab","endpoint":"https://<redacted>/api/v4","token":"***********","gitAuthor":"Renovate Bot <no-reply@<redacted>>","ignorePrAuthor":true},"msg":"Combined config"}
{"msg":"Adding trailing slash to endpoint"}
{"msg":"GitLab version is: 13.12.10"}
{"msg":"Using configured gitAuthor (Renovate Bot <no-reply@<redacted>>)"}
{"msg":"Adding token authentication for <redacted> to hostRules"}
{"msg":"Using configured baseDir: <redacted>/renovate-bot/renovate"}
{"msg":"Using cacheDir: <redacted>/renovate-bot/renovate/cache"}
{"msg":"Initializing Renovate internal cache into <redacted>/renovate-bot/renovate/cache/renovate/renovate-cache-v1"}
{"msg":"Commits limit = null"}
{"err":{"name":"HTTPError","timings":{"start":1630400096112,"socket":1630400096180,"lookup":1630400096180,"connect":1630400096180,"secureConnect":1630400096183,"upload":1630400096184,"response":1630400096189,"end":1630400096190,"phases":{"wait":68,"dns":0,"tcp":0,"tls":3,"request":1,"firstByte":5,"download":1,"total":78}},"message":"Response code 403 (rate limit exceeded)","stack":"HTTPError: Response code 403 (rate limit exceeded)\n    at Request.<anonymous> (/usr/src/app/node_modules/got/dist/source/as-promise/index.js:117:42)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)","options":{"headers":{"user-agent":"RenovateBot/26.10.0 (https://github.com/renovatebot/renovate)","accept":"application/vnd.github.v3+json","accept-encoding":"gzip, deflate, br"},"url":"https://api.github.com/repos/whitesource/merge-confidence/contents/beta.json","username":"","password":"","method":"GET","http2":false},"response":{"statusCode":403,"statusMessage":"rate limit exceeded","body":{"message":"API rate limit exceeded for <redacted>. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"},"headers":{"date":"Tue, 31 Aug 2021 08:54:56 GMT","server":"Varnish","strict-transport-security":"max-age=31536000; includeSubdomains; preload","x-content-type-options":"nosniff","x-frame-options":"deny","x-xss-protection":"1; mode=block","content-security-policy":"default-src 'none'; style-src 'unsafe-inline'","access-control-allow-origin":"*","access-control-expose-headers":"ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-RateLimit-Used, X-RateLimit-Resource, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset","content-type":"application/json; charset=utf-8","referrer-policy":"origin-when-cross-origin, strict-origin-when-cross-origin","x-github-media-type":"github.v3; format=json","x-ratelimit-limit":"60","x-ratelimit-remaining":"0","x-ratelimit-reset":"1630400264","x-ratelimit-resource":"core","x-ratelimit-used":"60","content-length":"278","x-github-request-id":"2761:32CC:413FE9A:4297457:612DEE60","connection":"close"},"httpVersion":"1.1"}},"msg":"GitHub failure: rate limit"}
{"url":"https://api.github.com/repos/whitesource/merge-confidence/contents/beta.json","msg":"Failed to retrieve beta.json from repo"}
{"preset":"github>whitesource/merge-confidence:beta","err":{"message":"dep not found","stack":"Error: dep not found\n    at fetchJSONFile (/usr/src/app/node_modules/renovate/lib/config/presets/github/index.ts:33:11)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at Object.fetchPreset (/usr/src/app/node_modules/renovate/lib/config/presets/util.ts:47:19)\n    at getPreset (/usr/src/app/node_modules/renovate/lib/config/presets/index.ts:180:22)\n    at Object.resolveConfigPresets (/usr/src/app/node_modules/renovate/lib/config/presets/index.ts:255:27)\n    at validatePresets (/usr/src/app/node_modules/renovate/lib/workers/global/index.ts:77:5)\n    at Object.start (/usr/src/app/node_modules/renovate/lib/workers/global/index.ts:91:5)\n    at /usr/src/app/node_modules/renovate/lib/renovate.ts:16:22"},"msg":"Preset fetch error"}
{"validationError":"Cannot find preset's package (github>whitesource/merge-confidence:beta)","msg":"Throwing preset error"}
{"name":"renovate","hostname":"<redacted>","pid":39,"level":60,"logContext":"4O8RqOsYX","err":{"message":"config-presets-invalid","stack":"Error: config-presets-invalid\n    at validatePresets (/usr/src/app/node_modules/renovate/lib/workers/global/index.ts:79:11)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at Object.start (/usr/src/app/node_modules/renovate/lib/workers/global/index.ts:91:5)\n    at /usr/src/app/node_modules/renovate/lib/renovate.ts:16:22"},"msg":"Fatal error: config-presets-invalid"}
{"msg":"Renovate exiting"}
{"loggerErrors":[{"name":"renovate","level":60,"logContext":"4O8RqOsYX","err":{"message":"config-presets-invalid","stack":"Error: config-presets-invalid\n    at validatePresets (/usr/src/app/node_modules/renovate/lib/workers/global/index.ts:79:11)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at Object.start (/usr/src/app/node_modules/renovate/lib/workers/global/index.ts:91:5)\n    at /usr/src/app/node_modules/renovate/lib/renovate.ts:16:22"},"msg":"Fatal error: config-presets-invalid"}],"msg":"Renovate is exiting with a non-zero code due to the following logged errors"}

[rarkins]

Do you know where the github>whitesource/merge-confidence:beta preset is coming from?

[Tanemahuta]

Honestly, I do not have a clue.
This is my configuration:

module.exports = {
    // Autodiscover all projects
    autodiscover: true,
    autodiscoverFilter: "**/*",
    // Make sure we use our dockerhub mirror
    dockerImagePrefix: "<redacted>/renovate",
    // Enforce auto-merging changes
    force: {gitLabAutomerge: true, automerge: true},
    onboardingConfig: {
        extends: [
            'config:base',
            'group:unitTest',
            'group:definitelyTyped',
            'group:linters',
            'group:monorepos',
            ':prConcurrentLimit20',
            ':prHourlyLimitNone',
            ':automergeMinor',
            'docker:enableMajor'
        ]
    },
    hostRules: [{
        "matchHost": "api.github.com",
        "token": process.env.GITHUB_COM_TOKEN
    }],
    customEnvVariables: {
        "HTTP_PROXY": process.env.HTTP_PROXY,
        "HTTPS_PROXY": process.env.HTTPS_PROXY,
        "NO_PROXY": process.env.NO_PROXY,
        "http_proxy": process.env.HTTP_PROXY,
        "https_proxy": process.env.HTTPS_PROXY,
        "no_proxy": process.env.NO_PROXY,
        "GITHUB_COM_TOKEN": process.env.GITHUB_COM_TOKEN,
        "GITHUB_TOKEN": process.env.GITHUB_TOKEN
    },
    gitLabAutomerge: true,
    automerge: true
};

I guess one of the presets is referencing it.

[rarkins]

@viceice can you figure it out? I don't see it in our source code:

[viceice]

I guess he's using our https://gitlab.com/renovate-bot/renovate-runner templates ?

[Tanemahuta]

Aye. That's where it' coming from.
Still... I might be wrong, but...the credentials should be configured before using a provider to load a prefix, isn't it?

[pataar]

Yep, experiencing the same problem. Rate limit has been hit because of the unauthorised calls. Also using the gitlab renovate-runner.

Disabling the extends works for now. Although the merge-confidence part is pretty nice. So it'd be nice to use it.

[rarkins]

I guess we need to do a hostRules.add() call before resolving presets

[pataar]

Just a minor update, the preset resolvement also ignores hostRules that are set explicitly.

[xoxys]

Its not only the preset fetching, somehow also dependency checks are running into zthe rate limit now (as unauthenticated "x-ratelimit-limit": "60":

"message": "Response code 403 (rate limit exceeded)",
         "stack": "HTTPError: Response code 403 (rate limit exceeded)\n    at Request.<anonymous> (/usr/src/app/node_modules/got/dist/source/as-promise/index.js:117:42)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)",
         "options": {
           "headers": {
             "user-agent": "RenovateBot/26.19.1 (https://github.com/renovatebot/renovate)",
             "accept": "application/vnd.github.v3+json",
             "accept-encoding": "gzip, deflate, br"
           },
           "url": "https://api.github.com/repos/AlecAivazis/survey/tags?per_page=100",
           "hostType": "github-tags",
           "username": "",
           "password": "",
           "method": "GET",
           "http2": false
         },
         "response": {
           "statusCode": 403,
           "statusMessage": "rate limit exceeded",
           "body": {
             "message": "API rate limit exceeded for 78.47.51.41. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
             "documentation_url": "https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"
           },
           "headers": {
             "date": "Mon, 06 Sep 2021 12:33:31 GMT",
             "server": "Varnish",
             "strict-transport-security": "max-age=31536000; includeSubdomains; preload",
             "x-content-type-options": "nosniff",
             "x-frame-options": "deny",
             "x-xss-protection": "1; mode=block",
             "content-security-policy": "default-src 'none'; style-src 'unsafe-inline'",
             "access-control-allow-origin": "*",
             "access-control-expose-headers": "ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-RateLimit-Used, X-RateLimit-Resource, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset",
             "content-type": "application/json; charset=utf-8",
             "referrer-policy": "origin-when-cross-origin, strict-origin-when-cross-origin",
             "x-github-media-type": "github.v3; format=json",
             "x-ratelimit-limit": "60",
             "x-ratelimit-remaining": "0",
             "x-ratelimit-reset": "1630933265",
             "x-ratelimit-resource": "core",
             "x-ratelimit-used": "60",
             "content-length": "277",
             "x-github-request-id": "A8A8:0624:4B8685:4D9630:61360A9B",
             "connection": "close"
           },
           "httpVersion": "1.1"
         }
       }

[xoxys]

Something is really strange... With GITHUB_COM_TOKEN set everything was working a while ago, now dependency lookups e.g. for go resources (see my last comment) are unauthenticated and results in rate limits. The GITHUB_COM_TOKEN var adds {"hostType": "github", "matchHost": "github.com", "token": "***********"} to the hostrules, but that is not enough... I have added a broader hostrule manually now {"matchHost": "api.github.com", "token": "********"} and now the rate limiting issue is gone and everything works again as expected. But I guess that's just a workaround, as setting GITHUB_COM_TOKEN should "just work" 🤷‍♂️

[viceice]

@xoxys Your issue will be fixed in #11605