Whitesource Renovate stopped working well

[suzuki-shunsuke]

What

Use Self hosted Renovate instead of Whitesource Renovate

Problem to solve

Currently, Self hosted Renovate often stops working due to API rate limiting.

https://app.renovatebot.com/dashboard#github/aquaproj/aqua-registry/1074940570

DEBUG: No repository returned
{
  "res": {
    "errors": [
      {
        "type": "RATE_LIMITED",
        "message": "API rate limit exceeded for installation ID ***."
      }
    ]
  }
}

Once GitHub App reaches API rate limiting, GitHub App stops working until we suspend and unsuspend the App.

https://github.com/apps/renovate

What I tried

To avoid API rate limiting, I tried several things.

• Restrict prConcurrentLimit

aqua-registry/renovate.json5

Lines 10 to 12 in bb7be1a

	branchConcurrentLimit: 10,
	prConcurrentLimit: 10,
	prHourlyLimit: 0,

• Restrict schedule

aqua-registry/renovate.json5

Lines 15 to 32 in bb7be1a

	packageRules: [
	// Separate packages with schedule.
	// Otherwise, Renovate doesn't work because there are too many dependencies.
	// https://github.com/renovatebot/renovate/discussions/21410
	// https://github.com/aquaproj/aqua-registry/issues/11363
	{
	matchPackagePatterns: ["^[a-g]"],
	schedule: ["* 0-7 * * *"],
	},
	{
	matchPackagePatterns: ["^[h-o]"],
	schedule: ["* 8-15 * * *"],
	},
	{
	matchPackagePatterns: ["^[^a-o]"],
	schedule: ["* 16-23 * * *"],
	},
	],

• Change rebaseWhen to conflicted

aqua-registry/renovate.json5

Line 14 in bb7be1a

rebaseWhen: "conflicted", // To decrease API call and avoid API rate limiting

But the issue couldn't be solved.

What I'll try

I'd like to try GitHub Actions.

https://github.com/renovatebot/github-action
https://docs.renovatebot.com/self-hosted-configuration

The experience of GitHub Actions is worse than Whitesource Renovate, but we have no choice.

[sheldonhull]

So I use the cli to invoke for Azure DevOps repos. The github app might still be possible for you with the caveat of modifying some behavior. I'd suggest that before flipping to using actions.

This may be a problem again in the future if the package size grows even with the self-hosted, so consider app behavior modification.

On the GitHub hosted app Mend checks each active repository roughly every three hours, if no activity has been seen before then (merged PRs, etc). source

Additionally the docs talk about only one merge occurs at a time due to needing to make sure conflicts don't occur.

These are just ideas because moving away from the app might result in a less optimal experience.

• Scheduling. I think it's perfectly legitimate to not have near realtime PR's being created for tooling. Consider throttling to a schedule like hourly, and make sure you setup prHourlyLimit: 0 to something reasonable to help protect against over loading your tokens.
• PR Creation, consider not-pending as it won't create the pr till the status checks pass. I prefer immediate, but maybe that would help. This might cut down dramatically on your api calls just by doing every 2-3 hours as a schedule instead of what might be every merge resulting in new PR's being created.
• Disable dependency dashboard. I love that feature, but with your scale I see it's truncated. I bet updating that issue also costs more api calls and not sure it's useful for you as it is.

After that, if it still persists, I'd also suggest opening a discussion on renovate's github since there's some interesting posts about caching and other general improvements. Maybe you are hitting a unique scale issue that will help them if they are making excessive calls that a schedule change doesn't fix.

quick observations on the status checks

This is just me reading through the check and giving some feedback, so unlikely to help with the main issue. I can move this off into a separate issue if you like some of the ideas.
Probably works great as is, but maybe there's something useful in here.

• The status checks run on every push that impacts the registry, right? If I'm reading the test.yaml correctly that means every single cli tool == full test runs. Maybe consider adding your tibdex/github-app-token that exports a cross job secret env variable that other tasks like earthly use IF those api calls go against the GITHUB_TOKEN that might use.
• First if you have your github status checks setup, you could cut out the need for gh merge via command line (only one call perhaps but still)... Enabling renovate's automerge and if the status checks pass and renovatebot-autoapproves then it will automerge with no extra code.
  In fact, as I'm typing I see this and while the github app token generator is pretty cool, I don't think you need this if you just the platform native automerge. This cuts out more calls by tooling. example from something i setup.
  The key is to just do branch protection with status checks passed and one approval required, which can be renovatebotapprove. I do like your approach, just giving you another idea.

[suzuki-shunsuke]

PR Creation

test is triggered by pull_request event, so I think the prcreation must be immediate (default) because until pull request is created status check hasn't completed.
To set prcreation: not-pending, we should change the trigger to push event, but I don't want to do it.

quick observations on the status checks

We face the API rate limiting of Renovate GitHub App, but this app is different from GitHub App we use in GitHub Actions.
So GitHub API call in GitHub Actions has nothing to do with this issue.

[suzuki-shunsuke]

In fact, as I'm typing I see this and while the github app token generator is pretty cool, I don't think you need this if you just the platform native automerge.

We set up our GitHub Actions based on the guide.
We know Renovate's platformAutomerge but we don't use it intentionally (ref).
Furthermore, platformAutomerge consumes Renovate GitHub App's API rate limiting so the situation gets worse.

[suzuki-shunsuke]

Disable dependency dashboard. I love that feature, but with your scale I see it's truncated. I bet updating that issue also costs more api calls and not sure it's useful for you as it is.

Good suggestion. I'll try to disable the dashboard. Thanks.

https://docs.renovatebot.com/configuration-options/#dependencydashboard

• chore(renovate): disable Dependency Dashboard to avoid API rate limiting #12295

I use Dependency Dashboard to trigger Awaiting Schedule when Renovate doesn't work for a while, so it may be inconvenient to disable the dashboard, but let's try.

[suzuki-shunsuke]

If my understanding is correct, even if we limit branchConcurrentLimit and prConcurrentLimit Renovate seems to check all dependencies, which consumes many API rate limiting unnecessarily.

[suzuki-shunsuke]

Limit prHourlyLimit to 50.

• chore(renovate): limit prHourlyLimit #12313

[suzuki-shunsuke]

We are working on updating packages by aqua-registry-updater instead of Renovate.

[suzuki-shunsuke]

aqua-registry-updater starts working!

https://github.com/aquaproj/aqua-registry/actions/workflows/update.yaml

https://github.com/aquaproj/aqua-registry/commits/main

[suzuki-shunsuke]

Memo

• TODO
  • Support ignore
    • chore: update grafana/xk6 v0.8.1 to v0.9.1 #12427
  • check error_message and no_asset
  • Ignore downgrade: chore: update rancher/cli v2.7.0 to v2.6.11 #12464
• chore: update neovim/neovim v0.9.0 to stable #12456
• chore: update linkerd/linkerd2 stable-2.12.0 to edge-23.5.2 #12449
• latest: chore: update kubemq-io/kubemqctl v3.7.2 to latest #12443
• golang/go chore: update golang/go 1.20.4 to go1.20.4 #12415
• disable auto-merge
  • chore: update mongodb/mongodb-atlas-cli/mongocli mongocli/v1.27.0 to mongocli/v1.29.0 #12453 has prefix
• 🤔
  • chore: update kubernetes/kubectl-convert v1.27.2 to v1.25.10 #12446
  • chore: update kubernetes/kubectl v1.27.2 to v1.25.10 #12445

no match

• tamasfe/taplo/full
• temporalio/temporal
• sigstore/sget
• gokcehan/lf
• golang/vuln/govulncheck
• golang.org/x/perf/cmd/benchstat
• google/pprof

latest not found

• weaveworks/tf-controller
• suzuki-shunsuke/checkout-merged-branch-with-ci-info
• suzuki-shunsuke/ci-renovate-config-validator
• suzuki-shunsuke/discussion-slack-notifier
• rancher/kim
• siderolabs/conform
• aquaproj/aqua-registry-updater

ip restriction

• bridgecrewio/yor

[suzuki-shunsuke]

• chore: stop updating pkgs by Renovate #12533

[suzuki-shunsuke]

Resolved.

[sheldonhull]

This deserves a blog post. Great example of creative solving of issues with Go and a fresh look. Would love to hear more when you get time.