Repology integration didn't work anymore #20152
Replies: 30 comments 3 replies
-
Logs
|
Beta Was this translation helpful? Give feedback.
-
Logs next
|
Beta Was this translation helpful? Give feedback.
-
Logs next
|
Beta Was this translation helpful? Give feedback.
-
Same here on e.g. https://github.com/swissgrc/docker-azure-pipelines-renovate When accessing the URL which fails (e.g. https://repology.org/api/v1/project/findutils) in a browser it works for me. |
Beta Was this translation helpful? Give feedback.
-
Repology lookups also still work for me on a self hosted instance running Renovate 32.216.0. Maybe this is caused by some IP blocking for the GitHub hosted app on Repology side. |
Beta Was this translation helpful? Give feedback.
-
Hi there, Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible. Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this. To get started, please read our guide on creating a minimal reproduction. We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment. Good luck, The Renovate team |
Beta Was this translation helpful? Give feedback.
-
Can someone create a one dependency reproduction? |
Beta Was this translation helpful? Give feedback.
-
@rarkins Does this work for you: https://github.com/pascalberger/renovate-18144 ? |
Beta Was this translation helpful? Give feedback.
-
Forked into our |
Beta Was this translation helpful? Give feedback.
-
It looks like the renovate github app is getting a 403 from Repology for the same urls which I am able to resolve from my own system. Definitely appears to be a change in behavior based on the source IP. I've had to disable repology integration for my renovate app as the failure to resolve fails the renovate build. |
Beta Was this translation helpful? Give feedback.
-
The Repology service is blocking the hosted app's user-agent string. |
Beta Was this translation helpful? Give feedback.
-
@AMDmi3 is there anything we can do to fix to unblock the app? Apologies if we caused an overload at any point in the past. |
Beta Was this translation helpful? Give feedback.
-
I've started to block renovate instances doing more than 1RPS. For unblock, please make sure to configure a corresponding delay between requests and tell me IP address/user agent of the instance. |
Beta Was this translation helpful? Give feedback.
-
@AMDmi3 thank you for your time replying, and apologies again for any overloads caused. Do you mean that you have unblocked the user agent string, or do I misunderstand? You may recall we didn't fully implement RPS limiting, instead we made sure Renovate limits itself to 1 concurrent request. The issue for RPS is #6359 which we'll try to prioritize |
Beta Was this translation helpful? Give feedback.
-
I need a user agent and IP to unblock as I've blocked a specific bunch of instances.
Thank you! |
Beta Was this translation helpful? Give feedback.
-
@AMDmi3 it looks like you've blocked the hosted Renovate app's user agent completely. The hosted app uses ephemeral EC2 instances and it looks they're always blocked, which implies it's a user agent string blocking and not IP-based. |
Beta Was this translation helpful? Give feedback.
-
I've added config to disable the |
Beta Was this translation helpful? Give feedback.
-
Was it |
Beta Was this translation helpful? Give feedback.
-
@AMDmi3 @rarkins any updates here? We've been unable to use the renovate repology integration for the last month. From the last comment by @AMDmi3 it sounds like it has been unbanned. Are we ready to re-enable the integration? |
Beta Was this translation helpful? Give feedback.
-
Sorry, I still had a server-side block of my own to try to avoid querying the Repology API when the ban was in place. I will remove it tomorrow at a time where I can monitor its reintroduction. We now also have a requests per second limit capability which I will enable for Repology (maybe in the Renovate source code itself by default). |
Beta Was this translation helpful? Give feedback.
-
I'm running Renovate Self-hosted via Github Actions (using version
@AMDmi3 I appreciate any concerns that Renovate is increasing the load on Repology, but I'm not sure how this can be resolved when not using Renovate's Github App (i.e. my requests will have a different Github App ID). Is there a reasonable RequestsPerSecond which could be configured to re-gain access? @rarkins does this essentially make Repology an unusable source for anyone running Renovate self-hosted? If that's the case, would it be useful to have a flag to disable/enable Repology as a data source in the self-hosted configuration - this would avoid existing renovate.json configurations that use Repology from blocking any other repositories being updated. |
Beta Was this translation helpful? Give feedback.
-
Repology datasource should not be enabled by default for anyone self-hosted. If rate limits for self-hosted users make it unreliable then we could/should note it in documentation, but I don't think it makes sense to add a feature to disable a datasource which itself isn't turned on unless you configure it. You could easily write a package rule to disable the datasource but in that case better to remove the regex manager which is leading to it. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
We had the same issue with timeouts for Repology recently on a self-hosted Renovate instance running in an Azure scale set, which run fine for several months before. I can try to find out and report IP address next time it happens. |
Beta Was this translation helpful? Give feedback.
-
Note that I've banned a lot of Microsoft networks lately, because of bots disguising themselves as normal user agents |
Beta Was this translation helpful? Give feedback.
-
@rarkins I'm not sure I understand what you mean by this - we run Renovate against a number of internal repositories where each team manages their own renovate.json per repository, but the Renovate "bot" is managed by a central IT team. Our teams often use Repology as a datasource (via a regex manager), and we would need to update a number of
@AMDmi3 Ah.. this is probably the issue then.. we're using Renovate via the Github Action which means the IP is not fixed and will be shared by anyone else using Renovate in this manner (or anyone accessing Repology in another way via Github Actions). I think Renovate supports setting an explicit User Agent - is there a particular User Agent we could use to avoid the blocks which are applied in this manner? |
Beta Was this translation helpful? Give feedback.
-
I've converted this to a discussion as it does not appear to be a Renovate bug |
Beta Was this translation helpful? Give feedback.
-
I've unbanned microsoft networks. Will have to hunt down the violating bot or ban it by UA instead of IP.
Changing user agent won't help against IP ban. |
Beta Was this translation helpful? Give feedback.
-
@AMDmi3 hello! First of all, thanks for repology. Is an awesome tool. How can I send you my organization's ips? We would like to see if we can be unbanned. Thanks for your help! |
Beta Was this translation helpful? Give feedback.
-
How are you running Renovate?
Mend Renovate hosted app on github.com
If you're self-hosting Renovate, tell us what version of Renovate you run.
No response
If you're self-hosting Renovate, select which platform you are using.
No response
If you're self-hosting Renovate, tell us what version of the platform you run.
No response
Was this something which used to work for you, and then stopped?
It used to work, and then stopped
Describe the bug
On dependency on reposobogy I get the following warn message
At the end I get the following info messages:
And the job fail with the result external-host-error
Reproducible here: https://github.com/camptocamp/c2cciutils
Relevant debug logs
Logs
see belowHave you created a minimal reproduction repository?
No reproduction, but I have linked to a public repo where it occurs
Beta Was this translation helpful? Give feedback.
All reactions