Lazy validation of versions

[rjatkins]

What would you like Renovate to be able to do?
I'm running a hosted renovate on a monoversioned multi-module maven project where it can see over 1000 versions for some dependencies, and checking their versions is taking an extraordinarily long time, even though response times are averaging around 100ms. This appears to be due to a low, hard coded concurrency limit on the number of HTTP HEAD requests run at once, coupled with executing a request per available version. The vast majority of these checks are redundant, because none of these versions will be used for an upgrade.

Renovate should avoid as many network calls as it can, and only validate versions for packages if these are allowed by allowedVersions filters, and if they would be selected for a major/minor/patch upgrades.

Describe the solution you'd like
Renovate should defer version validation checks until after passing the versions through the allowedVersions filters, and after a subset of versions have been selected for upgrades. It should stop checking for validity after a valid version that is a suitable upgrade candidate is found.

Describe alternatives you've considered
Renovate could possibly simply not do validation checks for versions in maven, and assume that a pom will show up eventually. The validation checks also don't ensure that the actual artifact required exists. For most maven dependencies, this is a jar without any classifier, but there are a lot of possible variations. A config flag to avoid this would speed up renovate run times for the project I work on dramatically.

Additional context
Relevant runtimes I'm seeing today with renovate@21.16.0:

DEBUG: Repository timing splits (milliseconds) (repository=****)
       "splits": {"init": 31462, "extract": 23950, "lookup": 22258530, "update": 390100},
       "total": 22704624
DEBUG: http statistics (repository=****)
       "hostStats": [
         "api.github.com, 507 requests, 210ms average",
         "gitlab.ow2.org, 5 requests, 525ms average",
         "<artifactory maven server>, 295692 requests, 116ms average",
         "registry.npmjs.org, 152 requests, 274ms average",
         "<bitbucket server>, 20 requests, 808ms average"
       ],
       "totalRequests": 296376
INFO: Repository finished (repository=****)
       "durationMs": 22704624

[rjatkins]

Related to this, implementing #6573 would help extend the usefulness of the cache when checking for updates in one maven module takes long enough that the cache expires before the next module can be checked. I wouldn't mind if the maven-metadata.xml cache defaulted to 1 hour expiry, since that would avoid a lot of other work.

Also, implementing #6440 would allow us to avoid version checks for mono-versioned multi-module maven projects, skipping any validation for all the milestones and betas for our own project, reducing the number of artifacts considered by a healthy margin. I can't easily disable upgrade checks on these modules, as some artifacts outside the repo share the same groupId, and there are enough modules involved that manually listing them all is no fun either.

[rarkins]

We've been going around in circles a little bit on this. It seems there's a lot of inconsistency in Maven hosting, which is what led us to make this check in the first place. i.e. there'd be versions in the metadata xml that actually didn't exist. I also don't object to a user-configurable option to bypass individual release checks and just assume they're all ok.

BTW it was our intention that these checks would be cached for a long time. Are you seeing a dramatic reduction in requests on subsequent runs?

CC @zharinov

[rjatkins]

The way our artifactory is configured, it's lazily mirroring packages from other repos, plus hosting artifacts of its own. If we keep pulling the same files, it should eventually have a local copy every time we ask for every version that's properly defined. But you're absolutely right that many versions might not have pom files, or other parts of the artifact that are supposed to be there.

That doesn't change that for the vast majority of versions that you'll see, none of that matters: only the recent releases, newer than the version in use need to be checked, and then only for the fix versions renovate would select for upgrades.

Regarding next run speed: For this particularly slow repo, we've been using an ephemeral instance for all runs so far, and running it at a schedule of once a day. As far as I can see, none of the caches last longer than that, apart from the 3 day cache in lib/manager/bazel/update. Very few survive the 6 hours it takes to complete the first run. However, I'll see about ensuring the cache dir survives between builds, and changing the build to run every 8 hours, or so, and report back any speed improvements, and artifactory network traffic changes in particular.

[rarkins]

I added an experimental workaround in 5b24943

If you define RENOVATE_EXPERIMENTAL_NO_MAVEN_POM_CHECK in your env then it should bypass the pom checks. Let me know if this works well for you.

[rjatkins]

Okay, I've enabled the experimental flag, and I have some confusing results:

DEBUG: Repository timing splits (milliseconds) (repository=****)
       "splits": {"init": 16783, "extract": 25628, "lookup": 20064815},
       "total": 20109272
DEBUG: http statistics (repository=****)
       "hostStats": [
         "api.github.com, 516 requests, 158ms average",
         "gitlab.ow2.org, 5 requests, 428ms average",
         "<artifactory maven server>, 1440 requests, 5009ms average",
         "registry.npmjs.org, 152 requests, 477800ms average",
         "<bitbucket server>, 2 requests, 389ms average"
       ],
       "totalRequests": 2115
INFO: Repository finished (repository=****)
       "durationMs": 20109272

So the number of artifactory requests are down by 300 times, but actually getting all the maven-metadata.xml seems to be particularly expensive for artifactory (5 seconds per file!), which makes me think we might be better off querying artifactory's rest api for available versions instead. Additionally, I'm now hitting npmjs rate limiting, so the total build time is still over 5 hours. I'll try to reconfigure my builds to also use our artifactory mirror for that to further speed up the builds. I'm really glad I'm not seeing github rate limiting here, though that may just be a matter of time. I'm not sure if there's any way to avoid that...

Adding to my build woes: the caching plugin I'm trying to use only works on green builds, and renovate is almost always red for me, due to "Unexpected end of JSON input" errors from got 9.x.

[rarkins]

Additionally, I'm now hitting npmjs rate limiting,

What leads you to conclude that? I've never noticed npmjs rate limiting including in the hosted app which performs millions of daily requests. Something obviously is very weird with the 8 minutes per request though - I don't even know how that's possible. 152 requests is not much at all.

[rjatkins]

My company happens to hit npm pretty hard from a handful of addresses, across dozens of different builds, which npm detects and rate limits. We added a proxy for it to try to avoid the problem. I guess this hasn't been rolled out to enough of our lockfiles for it to resolve everything though. The build logs for renovate, while it's hitting the npm repo directly, are full of logs like this:

WARN: Retrying npm error (repository=****)
       "pkgUrl": "https://registry.npmjs.org/@storybook%2Faddons",
       "errName": "ParseError
       "delaySeconds": 21

I suspect that all the retries are not being counted in the request stats, but the delays are? That would explain why they're averaging nearly 8 mins per request. My next run which should be reconfigured to use our npm proxy is still going, but I'll let you know if that speeds things up further.

[HonkingGoose]

@rarkins I'm not sure what priority to give this one. Also did we figure out what we need already or not? It still labeled status:requirements after all this time.

[rarkins]

Marking it as low until we have a better idea of what's needed