Cache pom.xml content of non-SNAPSHOT versions to improve Maven cache efficiency

[Lucas-C]

What would you like Renovate to be able to do?
We have around 350 to 450 -SNAPSHOT versions of a dozen internal artifacts,
and it takes Renovate 2 hours to analyze a single git repository containing a pom.xml with dependencies to those artifacts.
Hence, we would like Renovate cache mechanism to be improved in order to limit numerous pom.xml downloads & parsing.

Describe the solution you'd like
Based on datasource/maven/index.ts code source, it looks like currently Renovate retrieve maven-metadata.xml files in order to list all existing versions of an artifact. It also uses a cache with a TTL of 10min.

We suggest to improve the caching mechanism in order to avoid unncessary HTTP calls.
This could be done by using the <lastUpdated> field of maven-metadata.xml to only download pom.xml files if they were updated since Renovate last downloaded them and stored them in cache.
The idea would be to cache all non-SNAPSHOT versions pom.xml content.

Describe alternatives you've considered
Our Renovate instance use JFrog Artifactory as its main registry.
We considered playing with its zone / repository system to limit the number of artifacts listed,
or even simply purging old -SNAPSHOT versions.
This may be a valid workaround for us on short term, but I thought it may still be worth suggesting this feature.

[rarkins]

I would be very happy to receive any PR that helps improve this experience for yourself and others. If so, I recommend we discuss the choices of approaches here before you start coding, just to make sure the final implementation is acceptable.

First of all I'd like to make sure I understand what is causing so many requests. Is it because you have a lot of dependencies, or because certain dependencies have a lot of available versions available? i.e. if we have an O(n) problem I want to understand which n it is.

I'd also like you to elaborate a bit more about your caching idea, as I'm not sure I understand it.

FYI I also created #6573 as I think that's a separate approach that may help too.

[Lucas-C]

First of all I'd like to make sure I understand what is causing so many requests. Is it because you have a lot of dependencies, or because certain dependencies have a lot of available versions available?

It's because certain dependencies have a lot of available versions available.
Also, some of them are SNAPSHOT versions, meaning they can be overriden, contrary to release versions.

On second though the <lastUpdated> field won't help much with caching,
as it describes the maven-metadata.xml, and tells nothing about the artifacts pom.xml "last update time".

Currently caching happens in the getVersionsFromMetadata function, and only maven-metadata.xml content is cached, for 10min (correct me if I'm wrong).

To improve caching, I suggest to introduce some pom.xml content cache for non-SNAPSHOT versions, with a much longer TTL (several days, ideally configurable).

I'm going to rename this issue name accordingly.

[rarkins]

I added an experimental workaround in 5b24943

If you define RENOVATE_EXPERIMENTAL_NO_MAVEN_POM_CHECK in your env then it should bypass the pom checks. Let me know if this works well for you.

[Lucas-C]

Awesome !
I won't be able to test this in the next days as I'm on holidays, but thank you !

[Lucas-C]

This really improved our processing time !

This graph displays our processing time per repo (we extract this data after execution by parsing the resulting --log-file).
Vertical unit is: 1 line = 500s

[rarkins]

@zharinov let's work out how to address this. First of all, should the metadata file always be "correct", and the only way it's not correct is due to some type of mistake in the registry's data?

Even if it's only due to a mistake, we then need to evaluate if it happens often enough that we should protect against it anyway (like we have tried to do). Or if for example we think it's ok to require users to fix it themselves with allowedVersions restrictions (some of which we could put into our default presets if it's public packages).

Finally, if we still need to protect against it, we need to work out the best approach. One example is "lazy" verification as suggested in #6591. This would mean a significant refactoring of our lookup/evaluate logic so we'd do something like this instead:

• getReleases() returns non-validated results
• Our lookup processing would do filtering to reduce the list substantially (e.g. only newer, only stable, etc)
• We'd then need to call a new datasource method for "validation" of the filtered versions only, which could possibly remove some more

[zharinov]

Seems like this is solved