feat(maven): lookup parent homepage/scm information from pom

[twendelmuth]

Changes:

Changed the way maven is looking up sourceUrl & homepage. It will now also traverse up to the parents to look for this information if not given in the original dependency.
Also updated sbt to make use of the maven dependency lookup instead of trying it's own implementation.
This also led to me taking over the sbt parts of sourceUrl escaping for the maven implementation:

        const sourceUrl = pomXml.valueWithPath('scm.url');
        if (sourceUrl) {
          result.sourceUrl = sourceUrl
            .replace(/^scm:/, '')
            .replace(/^git:/, '')
            .replace(/^git@github.com:/, 'https://github.com/')
            .replace(/\.git$/, '');
        }

This has been extended to also support <url>git://github.com/child-scm/child</url> vs previously only supported <url>git://github.com:child-scm/child</url>. I had seen this in some poms and thought it makes sense to include this (reference poms are commented in the test poms)

Context:

In the java world parent dependency declarations are quiet common. A maven pom.xml will always inherit it's parents properties if they're not overridden in the original pom.xml. Since this is the case we're losing information by not parsing this information from the given parents.
The downside I see for this is, that we could end up with sourceUrl/homepage of an external framework if that's used as parent and no own scm-url/url information was provided in the pom.xml - spring-boot-starter-parent comes to mind here. I'd however assume that this is mainly an issue that internal libaries might be facing and can easily be mitigated by declaring correct properties for scm-url/url.

I wasn't able to test how the gradle implementation handles this since I didn't manage to get this running locally :-/

Closes #6742

PR comparisons

dependency: current code pr -> new code pr (potential comment)

Simple build tool
scala: twendelmuth/twendelmuth-testing-sbt-vanilla#6 -> twendelmuth/twendelmuth-testing-sbt#4 (nothing should have changed here)
ro.nextreports: twendelmuth/twendelmuth-testing-sbt-vanilla#5 -> twendelmuth/twendelmuth-testing-sbt#6 (finds correct source url)
xhtmlrenderer: twendelmuth/twendelmuth-testing-sbt-vanilla#3 -> twendelmuth/twendelmuth-testing-sbt#2 (finds correct source url, can parse github release notes)

maven
maven-compiler: twendelmuth/twendelmuth-testing-mvn-vanilla#4 -> twendelmuth/twendelmuth-testing-mvn#3 (nothing has changed here)
ro.nextreports: twendelmuth/twendelmuth-testing-mvn-vanilla#5 -> twendelmuth/twendelmuth-testing-mvn#5 (finds correct source url)
xhtmlrenderer: twendelmuth/twendelmuth-testing-mvn-vanilla#3 -> twendelmuth/twendelmuth-testing-mvn#2 (finds correct source url, can parse github release notes)

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

How I've tested my work (please tick one)

I have verified these changes via:

• Code inspection only, or
• Newly added unit tests, or
• No new tests but ran on a real repository, or
• Both unit tests + ran on a real repository

[zharinov]

How I see the roadmap for this PR:

• I refactor Maven datasource tests to be more clear and use HTTP mocks instead of file protocol
• Resolve conflicts and merge Maven-related part
• Merge SBT-related changes in the separate PR

@twendelmuth What do you think?

[twendelmuth]

How I see the roadmap for this PR:

• I refactor Maven datasource tests to be more clear and use HTTP mocks instead of file protocol
• Resolve conflicts and merge Maven-related part
• Merge SBT-related changes in the separate PR

@twendelmuth What do you think?

Works for me.
I used the file protocol base mainly as convience because I'm still struggling to work productively in the IDE I'm using for typescript 👼 I guess it would make sense to introduce to rework that setup a bit to make it easier to add new dependencies (like nocking head & get, maven-metadata.xml and the directory setup).

I tried to touch the SBT related changes as little as possible - so that they also gain the same approach without potentially breaking their functionality. In general I think at least parts of the logic could also make sense in the maven-utils part - however that would have made my changeset even bigger and less readable. The main goal was to be able to pass the repoRoot to the maven-utils function and actually using this (so we don't deviate in handling of maven.poms like we do today).
I could have done this also purely in the maven-util part by starting to split off things of the repoUrl if applicable but that felt more hackish.
(The issue is that SBT would pass https://repo.maven.apache.org/maven2/org/elasticsearch/ as it's root and I'm potentially needing to looku org/elasticsearch/parent and therefore needing to get rid of org/elasticsearch from the root).

The whole logic of trying different directory structures seem to be a fallback to deal with mistakes that have been made in uploading the maven artifacts. In the public repos usually you'd find a "normal" version anyway though but obviously I cannot speak for internally hosted maven repositories.

Examples of this:
https://repo.maven.apache.org/maven2/org/elasticsearch/client/rest/5.0.0-beta1/
https://repo.maven.apache.org/maven2/org.elasticsearch.client/rest/5.0.0-beta1/

Anyway. I'd for now split off the commits for sbt and put them in a different branch (if you haven't done that already 😄)

[zharinov]

Okay, just don't spend too much effort on tests for now :)

[twendelmuth]

So I've split the things off:
https://github.com/twendelmuth/renovate/tree/feat/feature_6742_mvn_lookup_parents
https://github.com/twendelmuth/renovate/tree/feat/sbt_lookup_parents

With the sbt one based on the mvn lookup things. How do you want me to proceed?
I explicitly didn't want to push to the branch with the PR already open - but could do so 😄

[zharinov]

Thanks, okay let's wait until my refactoring is in main branch, then let's see what's next.

[twendelmuth]

@HonkingGoose the refactorings are in. I now need to adapt the tests to run with the new setup, blocking is okay I guess. I could also revert to a draft PR 👼

(#9745)

[HonkingGoose]

@HonkingGoose the refactorings are in. I now need to adapt the tests to run with the new setup, blocking is okay I guess. I could also revert to a draft PR 👼

I've removed the "blocked" label.

I think it's a good idea to mark this PR as a draft again, that way it's clear that more work is still needed before it's in a ready-to-merge state again.

[zharinov]

Hi @twendelmuth, I'm about to adapt your tests for recent test refactoring right now. Once done, we only need to split Maven/Sbt parts

[twendelmuth]

@zharinov I've removed the sbt parts - we can add them in a different PR imo.

[twendelmuth]

Anything that is missing here?
Anyway I can help to bring this live? :-)

[zharinov]

Hi @twendelmuth, sorry for the long delay. Let's do this!
Please, merge my PR with null guards before recursion: twendelmuth#4

[zharinov]

Hi @twendelmuth, thanks for merging my changes. Finally, we need to limit the recursion depth and test it on self-referencing pomfile. I would limit to 5, what do you think?

[twendelmuth]

5 sounds reasonable - I could see certain projects might go a bit deeper than that but it's a good starting point imo.

Regarding the self-reference:
I guess you're referring to circular dependencies like POM1 -> POM2 -> POM1? A POM is not allowed to be it's own parent.

[zharinov]

Yes, I was a bit lazy for creating circle with 2 participants 🙃

[viceice]

needs deconflicting

[renovate-release]

🎉 This PR is included in version 25.51.0 🎉

The release is available on:

• GitHub release
• 25.51.0

Your semantic-release bot 📦🚀