Unable to properly resolve dynamic dependencies from mavenLocal() repo #11321

lkoe · 2019-11-12T07:08:47Z

Gradle 6.0 fails to resolve artifacts from the mavenLocal() repository.
The artifact was published using the maven-publish plugin.
In the local maven repository metadata was published as maven-metadata-local.xml (as before).

However Gradle 6.0 seems to be exclusively looking for a file named maven-metadata.xml - which isn't there.

A problem occurred configuring root project 'common'.
> Could not resolve all artifacts for configuration ':classpath'.
   > Could not find any matches for de.xxx.xxx.gradle.plugins:xxx-gradle-plugins:2.0.+ as no versions of de.xxx.xxx.gradle.plugins:geco-gradle-plugins are available.
     Searched in the following locations:
       - file:/C:/Users/xxx/.m2/repository/de/xxx/xxx/gradle/plugins/xxx-gradle-plugins/maven-metadata.xml
     Required by:
         project :

If the maven-metadata-local.xml files are renamed to maven-metadata.xml, resolution works.

Expected Behavior

The dependency should be properly resolved like with Gradle 5 and before.

Current Behavior

The dependency was not resolved from the local Maven repository.

Context

For inhouse gradle plugin development I publish the plugin locally first in order to test it with other projects.
Publishing to our artifactory repo would not be feasible as faulty plugin versions might get picked up by others.

Steps to Reproduce

Your Environment

Gradle 6.0
java version "1.8.0_202"
Java(TM) SE Runtime Environment (build 1.8.0_202-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode)
Windows 10
Build scan URL:

The text was updated successfully, but these errors were encountered:

jjohannes · 2019-11-12T13:31:56Z

Thanks for reporting @lkoe. Looks like Gradle never looked for maven-metadata-local.xml but always did a fall back to listing files. So we broke this by "not looking for artifacts by default" anymore if metadata is missing.
We will work on fixing this. For now you can work around it with this:

repositories {
    mavenLocal() {
        metadataSources {
            mavenPom()
            artifact()
        }
    }
}

Yg0R2 · 2019-11-13T14:49:36Z

Hi @jjohannes ,

I wanted to mention, this behavior appears only if you use dynamic versioning.
In that case, if you define a fixed version like 2.0.1-SNAPSHOT it works properly.

jjohannes · 2019-11-13T15:16:52Z

Good point @Yg0R2. If only the maven-metadata.xml is "missing", but pom.xml files all exist, a repository without artifact() still works fine if no dynamic versions are used. Because then Gradle directly checks the selected versions by checking if the corresponding pom.xml exists.

ljacomet · 2019-11-13T15:36:40Z

So doing a bit of research on this topic.

First of all, the presence of a maven-metadata-*.xml file is not guaranteed in a local repository.

Recent Maven versions will download the one from the remote repository, naming it maven-metadata-<repo-id>.xml.
A maven-metadata-local.xml only exists if you install that module from a local build.

Given all of that, my take is we should ignore the metadata sources configuration for Maven local version listing and always do file listing.

jjohannes · 2019-11-13T15:45:27Z

We can also restore the "old" behavior for Maven local. I.e have it default to

metadataSources {
  mavenPom()
  artifact()
}

It's probably less "special case", easier to implement, and with that less risky, as we will restore the 5.x behavior.

So something like adding this

        mavenRepository.metadataSources(new Action<MavenArtifactRepository.MetadataSources>() {
            @Override
            public void execute(MavenArtifactRepository.MetadataSources metadataSources) {
                metadataSources.mavenPom();
                metadataSources.artifact();
            }
        });

here:

gradle/subprojects/dependency-management/src/main/java/org/gradle/api/internal/artifacts/repositories/DefaultBaseRepositoryFactory.java

Lines 143 to 147 in 1f259cf

    
           public MavenArtifactRepository createMavenLocalRepository() { 
        
               MavenArtifactRepository mavenRepository = instantiator.newInstance(DefaultMavenLocalArtifactRepository.class, fileResolver, transportFactory, locallyAvailableResourceFinder, instantiatorFactory, artifactFileStore, pomParser, metadataParser, createAuthenticationContainer(), fileResourceRepository, mavenMetadataFactory, isolatableFactory, objectFactory, urlArtifactRepositoryFactory); 
        
               File localMavenRepository = localMavenRepositoryLocator.getLocalMavenRepository(); 
        
               mavenRepository.setUrl(localMavenRepository); 
        
               return mavenRepository;

melix · 2019-11-13T15:47:37Z

OR, the 3rd way. Currently "listing" and "metadata sources" have been bound together, but we could decide to have a different kind of lister for Maven local, which doesn't care about the presence of artifact() or not in metadata sources.

ljacomet · 2019-11-13T15:49:47Z

That's the current idea I am looking at, have a specialized MavenVersionLister for Maven local which would not even attempt a maven-metadata.xml but directly do the basic version listing.

Effectively changing the configuration is a simpler change, but it also makes it harder to explain the difference. And finally, the current MavenVersionLister is just useless code for Maven local, as it will never find a file.

There is no way to rely on a `maven-metadata.xml` for Maven local version listing. Since Gradle 6.0 removed the default `artifact()` metadata source, this causes all dynamic version resolution to fail with Maven local. This commit changes the way Maven local is handled to _always_ do version listing through directory listing. Fixes #11321

Issue #11321

jjohannes · 2019-11-15T11:03:52Z

@lkoe we have fixed the issue for a 6.0.1 patch release. Here is a nightly containing the fix:
https://services.gradle.org/distributions-snapshots/gradle-6.0.1-20191115103811+0000-bin.zip (link updated)
It would be very much appreciated if you can acknowledge that it works for you.

lkoe · 2019-11-15T11:24:57Z

Thanks @jjohannes, unfortunately due to a business trip I can only manage to test next wednesday. Hope that's still inside the 6.0.1 timeframe...

ljacomet · 2019-11-15T11:52:05Z

There was a copy/paste mistake in the nightly version. Please use 6.0.1-20191115103811+0000 as the version, or https\://services.gradle.org/distributions-snapshots/gradle-6.0.1-20191115103811+0000-bin.zip for the gradle-wrapper.properties

We hope to have 6.0.1 out before next wednesday, but no worry if you cannot validate before then. We will have to trust our own testing 😉

lkoe · 2019-11-20T17:03:13Z

Confirmed, works for me with 6.0.1. Thx!

kristian · 2019-12-01T13:06:34Z

Sorry just found out, "dynamic plugin versions are not supported". Thanks anyways!

lkoe · 2019-12-01T14:14:38Z

We're using a dynamic version as well for our inhouse plugins. This worked fine in the past and - with this issue fixed - also with 6.0.1

kristian · 2019-12-01T15:20:35Z

We're using a dynamic version as well for our inhouse plugins. This worked fine in the past and - with this issue fixed - also with 6.0.1

Hey @lkoe, interesting! Can you tell me how you did it? Maybe just to clarify, I was speaking about referencing the plugin versions dynamically like:

plugins {
    id 'anyplugin' version '0.+'
}

Not referencing libraries used by the plugin dynamically using a dependency block.

lkoe · 2019-12-01T15:38:26Z

We're defining our plugin as dependency in the buildscript block in ´init.gradle´

allprojects {
    buildscript {

        repositories {
            mavenLocal()
            maven {
                url System.properties["artifactoryUrl"] + "libs/"
                credentials {
                    username = System.properties["artifactoryUser"]
                    password = System.properties["artifactoryPassword"]
                }
            }
        }

        def pluginVersion = (gradle.gradleVersion[0] as int) > 4 ? '2.0.+' : '1.0.+'
         
        dependencies {
            classpath "xxx.xxx.xxx.gradle.plugins:xxx-gradle-plugins:$pluginVersion"
        }
    }
}

In the projects we then just apply our plugins without version.

kristian · 2019-12-01T20:03:28Z

Thanks so much @lkoe and sorry for spamming this issue with a "help request". I didn't came to the idea of using the legacy buildscript block, but it worked flawelessly! By adding the buildscript block to the settings.gradle, we not even had to use the allprojects closure block! Thanks again Lars!

mkhludnev · 2019-12-04T20:53:16Z

`

Could not resolve all files for configuration ':int-test:testCompileClasspath'.
Could not find org.apache.solr:solr-test-framework:8.4.0-SNAPSHOT.
Searched in the following locations:
- file:/C:/Users/noob/.m2/repository/org/apache/solr/solr-test-framework/8.4.0-SNAPSHOT/maven-metadata.xml
- file:/C:/Users/noob/.m2/repository/org/apache/solr/solr-test-framework/8.4.0-SNAPSHOT/solr-test-framework-8.4.0-SNAPSHOT.pom
- https://jcenter.bintray.com/org/apache/solr/solr-test-framework/8.4.0-SNAPSHOT/maven-metadata.xml
- https://jcenter.bintray.com/org/apache/solr/solr-test-framework/8.4.0-SNAPSHOT/solr-test-framework-8.4.0-SNAPSHOT.pom
Required by:
project :int-test

..
See https://docs.gradle.org/6.0.1-20191115103811+0000/userguide/command_line_interface.html#sec:command_line_warnings
`
I have maven-metadata-local.xml at that location, also notice gradle version int the url.

how I can resolve it?

ljacomet · 2019-12-05T09:02:49Z

6.0.1 has been released. Can you try with that version?

If that does not work, can you share:

The way you declare the dependency
the content of file:/C:/Users/noob/.m2/repository/org/apache/solr/solr-test-framework and file:/C:/Users/noob/.m2/repository/org/apache/solr/solr-test-framework/8.4.0-SNAPSHOT

mkhludnev · 2019-12-05T11:17:31Z

@ljacomet thanks for your response. Originally SNAPSHOTs were installed by Apache Solr ant script. Turns out, after I manually invoked $mvn install-file for all snapshot artefacts gradle successfully find local dependencies. I also noticed that mvn insall-file put some _remote.repositories which is starts from line: #NOTE: This is an Aether internal implementation file,. So, my issue is about outdated install routine, and not worth to worry about.
fwiw: there's a movement for migrating Solr to gradle.

ljacomet · 2019-12-05T13:03:17Z

Good to know the issue is solved.
And of course it is exciting to learn about a project considering Gradle!

henryju · 2021-05-11T15:40:42Z

Ithink I'm still affected by this issue. Let me know if you prefer that I create a separate issue.

./gradlew --version

------------------------------------------------------------
Gradle 6.8.2
------------------------------------------------------------

Build time:   2021-02-05 12:53:00 UTC
Revision:     b9bd4a5c6026ac52f690eaf2829ee26563cad426

Kotlin:       1.4.20
Groovy:       2.5.12
Ant:          Apache Ant(TM) version 1.10.9 compiled on September 27 2020
JVM:          11.0.11 (Red Hat, Inc. 11.0.11+9)
OS:           Linux 5.11.18-300.fc34.x86_64 amd64

This is when using custom configurations:

allprojects {
  repositories {
        mavenLocal()
  }
}
configurations {
    create("sqplugins") { isTransitive = false }
}
dependencies {
    "sqplugins"("org.sonarsource.dotnet:sonar-csharp-plugin:8.23-SNAPSHOT@jar")
}
tasks.prepareSandbox {
    doLast {
        copy {
            from(project.configurations.get("sqplugins"))
            into(file("$destinationDir/$pluginName/plugins"))
        }
    }
}

Running the build fails to find the dependency:

./gradlew buildPlugin
> Task :prepareSandbox FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':prepareSandbox'.
> Could not resolve all files for configuration ':sqplugins'.
   > Could not find org.sonarsource.dotnet:sonar-csharp-plugin:8.23-SNAPSHOT.
     Searched in the following locations:
       - file:/home/julien/.m2/repository/org/sonarsource/dotnet/sonar-csharp-plugin/8.23-SNAPSHOT/maven-metadata.xml
       - file:/home/julien/.m2/repository/org/sonarsource/dotnet/sonar-csharp-plugin/8.23-SNAPSHOT/sonar-csharp-plugin-8.23-SNAPSHOT.pom
       - ...

but the SNAPSHOT is there:

ll -R ~/.m2/repository/org/sonarsource/dotnet/sonar-csharp-plugin/
~/.m2/repository/org/sonarsource/dotnet/sonar-csharp-plugin/:
total 8
drwxrwxr-x. 2 julien julien 4096 11 mai   15:45 8.23-SNAPSHOT
-rw-rw-r--. 1 julien julien  300 11 mai   16:30 maven-metadata-local.xml

~/.m2/repository/org/sonarsource/dotnet/sonar-csharp-plugin/8.23-SNAPSHOT:
total 6812
-rw-rw-r--. 1 julien julien     928 11 mai   16:30 maven-metadata-local.xml
-rw-rw-r--. 1 julien julien     269 11 mai   16:30 _remote.repositories
-rw-rw-r--. 1 julien julien 5163463 11 mai   16:30 sonar-csharp-plugin-8.23-SNAPSHOT.jar
-rw-rw-r--. 1 julien julien   15923 11 mai   14:09 sonar-csharp-plugin-8.23-SNAPSHOT.pom
-rw-rw-r--. 1 julien julien 1785415 11 mai   16:30 sonar-csharp-plugin-8.23-SNAPSHOT-sources.jar

The workaround worked:

        mavenLocal() {
            metadataSources {
                mavenPom()
                artifact()
            }
        }

I can't understand why the log says that the pom (file:/home/julien/.m2/repository/org/sonarsource/dotnet/sonar-csharp-plugin/8.23-SNAPSHOT/sonar-csharp-plugin-8.23-SNAPSHOT.pom) is not there, while it obviously is.

devmattrick · 2021-06-17T01:04:13Z

I'm in the same boat as @henryju. My gradle info:

------------------------------------------------------------
Gradle 7.0
------------------------------------------------------------

Build time:   2021-04-09 22:27:31 UTC
Revision:     d5661e3f0e07a8caff705f1badf79fb5df8022c4

Kotlin:       1.4.31
Groovy:       3.0.7
Ant:          Apache Ant(TM) version 1.10.9 compiled on September 27 2020
JVM:          16.0.1 (AdoptOpenJDK 16.0.1+9)
OS:           Windows 10 10.0 amd64

Workaround also works for me.

ljacomet · 2021-07-16T10:03:14Z

@henryju If you can craft a reproducer, please file an issue.
org.sonarsource.dotnet:sonar-csharp-plugin:8.23-SNAPSHOT@jar is actually an artifact dependency that should not even look at metadata.

henryju · 2021-07-16T14:18:10Z

I created a reproducer here: https://github.com/henryju/repro_gradle_11321
Simply install the Maven project:

cd repro-gradle-dep
mvn install

and then try to build the Gradle project:

cd repro-gradle-main
./gradlew build

./gradlew build

FAILURE: Build failed with an exception.

* Where:
Build file '/home/julien/tmp/repro_gradle_11321/repro-gradle-main/build.gradle' line: 25

* What went wrong:
Could not determine the dependencies of task ':jar'.
> Could not resolve all files for configuration ':extraLibs'.
   > Could not find org.sonarsource:repro-gradle-dep:1.0-SNAPSHOT.
     Searched in the following locations:
       - https://repo.maven.apache.org/maven2/org/sonarsource/repro-gradle-dep/1.0-SNAPSHOT/maven-metadata.xml
       - https://repo.maven.apache.org/maven2/org/sonarsource/repro-gradle-dep/1.0-SNAPSHOT/repro-gradle-dep-1.0-SNAPSHOT.pom
       - file:/home/julien/.m2/repository/org/sonarsource/repro-gradle-dep/1.0-SNAPSHOT/maven-metadata.xml
       - file:/home/julien/.m2/repository/org/sonarsource/repro-gradle-dep/1.0-SNAPSHOT/repro-gradle-dep-1.0-SNAPSHOT.pom
     Required by:
         project :

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 1s

I don't have file /home/julien/.m2/repository/org/sonarsource/repro-gradle-dep/1.0-SNAPSHOT/maven-metadata.xml but I have /home/julien/.m2/repository/org/sonarsource/repro-gradle-dep/1.0-SNAPSHOT/maven-metadata-local.xml:

<?xml version="1.0" encoding="UTF-8"?>
<metadata modelVersion="1.1.0">
  <groupId>org.sonarsource</groupId>
  <artifactId>repro-gradle-dep</artifactId>
  <version>1.0-SNAPSHOT</version>
  <versioning>
    <snapshot>
      <localCopy>true</localCopy>
    </snapshot>
    <lastUpdated>20210716135756</lastUpdated>
    <snapshotVersions>
      <snapshotVersion>
        <extension>jar</extension>
        <value>1.0-SNAPSHOT</value>
        <updated>20210716135756</updated>
      </snapshotVersion>
      <snapshotVersion>
        <extension>pom</extension>
        <value>1.0-SNAPSHOT</value>
        <updated>20210716135756</updated>
      </snapshotVersion>
    </snapshotVersions>
  </versioning>
</metadata>

File /home/julien/.m2/repository/org/sonarsource/repro-gradle-dep/1.0-SNAPSHOT/repro-gradle-dep-1.0-SNAPSHOT.pom is there, contrary to what the log is saying.

The issue seems to come from the Maven packaging type that is sonar-plugin and not jar. I'm not sure what is different in produced metadata, but Gradle seems to be unable to locate such artifacts in the local repo. Maybe there is a confusion between packaging type and artifact extension?

lkoe added a:bug from:contributor labels Nov 12, 2019

jjohannes added the in:dependency-management DO NOT USE label Nov 12, 2019

ljacomet modified the milestones: 6.1 RC1, 6.0.1 Nov 12, 2019

veita mentioned this issue Nov 12, 2019

Gradle 6.0 seems to ignore project-local Maven repository #11327

Closed

big-guy added the affects-version:6.0 label Nov 12, 2019

ljacomet self-assigned this Nov 13, 2019

ljacomet added a commit that referenced this issue Nov 14, 2019

Add Maven local version listing test

049fbe0

Issue #11321

ljacomet added a commit that referenced this issue Nov 15, 2019

Add Maven local version listing test

d478773

Issue #11321

ljacomet added a commit that referenced this issue Nov 15, 2019

Rename creation method to create*

c591dc9

Issue #11321

ljacomet changed the title ~~Gradle 6.0 does not properly resolve dependencies from mavenLocal() repo~~ Gradle 6.0 does not properly resolve dynamic dependencies from mavenLocal() repo Nov 15, 2019

ljacomet closed this as completed in 781e12f Nov 18, 2019

big-guy changed the title ~~Gradle 6.0 does not properly resolve dynamic dependencies from mavenLocal() repo~~ Unable to properly resolve dynamic dependencies from mavenLocal() repo Nov 18, 2019

This comment has been minimized.

Sign in to view

wind57 mentioned this issue Oct 27, 2020

transitive dependencies and mavenPom() #14996

Closed

andrewparmet mentioned this issue Aug 25, 2023

buf breaking can't find latest version? bufbuild/buf-gradle-plugin#151

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to properly resolve dynamic dependencies from mavenLocal() repo #11321

Unable to properly resolve dynamic dependencies from mavenLocal() repo #11321

lkoe commented Nov 12, 2019

jjohannes commented Nov 12, 2019

Yg0R2 commented Nov 13, 2019

jjohannes commented Nov 13, 2019

ljacomet commented Nov 13, 2019

jjohannes commented Nov 13, 2019

melix commented Nov 13, 2019

ljacomet commented Nov 13, 2019 •

edited

jjohannes commented Nov 15, 2019 •

edited

lkoe commented Nov 15, 2019

ljacomet commented Nov 15, 2019

lkoe commented Nov 20, 2019

This comment has been minimized.

kristian commented Dec 1, 2019

lkoe commented Dec 1, 2019

kristian commented Dec 1, 2019

lkoe commented Dec 1, 2019 •

edited

kristian commented Dec 1, 2019

mkhludnev commented Dec 4, 2019

ljacomet commented Dec 5, 2019

mkhludnev commented Dec 5, 2019

ljacomet commented Dec 5, 2019

henryju commented May 11, 2021

devmattrick commented Jun 17, 2021

ljacomet commented Jul 16, 2021

henryju commented Jul 16, 2021

Unable to properly resolve dynamic dependencies from mavenLocal() repo #11321

Unable to properly resolve dynamic dependencies from mavenLocal() repo #11321

Comments

lkoe commented Nov 12, 2019

Expected Behavior

Current Behavior

Context

Steps to Reproduce

Your Environment

jjohannes commented Nov 12, 2019

Yg0R2 commented Nov 13, 2019

jjohannes commented Nov 13, 2019

ljacomet commented Nov 13, 2019

jjohannes commented Nov 13, 2019

melix commented Nov 13, 2019

ljacomet commented Nov 13, 2019 • edited

jjohannes commented Nov 15, 2019 • edited

lkoe commented Nov 15, 2019

ljacomet commented Nov 15, 2019

lkoe commented Nov 20, 2019

This comment has been minimized.

kristian commented Dec 1, 2019

lkoe commented Dec 1, 2019

kristian commented Dec 1, 2019

lkoe commented Dec 1, 2019 • edited

kristian commented Dec 1, 2019

mkhludnev commented Dec 4, 2019

ljacomet commented Dec 5, 2019

mkhludnev commented Dec 5, 2019

ljacomet commented Dec 5, 2019

henryju commented May 11, 2021

devmattrick commented Jun 17, 2021

ljacomet commented Jul 16, 2021

henryju commented Jul 16, 2021

ljacomet commented Nov 13, 2019 •

edited

jjohannes commented Nov 15, 2019 •

edited

lkoe commented Dec 1, 2019 •

edited