Problem when adding Caffeine to a Tar in Gradle

[alkamo]

Gradle Version: 6.3
Java Version: 11.0.11 (Amazon.com Inc. 11.0.11+9-LTS)
Caffeine Version: 3.0.3

This is probably a configuration problem on my part, but I'm not sure where else to ask about it.

I have a gradle sub-project that gets caffeine as a transitive dependency (from jdbi). It builds without issue. However, I have a separate gradle sub-project that pulls the first artifact and its dependencies into a tar using the distribution plugin, using the following configuration:

configurations {
    library
}

dependencies {
    library(project(':my-project:project-jar'))
}

distributions {
    main {
        contents {
            into('/lib') {
                from configurations.library.files
            }
        }
    }
}

When the caffeine version is 3.x and I run the distTar task for this configuration, I get the following error:

Could not resolve com.github.ben-manes.caffeine:caffeine:3.0.3.
      Required by:
          project :my-project:project-tar > project :my-project:project-jar > org.jdbi:jdbi3-core:3.29.0
       > Cannot choose between the following variants of com.github.ben-manes.caffeine:caffeine:3.0.3:
           - runtimeElements
           - shadowRuntimeElements

If I bring the dependency down to the point where the Caffeine dependency resolves to 2.x, this issue goes away.

[ben-manes]

hmm.. that's something I can't answer. The shadowRuntimeElements stands out and I know that the shadow plugin does use or manipulate the distribution plugin. If you don't need it then that would be the easiest fix.

[alkamo]

This project isn't using the shadow plugin at all, but maybe it goes the other way and the distribution plugin builds on the shadow plugin.

[ben-manes]

hmm, then I don't know why that configuration would exist? The shadow plugin is external, whereas the distribution is internal.

It is used by the JMH plugin for benchmarks, which might mean it leaks into Caffeine's gradle metadata file. I'm not sure why that would cause breaks though.

[ben-manes]

yeah, I see it in the module descriptor. I don't know enough about how Gradle uses this and why it would conflict.

{
  "name": "shadowRuntimeElements",
  "attributes": {
    "org.gradle.category": "library",
    "org.gradle.dependency.bundling": "shadowed",
    "org.gradle.libraryelements": "jar",
    "org.gradle.usage": "java-runtime"
  },
  "files": [
    {
      "name": "caffeine-3.0.3-all.jar",
      "url": "caffeine-3.0.3-all.jar",
      "size": 1098775,
      "sha512": "8a42cff3abb1ffa60616504f0af75cac75dbe15137e777188926d76b23d1865a36f3f773946426c3f11e834932bc1161d9828a209885799b41d9f32241386119",
      "sha256": "a2d5edfb572bf47da90c68b55195baa84f9f1ae81a4fa13e2151b29e44f8021b",
      "sha1": "206aa3f4706839ac2970c447ce6575f716a99e06",
      "md5": "9597edf3a8a476607c33d14582621c7d"
    }
  ]
}

[ben-manes]

Trying to remove the shadow plugin from the build and it seems fine. Unfortunately I did not describe the issue very well in the git commit. I think it was an issue with Ehcache2 + Ehcache3, which later became incompatible dependencies. Whatever was wrong, a jmh benchmark with Ehcache3 seems to run fine so I could remove that going forward. We could try a snapshot build and see if that resolves your problem. Wdyt?

[ben-manes]

@alkamo please try 3.1.1-SNAPSHOT and let me know if that resolves the problem

[alkamo]

3.1.1-SNAPSHOT did resolve the problem.

---

Looking at the full error message, I think I might have been able to get around this by telling Gradle to require the Java 11 version or the "external" bundling for the tar, since those are the attributes that differ between the two variants (I just haven't been able to figure out how to do that).

> Could not resolve all files for configuration ':my-project:my-project-tar:library'.
   > Could not resolve com.github.ben-manes.caffeine:caffeine:3.0.3.
     Required by:
         project :cdars-match:cdars-match-tar > project :my-project:my-project-jar > org.jdbi:jdbi3-core:3.29.0
      > Cannot choose between the following variants of com.github.ben-manes.caffeine:caffeine:3.0.3:
          - runtimeElements
          - shadowRuntimeElements
        All of them match the consumer attributes:
          - Variant 'runtimeElements' capability com.github.ben-manes.caffeine:caffeine:3.0.3:
              - Unmatched attributes:
                  - Found org.gradle.category 'library' but wasn't required.
                  - Found org.gradle.dependency.bundling 'external' but wasn't required.
                  - Found org.gradle.jvm.version '11' but wasn't required.
                  - Found org.gradle.libraryelements 'jar' but wasn't required.
                  - Found org.gradle.status 'release' but wasn't required.
                  - Found org.gradle.usage 'java-runtime' but wasn't required.
          - Variant 'shadowRuntimeElements' capability com.github.ben-manes.caffeine:caffeine:3.0.3:
              - Unmatched attributes:
                  - Found org.gradle.category 'library' but wasn't required.
                  - Found org.gradle.dependency.bundling 'shadowed' but wasn't required.
                  - Found org.gradle.libraryelements 'jar' but wasn't required.
                  - Found org.gradle.status 'release' but wasn't required.
                  - Found org.gradle.usage 'java-runtime' but wasn't required.

[ben-manes]

I find the new resolution behavior very confusing and frustrating. It breaks the gradle-versions-plugin even though that uses apis since 1.0 and does simple version lookups. I think they might have tried to evolve to solve too many problems (android, native).

I'll try to have this out by next Monday. Just waiting to see if the fix for #715 can be verified by the user before cutting the release.

[ben-manes]

Released 3.1.1

[manishkrishan]

same issue with 2.9.3 with java 8, any workaround ?

[ben-manes]

I think you can tell gradle which to select.

[manishkrishan]

Thankyou, can you pls give an example of selecting Variant 'runtimeElements' capability in
configurations {
all {
resolutionStrategy.capabilitiesResolution.withCapability('com.github.ben-manes.caffeine:caffeine:2.9.3') {
. . .
. . .

[ben-manes]

It might take me a while to have free time to look into that. A broader approach that I have used to workaround gradle plugin portal outages is to disable the metadata entirely. It isn't useful for server-side builds most of the time and seems to be mostly for kotlin and android quirkiness.

repositories {
    maven {
        url "http://repo.mycompany.com/repo"
        metadataSources {
            mavenPom()
            artifact()
        }
    }
}

[ben-manes]

Here is what chatgpt said, might work too.

In Gradle, when you encounter dependency resolution issues like the one you've described, where there are multiple variants of a dependency available and > the build system cannot decide which one to use, you can use dependency constraints to guide Gradle's selection process. To force Gradle to select the > runtimeElements variant of the com.github.ben-manes.caffeine:caffeine:3.0.3 dependency, you can add a dependency constraint to your project's build.> gradle file.

Here's an example of how you can do that:

dependencies {
    // Add a dependency constraint for caffeine to force using the runtimeElements variant
    constraints {
        implementation("com.github.ben-manes.caffeine:caffeine:3.0.3") {
            because "We want to use the runtimeElements variant"
        }
    }
    
    // Other dependencies...
}

By adding this constraint, you're explicitly telling Gradle that you want to use the implementation (which corresponds to the runtimeElements variant) > of the caffeine library with version 3.0.3. The because clause is optional but can be helpful to provide a reason for the constraint.

After adding this constraint, Gradle should prioritize the runtimeElements variant when resolving dependencies and resolving conflicts.

Remember that dependency resolution can be complex, and conflicts might still arise due to other factors in your project configuration. If you're still > facing issues after adding the constraint, you might need to further investigate your project's dependencies and configurations to ensure there are no > conflicting dependencies or other issues causing the resolution problem.

[ben-manes]

and when confirming, it said,

Apologies for any confusion in my previous response. If you want to specifically exclude the shadowRuntimeElements variant and ensure that only the > runtimeElements variant of the com.github.ben-manes.caffeine:caffeine:3.0.3 dependency is used, you can use the following approach:

dependencies {
    // Exclude shadowRuntimeElements variant of caffeine
    implementation("com.github.ben-manes.caffeine:caffeine:3.0.3") {
        exclude group: 'com.github.ben-manes.caffeine', module: 'caffeine', version: '3.0.3', classifier: 'shadowRuntimeElements'
    }
    
    // Other dependencies...
}

In this case, you're explicitly excluding the shadowRuntimeElements variant of the caffeine dependency from being resolved. This should ensure that > only the runtimeElements variant is used in your project.

Remember that the exact syntax might vary based on your Gradle version and project structure, but the concept of excluding a specific variant from a > dependency remains the same.

[manishkrishan]

thankyou for response but the above solution from ChatGPT does works with gradle 6.7.1. Error "Could not set unknown property 'classifier' for object of type org.gradle.api.internal.artifacts.DefaultExcludeRule"

[ben-manes]

how about the metadataSources approach? That should just use the poms which is almost always what you want in a server project

[manishkrishan]

It was already there under "buildscript" in my build.gradle

buildscript {
  repositories {
    maven {
        url repo1
        metadataSources {
            ignoreGradleMetadataRedirection()
            mavenPom()
            artifact()
        }
    }
 }
}

I also tried adding it outside buildscript scope like

   repositories {
      maven { 
         url repo1
         metadataSources {
            mavenPom()
            artifact()
         }
      }
      maven { 
         url repo2 
         metadataSources {
            mavenPom()
            artifact()
         }
      }
   }

[ben-manes]

odd.. did you try with --refresh-dependencies or delete your ~/.gradle/caches to make sure it did not pick up somewhere else?

I can't reproduce this problem so unfortunately I don't know how to debug this.

[manishkrishan]

does not work with --refresh-dependencies or after deleting gradle cache

[ben-manes]

since all I can do is guess, what about using an attribute to force the selection?

dependencies {
  implementation('com.github.ben-manes.caffeine:caffeine:2.9.3') {
    attributes {
      attribute(Attribute.of("org.gradle.dependency.bundling", String), "external")
    }
  }
}

[manishkrishan]

finally works as below:

 configurations { 
  . . . . . . 
  . . . . . . 
  caffeine
  . . . . . . 
  . . . . . . 
 }

 dependencies {
  . . . . . . 
  . . . . . . 
  caffeine ('com.github.ben-manes.caffeine:caffeine:2.9.3')  {
         attributes {
            attribute(Attribute.of("org.gradle.dependency.bundling", String), "external")
         }	 
  }
  . . . . . . 
  . . . . . . 
 }