Hibernate service loading logs HHH000505 warnings for ServiceConfigurationError with Gradle-built jars since 2.5.10 when using Java 11 or later

[dreis2211]

Hi,

after upgrading an internal customer project today from Spring-Boot 2.5.9 to 2.5.11 I noticed a malfunction of what I believe is a regression caused by #28562 with stacktraces similar to the following:

java.util.ServiceConfigurationError: com.example.demo.TestInterface: Provider com.example.demo.TestImplementation not found
        at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:589) ~[na:na]
        at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1212) ~[na:na]
        at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1221) ~[na:na]
        at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265) ~[na:na]
        at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300) ~[na:na]
        at java.base/java.util.ServiceLoader$ProviderSpliterator.tryAdvance(ServiceLoader.java:1484) ~[na:na]
        at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681) ~[na:na]

I created a minimal reproduction project https://github.com/dreis2211/hibernate-service-loader-meta-inf-demo that showcases the problem. (We're not using the internals directly in reality, it's just the most simple way to showcase things ;-) ) Essentially the service loading of Hibernate cannot find services anymore when run as a JAR.

I'm still not quite sure if this a problem in Hibernate, after all (because a vanilla ServiceLoader.load doesn't complain.). But between 2.5.9 and 2.5.11 no Hibernate upgrade happened (version 5.4.33 is used throughout).

I wonder if the following should ignore META-INF/services folders.

spring-boot/spring-boot-project/spring-boot-tools/spring-boot-gradle-plugin/src/main/java/org/springframework/boot/gradle/tasks/bundling/BootJar.java

Line 116 in a5ec85b

if (path.startsWith("META-INF/") && !path.equals("META-INF/aop.xml") && !path.endsWith(".kotlin_module")) {

Let me know if you need anything from me to resolve the issue.

Cheers,
Christoph

[wilkinsona]

Thanks for the sample, @dreis2211, and apologies for the change in behaviour. As far as I can tell, this has alway been broken with Maven and, unfortunately, #28562 made things behave the same way with Gradle.

I don't think it makes sense to leave META-INF/services in the root of the jar. In that location, the file will only be visible to the system class loader which won't be able to load the types to which it refers. That said, I am a little surprised that a Maven user hasn't encountered and reported the problem and a bit wary of a regression in the other direction on the Maven side if we change its behaviour.

[dreis2211]

No need to apologize - I could apologize for not noticing earlier.

I'm not sure I get the following:

In that location, the file will only be visible to the system class loader which won't be able to load the types to which it refers

Isn't that what happened before #28562 - where things have worked?
Nevermind. You meant root of the jar...

Should we maybe involve someone from Hibernate? After all, with a vanilla ServiceLoader.load() I couldn't reproduce things. (I must admit I haven't looked to deep on the Hibernate side of things and what the problem might be - maybe a different classloader not being able to load the particular services (!?) )

[wilkinsona]

In 2.5.9, prior to #28562, the Gradle plugin moved META-INF beneath BOOT-INF/classes. In your sample, that results in this entry in the jar:

BOOT-INF/classes/META-INF/services/com.example.demo.TestInterface

In 2.5.10 and later, it gets left in the root:

META-INF/services/com.example.demo.TestInterface

2.5.10's behaviour doesn't make sense as we now have a file that's visible to the system class loader that refers to types that are only visible to Boot's ClassLoader. META-INF/aop.xml has the same characteristics which is why it's excluded from being moved to the root.

So, I think it makes sense to move META-INF/services/* files beneath BOOT-INF/classes. We should do this in 2.5.x for Gradle to restore 2.5.9's behaviour. I'm less sure about when we should do this for Maven as I'm surprised that a Maven user hasn't already reported the problem. We could defer the changes on the Maven side to 2.6.x or even 2.7, back porting them if needs be in the future. I'll flag this for team attention to see what everyone else thinks about the timing of the change on the Maven side.

[wilkinsona]

Should we maybe involve someone from Hibernate? After all, with a vanilla ServiceLoader.load() I couldn't reproduce things. (I must admit I haven't looked to deep on the Hibernate side of things and what the problem might be - maybe a different classloader not being able to load the particular services (!?) )

Using Boot's LaunchedURLClassLoader, ServiceLoader finds resources for META-INF/services files via its parent ClassLoader. These resources are then parsed and their entries are loaded by LaunchedURLClassLoader.

Using Hibernate's AggregatedClassLoader, ServiceLoader finds resources for META-INF/services files but it's then unable to load their entries. The JDK behaves in the same way if you pass in LaunchedURLClassLoader's parent when creating the ServiceLoader:

ServiceLoader<TestInterface> loader = ServiceLoader.load(TestInterface.class, getClass().getClassLoader().getParent());
loader.forEach(System.out::println);

This leaves the ServiceLoader working with a ClassLoader hierarchy that can load the META-INF/services files but cannot load their entries.

It's quite an odd arrangement to have META-INF/services files in an ancestor class loader with entries that are only loadable by one of its descendents so, while it works with a more traditional ClassLoader hierarchy, it doesn't seem unreasonable to me for it not to work with Hibernate's AggregatedClassLoader.

[dreis2211]

I'm afraid I overlooked that the service loading seems to be still working. It's only logged as a big warning apparently, but the class is still loaded properly (probably by at least one of the involved classloaders). This changes the urgency drastically. Feel free to rephrase the ticket as needed - I'm sorry I didn't notice that earlier.

It might still makes sense to ignore META-INF/services in BootJar, but at least it's not breaking anything drastically as things stand (which might explain why there was no such report so far)

[wilkinsona]

Thanks very much for the clarification, Christoph. There's no need to apologise!

I've just confirmed with your sample that it's "just" a warning message and also that the problem doesn't occur with Java 8. I believe the latter's because Hibernate doesn't use its AggregatedClassLoader unless you're using Java 9 or later.

[philwebb]

We're going to align both Maven and Gradle to put the file under BOOT-INF/classes/META-INF/services/com.example.demo.TestInterface