Class loading fails on an interrupted thread causing com.mongodb.event.ServerClosedEvent to fail to load when Mongo detects a cluster change

[manjian14]

We've encountered a connectivity problem after migrating from Spring Boot 3.1.5/3.1.6 to 3.2.0. The error log points to a missing class:

java.lang.ClassNotFoundException: com.mongodb.event.ServerClosedEvent

This is affecting our connection to MongoDB "AWS DocumentDB." Any guidance or resolution on this matter would be greatly appreciated.

[mhalbritter]

Hello!

I can't see any code from Boot in the stacktrace, besides the JAR loader. I'm not sure if this has something to do with the Uber JAR rewrite. Can you please try with the classic loader implementation and report back if that fixes the issue?

See here for details: https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.2-Release-Notes#nested-jar-support

[manjian14]

The connectivity issue with MongoDB "AWS DocumentDB" appears resolved after implementing the classic loader in Spring Boot 3.2.0:

bootJar {
    archiveFileName = "${rootProject.name}.${archiveExtension.get()}"
    loaderImplementation = LoaderImplementation.CLASSIC
}

Any additional insights or recommendations moving forward would be appreciated.

[mhalbritter]

cc @philwebb

[wilkinsona]

@manjian14 The stack trace would suggest that the problem occurs when there's a change in the topology of the cluster but I haven't been able to reproduce it. Can you please provide a minimal sample and the steps to follow to cause the failure?

[manjian14]

I've configured my application to connect to an AWS DocDB cluster using the standard driver configuration. Below is a snippet from my MongoDbConfiguration class:

@Slf4j
@Configuration
@EnableMongoRepositories(value = "com.????")
@ConditionalOnProperty(prefix = "mongo.connection", value = "string")
public class MongoDbConfiguration extends AbstractMongoClientConfiguration {

    @Value("${mongo.connection.string}")
    private String connectionString;

    @Override
    @Primary
    @Bean
    public @NonNull MongoClient mongoClient() {
        return MongoClients.create(connectionString);
    }

    @Override
    protected @NonNull String getDatabaseName() {
        return "???";
    }
}

In this setup, I've embedded the username and password within the cluster URL. The cluster has two replica sets, one for reading and one for writing. Additionally, the database is not accessible from outside; access is restricted either by whitelisted IP addresses or by project roles within the ECS cluster.

The AWS DocDB version is 5.0.0, the latest version as per AWS recommendations. The project is using Gradle version 8.5 and Java version 17. The base image is amazoncorretto:17-al2023-headful, and during the image build process, I've imported the SSL certificate using the keytool tool.

If you have any specific questions or if there's anything else you'd like to know, feel free to ask!

[philwebb]

@manjian14 Is there any chance you could switch back to the DEFAULT loader implementation and try 3.2.1-SNAPSHOT? We've fixed a few bugs with the nested loader. I'm doubtful that we've caught this one, but I'd like to check since it seems hard for us to replicate the problem locally.

[manjian14]

I tried using 3.2.1-SNAPSHOT, but ran into a problem—' springboot binaries are incompatible with your platform'

[wilkinsona]

What platform are you using? It's rather unfortunate that it apparently won't allow you to test snapshot artifacts.

[msievers]

Seeing the same error with our own MongoDB cluster(s) deployed to our own Kubernetes starting with Spring Boot 3.2.0.

java.lang.NoClassDefFoundError: com/mongodb/event/ServerClosedEvent

I'll give it a try with 3.2.1-SNAPSHOT.

[msievers]

Same thing with 3.2.1-SNAPSHOT

java.lang.NoClassDefFoundError: com/mongodb/event/ServerClosedEvent
	at com.mongodb.internal.connection.DefaultServer.close(DefaultServer.java:172)
	at com.mongodb.internal.connection.AbstractMultiServerCluster.removeExtraHosts(AbstractMultiServerCluster.java:436)
	at com.mongodb.internal.connection.AbstractMultiServerCluster.ensureServers(AbstractMultiServerCluster.java:407)
	at com.mongodb.internal.connection.AbstractMultiServerCluster.handleReplicaSetMemberChanged(AbstractMultiServerCluster.java:248)
	at com.mongodb.internal.connection.AbstractMultiServerCluster.lambda$onChange$3(AbstractMultiServerCluster.java:197)
	at com.mongodb.internal.Locks.lambda$withInterruptibleLock$1(Locks.java:53)
	at com.mongodb.internal.Locks.checkedWithInterruptibleLock(Locks.java:66)
	at com.mongodb.internal.Locks.withInterruptibleLock(Locks.java:59)
	at com.mongodb.internal.Locks.withInterruptibleLock(Locks.java:52)
	at com.mongodb.internal.connection.BaseCluster.withLock(BaseCluster.java:226)
	at com.mongodb.internal.connection.AbstractMultiServerCluster.withLock(AbstractMultiServerCluster.java:54)
	at com.mongodb.internal.connection.AbstractMultiServerCluster.onChange(AbstractMultiServerCluster.java:165)
	at com.mongodb.internal.connection.DefaultSdamServerDescriptionManager.updateDescription(DefaultSdamServerDescriptionManager.java:113)
	at com.mongodb.internal.connection.DefaultSdamServerDescriptionManager.lambda$update$0(DefaultSdamServerDescriptionManager.java:75)
	at com.mongodb.internal.Locks.lambda$withInterruptibleLock$1(Locks.java:53)
	at com.mongodb.internal.Locks.checkedWithInterruptibleLock(Locks.java:66)
	at com.mongodb.internal.Locks.withInterruptibleLock(Locks.java:59)
	at com.mongodb.internal.Locks.withInterruptibleLock(Locks.java:52)
	at com.mongodb.internal.connection.BaseCluster.withLock(BaseCluster.java:226)
	at com.mongodb.internal.connection.AbstractMultiServerCluster.withLock(AbstractMultiServerCluster.java:54)
	at com.mongodb.internal.connection.DefaultSdamServerDescriptionManager.update(DefaultSdamServerDescriptionManager.java:60)
	at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:169)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ClassNotFoundException: com.mongodb.event.ServerClosedEvent
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
	at org.springframework.boot.loader.net.protocol.jar.JarUrlClassLoader.loadClass(JarUrlClassLoader.java:104)
	at org.springframework.boot.loader.launch.LaunchedClassLoader.loadClass(LaunchedClassLoader.java:91)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
	... 23 more 

[msievers]

The errors are gone when using <loaderImplementation>CLASSIC</loaderImplementation> with Spring Boot 3.2.0

[philwebb]

@msievers Are you able to provide a sample application that we can debug?

[msievers]

@philwebb I'll give it a try and ping you once I got it. As I cannot take one of our company apps where I see this happen, I hope it can be reproduced with some simple playground app.

[wilkinsona]

Thanks very much, @msievers. I've now been able to reproduce the ClassNotFoundException.

[ANigam123]

@msievers actually it did solve the problem for me. I moved the configuration to top-level and it started working. Not sure what's the difference. Thanks for the help!

[wilkinsona]

As an alternative to switching back to the classic loader implementation, another workaround is to load the class up front:

@SpringBootApplication
public class Application {

    public static void main(String[] args) throws Exception {
        Class.forName("com.mongodb.event.ServerClosedEvent");
        SpringApplication.run(Application.class, args);
    }

}

[wilkinsona]

This is another variant of #38154, but this time allowing the interruption to continue by rethrowing the ClosedByInterruptException is insufficient. We really need to perform the read in an uninterruptible manner. Unfortunately, the JDK's support for that is an implementation detail of sun.nio.ch.FileChannelImpl so we'll need to implement our own.

[wilkinsona]

Possible fix: https://github.com/wilkinsona/spring-boot/tree/gh-38611.

[philwebb]

We've just pushed a fix for this. It would be much appreciated if someone could try the latest 3.2.1 SNAPSHOT with the default loader to see if we've fixed the problem.

[msievers]

@philwebb I'll give it a try and report back.

[msievers]

@philwebb It's still failing with 3.2.1-SNAPSHOT. Was the last CI build completed successfully? It's orange, maybe the changes were not published/pushed, see

https://ci.spring.io/teams/spring-boot/pipelines/spring-boot-3.2.x/jobs/build/builds/802

[philwebb]

Gahh, those failures are a pain. It wasn't published but I also didn't get a notification. I've kicked it off again and I'll comment back here when it's done. Sorry about that.

[philwebb]

Sorry @msievers, we're having some trouble with our CI right now and we haven't managed to publish jars yet.

[wilkinsona]

@msievers Our CI's been fixed and a new snapshot 3.2.1-SNAPSHOT build has been published. Could you please give it a try if you have the time?

[msievers]

@philwebb @wilkinsona I can confirm that for my two use cases, one being our internal application and the other the application I created on GitHub to reproduce the error, the issue seems to have disappeared with the use of the current Spring Boot 3.2.1-SNAPSHOT version.

[wilkinsona]

Thank you, @msievers. That's some greatly appreciated good news.