Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PubSub] Caused by: java.lang.ClassNotFoundException: io.grpc.internal.BaseDnsNameResolverProvider #217

Closed
lpellegr opened this issue May 24, 2020 · 7 comments
Closed

[PubSub] Caused by: java.lang.ClassNotFoundException: io.grpc.internal.BaseDnsNameResolverProvider #217

lpellegr opened this issue May 24, 2020 · 7 comments
Assignees
@hannahrogers-google
Labels
api: pubsub Issues related to the googleapis/java-pubsub API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@lpellegr
Copy link

Upgrading to version 1.105.1 from version 1.104.1 for package com.google.cloud:google-cloud-pubsub, when executing the following piece of code:

try {
    subscriber.stopAsync().awaitTerminated();
} catch (IllegalStateException e) {
    log.error("Error while terminating subscriber", e);
}

I get the following exception:

java.lang.IllegalStateException: Expected the service InnerService [FAILED] to be TERMINATED, but the service has FAILED
	at com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379) ~[guava-29.0-jre.jar:na]
	at com.google.common.util.concurrent.AbstractService.awaitTerminated(AbstractService.java:336) ~[guava-29.0-jre.jar:na]
	at com.google.api.core.AbstractApiService.awaitTerminated(AbstractApiService.java:104) ~[api-common-1.9.0.jar:na]
	at co.x.api.usage_handler.UsageHandlerService.onDestroy(UsageHandlerService.java:198) ~[main/:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:564) ~[na:na]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeDestroyMethods(InitDestroyAnnotationBeanPostProcessor.java:347) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeDestruction(InitDestroyAnnotationBeanPostProcessor.java:177) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:242) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:579) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:551) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1091) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:512) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1084) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1060) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1029) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:948) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
Caused by: java.lang.NoClassDefFoundError: io/grpc/internal/BaseDnsNameResolverProvider
	at java.base/java.lang.ClassLoader.defineClass1(Native Method) ~[na:na]
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[na:na]
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:151) ~[na:na]
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:821) ~[na:na]
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:719) ~[na:na]
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:642) ~[na:na]
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:600) ~[na:na]
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[na:na]
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
	at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
	at java.base/java.lang.Class.forName(Class.java:427) ~[na:na]
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1211) ~[na:na]
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1222) ~[na:na]
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1266) ~[na:na]
	at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1301) ~[na:na]
	at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1386) ~[na:na]
	at io.grpc.ServiceProviders.loadAll(ServiceProviders.java:67) ~[grpc-api-1.29.0.jar:1.29.0]
	at io.grpc.NameResolverRegistry.getDefaultRegistry(NameResolverRegistry.java:101) ~[grpc-api-1.29.0.jar:1.29.0]
	at io.grpc.internal.AbstractManagedChannelImplBuilder.<init>(AbstractManagedChannelImplBuilder.java:107) ~[grpc-core-1.29.0.jar:1.29.0]
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.<init>(NettyChannelBuilder.java:136) ~[grpc-netty-shaded-1.28.1.jar:1.28.1]
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.<init>(NettyChannelBuilder.java:131) ~[grpc-netty-shaded-1.28.1.jar:1.28.1]
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.forAddress(NettyChannelBuilder.java:117) ~[grpc-netty-shaded-1.28.1.jar:1.28.1]
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForAddress(NettyChannelProvider.java:37) ~[grpc-netty-shaded-1.28.1.jar:1.28.1]
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForAddress(NettyChannelProvider.java:23) ~[grpc-netty-shaded-1.28.1.jar:1.28.1]
	at io.grpc.ManagedChannelBuilder.forAddress(ManagedChannelBuilder.java:39) ~[grpc-api-1.29.0.jar:1.29.0]
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:270) ~[gax-grpc-1.56.0.jar:1.56.0]
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.access$1500(InstantiatingGrpcChannelProvider.java:71) ~[gax-grpc-1.56.0.jar:1.56.0]
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$1.createSingleChannel(InstantiatingGrpcChannelProvider.java:202) ~[gax-grpc-1.56.0.jar:1.56.0]
	at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:72) ~[gax-grpc-1.56.0.jar:1.56.0]
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:209) ~[gax-grpc-1.56.0.jar:1.56.0]
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:192) ~[gax-grpc-1.56.0.jar:1.56.0]
	at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:155) ~[gax-1.56.0.jar:1.56.0]
	at com.google.cloud.pubsub.v1.stub.GrpcSubscriberStub.create(GrpcSubscriberStub.java:263) ~[google-cloud-pubsub-1.105.1.jar:1.105.1]
	at com.google.cloud.pubsub.v1.Subscriber.doStart(Subscriber.java:271) ~[google-cloud-pubsub-1.105.1.jar:1.105.1]
	at com.google.api.core.AbstractApiService$InnerService.doStart(AbstractApiService.java:148) ~[api-common-1.9.0.jar:na]
	at com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:249) ~[guava-29.0-jre.jar:na]
	at com.google.api.core.AbstractApiService.startAsync(AbstractApiService.java:120) ~[api-common-1.9.0.jar:na]
	at com.google.cloud.pubsub.v1.Subscriber.startAsync(Subscriber.java:263) ~[google-cloud-pubsub-1.105.1.jar:1.105.1]
	at co.x.api.usage_handler.UsageHandlerService.subscribe(UsageHandlerService.java:96) ~[main/:na]
	at co.x.api.usage_handler.Application.run(Application.java:32) ~[main/:na]
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:795) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
	at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:779) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:322) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
	at co.x.api.usage_handler.Application.main(Application.java:27) ~[main/:na]
Caused by: java.lang.ClassNotFoundException: io.grpc.internal.BaseDnsNameResolverProvider
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:602) ~[na:na]
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[na:na]
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
	... 44 common frames omitted

The root cause is:

Caused by: java.lang.ClassNotFoundException: io.grpc.internal.BaseDnsNameResolverProvider

In the classpath I have protobuf, guava, lettuce and spring-boot-starter (all up-to-date). Java 14 is used.

A workaround is to downgrade to version 1.104.1.
@danoscarmike danoscarmike transferred this issue from googleapis/google-cloud-java May 27, 2020
@product-auto-label product-auto-label bot added the api: pubsub Issues related to the googleapis/java-pubsub API. label May 27, 2020
@yoshi-automation yoshi-automation added triage me I really want to be triaged. 🚨 This issue needs some love. labels May 27, 2020
@kamalaboulhosn
Copy link
Contributor

@hannahrogers-google worth making sure there isn't some dependency issue in the 1.105 release.

@hannahrogers-google hannahrogers-google added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p2 Moderately-important priority. Fix may not be included in next release. and removed 🚨 This issue needs some love. triage me I really want to be triaged. labels Jun 1, 2020
@hannahrogers-google
Copy link
Contributor

It looks like there were a number of dependency updates in 1.105.1:

https://github.com/googleapis/java-pubsub/releases/tag/v1.105.1

I will try and reproduce this issue.

@hannahrogers-google
Copy link
Contributor

hannahrogers-google commented Jun 4, 2020 •
edited

It looks like we upgraded to grpc v1.29.0 in this release, and io.grpc.internal.DnsNameResolverProvider.enable_grpclb was removed in this version:

https://github.com/grpc/grpc-java/releases/tag/v1.29.0

However, with version 1.105.1, I was able to successfully shutdown a subscriber without running into the above issue. @lpellegr, can you tell me more about the environment you are running in?

@davinchia
Copy link

davinchia commented Jun 5, 2020 •
edited

fwiw, I'm also running into this while trying to run the basic Java examples from https://github.com/googleapis/google-cloud-java/blob/master/google-cloud-examples/src/main/java/com/google/cloud/examples/pubsub/snippets/CreateTopicAndPublishMessages.java.

My gradle file only has one dependency:

dependencies {
  implementation 'com.google.cloud:google-cloud-pubsub:1.106.1'
}

Here is my stacktrace -

Exception in thread "main" java.lang.NoClassDefFoundError: io/grpc/internal/BaseDnsNameResolverProvider
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	at io.grpc.ServiceProviders.loadAll(ServiceProviders.java:67)
	at io.grpc.NameResolverRegistry.getDefaultRegistry(NameResolverRegistry.java:101)
	at io.grpc.internal.AbstractManagedChannelImplBuilder.<init>(AbstractManagedChannelImplBuilder.java:107)
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.<init>(NettyChannelBuilder.java:136)
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.<init>(NettyChannelBuilder.java:131)
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.forAddress(NettyChannelBuilder.java:117)
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForAddress(NettyChannelProvider.java:37)
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForAddress(NettyChannelProvider.java:23)
	at io.grpc.ManagedChannelBuilder.forAddress(ManagedChannelBuilder.java:39)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:270)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.access$1500(InstantiatingGrpcChannelProvider.java:71)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$1.createSingleChannel(InstantiatingGrpcChannelProvider.java:202)
	at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:72)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:209)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:192)
	at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:155)
	at com.google.cloud.pubsub.v1.stub.GrpcPublisherStub.create(GrpcPublisherStub.java:164)
	at com.google.cloud.pubsub.v1.Publisher.<init>(Publisher.java:187)
	at com.google.cloud.pubsub.v1.Publisher.<init>(Publisher.java:88)
	at com.google.cloud.pubsub.v1.Publisher$Builder.build(Publisher.java:798)

I tracked this down to grpc/grpc-java#7002 before figuring out downgrading to 1.104.1 worked for me.

Trying to proposed solutions in the linked grpc-java issue did not work for me.

@hannahrogers-google
Copy link
Contributor

Thanks davinchia, I was able to reproduce this issue running the above sample with gradle. Looking at ./gradlew dependencies, gax (version 1.56.0) depends on grpc-alts 1.28.1, which is why we are seeing this issue. Forcing the 'io.grpc:grpc-alts' to 1.29.0 seems to resolve the issue.

@chingor13, any plans to upgrade the grpc dependency for gax? Or would it be better for us to revert back to grpc 1.28.1?

@chingor13 chingor13 mentioned this issue Jun 9, 2020
Merged
@chingor13
Copy link
Contributor

Ideally, the user should use on of our BOMs for dependency conflict resolution like com.google.cloud:libraries-bom: https://github.com/googleapis/java-pubsub#quickstart

Also opened a PR to update grpc/protobuf in gax: googleapis/gax-java#1125

@hannahrogers-google
Copy link
Contributor

Fixed in v1.108.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hannahrogers-google hannahrogers-google

Labels
api: pubsub Issues related to the googleapis/java-pubsub API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@chingor13 @lpellegr @kamalaboulhosn @davinchia @yoshi-automation @hannahrogers-google