PubSub: Java - https.proxyHost no longer works after version 1.88.0

[andreasmh]

Hello!
I am trying to connect to PubSub via a proxy.
I am using java and the google-cloud-pubsub dependency.
Initial testing shows it is working as expected in version 1.87.0 and earlier
On version 1.88.0 and later it is not working.

Steps to reproduce

1. Provide the properties https.proxyHost and https.proxyPort when invoking the JVM
   OR the environment variable GRPC_PROXY_EXP
2. Run any code trying to connect to pubsub using version 1.88.0 or later.

    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-pubsub</artifactId>
      <version>1.88.0</version>
    </dependency>

I was able to reproduce this with PublisherExample

You will now get a an error saying DEADLINE_EXCEEDED:

Stack trace

Exception in thread "main" java.util.concurrent.ExecutionException: com.google.api.gax.rpc.DeadlineExceededException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9965016637ns. [buffered_nanos=9973098551, waiting_for_connection]
	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:552)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:533)
	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:90)
	at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:62)
	at com.example.pubsub.PublisherExample.main(PublisherExample.java:69)
Caused by: com.google.api.gax.rpc.DeadlineExceededException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9965016637ns. [buffered_nanos=9973098551, waiting_for_connection]
	at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:51)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
	at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1056)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1139)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748)
	at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:515)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:490)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:510)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:630)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:518)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:692)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:681)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9965016637ns. [buffered_nanos=9973098551, waiting_for_connection]
	at io.grpc.Status.asRuntimeException(Status.java:533)
	... 23 more

External references such as API reference guides used

Setting up proxy as explained here:
https://github.com/googleapis/google-cloud-java#using-a-proxy

Thanks!

[andreasmh]

I found this issue in the grpc github: grpc/grpc-java#6118
I guess that could be the culprit?
The grpc dependency was bumped to 1.23.0 from 1.21.0 in version 0.106.0

[chingor13]

Sorry for the delay on responding to this. Yes, grpc had an issue with proxy usage in grpc 1.22.0 and 1.23.0. Upgrading grpc to 1.24.1 or lager (or pubsub to 1.99.0 or later) will fix this issue.

[rednitish]

Hi @chingor13
I am getting same issue with version 1.26.0 and 1.25.0, Is this fix not available in these version ?

[waytoalpit]

@chingor13 i am also getting same issue with grpc-netty-shaded/1.25.0/grpc-netty-shaded-1.25.0.jar

[waytoalpit]

Can someone update me if there is any fix out?