Cannot use Google Cloud Logging and Google Cloud Observability together (IllegalStateException: Set cannot be called after any get call)

[pkbriggs]

What version of gRPC-Java are you using?

1.62.2

What is your environment?

Ubuntu 20.04.6 LTS, openjdk 17.0.10

What did you expect to see?

Be able to successfully run an application with Google Cloud Logging + Google Cloud Observability

What did you see instead?

An exception is thrown when running the application:

Exception in thread "main" java.lang.IllegalStateException: Set cannot be called after any get call
	at io.grpc.GlobalInterceptors.setInterceptorsTracers(GlobalInterceptors.java:61)
	at io.grpc.InternalGlobalInterceptors.setInterceptorsTracers(InternalGlobalInterceptors.java:29)
	at io.grpc.gcp.observability.GcpObservability.setProducer(GcpObservability.java:164)
	at io.grpc.gcp.observability.GcpObservability.grpcInit(GcpObservability.java:116)
	at io.grpc.gcp.observability.GcpObservability.grpcInit(GcpObservability.java:98)

Steps to reproduce the bug

Minimal app showing issue: https://github.com/pkbriggs/gcp-logging-observability-issue

1. Create new Java app that uses both Google Cloud Logging (https://cloud.google.com/logging/docs/setup/java) and Google Cloud Observability (https://cloud.google.com/stackdriver/docs/solutions/grpc/set-up-observability)
2. Add a logback.xml as described in the GCP Logging docs
3. Run app (GRPC_GCP_OBSERVABILITY_CONFIG="{}" ./gradlew run) and observe that an exception is thrown when GcpObservability.grpcInit() is called

If you delete/rename src/main/resources/logback.xml and run the app, it will run successfully. This seems to indicate there is some issue with com.google.cloud.logging.logback.LoggingAppender and io.grpc.gcp.observability.GcpObservability.

[ejona86]

For the reproduction, the "simple solution" is "don't log before GcpObservability.grpcInit()". It might even require delay calling LoggerFactory. But I understand that is easier said than done in a real app, and may not be desired.

Even if you get it running, there are serious risks of forming a cycle between GCP Logging and GcpObservability.

@DNVindhya, do you have ideas?

[pkbriggs]

Thanks for the reply! Unfortunately the issue still persists if the logger.info call is moved after GcpObservability.grpcInit(). Deferring LoggerFactory.getLogger does seem to fix the issue but, as you pointed out, would be burdensome in our app.

I'm a bit confused about the intended use case for the Observability and Logging products: given Observability hooks up to GCP Logging, does that mean we can remove our direct dependency on GCP Logging? Is Observability meant to supersede the individual Logging and Metrics integrations? Are they intended to work together? It would be a bit silly to say "you can either have observability or logging, but not both!"