WatchDog shuts down ExecutorService from FixedExecutorProvider

[chrisribble]

#838 modified WatchDog to implement BackgroundResource and to track a Future for the work scheduled via the ExecutorService so that it can be cancelled. This is perfectly reasonable, but WatchDog.shutdown also calls ExecutorService.shutdown

This is problematic when you provide a FixedExecutorProvider to ClientContex.create since it uses the same ExecutorService instance (from the ExecutorProvider) for the WatchDog, which makes FixedExecutorProvider practically useless (since the ExecutorService that it provides can no longer be shared between clients, as shutting one down will shut down the ExecutorService)

I noticed this issue in our application while attempting to upgrade to google-cloud-core 1.92.5. We use java-pubsub 1.102.1 and we share the same ExecutorService among multiple dynamically-created Publisher instances. As soon as one is shut down, they all become inoperable since the underlying ExecutorService that they share is shut down.

Obviously, I could work around this by using an InstantiatingExecutorProvider, but I don't want to allocate new threads for every Publisher that my application creates (since the java-pubsub library requires the creation of a Publisher per topic and we publish to dynamically-created topics).

Possible solutions:

1. Remove the call to executor.shutdown in WatchDog.shutdown
2. Modify WatchDogProvider such that it accepts (and uses) an ExecutorProvider and checks ExecutorProvider.shouldAutoClose before calling shutdown on the ExecutorService
3. Something else?

[igorbernstein2]

I think the first option makes the most sense. ClientContext should be the primary owner of the executor

[rgrebski]

Should executor.shutdownNow() be removed from Watchdog.shutdownNow() as well ?

Since fix of #22, exactly this line, Subscriber.doStop is calling background resources shutdown, resulting with Watchdog.shutdownNow() being executed and ExecutorService being shutdown.

[igorbernstein2]

Yes, that was an oversight. Thanks for pointing it out.