Allow BoundedExecutor to recover from failures

[electrum]

Previously submitted tasks will eventually run if the underlying
executor recovers and additional tasks are submitted.

[findepi]

if coreExecutor.execute(this::drainQueue) throws immediately, you can end up having 1 task in queue and queueSize=0.

To me, it looks like you want something like "current thread count" instead of "queue size". Question is how to do this without losing an incoming task.

[findepi]

unnecessary parenthesis

[findepi]

With maxThreads=1 consider

1. thread A calls execute
   a. adds task
   b. queueSize := 1
   c. starts a thread A'
2. thread A' removes the task from the queue and executes it
3. thread A' calls decrementAndGetQueueSize, i.e. queueSize := 0, but does not exit because task was not null
4. thread B (or A) calls execute
   a. adds task
   b. queueSize := 1
   c. starts a thread B', because queueSize <= maxThreads
5. thread C calls execute
   a. adds task
   b. queueSize := 2
   c. doesn't start a thread

At this point, 2 threads A' and B' are executing within coreExecutor even though maxThreads is 1 and both will attempt to poll & execute next task.

[findepi]

(updated)

[Praveen2112]

But if we could rearrange the if condition to task == null && decrementAndGetQueueSize() == 0 then threadB' won't be created as queueSize := 0 only if the task == null or maybe we can peek into the queue to check for the next task
decrementAndGetQueueSize == 0 && queue.peek() == null

[erichwang]

@electrum: to achieve the desired behavior, we really only have 3 possible options here:

1. Make the bounded executor fully synchronized for all task submissions (only allowing one submission at a time) -- but this might add unwanted overhead in high performance cases since none of the existing code or dependencies use any locks.
2. Add a scheduled background thread pool whose job is to keep retrying failed task submission retries forever until success
3. Consume the calling thread as one of the draining tasks in the event of thread pool failure -- while technically this wouldn't corrupt the data structure, this would be terrible for callers and so this isn't really a desirable path.

There is no way to make this magically work without one of these three options. For me, I would suggest (2).