Fix Glue metastore deadlock when getting statistics #21954
Conversation
findepi
force-pushed
the
findepi/fix-glue-metastore-deadlock-when-getting-statistics-dc0613
branch
from
4931ddd
to
e6446fa
Compare
| import static java.util.Collections.nCopies; | ||
| import static java.util.Objects.requireNonNull; | ||
|
|
||
| public final class HiveExecutorUtil |
There was a problem hiding this comment.
Let's not add a new utility method for this. We can extract this later when we need to reuse the code.
There was a problem hiding this comment.
i find utility class better isolated and thus easier to reason about from algorithmic perspective
i will make this package private and move to glue package though
| // process ready results to fail fast on exceptions | ||
| for (Future<TaskResult<T>> ready = completionService.poll(); ready != null; ready = completionService.poll()) { | ||
| TaskResult<T> taskResult = ready.get(); | ||
| if (taskResult != null) { |
There was a problem hiding this comment.
Can you add a comment here explaining that null means that the task was completed inline so the result is already set?
There was a problem hiding this comment.
added // Null result means task was processed by the calling thread
| } | ||
| catch (InterruptedException e) { | ||
| Thread.currentThread().interrupt(); | ||
| futures.forEach(future -> future.cancel(true)); |
There was a problem hiding this comment.
Move this to a finally block to ensure the futures are always cancled, so we don't do any unnecessary work (for example when we throw ExecutionException above)
findepi
force-pushed
the
findepi/fix-glue-metastore-deadlock-when-getting-statistics-dc0613
branch
2 times, most recently
from
d59c4eb
to
b61ccf5
Compare
Glue metastore uses `runParallel` utility to parallelize getting statistics across partitions, but also for one partition to parallelize getting stats for column batches. The underlying executor is fixed size, to this may lead to a dead-lock. This commit avoids the dead-lock by work stealing, i.e. asking the calling thread to participate in the work.
findepi
force-pushed
the
findepi/fix-glue-metastore-deadlock-when-getting-statistics-dc0613
branch
from
b61ccf5
to
1f9f747
Compare
findepi
deleted the
findepi/fix-glue-metastore-deadlock-when-getting-statistics-dc0613
branch
Glue metastore uses
runParallelutility to parallelize getting statistics across partitions, but also for one partition to parallelize getting stats for column batches. The underlying executor is fixed size, to this may lead to a dead-lock. This commit avoids the dead-lock by work stealing, i.e. asking the calling thread to participate in the work.