Batch read #10

horizonzy · 2023-11-15T09:01:17Z

Descriptions of the changes in this PR:

Motivation

(Explain: why you're making that change, what is the problem you're trying to solve)

Changes

(Describe: what changes you have made)

Master Issue: #

In order to uphold a high standard for quality for code contributions, Apache BookKeeper runs various precommit
checks for pull requests. A pull request can only be merged when it passes precommit checks.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

If this PR is a BookKeeper Proposal (BP):

Make sure the PR title is formatted like:
<BP-#>: Description of bookkeeper proposal
e.g. BP-1: 64 bits ledger is support

Attach the master issue link in the description of this PR.

Attach the google doc link if the BP is written in Google Doc.

Otherwise:

Make sure the PR title is formatted like:
<Issue #>: Description of pull request
e.g. Issue 123: Description ...

Make sure tests pass via mvn clean apache-rat:check install spotbugs:check.

Replace <Issue #> in the title with the actual Issue number.

# Conflicts: # bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieProtoEncoding.java # bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/ReadEntryProcessor.java

…AvailableException`.

hangc0276

We need record the READ_ENTRY_SCHEDULING_DELAY stat in PacketProcessorBase#run()

...eper-benchmark/src/main/java/org/apache/bookkeeper/benchmark/BenchReadThroughputLatency.java

hangc0276 · 2023-12-07T13:18:11Z

bookkeeper-proto/src/main/proto/BookkeeperProtocol.proto

@@ -66,6 +66,7 @@ enum OperationType {
    START_TLS = 9;
    FORCE_LEDGER = 10;
    GET_LIST_OF_ENTRIES_OF_LEDGER = 11;
+    BATCH_READ_ENTRY = 12;


We are not planning to add batch read support in the V3 protocol. I think this protocol change can leave to the next PR, which support batch read on V3 protocal

We define all the op code using V3 to avoid conflict. See https://github.com/apache/bookkeeper/blob/5be03cfdfb4dbe4ad37bd2502fe514d9b347cd26/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java#L945

bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ClientInternalConf.java

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieProtocol.java

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieProtoEncoding.java

hangc0276 · 2023-12-07T14:14:46Z

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieProtoEncoding.java

+                    for (int i = 0; i < brr.getData().size(); i++) {
+                        ByteBuf entryData = brr.getData().getBuffer(i);
+                        buf.writeInt(entryData.readableBytes());
+                        buf.writeBytes(entryData);


If the payloadSize is less than SMALL_ENTRY_SIZE_THRESHOLD, we can use buf.writeBytes(entryData) to copy byebuf data. Otherwise, we can use ByteBufList to avoid data copy.

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieProtoEncoding.java

hangc0276 · 2023-12-07T14:26:20Z

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BatchedReadEntryProcessor.java

+    protected ReferenceCounted readData() throws Exception {
+        ByteBufList data = null;
+        BatchedReadRequest batchRequest = (BatchedReadRequest) request;
+        long maxSize = batchRequest.getMaxSize();


We need to calculate the maxSize based on min(nettyMaxFrameSizeBytes - (36 + 4 * batchRequest.getMaxCount()), batchRequest.getMaxSize())

Thanks for the reminder, the nettyMaxFrameSizeBytes is for decode operation. So the server side nettyMaxFrameSizeBytes does not work.

The maxSize should be batchRequest.getMaxSize() - (36 + 4 * batchRequest.getMaxCount())

We should let the server decide the frame size. Because you never know how the client set it. If a client sets it too huge, the server will read a lot of entries and make it down.

hangc0276

I completed my first review of the protocol and server-side code. We will continue to review the client tomorrow.

...eper-benchmark/src/main/java/org/apache/bookkeeper/benchmark/BenchReadThroughputLatency.java

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BatchedReadEntryProcessor.java

bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ClientConfiguration.java

hangc0276 · 2023-12-08T10:13:04Z

bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ClientConfiguration.java

@@ -2077,6 +2080,10 @@ public long getClientConnectBookieUnavailableLogThrottlingMs() {
        return getLong(CLIENT_CONNECT_BOOKIE_UNAVAILABLE_LOG_THROTTLING, 5_000L);
    }

+    public boolean isBatchReadFailBackToSingleRead() {


Please add the set method and add explanation for the method.

bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java

bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BatchedReadOp.java

hangc0276 · 2023-12-08T15:30:09Z

Please test the fallback logic using the batch read API

bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java

hangc0276 · 2023-12-11T08:34:52Z

bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BatchedReadOp.java

+
+    abstract class BatchedLedgerEntryRequest extends LedgerEntryRequest {
+
+        final long lId;


Do we need to store the ledgerId?

It's useful for tracing problems. We can get it from the heap dump to know which ledger is being batch read.

make sense. Please add a comment on the lId to avoid being removed in the future.

horizonzy

addressed

...eper-benchmark/src/main/java/org/apache/bookkeeper/benchmark/BenchReadThroughputLatency.java

horizonzy · 2023-12-12T04:47:56Z

bookkeeper-proto/src/main/proto/BookkeeperProtocol.proto

@@ -66,6 +66,7 @@ enum OperationType {
    START_TLS = 9;
    FORCE_LEDGER = 10;
    GET_LIST_OF_ENTRIES_OF_LEDGER = 11;
+    BATCH_READ_ENTRY = 12;


We define all the op code using V3 to avoid conflict. See https://github.com/apache/bookkeeper/blob/5be03cfdfb4dbe4ad37bd2502fe514d9b347cd26/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java#L945

horizonzy · 2023-12-12T04:50:12Z

bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BatchedReadOp.java

+
+    abstract class BatchedLedgerEntryRequest extends LedgerEntryRequest {
+
+        final long lId;


It's useful for tracing problems. We can get it from the heap dump to know which ledger is being batch read.

bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ClientConfiguration.java

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieProtoEncoding.java

horizonzy · 2023-12-12T07:24:32Z

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieProtoEncoding.java

+                    for (int i = 0; i < brr.getData().size(); i++) {
+                        ByteBuf entryData = brr.getData().getBuffer(i);
+                        buf.writeInt(entryData.readableBytes());
+                        buf.writeBytes(entryData);


bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieProtoEncoding.java

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieProtocol.java

zymap · 2023-12-13T03:19:36Z

bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java

@@ -760,6 +798,106 @@ public CompletableFuture<LedgerEntries> readAsync(long firstEntry, long lastEntr
        return readEntriesInternalAsync(firstEntry, lastEntry, false);
    }

+    @Override
+    public CompletableFuture<LedgerEntries> readAsync(long startEntry, int maxCount, long maxSize) {


We shouldn't provide the fallback logic in the normal read method. We should provide a new method named with fallback. https://github.com/apache/bookkeeper/pull/4051/files#diff-4a1b643197f23e320d89550d6ae641179955e03f61332c69a82e10cacaa12390R62

zymap · 2023-12-13T03:23:54Z

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BatchedReadEntryProcessor.java

+    protected ReferenceCounted readData() throws Exception {
+        ByteBufList data = null;
+        BatchedReadRequest batchRequest = (BatchedReadRequest) request;
+        long maxSize = batchRequest.getMaxSize();


We should let the server decide the frame size. Because you never know how the client set it. If a client sets it too huge, the server will read a lot of entries and make it down.

hangc0276 · 2023-12-14T08:14:02Z

...eper-benchmark/src/main/java/org/apache/bookkeeper/benchmark/BenchReadThroughputLatency.java

@@ -121,6 +126,7 @@ private static void readLedger(ClientConfiguration conf, long ledgerId, byte[] p
                lh.close();
                lh = null;
                Thread.sleep(1000);
+                break;


Why add break here?

bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java

hangc0276 · 2023-12-14T08:32:36Z

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BatchedReadEntryProcessor.java

+        if (maxCount <= 0) {
+            maxCount = Integer.MAX_VALUE;
+        }
+        long maxSize = Math.min(batchRequest.getMaxSize(), maxBatchReadSize);


We need to check if the batchRequest.getMaxSize() or maxBatchReadSize < 0

Now, although the maxSize or maxBatchReadSize < 0, it will still read one entry.

If we check it and then find that the value < 0, what should we do?

For current logic, if the maxSize <0, it won't return any entries to the client and it will be hard to debug what happens. At least we need to add a debug level log.

No, we will at least add one entry to the response whatever how size of the maxSize.
We just check the maxSize when the ByteBufList is not null.

hangc0276 · 2023-12-14T08:56:24Z

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java

+        if (useV2WireProtocol) {
+            request = BookieProtocol.BatchedReadRequest.create(BookieProtocol.CURRENT_PROTOCOL_VERSION,
+                    ledgerId, startEntryId, (short) flags, masterKey, txnId, maxCount, maxSize);
+            completionKey = new V3CompletionKey(txnId, OperationType.BATCH_READ_ENTRY);


We are using the V2 protocol, why do we use V3CompletionKey?

We'd better define a V2BatchedCompletionKey?

How about renaming V3CompletionKey to TxnCompletionKey

hangc0276 · 2023-12-14T09:10:52Z

bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookKeeperTest.java

                    ledgerId = lh.getId();
                    for (int i = 0; i < numEntries; i++) {
                        lh.addEntry(data);
                    }
                }
                try (LedgerHandle lh = bkc.openLedger(ledgerId, digestType, "testPasswd".getBytes())) {
                    assertEquals(numEntries - 1, lh.readLastConfirmed());
-                    for (Enumeration<LedgerEntry> readEntries = lh.readEntries(0, numEntries - 1);
+                    for (Enumeration<LedgerEntry> readEntries = lh.readEntries(0, numEntries, -1);


We should create a new test instead of modifing the existing tests.

Ok, I will create more new tests for the batch read.

hangc0276 · 2023-12-14T09:28:21Z

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieProtoEncoding.java

+                        buffer.readerIndex(entryPos + entrySize);
+                    }
+                }
+                return new BookieProtocol.BatchedReadResponse(version, rc, ledgerId, entryId, requestId, data == null


Why not pass null directly instead of pass an empty ByteBufList when data is null?

Using empty ByteBufList can avoid NPE. And I check the existing code; it return null directly when the rc != Ok. I will follow the code style.

I checked it again; we would better use empty ByteBufList, or BatchedReadResponse#touch()) will throw NPE.

Make sense. Let's use empty ByteBufList

hangc0276 · 2023-12-14T09:29:36Z

bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BatchedReadOp.java

+        final ReadContext rctx = (ReadContext) ctx;
+        final BatchedLedgerEntryRequest entry = (BatchedLedgerEntryRequest) rctx.entry;
+
+        if (rc != BKException.Code.OK) {


If the rc is not OK but the bufList is an empty list, it may lead to a direct memory leak.

Ok. The empty bufList didn't occupy any memory. I will return null if the rc != OK

hangc0276 · 2023-12-14T09:30:50Z

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/ResponseBuilder.java

+            assert(r.getOpCode() == BookieProtocol.BATCH_READ_ENTRY);
+            return new BookieProtocol.BatchedReadResponse(r.getProtocolVersion(), errorCode,
+                    r.getLedgerId(), r.getEntryId(), ((BookieProtocol.BatchedReadRequest) r).getRequestId(),
+                    ByteBufList.get());


Why pass an empty ByteBufList here

Because it build error response, the reason is same with above, I will return null directly.

hangc0276 · 2023-12-14T09:31:37Z

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieClient.java

+        readEntries(address, ledgerId, startEntryId, maxCount, maxSize, cb, ctx, flags, masterKey, false);
+    }
+
+    void readEntries(BookieId address, long ledgerId, long startEntryId,


Please add the readEntryWithFallback API in LedgerHandle as we discussed.

hangc0276

Overall LGTM. Please fix the check style issues.

We can start to split PRs and push them to the open source BookKeeper repo.

hangc0276 · 2023-12-25T09:33:55Z

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BatchedReadEntryProcessor.java

+        if (maxCount <= 0) {
+            maxCount = Integer.MAX_VALUE;
+        }
+        long maxSize = Math.min(batchRequest.getMaxSize(), maxBatchReadSize);


For current logic, if the maxSize <0, it won't return any entries to the client and it will be hard to debug what happens. At least we need to add a debug level log.

### Motivation The auditor didn't close the ledger manager and ledger under replication manager the thread will leak. Run the test `AuditorPeriodicBookieCheckTest` and get the stack. ## Before this PR: ``` Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.0.15+8-LTS-149 mixed mode): Threads class SMR info: _java_thread_list=0x0000600003e44460, length=16, elements={ 0x00007fad34009000, 0x00007fad45032800, 0x00007fad14813800, 0x00007fad44010800, 0x00007fad44011800, 0x00007fad45037800, 0x00007fad3400a000, 0x00007fad45812800, 0x00007fad34944000, 0x00007fad4585f800, 0x00007fad45456000, 0x00007fad4444c000, 0x00007fad3431b800, 0x00007face5825800, 0x00007face5860800, 0x00007face4009800 } "main" #1 prio=5 os_prio=31 cpu=1801.68ms elapsed=20.85s tid=0x00007fad34009000 nid=0x2a03 waiting on condition [0x000000030a10d000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method) - parking to wait for <0x000000079f26a8a0> (a java.util.concurrent.FutureTask) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.15/LockSupport.java:234) at java.util.concurrent.FutureTask.awaitDone(java.base@11.0.15/FutureTask.java:444) at java.util.concurrent.FutureTask.get(java.base@11.0.15/FutureTask.java:203) at org.junit.internal.runners.statements.FailOnTimeout.getResult(FailOnTimeout.java:141) at org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:127) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69) at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38) at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11) at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35) at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235) at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54) Locked ownable synchronizers: - None "Reference Handler" #2 daemon prio=10 os_prio=31 cpu=2.14ms elapsed=20.82s tid=0x00007fad45032800 nid=0x4803 waiting on condition [0x000000030a823000] java.lang.Thread.State: RUNNABLE at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.15/Native Method) at java.lang.ref.Reference.processPendingReferences(java.base@11.0.15/Reference.java:241) at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.15/Reference.java:213) Locked ownable synchronizers: - None "Finalizer" #3 daemon prio=8 os_prio=31 cpu=1.33ms elapsed=20.82s tid=0x00007fad14813800 nid=0x5503 in Object.wait() [0x000000030a926000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.15/Native Method) - waiting on <0x00000007801b6f18> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(java.base@11.0.15/ReferenceQueue.java:155) - waiting to re-lock in wait() <0x00000007801b6f18> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(java.base@11.0.15/ReferenceQueue.java:176) at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.15/Finalizer.java:170) Locked ownable synchronizers: - None "Signal Dispatcher" #4 daemon prio=9 os_prio=31 cpu=2.69ms elapsed=20.81s tid=0x00007fad44010800 nid=0x5b03 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Service Thread" #5 daemon prio=9 os_prio=31 cpu=0.03ms elapsed=20.81s tid=0x00007fad44011800 nid=0x7403 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread0" #6 daemon prio=9 os_prio=31 cpu=1986.68ms elapsed=20.81s tid=0x00007fad45037800 nid=0x5d03 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE No compile task Locked ownable synchronizers: - None "C1 CompilerThread0" #9 daemon prio=9 os_prio=31 cpu=769.61ms elapsed=20.81s tid=0x00007fad3400a000 nid=0x7003 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE No compile task Locked ownable synchronizers: - None "Sweeper thread" #10 daemon prio=9 os_prio=31 cpu=0.08ms elapsed=20.81s tid=0x00007fad45812800 nid=0x6e03 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Common-Cleaner" apache#11 daemon prio=8 os_prio=31 cpu=11.62ms elapsed=20.73s tid=0x00007fad34944000 nid=0x6b03 in Object.wait() [0x000000030b049000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.15/Native Method) - waiting on <0x00000007801b7550> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(java.base@11.0.15/ReferenceQueue.java:155) - waiting to re-lock in wait() <0x00000007801b7550> (a java.lang.ref.ReferenceQueue$Lock) at jdk.internal.ref.CleanerImpl.run(java.base@11.0.15/CleanerImpl.java:148) at java.lang.Thread.run(java.base@11.0.15/Thread.java:834) at jdk.internal.misc.InnocuousThread.run(java.base@11.0.15/InnocuousThread.java:134) Locked ownable synchronizers: - None "Monitor Ctrl-Break" apache#12 daemon prio=5 os_prio=31 cpu=21.16ms elapsed=20.65s tid=0x00007fad4585f800 nid=0x6a03 runnable [0x000000030b14c000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(java.base@11.0.15/Native Method) at java.net.SocketInputStream.socketRead(java.base@11.0.15/SocketInputStream.java:115) at java.net.SocketInputStream.read(java.base@11.0.15/SocketInputStream.java:168) at java.net.SocketInputStream.read(java.base@11.0.15/SocketInputStream.java:140) at sun.nio.cs.StreamDecoder.readBytes(java.base@11.0.15/StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(java.base@11.0.15/StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(java.base@11.0.15/StreamDecoder.java:178) - locked <0x00000007801b81e8> (a java.io.InputStreamReader) at java.io.InputStreamReader.read(java.base@11.0.15/InputStreamReader.java:181) at java.io.BufferedReader.fill(java.base@11.0.15/BufferedReader.java:161) at java.io.BufferedReader.readLine(java.base@11.0.15/BufferedReader.java:326) - locked <0x00000007801b81e8> (a java.io.InputStreamReader) at java.io.BufferedReader.readLine(java.base@11.0.15/BufferedReader.java:392) at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:56) Locked ownable synchronizers: - None "Log4j2-TF-2-Scheduled-1" apache#13 daemon prio=5 os_prio=31 cpu=3.80ms elapsed=19.18s tid=0x00007fad45456000 nid=0x8c03 waiting on condition [0x000000030bb6a000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method) - parking to wait for <0x00000007805c1f78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.15/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.15/AbstractQueuedSynchronizer.java:2123) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:1182) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.15/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.15/Thread.java:834) Locked ownable synchronizers: - None "Time-limited test" apache#15 daemon prio=5 os_prio=31 cpu=1344.20ms elapsed=18.79s tid=0x00007fad4444c000 nid=0x9107 waiting on condition [0x000000030bd70000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@11.0.15/Native Method) at org.apache.bookkeeper.replication.AuditorPeriodicBookieCheckTest.tearDown(AuditorPeriodicBookieCheckTest.java:81) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.15/Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.15/NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.15/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(java.base@11.0.15/Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(java.base@11.0.15/FutureTask.java:264) at java.lang.Thread.run(java.base@11.0.15/Thread.java:834) Locked ownable synchronizers: - None "ZkLedgerManagerScheduler-48-1" apache#165 prio=5 os_prio=31 cpu=9.25ms elapsed=16.04s tid=0x00007fad3431b800 nid=0x25b03 waiting on condition [0x0000000314d1a000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method) - parking to wait for <0x000000079db6aed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.15/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:1170) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.15/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(java.base@11.0.15/Thread.java:834) Locked ownable synchronizers: - None "ZkLedgerManagerScheduler-49-1" apache#166 prio=5 os_prio=31 cpu=1.01ms elapsed=16.04s tid=0x00007face5825800 nid=0x25903 waiting on condition [0x0000000314e1d000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method) - parking to wait for <0x000000079db6b3f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.15/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:1170) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.15/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(java.base@11.0.15/Thread.java:834) Locked ownable synchronizers: - None "ForkJoinPool.commonPool-worker-19" apache#187 daemon prio=5 os_prio=31 cpu=1.52ms elapsed=15.64s tid=0x00007face5860800 nid=0x2ca03 waiting on condition [0x0000000316259000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method) - parking to wait for <0x000000079ef3f6e8> (a java.util.concurrent.ForkJoinPool) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194) at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.15/ForkJoinPool.java:1628) at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.15/ForkJoinWorkerThread.java:183) Locked ownable synchronizers: - None "Attach Listener" apache#313 daemon prio=9 os_prio=31 cpu=1.90ms elapsed=0.24s tid=0x00007face4009800 nid=0x3d53b waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "VM Thread" os_prio=31 cpu=122.89ms elapsed=20.84s tid=0x00007fad1405f000 nid=0x4e03 runnable "GC Thread#0" os_prio=31 cpu=67.63ms elapsed=20.85s tid=0x00007fad3480c000 nid=0x3b03 runnable "GC Thread#1" os_prio=31 cpu=60.78ms elapsed=20.16s tid=0x00007fad45203800 nid=0x8233 runnable "GC Thread#2" os_prio=31 cpu=56.71ms elapsed=20.16s tid=0x00007fad45039800 nid=0x8403 runnable "GC Thread#3" os_prio=31 cpu=59.80ms elapsed=20.16s tid=0x00007fad45341800 nid=0xa703 runnable "GC Thread#4" os_prio=31 cpu=60.21ms elapsed=20.16s tid=0x00007fad45342000 nid=0x8503 runnable "GC Thread#5" os_prio=31 cpu=60.89ms elapsed=20.16s tid=0x00007fad4536d000 nid=0xa503 runnable "GC Thread#6" os_prio=31 cpu=57.91ms elapsed=20.16s tid=0x00007fad4594a800 nid=0x8803 runnable "GC Thread#7" os_prio=31 cpu=56.48ms elapsed=20.16s tid=0x00007fad4594b800 nid=0xa403 runnable "GC Thread#8" os_prio=31 cpu=56.38ms elapsed=20.16s tid=0x00007fad4536e000 nid=0x8a03 runnable "G1 Main Marker" os_prio=31 cpu=2.03ms elapsed=20.85s tid=0x00007fad3482d000 nid=0x3a03 runnable "G1 Conc#0" os_prio=31 cpu=12.67ms elapsed=20.85s tid=0x00007fad4500d800 nid=0x5403 runnable "G1 Conc#1" os_prio=31 cpu=14.15ms elapsed=18.88s tid=0x00007fad45470800 nid=0x8d4f runnable "G1 Refine#0" os_prio=31 cpu=2.03ms elapsed=20.84s tid=0x00007fad3492d800 nid=0x4103 runnable "G1 Young RemSet Sampling" os_prio=31 cpu=5.41ms elapsed=20.84s tid=0x00007fad14008800 nid=0x5103 runnable "VM Periodic Task Thread" os_prio=31 cpu=14.86ms elapsed=20.64s tid=0x00007fad3494b000 nid=0x680f waiting on condition JNI global refs: 20, weak refs: 0 ``` ## After this PR: ``` Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.0.15+8-LTS-149 mixed mode): Threads class SMR info: _java_thread_list=0x00006000020b8540, length=14, elements={ 0x00007ff536808800, 0x00007ff53311f800, 0x00007ff533021800, 0x00007ff53485f000, 0x00007ff53500f000, 0x00007ff535010000, 0x00007ff53500c000, 0x00007ff53400d800, 0x00007ff536861800, 0x00007ff536053000, 0x00007ff53621c800, 0x00007ff536178800, 0x00007ff5357e2000, 0x00007ff4d3009800 } "main" #1 prio=5 os_prio=31 cpu=1768.54ms elapsed=23.32s tid=0x00007ff536808800 nid=0x2a03 waiting on condition [0x000000030630b000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method) - parking to wait for <0x000000079f31ab80> (a java.util.concurrent.FutureTask) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.15/LockSupport.java:234) at java.util.concurrent.FutureTask.awaitDone(java.base@11.0.15/FutureTask.java:444) at java.util.concurrent.FutureTask.get(java.base@11.0.15/FutureTask.java:203) at org.junit.internal.runners.statements.FailOnTimeout.getResult(FailOnTimeout.java:141) at org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:127) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69) at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38) at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11) at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35) at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235) at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54) Locked ownable synchronizers: - None "Reference Handler" #2 daemon prio=10 os_prio=31 cpu=1.79ms elapsed=23.29s tid=0x00007ff53311f800 nid=0x4d03 waiting on condition [0x0000000306a21000] java.lang.Thread.State: RUNNABLE at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.15/Native Method) at java.lang.ref.Reference.processPendingReferences(java.base@11.0.15/Reference.java:241) at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.15/Reference.java:213) Locked ownable synchronizers: - None "Finalizer" #3 daemon prio=8 os_prio=31 cpu=1.34ms elapsed=23.29s tid=0x00007ff533021800 nid=0x5503 in Object.wait() [0x0000000306b24000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.15/Native Method) - waiting on <0x00000007801c5b68> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(java.base@11.0.15/ReferenceQueue.java:155) - waiting to re-lock in wait() <0x00000007801c5b68> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(java.base@11.0.15/ReferenceQueue.java:176) at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.15/Finalizer.java:170) Locked ownable synchronizers: - None "Signal Dispatcher" #4 daemon prio=9 os_prio=31 cpu=1.44ms elapsed=23.27s tid=0x00007ff53485f000 nid=0x5b03 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Service Thread" #5 daemon prio=9 os_prio=31 cpu=0.03ms elapsed=23.27s tid=0x00007ff53500f000 nid=0x7503 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread0" #6 daemon prio=9 os_prio=31 cpu=1882.60ms elapsed=23.27s tid=0x00007ff535010000 nid=0x7403 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE No compile task Locked ownable synchronizers: - None "C1 CompilerThread0" #9 daemon prio=9 os_prio=31 cpu=768.63ms elapsed=23.27s tid=0x00007ff53500c000 nid=0x6003 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE No compile task Locked ownable synchronizers: - None "Sweeper thread" #10 daemon prio=9 os_prio=31 cpu=0.05ms elapsed=23.27s tid=0x00007ff53400d800 nid=0x6203 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Common-Cleaner" apache#11 daemon prio=8 os_prio=31 cpu=8.06ms elapsed=23.20s tid=0x00007ff536861800 nid=0x6403 in Object.wait() [0x0000000307245000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.15/Native Method) - waiting on <0x00000007801c6678> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(java.base@11.0.15/ReferenceQueue.java:155) - waiting to re-lock in wait() <0x00000007801c6678> (a java.lang.ref.ReferenceQueue$Lock) at jdk.internal.ref.CleanerImpl.run(java.base@11.0.15/CleanerImpl.java:148) at java.lang.Thread.run(java.base@11.0.15/Thread.java:834) at jdk.internal.misc.InnocuousThread.run(java.base@11.0.15/InnocuousThread.java:134) Locked ownable synchronizers: - None "Monitor Ctrl-Break" apache#12 daemon prio=5 os_prio=31 cpu=19.84ms elapsed=23.11s tid=0x00007ff536053000 nid=0x6c03 runnable [0x0000000307348000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(java.base@11.0.15/Native Method) at java.net.SocketInputStream.socketRead(java.base@11.0.15/SocketInputStream.java:115) at java.net.SocketInputStream.read(java.base@11.0.15/SocketInputStream.java:168) at java.net.SocketInputStream.read(java.base@11.0.15/SocketInputStream.java:140) at sun.nio.cs.StreamDecoder.readBytes(java.base@11.0.15/StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(java.base@11.0.15/StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(java.base@11.0.15/StreamDecoder.java:178) - locked <0x00000007801c7860> (a java.io.InputStreamReader) at java.io.InputStreamReader.read(java.base@11.0.15/InputStreamReader.java:181) at java.io.BufferedReader.fill(java.base@11.0.15/BufferedReader.java:161) at java.io.BufferedReader.readLine(java.base@11.0.15/BufferedReader.java:326) - locked <0x00000007801c7860> (a java.io.InputStreamReader) at java.io.BufferedReader.readLine(java.base@11.0.15/BufferedReader.java:392) at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:56) Locked ownable synchronizers: - None "Log4j2-TF-2-Scheduled-1" apache#13 daemon prio=5 os_prio=31 cpu=14.05ms elapsed=21.67s tid=0x00007ff53621c800 nid=0x9c03 waiting on condition [0x0000000307d66000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method) - parking to wait for <0x00000007805a1378> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.15/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.15/AbstractQueuedSynchronizer.java:2123) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:1182) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.15/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.15/Thread.java:834) Locked ownable synchronizers: - None "Time-limited test" apache#15 daemon prio=5 os_prio=31 cpu=1353.43ms elapsed=21.31s tid=0x00007ff536178800 nid=0x9603 waiting on condition [0x0000000307f6c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@11.0.15/Native Method) at org.apache.bookkeeper.replication.AuditorPeriodicBookieCheckTest.tearDown(AuditorPeriodicBookieCheckTest.java:81) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.15/Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.15/NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.15/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(java.base@11.0.15/Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(java.base@11.0.15/FutureTask.java:264) at java.lang.Thread.run(java.base@11.0.15/Thread.java:834) Locked ownable synchronizers: - None "ForkJoinPool.commonPool-worker-19" apache#187 daemon prio=5 os_prio=31 cpu=1.50ms elapsed=18.22s tid=0x00007ff5357e2000 nid=0x2d003 waiting on condition [0x0000000312455000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method) - parking to wait for <0x000000079f052a68> (a java.util.concurrent.ForkJoinPool) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194) at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.15/ForkJoinPool.java:1628) at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.15/ForkJoinWorkerThread.java:183) Locked ownable synchronizers: - None "Attach Listener" apache#313 daemon prio=9 os_prio=31 cpu=1.42ms elapsed=0.25s tid=0x00007ff4d3009800 nid=0x2a767 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "VM Thread" os_prio=31 cpu=122.92ms elapsed=23.30s tid=0x00007ff536863800 nid=0x5003 runnable "GC Thread#0" os_prio=31 cpu=62.12ms elapsed=23.31s tid=0x00007ff536815000 nid=0x3403 runnable "GC Thread#1" os_prio=31 cpu=57.82ms elapsed=22.63s tid=0x00007ff53580c800 nid=0x8207 runnable "GC Thread#2" os_prio=31 cpu=56.34ms elapsed=22.63s tid=0x00007ff535ad9800 nid=0xa703 runnable "GC Thread#3" os_prio=31 cpu=59.35ms elapsed=22.63s tid=0x00007ff53516f000 nid=0xa603 runnable "GC Thread#4" os_prio=31 cpu=57.35ms elapsed=22.63s tid=0x00007ff536015000 nid=0x8503 runnable "GC Thread#5" os_prio=31 cpu=57.46ms elapsed=22.63s tid=0x00007ff5351b5000 nid=0x8603 runnable "GC Thread#6" os_prio=31 cpu=52.34ms elapsed=22.63s tid=0x00007ff5351b6000 nid=0x8803 runnable "GC Thread#7" os_prio=31 cpu=57.02ms elapsed=22.63s tid=0x00007ff5351b6800 nid=0x8a03 runnable "GC Thread#8" os_prio=31 cpu=58.19ms elapsed=22.63s tid=0x00007ff5351b7800 nid=0xa403 runnable "G1 Main Marker" os_prio=31 cpu=1.36ms elapsed=23.31s tid=0x00007ff53482e800 nid=0x3603 runnable "G1 Conc#0" os_prio=31 cpu=11.69ms elapsed=23.31s tid=0x00007ff53482f000 nid=0x5403 runnable "G1 Conc#1" os_prio=31 cpu=12.95ms elapsed=21.39s tid=0x00007ff53695b000 nid=0x9103 runnable "G1 Refine#0" os_prio=31 cpu=2.18ms elapsed=23.31s tid=0x00007ff53311e800 nid=0x5303 runnable "G1 Young RemSet Sampling" os_prio=31 cpu=6.12ms elapsed=23.31s tid=0x00007ff534854000 nid=0x4203 runnable "VM Periodic Task Thread" os_prio=31 cpu=17.11ms elapsed=23.11s tid=0x00007ff535888000 nid=0x8003 waiting on condition JNI global refs: 20, weak refs: 0 ``` ### You can see the executor `ZkLedgerManagerScheduler` shutdown correctly.

### Motivation The auditor didn't close the ledger manager and ledger under replication manager the thread will leak. Run the test `AuditorPeriodicBookieCheckTest` and get the stack. ## Before this PR: ``` Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.0.15+8-LTS-149 mixed mode): Threads class SMR info: _java_thread_list=0x0000600003e44460, length=16, elements={ 0x00007fad34009000, 0x00007fad45032800, 0x00007fad14813800, 0x00007fad44010800, 0x00007fad44011800, 0x00007fad45037800, 0x00007fad3400a000, 0x00007fad45812800, 0x00007fad34944000, 0x00007fad4585f800, 0x00007fad45456000, 0x00007fad4444c000, 0x00007fad3431b800, 0x00007face5825800, 0x00007face5860800, 0x00007face4009800 } "main" #1 prio=5 os_prio=31 cpu=1801.68ms elapsed=20.85s tid=0x00007fad34009000 nid=0x2a03 waiting on condition [0x000000030a10d000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method) - parking to wait for <0x000000079f26a8a0> (a java.util.concurrent.FutureTask) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.15/LockSupport.java:234) at java.util.concurrent.FutureTask.awaitDone(java.base@11.0.15/FutureTask.java:444) at java.util.concurrent.FutureTask.get(java.base@11.0.15/FutureTask.java:203) at org.junit.internal.runners.statements.FailOnTimeout.getResult(FailOnTimeout.java:141) at org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:127) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69) at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38) at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11) at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35) at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235) at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54) Locked ownable synchronizers: - None "Reference Handler" #2 daemon prio=10 os_prio=31 cpu=2.14ms elapsed=20.82s tid=0x00007fad45032800 nid=0x4803 waiting on condition [0x000000030a823000] java.lang.Thread.State: RUNNABLE at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.15/Native Method) at java.lang.ref.Reference.processPendingReferences(java.base@11.0.15/Reference.java:241) at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.15/Reference.java:213) Locked ownable synchronizers: - None "Finalizer" #3 daemon prio=8 os_prio=31 cpu=1.33ms elapsed=20.82s tid=0x00007fad14813800 nid=0x5503 in Object.wait() [0x000000030a926000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.15/Native Method) - waiting on <0x00000007801b6f18> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(java.base@11.0.15/ReferenceQueue.java:155) - waiting to re-lock in wait() <0x00000007801b6f18> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(java.base@11.0.15/ReferenceQueue.java:176) at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.15/Finalizer.java:170) Locked ownable synchronizers: - None "Signal Dispatcher" #4 daemon prio=9 os_prio=31 cpu=2.69ms elapsed=20.81s tid=0x00007fad44010800 nid=0x5b03 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Service Thread" #5 daemon prio=9 os_prio=31 cpu=0.03ms elapsed=20.81s tid=0x00007fad44011800 nid=0x7403 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread0" #6 daemon prio=9 os_prio=31 cpu=1986.68ms elapsed=20.81s tid=0x00007fad45037800 nid=0x5d03 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE No compile task Locked ownable synchronizers: - None "C1 CompilerThread0" #9 daemon prio=9 os_prio=31 cpu=769.61ms elapsed=20.81s tid=0x00007fad3400a000 nid=0x7003 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE No compile task Locked ownable synchronizers: - None "Sweeper thread" #10 daemon prio=9 os_prio=31 cpu=0.08ms elapsed=20.81s tid=0x00007fad45812800 nid=0x6e03 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Common-Cleaner" apache#11 daemon prio=8 os_prio=31 cpu=11.62ms elapsed=20.73s tid=0x00007fad34944000 nid=0x6b03 in Object.wait() [0x000000030b049000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.15/Native Method) - waiting on <0x00000007801b7550> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(java.base@11.0.15/ReferenceQueue.java:155) - waiting to re-lock in wait() <0x00000007801b7550> (a java.lang.ref.ReferenceQueue$Lock) at jdk.internal.ref.CleanerImpl.run(java.base@11.0.15/CleanerImpl.java:148) at java.lang.Thread.run(java.base@11.0.15/Thread.java:834) at jdk.internal.misc.InnocuousThread.run(java.base@11.0.15/InnocuousThread.java:134) Locked ownable synchronizers: - None "Monitor Ctrl-Break" apache#12 daemon prio=5 os_prio=31 cpu=21.16ms elapsed=20.65s tid=0x00007fad4585f800 nid=0x6a03 runnable [0x000000030b14c000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(java.base@11.0.15/Native Method) at java.net.SocketInputStream.socketRead(java.base@11.0.15/SocketInputStream.java:115) at java.net.SocketInputStream.read(java.base@11.0.15/SocketInputStream.java:168) at java.net.SocketInputStream.read(java.base@11.0.15/SocketInputStream.java:140) at sun.nio.cs.StreamDecoder.readBytes(java.base@11.0.15/StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(java.base@11.0.15/StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(java.base@11.0.15/StreamDecoder.java:178) - locked <0x00000007801b81e8> (a java.io.InputStreamReader) at java.io.InputStreamReader.read(java.base@11.0.15/InputStreamReader.java:181) at java.io.BufferedReader.fill(java.base@11.0.15/BufferedReader.java:161) at java.io.BufferedReader.readLine(java.base@11.0.15/BufferedReader.java:326) - locked <0x00000007801b81e8> (a java.io.InputStreamReader) at java.io.BufferedReader.readLine(java.base@11.0.15/BufferedReader.java:392) at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:56) Locked ownable synchronizers: - None "Log4j2-TF-2-Scheduled-1" apache#13 daemon prio=5 os_prio=31 cpu=3.80ms elapsed=19.18s tid=0x00007fad45456000 nid=0x8c03 waiting on condition [0x000000030bb6a000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method) - parking to wait for <0x00000007805c1f78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.15/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.15/AbstractQueuedSynchronizer.java:2123) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:1182) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.15/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.15/Thread.java:834) Locked ownable synchronizers: - None "Time-limited test" apache#15 daemon prio=5 os_prio=31 cpu=1344.20ms elapsed=18.79s tid=0x00007fad4444c000 nid=0x9107 waiting on condition [0x000000030bd70000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@11.0.15/Native Method) at org.apache.bookkeeper.replication.AuditorPeriodicBookieCheckTest.tearDown(AuditorPeriodicBookieCheckTest.java:81) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.15/Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.15/NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.15/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(java.base@11.0.15/Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(java.base@11.0.15/FutureTask.java:264) at java.lang.Thread.run(java.base@11.0.15/Thread.java:834) Locked ownable synchronizers: - None "ZkLedgerManagerScheduler-48-1" apache#165 prio=5 os_prio=31 cpu=9.25ms elapsed=16.04s tid=0x00007fad3431b800 nid=0x25b03 waiting on condition [0x0000000314d1a000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method) - parking to wait for <0x000000079db6aed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.15/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:1170) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.15/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(java.base@11.0.15/Thread.java:834) Locked ownable synchronizers: - None "ZkLedgerManagerScheduler-49-1" apache#166 prio=5 os_prio=31 cpu=1.01ms elapsed=16.04s tid=0x00007face5825800 nid=0x25903 waiting on condition [0x0000000314e1d000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method) - parking to wait for <0x000000079db6b3f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.15/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:1170) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.15/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(java.base@11.0.15/Thread.java:834) Locked ownable synchronizers: - None "ForkJoinPool.commonPool-worker-19" apache#187 daemon prio=5 os_prio=31 cpu=1.52ms elapsed=15.64s tid=0x00007face5860800 nid=0x2ca03 waiting on condition [0x0000000316259000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method) - parking to wait for <0x000000079ef3f6e8> (a java.util.concurrent.ForkJoinPool) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194) at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.15/ForkJoinPool.java:1628) at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.15/ForkJoinWorkerThread.java:183) Locked ownable synchronizers: - None "Attach Listener" apache#313 daemon prio=9 os_prio=31 cpu=1.90ms elapsed=0.24s tid=0x00007face4009800 nid=0x3d53b waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "VM Thread" os_prio=31 cpu=122.89ms elapsed=20.84s tid=0x00007fad1405f000 nid=0x4e03 runnable "GC Thread#0" os_prio=31 cpu=67.63ms elapsed=20.85s tid=0x00007fad3480c000 nid=0x3b03 runnable "GC Thread#1" os_prio=31 cpu=60.78ms elapsed=20.16s tid=0x00007fad45203800 nid=0x8233 runnable "GC Thread#2" os_prio=31 cpu=56.71ms elapsed=20.16s tid=0x00007fad45039800 nid=0x8403 runnable "GC Thread#3" os_prio=31 cpu=59.80ms elapsed=20.16s tid=0x00007fad45341800 nid=0xa703 runnable "GC Thread#4" os_prio=31 cpu=60.21ms elapsed=20.16s tid=0x00007fad45342000 nid=0x8503 runnable "GC Thread#5" os_prio=31 cpu=60.89ms elapsed=20.16s tid=0x00007fad4536d000 nid=0xa503 runnable "GC Thread#6" os_prio=31 cpu=57.91ms elapsed=20.16s tid=0x00007fad4594a800 nid=0x8803 runnable "GC Thread#7" os_prio=31 cpu=56.48ms elapsed=20.16s tid=0x00007fad4594b800 nid=0xa403 runnable "GC Thread#8" os_prio=31 cpu=56.38ms elapsed=20.16s tid=0x00007fad4536e000 nid=0x8a03 runnable "G1 Main Marker" os_prio=31 cpu=2.03ms elapsed=20.85s tid=0x00007fad3482d000 nid=0x3a03 runnable "G1 Conc#0" os_prio=31 cpu=12.67ms elapsed=20.85s tid=0x00007fad4500d800 nid=0x5403 runnable "G1 Conc#1" os_prio=31 cpu=14.15ms elapsed=18.88s tid=0x00007fad45470800 nid=0x8d4f runnable "G1 Refine#0" os_prio=31 cpu=2.03ms elapsed=20.84s tid=0x00007fad3492d800 nid=0x4103 runnable "G1 Young RemSet Sampling" os_prio=31 cpu=5.41ms elapsed=20.84s tid=0x00007fad14008800 nid=0x5103 runnable "VM Periodic Task Thread" os_prio=31 cpu=14.86ms elapsed=20.64s tid=0x00007fad3494b000 nid=0x680f waiting on condition JNI global refs: 20, weak refs: 0 ``` ## After this PR: ``` Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.0.15+8-LTS-149 mixed mode): Threads class SMR info: _java_thread_list=0x00006000020b8540, length=14, elements={ 0x00007ff536808800, 0x00007ff53311f800, 0x00007ff533021800, 0x00007ff53485f000, 0x00007ff53500f000, 0x00007ff535010000, 0x00007ff53500c000, 0x00007ff53400d800, 0x00007ff536861800, 0x00007ff536053000, 0x00007ff53621c800, 0x00007ff536178800, 0x00007ff5357e2000, 0x00007ff4d3009800 } "main" #1 prio=5 os_prio=31 cpu=1768.54ms elapsed=23.32s tid=0x00007ff536808800 nid=0x2a03 waiting on condition [0x000000030630b000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method) - parking to wait for <0x000000079f31ab80> (a java.util.concurrent.FutureTask) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.15/LockSupport.java:234) at java.util.concurrent.FutureTask.awaitDone(java.base@11.0.15/FutureTask.java:444) at java.util.concurrent.FutureTask.get(java.base@11.0.15/FutureTask.java:203) at org.junit.internal.runners.statements.FailOnTimeout.getResult(FailOnTimeout.java:141) at org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:127) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69) at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38) at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11) at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35) at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235) at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54) Locked ownable synchronizers: - None "Reference Handler" #2 daemon prio=10 os_prio=31 cpu=1.79ms elapsed=23.29s tid=0x00007ff53311f800 nid=0x4d03 waiting on condition [0x0000000306a21000] java.lang.Thread.State: RUNNABLE at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.15/Native Method) at java.lang.ref.Reference.processPendingReferences(java.base@11.0.15/Reference.java:241) at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.15/Reference.java:213) Locked ownable synchronizers: - None "Finalizer" #3 daemon prio=8 os_prio=31 cpu=1.34ms elapsed=23.29s tid=0x00007ff533021800 nid=0x5503 in Object.wait() [0x0000000306b24000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.15/Native Method) - waiting on <0x00000007801c5b68> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(java.base@11.0.15/ReferenceQueue.java:155) - waiting to re-lock in wait() <0x00000007801c5b68> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(java.base@11.0.15/ReferenceQueue.java:176) at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.15/Finalizer.java:170) Locked ownable synchronizers: - None "Signal Dispatcher" #4 daemon prio=9 os_prio=31 cpu=1.44ms elapsed=23.27s tid=0x00007ff53485f000 nid=0x5b03 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Service Thread" #5 daemon prio=9 os_prio=31 cpu=0.03ms elapsed=23.27s tid=0x00007ff53500f000 nid=0x7503 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread0" #6 daemon prio=9 os_prio=31 cpu=1882.60ms elapsed=23.27s tid=0x00007ff535010000 nid=0x7403 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE No compile task Locked ownable synchronizers: - None "C1 CompilerThread0" #9 daemon prio=9 os_prio=31 cpu=768.63ms elapsed=23.27s tid=0x00007ff53500c000 nid=0x6003 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE No compile task Locked ownable synchronizers: - None "Sweeper thread" #10 daemon prio=9 os_prio=31 cpu=0.05ms elapsed=23.27s tid=0x00007ff53400d800 nid=0x6203 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Common-Cleaner" apache#11 daemon prio=8 os_prio=31 cpu=8.06ms elapsed=23.20s tid=0x00007ff536861800 nid=0x6403 in Object.wait() [0x0000000307245000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.15/Native Method) - waiting on <0x00000007801c6678> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(java.base@11.0.15/ReferenceQueue.java:155) - waiting to re-lock in wait() <0x00000007801c6678> (a java.lang.ref.ReferenceQueue$Lock) at jdk.internal.ref.CleanerImpl.run(java.base@11.0.15/CleanerImpl.java:148) at java.lang.Thread.run(java.base@11.0.15/Thread.java:834) at jdk.internal.misc.InnocuousThread.run(java.base@11.0.15/InnocuousThread.java:134) Locked ownable synchronizers: - None "Monitor Ctrl-Break" apache#12 daemon prio=5 os_prio=31 cpu=19.84ms elapsed=23.11s tid=0x00007ff536053000 nid=0x6c03 runnable [0x0000000307348000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(java.base@11.0.15/Native Method) at java.net.SocketInputStream.socketRead(java.base@11.0.15/SocketInputStream.java:115) at java.net.SocketInputStream.read(java.base@11.0.15/SocketInputStream.java:168) at java.net.SocketInputStream.read(java.base@11.0.15/SocketInputStream.java:140) at sun.nio.cs.StreamDecoder.readBytes(java.base@11.0.15/StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(java.base@11.0.15/StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(java.base@11.0.15/StreamDecoder.java:178) - locked <0x00000007801c7860> (a java.io.InputStreamReader) at java.io.InputStreamReader.read(java.base@11.0.15/InputStreamReader.java:181) at java.io.BufferedReader.fill(java.base@11.0.15/BufferedReader.java:161) at java.io.BufferedReader.readLine(java.base@11.0.15/BufferedReader.java:326) - locked <0x00000007801c7860> (a java.io.InputStreamReader) at java.io.BufferedReader.readLine(java.base@11.0.15/BufferedReader.java:392) at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:56) Locked ownable synchronizers: - None "Log4j2-TF-2-Scheduled-1" apache#13 daemon prio=5 os_prio=31 cpu=14.05ms elapsed=21.67s tid=0x00007ff53621c800 nid=0x9c03 waiting on condition [0x0000000307d66000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method) - parking to wait for <0x00000007805a1378> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.15/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.15/AbstractQueuedSynchronizer.java:2123) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:1182) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.15/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.15/Thread.java:834) Locked ownable synchronizers: - None "Time-limited test" apache#15 daemon prio=5 os_prio=31 cpu=1353.43ms elapsed=21.31s tid=0x00007ff536178800 nid=0x9603 waiting on condition [0x0000000307f6c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@11.0.15/Native Method) at org.apache.bookkeeper.replication.AuditorPeriodicBookieCheckTest.tearDown(AuditorPeriodicBookieCheckTest.java:81) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.15/Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.15/NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.15/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(java.base@11.0.15/Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(java.base@11.0.15/FutureTask.java:264) at java.lang.Thread.run(java.base@11.0.15/Thread.java:834) Locked ownable synchronizers: - None "ForkJoinPool.commonPool-worker-19" apache#187 daemon prio=5 os_prio=31 cpu=1.50ms elapsed=18.22s tid=0x00007ff5357e2000 nid=0x2d003 waiting on condition [0x0000000312455000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method) - parking to wait for <0x000000079f052a68> (a java.util.concurrent.ForkJoinPool) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.15/LockSupport.java:194) at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.15/ForkJoinPool.java:1628) at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.15/ForkJoinWorkerThread.java:183) Locked ownable synchronizers: - None "Attach Listener" apache#313 daemon prio=9 os_prio=31 cpu=1.42ms elapsed=0.25s tid=0x00007ff4d3009800 nid=0x2a767 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "VM Thread" os_prio=31 cpu=122.92ms elapsed=23.30s tid=0x00007ff536863800 nid=0x5003 runnable "GC Thread#0" os_prio=31 cpu=62.12ms elapsed=23.31s tid=0x00007ff536815000 nid=0x3403 runnable "GC Thread#1" os_prio=31 cpu=57.82ms elapsed=22.63s tid=0x00007ff53580c800 nid=0x8207 runnable "GC Thread#2" os_prio=31 cpu=56.34ms elapsed=22.63s tid=0x00007ff535ad9800 nid=0xa703 runnable "GC Thread#3" os_prio=31 cpu=59.35ms elapsed=22.63s tid=0x00007ff53516f000 nid=0xa603 runnable "GC Thread#4" os_prio=31 cpu=57.35ms elapsed=22.63s tid=0x00007ff536015000 nid=0x8503 runnable "GC Thread#5" os_prio=31 cpu=57.46ms elapsed=22.63s tid=0x00007ff5351b5000 nid=0x8603 runnable "GC Thread#6" os_prio=31 cpu=52.34ms elapsed=22.63s tid=0x00007ff5351b6000 nid=0x8803 runnable "GC Thread#7" os_prio=31 cpu=57.02ms elapsed=22.63s tid=0x00007ff5351b6800 nid=0x8a03 runnable "GC Thread#8" os_prio=31 cpu=58.19ms elapsed=22.63s tid=0x00007ff5351b7800 nid=0xa403 runnable "G1 Main Marker" os_prio=31 cpu=1.36ms elapsed=23.31s tid=0x00007ff53482e800 nid=0x3603 runnable "G1 Conc#0" os_prio=31 cpu=11.69ms elapsed=23.31s tid=0x00007ff53482f000 nid=0x5403 runnable "G1 Conc#1" os_prio=31 cpu=12.95ms elapsed=21.39s tid=0x00007ff53695b000 nid=0x9103 runnable "G1 Refine#0" os_prio=31 cpu=2.18ms elapsed=23.31s tid=0x00007ff53311e800 nid=0x5303 runnable "G1 Young RemSet Sampling" os_prio=31 cpu=6.12ms elapsed=23.31s tid=0x00007ff534854000 nid=0x4203 runnable "VM Periodic Task Thread" os_prio=31 cpu=17.11ms elapsed=23.11s tid=0x00007ff535888000 nid=0x8003 waiting on condition JNI global refs: 20, weak refs: 0 ``` ### You can see the executor `ZkLedgerManagerScheduler` shutdown correctly. (cherry picked from commit 1e3efa1)

codelipenghui and others added 18 commits May 10, 2023 11:16

Batch Read

3ff498e

Merge branch 'master' into batch_read

01e79fe

# Conflicts: # bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieProtoEncoding.java # bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/ReadEntryProcessor.java

Support batch read fail back to single read.

0e72b90

code clean.

5a3791f

Code clean.

7ae6631

Fix client and server compatible problem.

6f02306

Use another way to support compatibility.

0502a36

Try to read another bookie when one bookie encounter `BookieHandleNot…

a453f98

…AvailableException`.

Fix license check.

89e19d1

Fix ci.

af94443

Fix ci.

548f6d3

Fix bug.

a45ebe0

code clean.

b2960c3

Tune ci.

7d92be9

Fix ci.

c32ce7b

Fix ci.

4a3e9b3

Merge branch 'master' into batch_read

e4aa956

Merge branch 'master' into batch_read

d68fbf2

hangc0276 reviewed Dec 7, 2023

View reviewed changes

...eper-benchmark/src/main/java/org/apache/bookkeeper/benchmark/BenchReadThroughputLatency.java Outdated Show resolved Hide resolved

bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BatchedReadEntryProcessor.java Outdated Show resolved Hide resolved

hangc0276 reviewed Dec 8, 2023

View reviewed changes

Fix ci.

5af6483

hangc0276 reviewed Dec 11, 2023

View reviewed changes

address the comments.

b77c790

horizonzy commented Dec 12, 2023

View reviewed changes

horizonzy added 2 commits December 12, 2023 20:15

Fix checkstyle.

e7143ba

fix ci.

9d1990a

zymap reviewed Dec 13, 2023

View reviewed changes

horizonzy added 2 commits December 13, 2023 14:01

address the comments.

9914d4f

Fix ci.

52807ff

hangc0276 reviewed Dec 14, 2023

View reviewed changes

horizonzy added 5 commits December 14, 2023 18:28

address the comments.

238bb19

address the comments.

6c4fc96

stash commit.

7c96b68

fix ci.

cc924c8

fix ci.

496fd05

hangc0276 approved these changes Dec 25, 2023

View reviewed changes


		abstract class BatchedLedgerEntryRequest extends LedgerEntryRequest {

		final long lId;

Batch read #10

Are you sure you want to change the base?

Batch read #10

Conversation

horizonzy commented Nov 15, 2023

Motivation

Changes

hangc0276 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hangc0276 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hangc0276 commented Dec 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

horizonzy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hangc0276 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment