Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unrecycled RPC threads cause OOM #1104

Closed
Brokenice0415 opened this issue May 13, 2024 · 2 comments
Closed

Unrecycled RPC threads cause OOM #1104

Brokenice0415 opened this issue May 13, 2024 · 2 comments

Comments

@Brokenice0415
Copy link

Describe the bug

I started and stopped jraft server multiple times via ProcessBuilder and triggered the OOM.

My guess is that the cause of OOM is that I didn't call shutdown but killed the process when stopping it, and RPC-related resources weren't properly recycled.

Actual behavior

#  Out of Memory Error (os_linux.cpp:2787), pid=3492334, tid=3689298
#
# JRE version: OpenJDK Runtime Environment Zulu17.38+21-CA (17.0.5+8) (build 17.0.5+8-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu17.38+21-CA (17.0.5+8-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, parallel gc, linux-amd64)
#

---------------  S U M M A R Y ------------

Command Line: -Xmx1800m -XX:-OmitStackTraceInFastThrow -XX:+UseParallelGC -XX:+IgnoreUnrecognizedVMOptions -XX:+CriticalJNINatives -XX:+EnableDynamicAgentLoading --add-opens=java.base/java.lang=ALL-UNNAMED 

Host: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz, 16 cores, 62G, Ubuntu 22.04.3 LTS
Time: Fri May 10 02:29:21 2024 CST elapsed time: 9493.794026 seconds (0d 2h 38m 13s)

---------------  T H R E A D  ---------------

Current thread (0x00007ff6e4981c20):  JavaThread "JRaft-RPC-Processor-0" daemon [_thread_new, id=3689298, stack(0x00007feedbd00000,0x00007feedbe00000)]

Stack: [0x00007feedbd00000,0x00007feedbe00000],  sp=0x00007feedbdfe8f0,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xeffe02]  VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x1a2
V  [libjvm.so+0xf00aeb]  VMError::report_and_die(Thread*, char const*, int, unsigned long, VMErrorType, char const*, __va_list_tag*)+0x2b
V  [libjvm.so+0x601082]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0xc2
V  [libjvm.so+0xbcbf48]  os::pd_commit_memory(char*, unsigned long, bool)+0xd8
V  [libjvm.so+0xbc57ef]  os::commit_memory(char*, unsigned long, bool)+0x1f
V  [libjvm.so+0xbccd3e]  os::pd_create_stack_guard_pages(char*, unsigned long)+0x6e
V  [libjvm.so+0xdbc588]  StackOverflow::create_stack_guard_pages()+0x58
V  [libjvm.so+0xe83a23]  JavaThread::run()+0x23
V  [libjvm.so+0xe8680d]  Thread::call_run()+0x6d
V  [libjvm.so+0xbcece1]  thread_native_entry(Thread*)+0xe1

Java Threads: ( => current thread )
  0x0000560233e93d80 JavaThread "main" [_thread_blocked, id=3492334, stack(0x00007ffe066a1000,0x00007ffe067a1000)]
  0x0000560233ff84b0 JavaThread "Reference Handler" daemon [_thread_blocked, id=3492338, stack(0x00007ff7db92e000,0x00007ff7dba2e000)]
  0x0000560233ff98e0 JavaThread "Finalizer" daemon [_thread_blocked, id=3492339, stack(0x00007ff7db82e000,0x00007ff7db92e000)]
  0x0000560233ffff70 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3492340, stack(0x00007ff7daeb6000,0x00007ff7dafb6000)]
  0x0000560234001310 JavaThread "Service Thread" daemon [_thread_blocked, id=3492341, stack(0x00007ff7dadb6000,0x00007ff7daeb6000)]
  0x0000560234002710 JavaThread "Monitor Deflation Thread" daemon [_thread_blocked, id=3492342, stack(0x00007ff7dacb6000,0x00007ff7dadb6000)]
  0x00005602340040b0 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=3492343, stack(0x00007ff7dabb6000,0x00007ff7dacb6000)]
  0x00005602340055d0 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=3492344, stack(0x00007ff7daab6000,0x00007ff7dabb6000)]
  0x000056023400ea20 JavaThread "Sweeper thread" daemon [_thread_blocked, id=3492345, stack(0x00007ff7da9b6000,0x00007ff7daab6000)]
  0x0000560234054e60 JavaThread "Common-Cleaner" daemon [_thread_blocked, id=3492346, stack(0x00007ff7da8b6000,0x00007ff7da9b6000)]
  0x0000560234057170 JavaThread "Notification Thread" daemon [_thread_blocked, id=3492347, stack(0x00007ff7da7b6000,0x00007ff7da8b6000)]
  0x00005602340638b0 JavaThread "process reaper" daemon [_thread_in_native, id=3492352, stack(0x00007ff7dc46d000,0x00007ff7dc48f000)]
  0x00007ff7c4000ff0 JavaThread "Attach Listener" daemon [_thread_blocked, id=3492370, stack(0x00007ff7da4b4000,0x00007ff7da5b4000)]
  0x0000560234ec8020 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=3492694, stack(0x00007ff7d98b4000,0x00007ff7d99b4000)]
  0x0000560234ecb870 JavaThread "RMI Reaper" [_thread_blocked, id=3492695, stack(0x00007ff7d9ab4000,0x00007ff7d9bb4000)]
  0x0000560234ecd0f0 JavaThread "RMI GC Daemon" daemon [_thread_blocked, id=3492696, stack(0x00007ff7d9eb4000,0x00007ff7d9fb4000)]
  0x0000560234ec2da0 JavaThread "RMI TCP Accept-2599" daemon [_thread_in_native, id=3492697, stack(0x00007ff7d9db4000,0x00007ff7d9eb4000)]
  0x0000560234f51150 JavaThread "process reaper" daemon [_thread_in_native, id=3492701, stack(0x00007ff7d9703000,0x00007ff7d9725000)]
  0x0000560234f52a70 JavaThread "process reaper" daemon [_thread_in_native, id=3492704, stack(0x00007ff7d96e1000,0x00007ff7d9703000)]
  0x00007ff78c00f4f0 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=3492782, stack(0x00007ff7d94ee000,0x00007ff7d95ee000)]
  0x00007ff7187dfd90 JavaThread "RpcTaskScannerThread-8-thread-1" daemon [_thread_blocked, id=3493345, stack(0x00007ff740300000,0x00007ff740400000)]
  0x00007ff704005690 JavaThread "RpcTaskScannerThread-9-thread-1" daemon [_thread_blocked, id=3493346, stack(0x00007ff740200000,0x00007ff740300000)]
  0x00007ff7040420b0 JavaThread "bolt-netty-client-worker-1-thread-1" daemon [_thread_in_native, id=3493349, stack(0x00007ff7410fe000,0x00007ff7411fe000)]
  0x00007ff71859d120 JavaThread "bolt-netty-client-worker-1-thread-2" daemon [_thread_in_native, id=3493350, stack(0x00007ff740100000,0x00007ff740200000)]
  0x00007ff710040d60 JavaThread "Bolt-conn-event-executor-5-thread-1" daemon [_thread_blocked, id=3493353, stack(0x00007ff740000000,0x00007ff740100000)]
  0x00007ff70000d3b0 JavaThread "Bolt-conn-event-executor-4-thread-1" daemon [_thread_blocked, id=3493354, stack(0x00007ff6fbf00000,0x00007ff6fc000000)]
  0x00007ff710138e80 JavaThread "Bolt-default-executor-6-thread-1" daemon [_thread_blocked, id=3493358, stack(0x00007ff6f9dfe000,0x00007ff6f9efe000)]
  0x00007ff70000ff90 JavaThread "Bolt-default-executor-6-thread-2" daemon [_thread_blocked, id=3493359, stack(0x00007ff6f9cfe000,0x00007ff6f9dfe000)]
  0x00007ff70406d6b0 JavaThread "DefaultTimer10-10-thread-1" daemon [_thread_blocked, id=3493360, stack(0x00007ff6f9bfe000,0x00007ff6f9cfe000)]
  0x00007ff700012880 JavaThread "Bolt-default-executor-6-thread-3" daemon [_thread_blocked, id=3493363, stack(0x00007ff6f9afe000,0x00007ff6f9bfe000)]
  0x00007ff710139f00 JavaThread "Bolt-default-executor-6-thread-4" daemon [_thread_blocked, id=3493364, stack(0x00007ff6f99fe000,0x00007ff6f9afe000)]
  0x00007ff6d80012f0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493365, stack(0x00007ff6f98fe000,0x00007ff6f99fe000)]
  0x00007ff6e4003390 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493366, stack(0x00007ff6f97fe000,0x00007ff6f98fe000)]
  0x00007ff6dc004d90 JavaThread "RpcTaskScannerThread-13-thread-1" daemon [_thread_blocked, id=3493368, stack(0x00007ff6f95fe000,0x00007ff6f96fe000)]
  0x00007ff6dc014750 JavaThread "bolt-netty-client-worker-1-thread-3" daemon [_thread_in_native, id=3493369, stack(0x00007ff6f94fe000,0x00007ff6f95fe000)]
  0x00007ff6e4001ab0 JavaThread "Bolt-conn-event-executor-12-thread-1" daemon [_thread_blocked, id=3493371, stack(0x00007ff6f93fe000,0x00007ff6f94fe000)]
  0x00007ff6e4005fd0 JavaThread "Bolt-default-executor-6-thread-5" daemon [_thread_blocked, id=3493372, stack(0x00007ff6f82fd000,0x00007ff6f83fd000)]
  0x00007ff6e4007170 JavaThread "Bolt-default-executor-6-thread-6" daemon [_thread_blocked, id=3493374, stack(0x00007ff6f81fd000,0x00007ff6f82fd000)]
  0x00007ff6e8001700 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493375, stack(0x00007ff6f80fd000,0x00007ff6f81fd000)]
  0x00007ff700014560 JavaThread "RpcTaskScannerThread-16-thread-1" daemon [_thread_blocked, id=3493379, stack(0x00007ff6d7d00000,0x00007ff6d7e00000)]
  0x00007ff700015b00 JavaThread "bolt-netty-client-worker-1-thread-4" daemon [_thread_in_native, id=3493380, stack(0x00007ff6d7c00000,0x00007ff6d7d00000)]
  0x00007ff708001720 JavaThread "Bolt-conn-event-executor-15-thread-1" daemon [_thread_blocked, id=3493382, stack(0x00007ff6d7b00000,0x00007ff6d7c00000)]
  0x00007ff708002a50 JavaThread "Bolt-default-executor-6-thread-7" daemon [_thread_in_Java, id=3493383, stack(0x00007ff6d69ff000,0x00007ff6d6aff000)]
  0x00007ff708003c90 JavaThread "Bolt-default-executor-6-thread-8" daemon [_thread_blocked, id=3493385, stack(0x00007ff6d68ff000,0x00007ff6d69ff000)]
  0x00007ff70c003710 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493386, stack(0x00007ff6d67ff000,0x00007ff6d68ff000)]
  0x00007ff7000187e0 JavaThread "Bolt-default-executor-6-thread-9" daemon [_thread_blocked, id=3493388, stack(0x00007ff6d66ff000,0x00007ff6d67ff000)]
  0x00007ff6e4008360 JavaThread "Bolt-default-executor-6-thread-10" daemon [_thread_blocked, id=3493389, stack(0x00007ff6d65ff000,0x00007ff6d66ff000)]
  0x00007ff7080052b0 JavaThread "Bolt-default-executor-6-thread-11" daemon [_thread_blocked, id=3493392, stack(0x00007ff6d64ff000,0x00007ff6d65ff000)]
  0x00007ff710150720 JavaThread "Bolt-default-executor-6-thread-12" daemon [_thread_blocked, id=3493393, stack(0x00007ff6d63ff000,0x00007ff6d64ff000)]
  0x00007ff738003790 JavaThread "RpcTaskScannerThread-19-thread-1" daemon [_thread_blocked, id=3493746, stack(0x00007ff6d5fff000,0x00007ff6d60ff000)]
  0x00007ff738004b70 JavaThread "bolt-netty-client-worker-1-thread-5" daemon [_thread_in_native, id=3493747, stack(0x00007ff6d5eff000,0x00007ff6d5fff000)]
  0x00007ff73c003a70 JavaThread "Bolt-conn-event-executor-18-thread-1" daemon [_thread_blocked, id=3493749, stack(0x00007ff6d5dff000,0x00007ff6d5eff000)]
  0x00007ff73c004e00 JavaThread "Bolt-default-executor-6-thread-13" daemon [_thread_blocked, id=3493753, stack(0x00007ff6d4cfe000,0x00007ff6d4dfe000)]
  0x00007ff73c006090 JavaThread "Bolt-default-executor-6-thread-14" daemon [_thread_blocked, id=3493755, stack(0x00007ff6d4bfe000,0x00007ff6d4cfe000)]
  0x00007ff750002df0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493756, stack(0x00007ff6d4afe000,0x00007ff6d4bfe000)]
  0x00007ff73c007320 JavaThread "Bolt-default-executor-6-thread-15" daemon [_thread_blocked, id=3493778, stack(0x00007ff6d46fe000,0x00007ff6d47fe000)]
  0x00007ff768251010 JavaThread "RpcTaskScannerThread-22-thread-1" daemon [_thread_blocked, id=3493781, stack(0x00007ff6d44fe000,0x00007ff6d45fe000)]
  0x00007ff768c1b7d0 JavaThread "bolt-netty-client-worker-1-thread-6" daemon [_thread_in_native, id=3493782, stack(0x00007ff6d43fe000,0x00007ff6d44fe000)]
  0x00007ff770005d00 JavaThread "Bolt-conn-event-executor-21-thread-1" daemon [_thread_blocked, id=3493784, stack(0x00007ff6d42fe000,0x00007ff6d43fe000)]
  0x00007ff770002240 JavaThread "Bolt-default-executor-6-thread-16" daemon [_thread_blocked, id=3493785, stack(0x00007ff6d31fd000,0x00007ff6d32fd000)]
  0x00007ff770003270 JavaThread "Bolt-default-executor-6-thread-17" daemon [_thread_blocked, id=3493787, stack(0x00007ff6d30fd000,0x00007ff6d31fd000)]
  0x00007ff774020180 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493788, stack(0x00007ff6d2ffd000,0x00007ff6d30fd000)]
  0x00007ff770006f40 JavaThread "Bolt-default-executor-6-thread-18" daemon [_thread_blocked, id=3493790, stack(0x00007ff6d2efd000,0x00007ff6d2ffd000)]
  0x00007ff788025c50 JavaThread "RpcTaskScannerThread-25-thread-1" daemon [_thread_blocked, id=3493794, stack(0x00007ff6d2cfd000,0x00007ff6d2dfd000)]
  0x00007ff78804d350 JavaThread "bolt-netty-client-worker-1-thread-7" daemon [_thread_in_native, id=3493795, stack(0x00007ff6d2bfd000,0x00007ff6d2cfd000)]
  0x00007ff79000aa60 JavaThread "Bolt-conn-event-executor-24-thread-1" daemon [_thread_blocked, id=3493797, stack(0x00007ff6d2afd000,0x00007ff6d2bfd000)]
  0x00007ff790007eb0 JavaThread "Bolt-default-executor-6-thread-19" daemon [_thread_blocked, id=3493798, stack(0x00007ff6d19fc000,0x00007ff6d1afc000)]
  0x00007ff790009280 JavaThread "Bolt-default-executor-6-thread-20" daemon [_thread_blocked, id=3493800, stack(0x00007ff6d18fc000,0x00007ff6d19fc000)]
  0x00007ff79400ff80 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493801, stack(0x00007ff6d17fc000,0x00007ff6d18fc000)]
  0x00007ff79c002090 JavaThread "RpcTaskScannerThread-28-thread-1" daemon [_thread_blocked, id=3493805, stack(0x00007ff6d15fc000,0x00007ff6d16fc000)]
  0x00007ff79c0038f0 JavaThread "bolt-netty-client-worker-1-thread-8" daemon [_thread_in_native, id=3493806, stack(0x00007ff6d14fc000,0x00007ff6d15fc000)]
  0x00007ff7a40015c0 JavaThread "Bolt-conn-event-executor-27-thread-1" daemon [_thread_blocked, id=3493808, stack(0x00007ff6d13fc000,0x00007ff6d14fc000)]
  0x00007ff6d8002ba0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493810, stack(0x00007ff6d02fb000,0x00007ff6d03fb000)]
  0x00007ff7b80015c0 JavaThread "RpcTaskScannerThread-31-thread-1" daemon [_thread_blocked, id=3493814, stack(0x00007ff6d00fb000,0x00007ff6d01fb000)]
  0x00007ff7b8002eb0 JavaThread "bolt-netty-client-worker-1-thread-9" daemon [_thread_in_native, id=3493815, stack(0x00007ff6cfffb000,0x00007ff6d00fb000)]
  0x00007ff7c0ced620 JavaThread "Bolt-conn-event-executor-30-thread-1" daemon [_thread_blocked, id=3493817, stack(0x00007ff6cfefb000,0x00007ff6cfffb000)]
  0x00007ff6ec001720 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493819, stack(0x00007ff6cedfa000,0x00007ff6ceefa000)]
  0x00007ff7c4002720 JavaThread "RpcTaskScannerThread-34-thread-1" daemon [_thread_blocked, id=3493821, stack(0x00007ff6cebfa000,0x00007ff6cecfa000)]
  0x00007ff71013ecf0 JavaThread "Bolt-conn-event-executor-33-thread-1" daemon [_thread_blocked, id=3493823, stack(0x00007ff6ceafa000,0x00007ff6cebfa000)]
  0x00007ff71013f940 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493825, stack(0x00007ff6ce9fa000,0x00007ff6ceafa000)]
  0x00007ff7e0002fe0 JavaThread "RpcTaskScannerThread-37-thread-1" daemon [_thread_blocked, id=3493842, stack(0x00007ff6ce7fa000,0x00007ff6ce8fa000)]
  0x00007ff700019bd0 JavaThread "Bolt-conn-event-executor-36-thread-1" daemon [_thread_blocked, id=3493844, stack(0x00007ff6ce6fa000,0x00007ff6ce7fa000)]
  0x00007ff714003c70 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493846, stack(0x00007ff6ce5fa000,0x00007ff6ce6fa000)]
  0x00007ff6e4016330 JavaThread "RpcTaskScannerThread-40-thread-1" daemon [_thread_blocked, id=3493848, stack(0x00007ff6ce3fa000,0x00007ff6ce4fa000)]
  0x00007ff6e4021b40 JavaThread "Bolt-conn-event-executor-39-thread-1" daemon [_thread_blocked, id=3493850, stack(0x00007ff6ce2fa000,0x00007ff6ce3fa000)]
  0x00007ff71c0038b0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493852, stack(0x00007ff6ce1fa000,0x00007ff6ce2fa000)]
  0x00007ff6f0002e10 JavaThread "RpcTaskScannerThread-43-thread-1" daemon [_thread_blocked, id=3493860, stack(0x00007ff6cdffa000,0x00007ff6ce0fa000)]
  0x00007ff708006610 JavaThread "Bolt-conn-event-executor-42-thread-1" daemon [_thread_blocked, id=3493862, stack(0x00007ff6cdefa000,0x00007ff6cdffa000)]
  0x00007ff750004470 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493872, stack(0x00007ff6cddfa000,0x00007ff6cdefa000)]
  0x00007ff708008020 JavaThread "RpcTaskScannerThread-46-thread-1" daemon [_thread_blocked, id=3493876, stack(0x00007ff6cdbfa000,0x00007ff6cdcfa000)]
  0x00007ff790005100 JavaThread "Bolt-conn-event-executor-45-thread-1" daemon [_thread_blocked, id=3493878, stack(0x00007ff6cdafa000,0x00007ff6cdbfa000)]
  0x00007ff70c005450 JavaThread "RpcTaskScannerThread-49-thread-1" daemon [_thread_blocked, id=3493880, stack(0x00007ff6cd8fa000,0x00007ff6cd9fa000)]
  0x00007ff7a4002cc0 JavaThread "Bolt-conn-event-executor-48-thread-1" daemon [_thread_blocked, id=3493882, stack(0x00007ff6cd7fa000,0x00007ff6cd8fa000)]
  0x00007ff720003fa0 JavaThread "RpcTaskScannerThread-52-thread-1" daemon [_thread_blocked, id=3493884, stack(0x00007ff6cd5fa000,0x00007ff6cd6fa000)]
  0x00007ff7c0e96e90 JavaThread "Bolt-conn-event-executor-51-thread-1" daemon [_thread_blocked, id=3493886, stack(0x00007ff6cd4fa000,0x00007ff6cd5fa000)]
  0x00007ff794011240 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493889, stack(0x00007ff6cd3fa000,0x00007ff6cd4fa000)]
  0x00007ff7300060b0 JavaThread "RpcTaskScannerThread-55-thread-1" daemon [_thread_blocked, id=3493893, stack(0x00007ff6cd1fa000,0x00007ff6cd2fa000)]
  0x00007ff6e4022c70 JavaThread "Bolt-conn-event-executor-54-thread-1" daemon [_thread_blocked, id=3493895, stack(0x00007ff6cd0fa000,0x00007ff6cd1fa000)]
  0x00007ff73400cc70 JavaThread "RpcTaskScannerThread-58-thread-1" daemon [_thread_blocked, id=3493897, stack(0x00007ff6ccefa000,0x00007ff6ccffa000)]
  0x00007ff708009640 JavaThread "Bolt-conn-event-executor-57-thread-1" daemon [_thread_blocked, id=3493899, stack(0x00007ff6ccdfa000,0x00007ff6ccefa000)]
  0x00007ff744002d70 JavaThread "RpcTaskScannerThread-61-thread-1" daemon [_thread_blocked, id=3493901, stack(0x00007ff6ccbfa000,0x00007ff6cccfa000)]
  0x00007ff73c008cc0 JavaThread "Bolt-conn-event-executor-60-thread-1" daemon [_thread_blocked, id=3493902, stack(0x00007ff6ccafa000,0x00007ff6ccbfa000)]
  0x00007ff6e8003130 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493905, stack(0x00007ff6cc9fa000,0x00007ff6ccafa000)]
  0x00007ff714005b20 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493910, stack(0x00007ff6cc8fa000,0x00007ff6cc9fa000)]
  0x00007ff71c005780 JavaThread "JRaft-RPC-Processor-1" daemon [_thread_blocked, id=3493917, stack(0x00007ff6cbf00000,0x00007ff6cc000000)]
  0x00007ff77c001890 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493921, stack(0x00007ff6cbe00000,0x00007ff6cbf00000)]
  0x00007ff6e80045d0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493926, stack(0x00007ff6cbd00000,0x00007ff6cbe00000)]
  0x00007ff6e4024100 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493931, stack(0x00007ff6cbc00000,0x00007ff6cbd00000)]
  0x00007ff6e80055d0 JavaThread "JRaft-RPC-Processor-1" daemon [_thread_blocked, id=3493934, stack(0x00007ff6cbb00000,0x00007ff6cbc00000)]
  0x00007ff71013c380 JavaThread "JRaft-RPC-Processor-1" daemon [_thread_blocked, id=3493935, stack(0x00007ff6cba00000,0x00007ff6cbb00000)]
  0x00007ff70c008780 JavaThread "JRaft-RPC-Processor-1" daemon [_thread_blocked, id=3493937, stack(0x00007ff6cb900000,0x00007ff6cba00000)]
  0x00007ff714006fc0 JavaThread "JRaft-RPC-Processor-1" daemon [_thread_blocked, id=3493938, stack(0x00007ff6cb800000,0x00007ff6cb900000)]
  0x00007ff720005960 JavaThread "JRaft-RPC-Processor-1" daemon [_thread_blocked, id=3493939, stack(0x00007ff6cb700000,0x00007ff6cb800000)]
  0x00007ff71c006c30 JavaThread "JRaft-RPC-Processor-1" daemon [_thread_blocked, id=3493940, stack(0x00007ff6cb600000,0x00007ff6cb700000)]
  0x00007ff7440042d0 JavaThread "JRaft-RPC-Processor-2" daemon [_thread_blocked, id=3493947, stack(0x00007ff6cb500000,0x00007ff6cb600000)]
  0x00007ff750005b90 JavaThread "JRaft-RPC-Processor-2" daemon [_thread_blocked, id=3493948, stack(0x00007ff6cb400000,0x00007ff6cb500000)]
  0x00007ff778005fc0 JavaThread "JRaft-RPC-Processor-2" daemon [_thread_blocked, id=3493949, stack(0x00007ff6cb300000,0x00007ff6cb400000)]
  0x00007ff728003f80 JavaThread "JRaft-RPC-Processor-2" daemon [_thread_blocked, id=3493950, stack(0x00007ff6cb200000,0x00007ff6cb300000)]
  0x00007ff75c00bd80 JavaThread "JRaft-RPC-Processor-2" daemon [_thread_blocked, id=3493951, stack(0x00007ff6cb100000,0x00007ff6cb200000)]
  0x00007ff77401db50 JavaThread "JRaft-RPC-Processor-2" daemon [_thread_blocked, id=3493952, stack(0x00007ff6cb000000,0x00007ff6cb100000)]
  0x00007ff77c007df0 JavaThread "JRaft-RPC-Processor-2" daemon [_thread_blocked, id=3493953, stack(0x00007ff6caf00000,0x00007ff6cb000000)]
  0x00007ff79c005940 JavaThread "RpcTaskScannerThread-64-thread-1" daemon [_thread_blocked, id=3493968, stack(0x00007ff6cad00000,0x00007ff6cae00000)]
  0x00007ff790006700 JavaThread "Bolt-conn-event-executor-63-thread-1" daemon [_thread_blocked, id=3493969, stack(0x00007ff6cac00000,0x00007ff6cad00000)]
  0x00007ff6e80060f0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493970, stack(0x00007ff6cab00000,0x00007ff6cac00000)]
  0x00007ff7ac0020d0 JavaThread "RpcTaskScannerThread-67-thread-1" daemon [_thread_blocked, id=3493974, stack(0x00007ff6ca900000,0x00007ff6caa00000)]
  0x00007ff7a4004230 JavaThread "Bolt-conn-event-executor-66-thread-1" daemon [_thread_blocked, id=3493975, stack(0x00007ff6ca800000,0x00007ff6ca900000)]
  0x00007ff6ec0031a0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493976, stack(0x00007ff6ca700000,0x00007ff6ca800000)]
  0x00007ff7bc001cd0 JavaThread "RpcTaskScannerThread-70-thread-1" daemon [_thread_blocked, id=3493984, stack(0x00007ff6ca500000,0x00007ff6ca600000)]
  0x00007ff7c0d80530 JavaThread "Bolt-conn-event-executor-69-thread-1" daemon [_thread_blocked, id=3493985, stack(0x00007ff6ca400000,0x00007ff6ca500000)]
  0x00007ff7440054d0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493986, stack(0x00007ff6ca300000,0x00007ff6ca400000)]
  0x00007ff7d40109e0 JavaThread "RpcTaskScannerThread-73-thread-1" daemon [_thread_blocked, id=3494276, stack(0x00007ff6ca600000,0x00007ff6ca700000)]
  0x00007ff710046660 JavaThread "Bolt-conn-event-executor-72-thread-1" daemon [_thread_blocked, id=3494277, stack(0x00007ff6cecfa000,0x00007ff6cedfa000)]
  0x00007ff728005470 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3494282, stack(0x00007ff6d45fe000,0x00007ff6d46fe000)]
  0x00007ff6d8004de0 JavaThread "RpcTaskScannerThread-76-thread-1" daemon [_thread_blocked, id=3494292, stack(0x00007ff6d7e00000,0x00007ff6d7f00000)]
  0x00007ff70001c580 JavaThread "Bolt-conn-event-executor-75-thread-1" daemon [_thread_blocked, id=3494294, stack(0x00007ff7412fe000,0x00007ff7413fe000)]
  0x00007ff77c006900 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3494297, stack(0x00007ff6ccffa000,0x00007ff6cd0fa000)]
  0x00007ff6e8007640 JavaThread "RpcTaskScannerThread-79-thread-1" daemon [_thread_blocked, id=3494331, stack(0x00007ff6ce0fa000,0x00007ff6ce1fa000)]
  0x00007ff6e4024be0 JavaThread "Bolt-conn-event-executor-78-thread-1" daemon [_thread_blocked, id=3494333, stack(0x00007ff6cd9fa000,0x00007ff6cdafa000)]
  0x00007ff6e8008660 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3494335, stack(0x00007ff6cd6fa000,0x00007ff6cd7fa000)]
  0x00007ff6fc0031b0 JavaThread "RpcTaskScannerThread-82-thread-1" daemon [_thread_blocked, id=3494340, stack(0x00007ff6d01fb000,0x00007ff6d02fb000)]
  0x00007ff70800a8a0 JavaThread "Bolt-conn-event-executor-81-thread-1" daemon [_thread_blocked, id=3494342, stack(0x00007ff6ce4fa000,0x00007ff6ce5fa000)]
  0x00007ff710046ce0 JavaThread "RpcTaskScannerThread-85-thread-1" daemon [_thread_blocked, id=3494346, stack(0x00007ff6d2dfd000,0x00007ff6d2efd000)]
  0x00007ff73c00c700 JavaThread "Bolt-conn-event-executor-84-thread-1" daemon [_thread_blocked, id=3494348, stack(0x00007ff6caa00000,0x00007ff6cab00000)]
  0x00007ff714008ea0 JavaThread "RpcTaskScannerThread-88-thread-1" daemon [_thread_blocked, id=3494358, stack(0x00007ff6cdcfa000,0x00007ff6cddfa000)]
  0x00007ff7900022e0 JavaThread "Bolt-conn-event-executor-87-thread-1" daemon [_thread_blocked, id=3494360, stack(0x00007ff6d60ff000,0x00007ff6d61ff000)]
  0x00007ff728005e20 JavaThread "RpcTaskScannerThread-91-thread-1" daemon [_thread_blocked, id=3494363, stack(0x00007ff6ca200000,0x00007ff6ca300000)]
  0x00007ff7a40051b0 JavaThread "Bolt-conn-event-executor-90-thread-1" daemon [_thread_blocked, id=3494365, stack(0x00007ff6ca100000,0x00007ff6ca200000)]
  0x00007ff72c006400 JavaThread "RpcTaskScannerThread-94-thread-1" daemon [_thread_blocked, id=3494371, stack(0x00007ff6c9f00000,0x00007ff6ca000000)]
  0x00007ff710047dc0 JavaThread "Bolt-conn-event-executor-93-thread-1" daemon [_thread_blocked, id=3494373, stack(0x00007ff6c9e00000,0x00007ff6c9f00000)]
  0x00007ff73c00ec30 JavaThread "RpcTaskScannerThread-97-thread-1" daemon [_thread_blocked, id=3494378, stack(0x00007ff6c9c00000,0x00007ff6c9d00000)]
  0x00007ff70001ce30 JavaThread "Bolt-conn-event-executor-96-thread-1" daemon [_thread_blocked, id=3494380, stack(0x00007ff6c9b00000,0x00007ff6c9c00000)]
  0x00007ff750007080 JavaThread "RpcTaskScannerThread-100-thread-1" daemon [_thread_blocked, id=3494383, stack(0x00007ff6c9900000,0x00007ff6c9a00000)]
  0x00007ff6e400b990 JavaThread "Bolt-conn-event-executor-99-thread-1" daemon [_thread_blocked, id=3494385, stack(0x00007ff6c9800000,0x00007ff6c9900000)]
  0x00007ff744005fa0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3494389, stack(0x00007ff6c9700000,0x00007ff6c9800000)]
  0x00007ff7600062d0 JavaThread "RpcTaskScannerThread-103-thread-1" daemon [_thread_blocked, id=3494391, stack(0x00007ff6c9500000,0x00007ff6c9600000)]
  0x00007ff73c010290 JavaThread "Bolt-conn-event-executor-102-thread-1" daemon [_thread_blocked, id=3494393, stack(0x00007ff6c9400000,0x00007ff6c9500000)]
  0x00007ff778008980 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3494394, stack(0x00007ff6c9300000,0x00007ff6c9400000)]
...
@fengjiachun
Copy link
Contributor

fengjiachun commented May 13, 2024 •
edited

Did you call RaftGroupService#shutdown when your app stoped?

One more thing to note: if you have enabled the RPC server sharing, you need to close the RPC server separately.

@fengjiachun
Copy link
Contributor

I'm closing it now, if there are any updates, feel free to reopen it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fengjiachun @Brokenice0415