Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spring version 2.7.8 : Deadlock when the application context is closed and System.exit(int) is then called #35854

Closed
ssakshi0302 opened this issue Jun 12, 2023 · 3 comments
Closed

Spring version 2.7.8 : Deadlock when the application context is closed and System.exit(int) is then called #35854

ssakshi0302 opened this issue Jun 12, 2023 · 3 comments

Comments

@ssakshi0302
Copy link

Hi team, we are using Spring 2.7.8 version. Our application is going in deadlock state after System.exit(1) is called as part of handling exception.

package com.example.demo;

Below is the threaddump of the application for reference.

2023-06-08 19:38:55
Full thread dump OpenJDK 64-Bit Server VM (11.0.15+10-LTS mixed mode):

Threads class SMR info:
_java_thread_list=0x0000600000ec8a40, length=15, elements={
0x00007fef4e012800, 0x00007fef4e07b000, 0x00007fef4d822000, 0x00007fef4d808800,
0x00007fef4d825000, 0x00007fef4d823000, 0x00007fef4f042800, 0x00007fef4e016000,
0x00007fef4d824000, 0x00007fef4e07c800, 0x00007fef4d850000, 0x00007fef4e3ac800,
0x00007fef4db48000, 0x00007fef4f1d0000, 0x00007fef4f301800
}

"main" #1 prio=5 os_prio=31 cpu=210981.88ms elapsed=935.39s tid=0x00007fef4e012800 nid=0x3f03 in Object.wait()  [0x000000030501a000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.15/Native Method)
	- waiting on <0x000000043b3b9710> (a java.lang.Thread)
	at java.lang.Thread.join(java.base@11.0.15/Thread.java:1300)
	- waiting to re-lock in wait() <0x000000043b3b9710> (a java.lang.Thread)
	at java.lang.Thread.join(java.base@11.0.15/Thread.java:1375)
	at java.lang.ApplicationShutdownHooks.runHooks(java.base@11.0.15/ApplicationShutdownHooks.java:107)
	at java.lang.ApplicationShutdownHooks$1.run(java.base@11.0.15/ApplicationShutdownHooks.java:46)
	at java.lang.Shutdown.runHooks(java.base@11.0.15/Shutdown.java:130)
	at java.lang.Shutdown.exit(java.base@11.0.15/Shutdown.java:174)
	- locked <0x00000004000c1108> (a java.lang.Class for java.lang.Shutdown)
	at java.lang.Runtime.exit(java.base@11.0.15/Runtime.java:116)
	at java.lang.System.exit(java.base@11.0.15/System.java:1752)
	at ...
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.15/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.15/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.15/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@11.0.15/Method.java:566)
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389)
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333)
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:440)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
	at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$348/0x0000000800332c40.getObject(Unknown Source)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	- locked <0x000000043b001028> (a java.util.concurrent.ConcurrentHashMap)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
	at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$348/0x0000000800332c40.getObject(Unknown Source)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	- locked <0x000000043b001028> (a java.util.concurrent.ConcurrentHashMap)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
	- locked <0x000000043ab105a0> (a java.lang.Object)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:731)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:307)
	at com.salesforce.sa.jobs.JobsMain.main(JobsMain.java:47)

"Reference Handler" #2 daemon prio=10 os_prio=31 cpu=17.79ms elapsed=935.36s tid=0x00007fef4e07b000 nid=0x4c03 waiting on condition  [0x0000000305731000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.15/Native Method)
	at java.lang.ref.Reference.processPendingReferences(java.base@11.0.15/Reference.java:241)
	at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.15/Reference.java:213)

"Finalizer" #3 daemon prio=8 os_prio=31 cpu=2.48ms elapsed=935.36s tid=0x00007fef4d822000 nid=0x4a03 in Object.wait()  [0x0000000305834000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.15/Native Method)
	- waiting on <0x00000004005174f8> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.15/ReferenceQueue.java:155)
	- waiting to re-lock in wait() <0x00000004005174f8> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.15/ReferenceQueue.java:176)
	at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.15/Finalizer.java:170)

"Signal Dispatcher" #4 daemon prio=9 os_prio=31 cpu=0.61ms elapsed=935.34s tid=0x00007fef4d808800 nid=0x7a03 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #5 daemon prio=9 os_prio=31 cpu=0.05ms elapsed=935.34s tid=0x00007fef4d825000 nid=0x7903 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread0" #6 daemon prio=9 os_prio=31 cpu=904.61ms elapsed=935.34s tid=0x00007fef4d823000 nid=0x7703 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #10 daemon prio=9 os_prio=31 cpu=0.05ms elapsed=935.34s tid=0x00007fef4f042800 nid=0x7603 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #11 daemon prio=8 os_prio=31 cpu=30.24ms elapsed=935.27s tid=0x00007fef4e016000 nid=0x6403 in Object.wait()  [0x0000000305e52000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.15/Native Method)
	- waiting on <no object reference available>
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.15/ReferenceQueue.java:155)
	- waiting to re-lock in wait() <0x0000000400517af0> (a java.lang.ref.ReferenceQueue$Lock)
	at jdk.internal.ref.CleanerImpl.run(java.base@11.0.15/CleanerImpl.java:148)
	at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)
	at jdk.internal.misc.InnocuousThread.run(java.base@11.0.15/InnocuousThread.java:161)

"JDWP Transport Listener: dt_socket" #12 daemon prio=10 os_prio=31 cpu=159.18ms elapsed=935.23s tid=0x00007fef4d824000 nid=0x6703 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"JDWP Event Helper Thread" #13 daemon prio=10 os_prio=31 cpu=240.32ms elapsed=935.23s tid=0x00007fef4e07c800 nid=0x6903 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"JDWP Command Reader" #14 daemon prio=10 os_prio=31 cpu=76.77ms elapsed=935.23s tid=0x00007fef4d850000 nid=0x6f03 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"RMI TCP Accept-0" #16 daemon prio=5 os_prio=31 cpu=101.41ms elapsed=934.64s tid=0x00007fef4e3ac800 nid=0xa803 runnable  [0x000000030625e000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(java.base@11.0.15/Native Method)
	at java.net.AbstractPlainSocketImpl.accept(java.base@11.0.15/AbstractPlainSocketImpl.java:474)
	at java.net.ServerSocket.implAccept(java.base@11.0.15/ServerSocket.java:576)
	at java.net.ServerSocket.accept(java.base@11.0.15/ServerSocket.java:539)
	at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(jdk.management.agent@11.0.15/LocalRMIServerSocketFactory.java:52)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(java.rmi@11.0.15/TCPTransport.java:394)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(java.rmi@11.0.15/TCPTransport.java:366)
	at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"Attach Listener" #17 daemon prio=9 os_prio=31 cpu=11.76ms elapsed=934.38s tid=0x00007fef4db48000 nid=0xa30b waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"RMI Scheduler(0)" #19 daemon prio=5 os_prio=31 cpu=132.30ms elapsed=934.20s tid=0x00007fef4f1d0000 nid=0xfe03 waiting on condition  [0x0000000306e82000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.15/Native Method)
	- parking to wait for  <0x000000040051ea08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.15/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.15/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:1182)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.15/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.15/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.15/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.15/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"SpringApplicationShutdownHook" #23 prio=5 os_prio=31 cpu=31.62ms elapsed=553.13s tid=0x00007fef4f301800 nid=0xf653 waiting for monitor entry  [0x0000000306d7f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1021)
	- waiting to lock <0x000000043ab105a0> (a java.lang.Object)
	at org.springframework.boot.SpringApplicationShutdownHook.closeAndWait(SpringApplicationShutdownHook.java:145)
	at org.springframework.boot.SpringApplicationShutdownHook$$Lambda$789/0x000000080070ec40.accept(Unknown Source)
	at java.lang.Iterable.forEach(java.base@11.0.15/Iterable.java:75)
	at org.springframework.boot.SpringApplicationShutdownHook.run(SpringApplicationShutdownHook.java:114)
	at java.lang.Thread.run(java.base@11.0.15/Thread.java:829)

"VM Thread" os_prio=31 cpu=259.19ms elapsed=935.37s tid=0x00007fef4d14b800 nid=0x4e03 runnable  

"GC Thread#0" os_prio=31 cpu=85.60ms elapsed=935.39s tid=0x00007fef4e013800 nid=0x3803 runnable  

"GC Thread#1" os_prio=31 cpu=85.24ms elapsed=934.37s tid=0x00007fef4db49000 nid=0x883b runnable  

"GC Thread#2" os_prio=31 cpu=85.79ms elapsed=934.37s tid=0x00007fef4db07800 nid=0x8903 runnable  

"GC Thread#3" os_prio=31 cpu=83.17ms elapsed=934.37s tid=0x00007fef4e475000 nid=0x8a03 runnable  

"GC Thread#4" os_prio=31 cpu=91.33ms elapsed=934.37s tid=0x00007fef4e486000 nid=0x9e03 runnable  

"GC Thread#5" os_prio=31 cpu=86.94ms elapsed=934.37s tid=0x00007fef4e491000 nid=0x9c03 runnable  

"GC Thread#6" os_prio=31 cpu=80.28ms elapsed=934.37s tid=0x00007fef4e492000 nid=0x9a03 runnable  

"GC Thread#7" os_prio=31 cpu=76.96ms elapsed=934.37s tid=0x00007fef4e492800 nid=0x9903 runnable  

"GC Thread#8" os_prio=31 cpu=83.71ms elapsed=934.37s tid=0x00007fef4e495800 nid=0x8c03 runnable  

"G1 Main Marker" os_prio=31 cpu=1.38ms elapsed=935.39s tid=0x00007fef4f03c000 nid=0x3603 runnable  

"G1 Conc#0" os_prio=31 cpu=21.48ms elapsed=935.39s tid=0x00007fef4f03c800 nid=0x4103 runnable  

"G1 Conc#1" os_prio=31 cpu=18.43ms elapsed=933.86s tid=0x00007fef4e4c6000 nid=0xfc03 runnable  

"G1 Refine#0" os_prio=31 cpu=0.70ms elapsed=935.39s tid=0x00007fef4d13f800 nid=0x5303 runnable  

"G1 Young RemSet Sampling" os_prio=31 cpu=237.05ms elapsed=935.39s tid=0x00007fef4d809800 nid=0x5203 runnable  
"VM Periodic Task Thread" os_prio=31 cpu=490.62ms elapsed=934.63s tid=0x00007fef4d864800 nid=0xa7c3 waiting on condition  

JNI global refs: 70, weak refs: 10757

It's SpringApplicationShutdownHook and main that are deadlocked.
@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged label Jun 12, 2023
@ssakshi0302 ssakshi0302 changed the title Deadlock when the application context is closed and System.exit(int) is then called Spring version 2.7.8 : Deadlock when the application context is closed and System.exit(int) is then called Jun 12, 2023
@wilkinsona
Copy link
Member

Thanks for the report. It sounds very similar to #27049 but it's hard to be certain with only a thread dump to look at. I suspect there must be some subtle difference as #27049 should not occur with Spring Boot 2.7.8.

First of all, please try upgrading to the latest 2.7.x release. If the problem still occurs and you would like us to spend some more time investigating, please spend some time creating a complete yet minimal sample that reproduces the problem. You can share it with us by pushing it to a separate repository on GitHub or by zipping it up and attaching it to this issue.

@wilkinsona wilkinsona added the status: waiting-for-feedback We need additional information before we can continue label Jun 13, 2023
@spring-projects-issues
Copy link
Collaborator

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

@spring-projects-issues spring-projects-issues added the status: feedback-reminder We've sent a reminder that we need additional information before we can continue label Jun 20, 2023
@spring-projects-issues
Copy link
Collaborator

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

@spring-projects-issues spring-projects-issues closed this as not planned Won't fix, can't repro, duplicate, stale Jun 27, 2023
@spring-projects-issues spring-projects-issues removed status: waiting-for-feedback We need additional information before we can continue status: feedback-reminder We've sent a reminder that we need additional information before we can continue status: waiting-for-triage An issue we've not yet triaged labels Jun 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@wilkinsona @ssakshi0302 @spring-projects-issues