Document that the shutdown endpoint is not intended for use when deploying a war to a servlet container

[membersound]

When a *.war file is undeployed from Tomcat (eg via the tomcat web frontend), the following memory leak is detected:

02-Jul-2019 11:50:40.143 WARNING [http-nio-8080-exec-5] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [my-app] appears to have started a thread named [Thread-22] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.3/java.lang.Thread.sleep(Native Method)
 org.springframework.boot.actuate.context.ShutdownEndpoint.performShutdown(ShutdownEndpoint.java:67)
 org.springframework.boot.actuate.context.ShutdownEndpoint$$Lambda$3977/0x000000084242f040.run(Unknown Source)
 java.base@11.0.3/java.lang.Thread.run(Thread.java:834)

Could you fix that?

[wilkinsona]

The shutdown endpoint will only create a thread to close the application context if you have called it. In other words, I don't believe that undeploying a war file from Tomcat will be sufficient to recreate this problem. Furthermore, the shutdown endpoint is disabled by default and is not intended for use when deploying a war file to a container. Instead, you should use the container's standard mechanisms to shut down and undeploy the application.

[membersound]

I will try to create an example project.

But in the meantime: the documentation says, the /shutdown endpoint "Lets the application be gracefully shutdown". So, is the endpoint only for shutting down the app in case of running an embedded jar file, not a war container? If so, maybe this could be added to the docs as a note?
https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html

Then, what would be the correct way of shutting down a Spring war in a tomcat container?

[wilkinsona]

That's a fair point. We should improve the docs here.

Then, what would be the correct way of shutting down a Spring war in a tomcat container?

Use the container's standard mechanism for undeploying an application. As part of that, the container will call the contextDestroyed(ServletContext) method on the ContextLoaderListener that Boot automatically registers which will result in the application context being closed.

[membersound]

The root cause is management.endpoints.enabled-by-default=true. If set (which I use in conjunction with spring-boot-admin-starter to just expose any of my actuator endpoints to the SBA interface), then also the ShutdownEndpoint is initialized (though not exposed to SBA by default).

So, if the docs state enabled by default = No in the table for shutdown endpoint, then I would expect the application property ...enabled-by-default=true having the same effect, and the shutdown endpoint should still be disabled and must be explicit opt in.

Sidenote: I'm setting this property explicit due to different profiles. This way, I can disable the actuator completely in dev+test, and enable it in production by just overriding the property mentioned:

application.properties [dev]:
management.endpoints.enabled-by-default=false

application-test.properties
(inherits from application.properties)

application-production.properties
management.endpoints.enabled-by-default=true

[wilkinsona]

The default for enabled-by-default is true so either there's a bug here or there's more to it than you have described. A sample war that reproduces the problem would be useful.

[membersound]

Find example attached. It's a simple as adding the actuator dependency to pom.xml and setting the property management.endpoints.enabled-by-default=true.

Put your breakpoint into ShutdownEndpoint.setApplicationContext(). It is only invoked if property above is set. Remove the property completely and the ShutdownEndpoint is never called (though as you said, the default of that property is also =true).

shutdown-example.zip

[membersound]

Also, if you compile my example project with <packaging>war</packaging>, deploy it to a tomcat webserver, access the deployed app, then simply undeploy the app. Then the memory leak logs can be found in catalina.out logs indeed.

So, if the ShutdownEndpoint is started inside a tomcat container, it definitely creates a memory leak. Probably we have multiple different issues here?

[wilkinsona]

The default for enabled-by-default is true

I had misremembered this, sorry. The default for enabled-by-default is that it has no value which means that each individual endpoint's default enablement is used. That's false for the shutdown endpoint and true for all other endpoints as documented. When you set management.endpoints.enabled-by-default=true it configures the default enablement of every endpoint. The enablement can then be fine-tuned using the individual management.endpoint.<id>.enabled properties.

[wilkinsona]

The shutdown endpoint will only create a thread to close the application context if you have called it. In other words, I don't believe that undeploying a war file from Tomcat will be sufficient to recreate this problem.

I was wrong here too. Sorry again. The sample has allowed me to see what's happening. When the shutdown endpoint is enabled, it's exposed as a bean. Framework's support for inferring a bean's destroy method and wrapping it in a DisposableBeanAdapter finds the shutdown method and assumes that it's a destroy method that should be called when the context is being closed. When Tomcat undeploys the application the context is closed. This causes the endpoint's shutdown method to be called as it's considered to be a destroy method. As a result, a thread is created to shut down the app by closing the context. This thread blocks for the remainder of the close processing that is already in progress which causes Tomcat to detect it as a leaked thread. Once the first close attempt has completed, the second will then no-op and the thread will exit.

[membersound]

So in my case I'd have to explicit set management.endpoint.shutdown.enabled=false to disable the shutdown endpoint.

[freeman9397]

I am deploying Spring Boot application on Tomcat with the war packaging and also getting memory leak error message when I deploy it locally on Tomcat. BTW, I have implemented graceful shutdown in my application by referring to this document here.

Application Versions:

• Tomcat - 9.0.64
• Spring Boot - 2.7.1
• Packaging - War
• Java - 1.8
• Project - Gradle
• Gradle - 7.4.1

I have found another way to implementing it manually by getting the Tomcat's connector object and waiting for grace period to allow all inflight requests to complete before moving to the shutdown phase with the help of @PreDestroy annotation. However, I just wanted to know if there is a better way to do this as this is a trivial problem that many developers would have faced.

[philwebb]

@freeman9397 Graceful shutdown as documented in the section you linked to is for embedded web servers. If you're deploying a war file to Tomcat then shutdown is outside of Spring Boot's control.

[freeman9397]

@philwebb Thanks for the quick update. Do you know of a way how to handle graceful shutdown when application is deployed through a war file in the best way other than I mentioned?