Encoutering memory issues post Spring boot 2.7.1 version

[loveshjain]

After upgrading our applications to Spring boot 2.7.1 version, we have started encountering memory issues.

Our Observations

Changes were made in 2.7.1 in the org.springframework.boot.loader.jar package around closing jar files. A list of nested jars were added and that is what is holding on to objects in our heap dump.

We can see 22,000 JarFile objects in the heap dump where we'd expect about 118 as that is how many nested jars there are in the application. Note that apart from the expected jar files, the other 22,000-odd are all url = jar:file:/deployments/application-web-1.0.0.jar!/BOOT-INF/lib/swagger-ui-4.11.1.jar!/

Please find below the heap dump and Dynatrace screenshot as reference

Note :- No changes were made in org.springframework.boot.loader.jar package in 2.7.2 so I suspect the problem will still be there.

Can someone please look into this.

[wilkinsona]

Thanks for the report, @loveshjain. There are some similarities with #32042 and #31853. Are you using PropertiesLauncher to run your app? To allow us to diagnose the problem, please provide a minimal sample that reproduces the leak.

[philwebb]

@loveshjain I've pushed a fix for #31853 that removes the nested jars list. Once build 713 could you please try the latest SNAPSHOT and see if the memory issues persist?

[beggsj42]

Hi @philwebb - I work with @loveshjain and am looking at this. Our problem is that we haven't been able to replicate this in non-prod. I'm going to see what I can do in that regard in the next couple of days.

I'm speculating that this might have to do with the JarFile objects being removed by the GC (as they are soft referenced?) if there is memory pressure, and then being reloaded later, but never getting removed from the nested jars list? I've never really looked at custom class loader code though so I'm a bit in the dark.
Does that sound reasonable though - to maybe try to get the JVM close to the heap limit and see if that causes it? I'm just looking for things to try really.

It was fairly odd behaviour in production - it went for several days without us noticing an issue, and when we did hit the issue it wasn't in a high-load period and it seemed to self-correct. In the tenured gen picture above it looks like it started to have a GC issue about 10am, but we didn't notice until it started affecting response time badly at about 9:45pm.

We have had similar issues on multiple services though and rolling back to 2.7.0 resolved it.

[beggsj42]

I believe we're using the JarLauncher - we're just using the spring-boot-maven-plugin and we're not overriding the 'layout'.

[philwebb]

See #31853 (comment). We're going to revert #29356

[beggsj42]

Right, thanks - I'm fairly sure that will solve our issue. I'll still try to put some effort in to replicating the problem though hopefully it'll mostly just be for the intellectual exercise now.

[snicoll]

@beggsj42 upgrading to 2.7.3 contains the change that Phil mentioned if you want to double check.

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[beggsj42]

Sorry for the late reply. I still haven't been able to replicate the problem in 2.7.1 in non-prod, but 2.7.3 definitely has solved the issue for us in production.

We're happy for this issue to be closed, thanks.

[snicoll]

Thanks for following up.

[sm0217]

Apologies for commenting on a closed ticket but we are seeing similar issues upgrading from 2.4 to 2.7.16. May I know if there was any fix provided for this and it is present in the latest 2.7.x versions

[wilkinsona]

Please see above. A change was made in Spring Boot 2.7.3 that fixed the problem that was reported in this issue. That fix has not be removed and should be present in the latest 2.7.x versions.

[sushmitha-malae]

Hi @sm0217 Were you able to fix the issues? we are also facing similar issues when upgraded to 2.7.16.