Intermittent out of memory for Java 21 builds of Jenkins core on ci.jenkins.io #3874

MarkEWaite · 2023-12-20T20:49:43Z

Service(s)

ci.jenkins.io

Summary

The Java 21 builds of Jenkins core on ci.jenkins.io have been reporting "out of memory" failures as in https://github.com/jenkinsci/jenkins/pull/8793/checks?check_run_id=19835902208 . It is not a consistent failure every time, but they have happened at least several times per week.

Reproduction steps

Run a build of Jenkins core on ci.jenkins.io and see a Java 21 failure like:
- https://ci.jenkins.io/job/Core/job/jenkins/job/master/5682/pipeline-graph/
- https://ci.jenkins.io/job/Core/job/jenkins/job/master/5663/pipeline-graph/

dduportal · 2023-12-21T07:01:17Z

Adding the error message (in case the build is recycled):

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-war-plugin:3.4.0:war (default-war) on project jenkins-war: Error assembling WAR: Problem creating war: Execution exception: Java heap space -> [Help 1]

[1343](https://ci.jenkins.io/job/Core/job/jenkins/job/PR-8793/2/pipeline-console/?start-byte=1535&selected-node=210#log-1343)
19:54:18  org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-war-plugin:3.4.0:war (default-war) on project jenkins-war: Error assembling WAR: Problem creating war: Execution exception

Also (to search for memory metrics):

Running on [jnlp-maven-21-qgn7t](https://ci.jenkins.io/computer/jnlp%2Dmaven%2D21%2Dqgn7t/) in /home/jenkins/agent/workspace/Core_jenkins_PR-8793

dduportal · 2024-01-02T15:43:34Z

Different angles to diagnose:

Were the agent OOM-killed? If not, then check the memory usage (metrics)
Check if JDK21 has weird behavior inside container (limit/requests)
Check which cluster was used (is there a failing correlation with AWS or DOKS?)
Try to run on a VM agent instead of container to see if it disappear

basil · 2024-01-02T19:20:42Z

It is possible that the baseline memory requirements have increased beyond the 1.1 GB heap allocated to the Maven process in https://github.com/jenkinsci/jenkins/blob/95bff130c32c5c1fb049792d6419517748658633/.mvn/jvm.config. One way to prove this would be to set up a reproduction scenario (e.g., run the test in a loop and observe that it always fails within n iterations) and then slowly increase -Xmx until the build no longer fails within the same n iterations as in jenkinsci/jenkins#6652. I am not sure how plausible this theory is, since I never see the failure locally (though I have 16 GB of RAM), only in CI. If the theory is disproven, the next step would be to enable -XX:HeapDumpOnOutOfMemoryError, update the CI job to capture the heap dump, reproduce the problem, and then do traditional heap dump analysis to see where the memory is going.

Exploring the out of memory error report from the Java 21 agents when packaging the war file. * jenkins-infra/helpdesk#3874

MarkEWaite · 2024-01-09T17:58:16Z

I've started 4 draft pull requests to build with Java 21 on Linux. I'm also running a series of 16 builds of Jenkins core on my local computer with a command line that is very close to the original command line used in the Jenkins job.

basil · 2024-01-10T01:03:13Z

From https://ci.jenkins.io/job/Core/job/jenkins/job/master/5747/consoleFull

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-war-plugin:3.4.0:war (default-war) on project jenkins-war: Error assembling WAR: Problem creating war: Execution exception: Java heap space -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-war-plugin:3.4.0:war (default-war) on project jenkins-war: Error assembling WAR: Problem creating war: Execution exception
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:333)
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
	at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
	at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
	at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
	at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
	at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
	at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
	at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
	at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
	at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
	at java.lang.reflect.Method.invoke (Method.java:580)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: org.apache.maven.plugin.MojoExecutionException: Error assembling WAR: Problem creating war: Execution exception
	at org.apache.maven.plugins.war.WarMojo.execute (WarMojo.java:181)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
	at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
	at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
	at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
	at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
	at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
	at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
	at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
	at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
	at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
	at java.lang.reflect.Method.invoke (Method.java:580)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: org.codehaus.plexus.archiver.ArchiverException: Problem creating war: Execution exception
	at org.codehaus.plexus.archiver.AbstractArchiver.createArchive (AbstractArchiver.java:1052)
	at org.apache.maven.archiver.MavenArchiver.createArchive (MavenArchiver.java:707)
	at org.apache.maven.plugins.war.WarMojo.performPackaging (WarMojo.java:237)
	at org.apache.maven.plugins.war.WarMojo.execute (WarMojo.java:179)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
	at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
	at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
	at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
	at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
	at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
	at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
	at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
	at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
	at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
	at java.lang.reflect.Method.invoke (Method.java:580)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: java.io.IOException: Execution exception
	at org.codehaus.plexus.archiver.zip.AbstractZipArchiver.close (AbstractZipArchiver.java:798)
	at org.codehaus.plexus.archiver.AbstractArchiver.createArchive (AbstractArchiver.java:1039)
	at org.apache.maven.archiver.MavenArchiver.createArchive (MavenArchiver.java:707)
	at org.apache.maven.plugins.war.WarMojo.performPackaging (WarMojo.java:237)
	at org.apache.maven.plugins.war.WarMojo.execute (WarMojo.java:179)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
	at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
	at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
	at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
	at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
	at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
	at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
	at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
	at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
	at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
	at java.lang.reflect.Method.invoke (Method.java:580)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.codehaus.plexus.archiver.zip.ByteArrayOutputStream.needNewBuffer (ByteArrayOutputStream.java:153)
	at org.codehaus.plexus.archiver.zip.ByteArrayOutputStream.write (ByteArrayOutputStream.java:192)
	at org.apache.commons.io.output.ThresholdingOutputStream.write (ThresholdingOutputStream.java:232)
	at org.codehaus.plexus.archiver.zip.DeferredScatterOutputStream.writeOut (DeferredScatterOutputStream.java:44)
	at org.apache.commons.compress.archivers.zip.StreamCompressor$ScatterGatherBackingStoreCompressor.writeOut (StreamCompressor.java:83)
	at org.apache.commons.compress.archivers.zip.StreamCompressor.writeCounted (StreamCompressor.java:312)
	at org.apache.commons.compress.archivers.zip.StreamCompressor.deflate (StreamCompressor.java:203)
	at org.apache.commons.compress.archivers.zip.StreamCompressor.deflateUntilInputIsNeeded (StreamCompressor.java:230)
	at org.apache.commons.compress.archivers.zip.StreamCompressor.writeDeflated (StreamCompressor.java:322)
	at org.apache.commons.compress.archivers.zip.StreamCompressor.write (StreamCompressor.java:299)
	at org.apache.commons.compress.archivers.zip.StreamCompressor.deflate (StreamCompressor.java:221)
	at org.apache.commons.compress.archivers.zip.ScatterZipOutputStream.addArchiveEntry (ScatterZipOutputStream.java:181)
	at org.apache.commons.compress.archivers.zip.ParallelScatterZipCreator.lambda$createCallable$1 (ParallelScatterZipCreator.java:220)
	at org.apache.commons.compress.archivers.zip.ParallelScatterZipCreator$$Lambda/0x00007fdab8f25ec8.call (Unknown Source)
	at java.util.concurrent.FutureTask.run (FutureTask.java:317)
	at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith (Thread.java:1596)
	at java.lang.Thread.run (Thread.java:1583)

timja · 2024-01-13T16:24:30Z

Should be resolved by jenkinsci/jenkins#8856, builds are looking more stable.

MarkEWaite · 2024-01-14T14:27:31Z

Unfortunately, even with jenkinsci/jenkins#8856 , we are still seeing intermittent failures like a recent dependabot pull request https://ci.jenkins.io/job/Core/job/jenkins/job/PR-8865/1/execution/node/271/log/#1318

MarkEWaite added the triage Incoming issues that need review label Dec 20, 2023

jenkins-infra-helpdesk-app bot added the ci.jenkins.io label Dec 20, 2023

MarkEWaite mentioned this issue Dec 20, 2023

Restore JCasC compatibility for JNLPLauncher.tunnel jenkinsci/jenkins#8793

Merged

6 tasks

dduportal added this to the infra-team-sync-2024-01-02 milestone Dec 21, 2023

dduportal removed the triage Incoming issues that need review label Jan 2, 2024

dduportal modified the milestones: infra-team-sync-2024-01-02, infra-team-sync-2024-01-09 Jan 3, 2024

MarkEWaite mentioned this issue Jan 5, 2024

[JENKINS-66530] Change focus in the 'new item' page only if 'from' has a valid job name jenkinsci/jenkins#8807

Merged

dduportal mentioned this issue Jan 8, 2024

Possible change to ci.jenkins.io's linux agents causes core test failure #3890

Open

MarkEWaite added a commit to MarkEWaite/jenkins that referenced this issue Jan 9, 2024

Test Linux with JDK 21

12e5694

Exploring the out of memory error report from the Java 21 agents when packaging the war file. * jenkins-infra/helpdesk#3874

This was referenced Jan 9, 2024

Test Linux with JDK 21 jenkinsci/jenkins#8848

Closed

Test Linux with JDK 21 jenkinsci/jenkins#8849

Closed

Test Linux with JDK 21 jenkinsci/jenkins#8850

Closed

Test Linux with JDK 21 jenkinsci/jenkins#8851

Closed

dduportal modified the milestones: infra-team-sync-2024-01-09, infra-team-sync-2024-01-16 Jan 10, 2024

timja mentioned this issue Jan 11, 2024

Increase memory for war assembly jenkinsci/jenkins#8856

Merged

smerle33 modified the milestones: infra-team-sync-2024-01-16, infra-team-sync-2024-01-23 Jan 16, 2024

dduportal modified the milestones: infra-team-sync-2024-01-23, infra-team-sync-2024-02-13 Jan 24, 2024

dduportal modified the milestones: infra-team-sync-2024-02-13, infra-team-sync-2024-02-20 Feb 13, 2024

dduportal modified the milestones: infra-team-sync-2024-02-20, infra-team-sync-2024-03-12 Feb 22, 2024

dduportal modified the milestones: infra-team-sync-2024-03-12, infra-team-sync-next Mar 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Intermittent out of memory for Java 21 builds of Jenkins core on ci.jenkins.io #3874

Intermittent out of memory for Java 21 builds of Jenkins core on ci.jenkins.io #3874

MarkEWaite commented Dec 20, 2023

dduportal commented Dec 21, 2023

dduportal commented Jan 2, 2024

basil commented Jan 2, 2024

MarkEWaite commented Jan 9, 2024

basil commented Jan 10, 2024

timja commented Jan 13, 2024 •

edited

MarkEWaite commented Jan 14, 2024

Intermittent out of memory for Java 21 builds of Jenkins core on ci.jenkins.io #3874

Intermittent out of memory for Java 21 builds of Jenkins core on ci.jenkins.io #3874

Comments

MarkEWaite commented Dec 20, 2023

Service(s)

Summary

Reproduction steps

dduportal commented Dec 21, 2023

dduportal commented Jan 2, 2024

basil commented Jan 2, 2024

MarkEWaite commented Jan 9, 2024

basil commented Jan 10, 2024

timja commented Jan 13, 2024 • edited

MarkEWaite commented Jan 14, 2024

timja commented Jan 13, 2024 •

edited