Gradle 8 RC2 runs out of metaspace

[TWiStErRob]

Expected Behavior

Works as before.

Current Behavior

I'm consistently getting

    The Daemon will expire after the build after running out of JVM Metaspace.
    The project memory settings are likely not configured or are configured to an insufficient value.
    The daemon will restart for the next build, which may increase subsequent build times.
    These settings can be adjusted by setting 'org.gradle.jvmargs' in 'gradle.properties'.
    The currently configured max heap space is '512 MiB' and the configured max metaspace is '256 MiB'.
    For more information on how to set these values, visit the user guide at https://docs.gradle.org/8.0-rc-2/userguide/build_environment.html#configuring_jvm_memory
    To disable this warning, set 'org.gradle.daemon.performance.disable-logging=true'.
    Daemon will be stopped at the end of the build after running out of JVM Metaspace

Context

I have a project which is a set of Gradle plugins. This project is using GradleRunner from Gradle TestKit to execute builds in temporary folders. There's complex CI setup to run on all kinds of Android Gradle Plugin and Gradle combinations (Kotlin and Java versions are fixed for each combo). Here's an example workflow from master.

I'm in the process of adopting Gradle 8 RC 1/2 in PR. In this PR I'm trying get the latest AGP 7.4.0 + Gradle 8 RC2 combination to be green. Locally it's OK, but on CI it keeps timing out, because of GradleRunner daemons running out of metaspace.

This means that I'm not directly observing this behavior, but it shows up on trivial near-empty projects in integration tests. I'm in the process of trying narrow it down to prove that Gradle 8 RC 1/2 is the root cause or another change in the PR (it's a lot of commits 😞). In the meantime I'm reporting this in case there are others experiencing the same, or you have any idea why it could be.

Steps to Reproduce

Sadly the only thing I have is a branch which consistently fails on GitHub Actions. I was able to reproduce it once locally, it's not as consistent.

Your Environment

Gradle 8.0 RC2 is definitely failing, so does RC1. Gradle 7.5.1 and Gradle 7.6 is not affected at all as can be seen here in the sidebar.

Build scan URL: Not possible, Gradle build doesn't finish, GitHub Actions terminates after timeout.

[TWiStErRob]

I was able to "fix" all the tests that were failing by increasing metaspace from 256M to 384M in these tests:

• AndroidSigningPluginIntgTest.kt
• AndroidMinificationPluginIntgTest.kt
• KotlinPluginIntgTest.kt
• JavaPluginIntgTest.kt

The commonalities between them is that most of them do Kotlin or Java compilation.

---

In the meantime I also noticed that I've given more memory to small manual integration example projects too, but strangely I haven't been able to reproduce it on those projects when I was investigating this issue. These projects apply AGP and one of my plugins, otherwise empty.

[ljacomet]

Thank you for your interest in Gradle!

This issue needs a decision from the team responsible for that area. They have been informed, response time may vary.

[TWiStErRob]

@ljacomet I don't think it's only plugin development affected, as all the builds running are simple projects. I have an idea to confirm this.

[TWiStErRob]

Potentially related https://youtrack.jetbrains.com/issue/KT-56093

[MGaetan89]

Might be fixed by #23749?

[SimonMarquis]

The warning message still appears on rc-3 on a trivial build https://github.com/SimonMarquis/SealedObjectInstances/actions/runs/4103117251/jobs/7076901164#step:4:125

[TWiStErRob]

I confirmed it too, #23749 does not fix this.

TWiStErRob/net.twisterrob.gradle#399

• RC2: TWiStErRob/net.twisterrob.gradle@7c8052e -> timeout because tests thrash after running out of JVM Metaspace
• RC3: TWiStErRob/net.twisterrob.gradle@385830f -> timeout because tests thrash after running out of JVM Metaspace

[TWiStErRob]

@ljacomet would changing the default be a long shot?

-org.gradle.jvmargs=-Xmx512M -XX:MaxMetaspaceSize=256M
+org.gradle.jvmargs=-Xmx512M -XX:MaxMetaspaceSize=384M

(Related docs: https://github.com/gradle/gradle/blame/v8.0.0-RC3/subprojects/docs/src/docs/userguide/running-builds/build_environment.adoc#L104)

[big-guy]

@SimonMarquis It looks like your build was reporting the low memory situation in 7.5.1 and 7.6. In 8.0, we made the message more elaborate/longer.

What's suspicious to me is that I don't think there's any reason for a build of this size to be running low on memory with our defaults, so it makes me think that either the heap size isn't what we think it is or we're misreporting the memory situation. I tried reproducing this locally and I don't see that warning. The build scan seems to say the max heap memory is 512MB, which is our default, so misreporting seems more likely.

[SimonMarquis]

My project includes tests which compile Kotlin source code snippets (in task :processor:test) (using tschuchortdev/kotlin-compile-testing), so this might be the reason why we see these logs here.

[TWiStErRob]

so it makes me think that either the heap size isn't what we think it is or we're misreporting the memory situation

I don't think there's a misreport, see my log below. There may be just more than 256M classes on the classpath, I guess; or some classloader leaks that shouldn't.

I tried reproducing this locally and I don't see that warning.

I can't repro it locally either, but on GitHub Actions I have a consistent failure. I'm trying to minimize a repro. Also note that @SimonMarquis' build is also running out of metaspace on the CI. @SimonMarquis can you repro locally consistently?

The build scan seems to say the max heap memory is 512MB, which is our default, so misreporting seems more likely.

The heap is 512, that's fine, the problem is the metaspace, which is 256... I have diagnostic info in my tests to confirm that everything is as expected:

// from the test running the testkit build:
Test Java: Oracle Corporation OpenJDK Runtime Environment 11.0.2 (11.0.2+9 2019-01-15) from P:\tools\lang\java-11.0.2-x64-openjdk.
Requesting Gradle 8.0-rc-5 in worker #1 at C:\Users\TWiStEr\AppData\Local\Temp\.gradle-test-kit-TWiStEr-1.

// beginning of build

// from an init script's rootProject { }
Gradle 8.0-rc-5 worker #1 at C:\Users\TWiStEr\AppData\Local\Temp\.gradle-test-kit-TWiStEr-1 with P:\caches\gradle\wrapper\dists\gradle-8.0-rc-5-all\7yhhfnupylrv44qmug3cozi6u\gradle-8.0-rc-5.
Gradle Java: Oracle Corporation OpenJDK Runtime Environment 11.0.2 (11.0.2+9 2019-01-15) from P:\tools\lang\java-11.0.2-x64-openjdk.

// from an init script's rootProject { }
Gradle memory: heap = 79MB/512MB (432MB free), metaspace = 122MB/256MB (133MB free).

Running `gradle assembleRelease`
// build output

// from an init script's buildFinished { }
Gradle memory: heap = 143MB/512MB (368MB free), metaspace = 241MB/256MB (14MB free).

This is a local (Windows) execution, you can see it barely passes. On CI (linux) something must be different.

[TWiStErRob]

I managed to reduce it to a smaller example:
CI: https://github.com/TWiStErRob/net.twisterrob.gradle/actions/runs/4150809580/jobs/7180706136
Steps: https://github.com/TWiStErRob/net.twisterrob.gradle/actions/runs/4150809580/workflow#L5-L27
Files: https://github.com/TWiStErRob/net.twisterrob.gradle/tree/6c5c119/docs/examples/release (originally, but now deleted the branch)

The repro has no dependencies other than mavenCentral().

It reproduces consistently on CI, but not locally. I think this might be because of the fashion of CI, that it starts from scratch every time, and for example it needs to generate gradle-api/dsl jars.

Locally I see this warning / run into metaspace quite often now that more of my projects are using Gradle 8 RC. (I switch between them multiple times a day.)

@big-guy what could we procure to move this along and figure out what's wrong? When I get a metaspace OOM, it dumps heap, but the .hprof file doesn't contain Metaspace 🙄

[eskatos]

@TWiStErRob, could you please try with 8.0.1?

[TWiStErRob]

Ok will do, what has changed?

[eskatos]

Looking at it, nothing changed wrt memory since RC5.
We'll get our folks to look at your reproducer.

[TWiStErRob]

I agree with that, but there's also an expectation that things that were building before, will build on future platforms too; Gradle is extremely good at this. I had the same test running on Gradle 4-8, and it only failed on latest.

If the platform takes up 50-100MB of extra metaspace (probably acceptable considering the 4-version Kotlin jump), then the default is increased to accommodate that. I think this is why they bumped it up in the end. I'm sure there exist other combinations of plugins where this might come up (e.g. Spring, GraphQL, Kotlin, Dokka, Detekt, ktfmt)

Actually, I'm surprised that this didn't come up in internal integration tests, as far as I know Gradle runs AGP-specific tests.

[mklueh]

This will be "resolved" in Gradle 8.0.2 by raising default metaspace, similar to this suggestion.

Note that the Gradle team has plans to look at memory consumption in upcoming 8.x releases.

Hi. I'm running into this issue with 8.0.2 - are you sure this was resolved?

[TWiStErRob]

@mklueh it'll never really be fully resolved, as Zac said as well. There'll always be project which need more heap and meta space. Gradle added 128MB extra metaspace to the default as a workaround to reduce the chance of this happening on trivial projects. Also as you quoted, the story doesn't end here, there'll be more investigations.

Is it possible you're overriding the jvmargs, because this change won't affect anyone with explicit memory, but the problem exists because the memory usage grew overall, so you might need to increase it a bit as well. (Just guessing here, as I don't know on what project you experienced this.)