Increased memory usage (with -p option)

[tomasAlabes]

Expected Behavior

We should be able to build a project in the multimodule project by specifying the -p <project> option.

Current Behavior

This isn't reproducible with Gradle 7.5.1, only with 7.6:
Using gradle's sample project and updating gradle with:

./gradlew wrapper --gradle-version=7.6-rc-2

./gradlew -p server-application/ build
> Task :build-logic:commons:compileKotlin FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':build-logic:commons:compileKotlin'.
> A failure occurred while executing org.jetbrains.kotlin.compilerRunner.GradleCompilerRunnerWithWorkers$GradleKotlinCompilerWorkAction
   > Could not isolate value org.jetbrains.kotlin.compilerRunner.GradleCompilerRunnerWithWorkers$GradleKotlinCompilerWorkParameters_Decorated@fa8c669 of type GradleCompilerRunnerWithWorkers.GradleKotlinCompilerWorkParameters
      > Java heap space

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.6-rc-2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 1m 4s
6 actionable tasks: 6 executed

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':build-logic:commons:compileKotlin'.
> A failure occurred while executing org.jetbrains.kotlin.compilerRunner.GradleCompilerRunnerWithWorkers$GradleKotlinCompilerWorkAction
   > Could not isolate value org.jetbrains.kotlin.compilerRunner.GradleCompilerRunnerWithWorkers$GradleKotlinCompilerWorkParameters_Decorated@fa8c669 of type GradleCompilerRunnerWithWorkers.GradleKotlinCompilerWorkParameters
      > Java heap space

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Java heap space

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.6-rc-2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 1m 4s

FAILURE: Build failed with an exception.

* What went wrong:
Could not receive a message from the daemon.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Context

In my own project, I can't do this either. I stopped all Gradle daemons, but it didn't help. I created a similar bug for 7.6, maybe it is related: #21708

Steps to Reproduce

Download gradle's sample project: https://docs.gradle.org/current/samples/zips/sample_structuring_software_projects-kotlin-dsl.zip
And run ./gradlew wrapper --gradle-version=7.6-rc-2 && ./gradlew -p server-application/ build

Your Environment

Gradle: 7.6-rc2
Kotlin: 1.6.21 / 1.7.10
OS: MacOS Monterey 12.5 (M1 Pro)

[jbartok]

Sorry that you're having trouble with Gradle!

Your issue lacks information about how to reproduce the problem you're having. A reproducer project can really help us track down and fix your problem quicker. We may also be able to suggest workarounds or ways to avoid the problem if we can reproduce it.

You can use the following as a base for your reproducer: https://github.com/gradle/gradle-issue-reproducer

This issue will be closed after 7 days, unless you can provide more information.

---

The current steps to reproduce don't reproduce it for us and it seems more like a memory setting problem, so pls. check those.

[tomasAlabes]

Indeed I tried it on another machine and it worked. Will re-open it if I can reproduce it again with the template.
Thanks!

[tomasAlabes]

Sorry, actually I tested it in the wrong branch. I'm still able to reproduce it in my project. It might be a problem with big multi-project modules (~80 subprojects). Do you have one to test it on? I can't share my project, and the reproducer isn't big enough. This is a blocker for us.

• I increased the java heap space from 4g to 6g, but didn't work, it might signal a problem on memory management.
• I use the -p myProject/ --parallel options
• Not reproducible with colon notation :myProject:build --parallel

I can share the pictures of the heap dump on error, to see if it helps:

[tomasAlabes]

Ok, I was able to reproduce it with 7.5.1. The regression label can be taken off. I'll leave it open in case the screenshots help to figure out the problem.

[ljacomet]

I renamed the issue to remove the link with 7.6 RCs

[xaviarias]

Thank you for providing a valid reproducer.

The issue is in the backlog of the relevant team but the existence of a workaround makes it non-critical so it might take a while before a fix is made.

---

Workaround is to increase the memory given to Gradle.

[tomasAlabes]

In my project I increased it from 4g to 8g, and it wasn't enough.

[eskatos]

Thank you for the provided information.
Unfortunately we're still not able to reproduce.
In order to make it actionable for us you would need to provide a reproducer build.

Another thing to try would be to use this Gradle version which contains an attempt at reducing memory usage:

gradle wrapper --gradle-version=7.6-branch-sg_76_memory_tweak-20221122202632+0000

[tomasAlabes]

Thanks for trying @eskatos. I tried again with that build and I reproduced it again. I'm attaching some reports:
java_pid3371_Leak_Suspects.zip
java_pid3371_Top_Components.zip
java_pid3371_Top_Consumers.zip

To see if I can help with these details:

• It's a big multi-modules project with around ~80 subprojects, and includedBuilds
• Mixes Java and Kotlin
• It's based on the gradle's sample project, so we generate plugins to reuse configurations
• It's using Version Catalogs

[eskatos]

Thanks @tomasAlabes. Still hard to reproduce with just the above instructions. Please provide a complete reproducer

[tomasAlabes]

I don't think I will be able to give you the reproducer, it's a pretty big project. If you don't have one in your suite, then you can close this or leave it open in case someone can. I will have to go back to the colon syntax to make it work. Although it's annoying because the commands become larger

./gradlew -p myProject/ clean build -x test -x check
# vs
./gradlew :myProject:clean :myProject:build -x :myProject:test -x :myProject:check

[ljacomet]

Slightly off topic, but you could have a much shorter command altogether.
Unless you have specific tasks bound to build, assemble does exactly what you ask when doing build without check and test.

And if it does not, consider adding a lifecycle task (a task without an action) that does what you need and nothing more.

Excluding tasks is not a recommended practice in Gradle.

[AlanMasciangelo]

I'm seeing similar symptoms with a large multiproject build after upgrading from 7.5.1 to 7.6.

Attaching a profiling shows a huge increase in instances of org.gradle.internal.component.external.model.ConfigurationBoundExternalDependencyMetadata which only accounts for <1% of heap at any time in previous versions

[mikethecalamity]

I am also seeing this in very large multiproject builds (~50 subprojects). In 7.4.2, things work ok. In 7.5.1, it gets a bit worse, had to bump up Gradle's memory. But then in 7.6, it really hits the fan, the build need over 12gb of memory to run, which seems insane.

[ParanoidUser]

Came here to say the same thing as @mikethecalamity pointed out. In my case, it hits the OOM through the Gradle plugin for IntelliJ. If any changes are made to the project the IDE asks to reload the project to rebuild the structure. With the extended JVM args in gradle.properties:

org.gradle.jvmargs=-Xmx2g -XX:MaxMetaspaceSize=512m -XX:+HeapDumpOnOutOfMemoryError

it used to work on Gradle 7.4.2 / 7.5.1, but crashes constanly on Gradle 7.6.0.

> Task :prepareKotlinBuildScriptModel UP-TO-DATE

FAILURE: Build failed with an exception.

* What went wrong:
Java heap space

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.

* Get more help at https://help.gradle.org

BUILD FAILED in 53s

Note: stretching out the heap size to 3GB helps to bring it back to work. But, it would be nice to have constant memory utilization across the builds.

[big-guy]

Could some of the reporters here try 7.6.1-branch-sg_761_reduce_memory_consumption-20230118002823+0000? This is a nightly build based off a branch. If this looks better, I'll get it merged for 7.6.1

[tomasAlabes]

Tried with my project and still getting to "Expiring Daemon because JVM heap space is exhausted" and dying.

…

------- Original Message -------

On Wednesday, January 18th, 2023 at 2:12 AM, Sterling Greene ***@***.***> wrote: Could some of the reporters here try 7.6.1-branch-sg_761_reduce_memory_consumption-20230118002823+0000? This is a nightly build based off a branch. If this looks better, I'll get it merged for 7.6.1 — Reply to this email directly, [view it on GitHub](#22688 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAFKYKA7NTYP6VXKTXD6JFLWS47RNANCNFSM6AAAAAAR5VDQGM). You are receiving this because you were mentioned.Message ID: ***@***.***>

[ParanoidUser]

@big-guy, thank you for looking at this! In my tests, the night build looks very promising. Here is a quick summary of the durations it took to rebuild the structure:

Build	-Xmx2g	-Xmx3g
7.6.1-reduce_memory	✔ 1m 59s	✔ 1m 28s
7.6.0	❌ OOM	✔ 1m 47s
7.5.1	✔ 1m 41s	✔ 1m 17s

[big-guy]

@AlanMasciangelo @mikethecalamity @tomasAlabes Could you give 7.6.1-branch-sg_761_reduce_memory_consumption_2-20230123043022+0000 a try? This has the previous changes plus a few more.

[ParanoidUser]

@big-guy, do you know if these memory optimizations made it into 8.0?

[ljacomet]

A number of improvements have been released in Gradle 8 and will be part of an upcoming Gradle 7.6.1 release.

Closing as resolved for now.

[mikethecalamity]

@big-guy @ljacomet I tried running with 8.1.1 recently, it's definitely significantly better but there are still some cases that fail compared to 7.5.1.

In the example below, we have 85 total projects and 36 of those are Quarkus services using -Xmx4g. The first was run as gradlew -p projects build (building everything) and the second is gradlew -p projects quarkusBuild (only building the services).

Build	build	quarkusBuild
8.1.1	❌ OOM	✔ 10:03
7.6.0	❌ OOM	❌ OOM
7.5.1	✔ 37:25	✔ 10:17

You'll note that even in 8.1.1 the full build still fails, even though it succeeds in 7.5.1 with 4GB of memory.

[ljacomet]

@mikethecalamity Could you try 8.2-milestone-1?

There are further improvements in there.

[mikethecalamity]

@ljacomet It worked with 8.2-milestone-1 and it was faster!

Do you know when 8.2 is planned to be released?