Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed)

[NikhilRamavath]

Github action workflow with default runners while building android application getting failed most of the time with below error

• What went wrong:
  Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed)

Runner used:
ubuntu-20.04

jvm args: -Xmx5120m -XX:MaxPermSize=4096m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8

[ChumachenkoRoman]

Same problem.
ubuntu-20.04
Xmx4G -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
Failed on:
./gradlew bundleRelease
Also we use concurrency to cancel running builds and build the freshest commit. In our case problem is triggered when multiple such builds are running in parallel. Imagine this:
release/29: commit 1 is running
release/29: commit 2 is pending
release/29: commit 1 is cancelled
release/29: commit 2 is running
release/29: commit 2 failed (because of specified reason)

Sometimes builds work just fine in this case. And sometimes they would fail

[VitaSokolova]

We also experience daemon disappearance too often recently when two or more workflows are running in parallel. This is our workflow:

name: Run tests and validate

on:
  pull_request:
    branches:
      ...

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  run-tests-and-validate:
    runs-on: ubuntu-latest

    env:
      GRADLE_OPTS: -Dorg.gradle.jvmargs="-Xmx4g -Xms1g -XX:MaxPermSize=2048m -XX:MaxMetaspaceSize=1024m -XX:+HeapDumpOnOutOfMemoryError" -Dfile.encoding="UTF-8" -Dorg.gradle.daemon=false -Dorg.gradle.parallel=true

    steps:
      - name: Checkout the project
        uses: actions/checkout@v3

      - name: Set up JDK 11
        uses: actions/setup-java@v3
        with:
          java-version: '11'
          distribution: 'temurin'

      - name: Setup Gradle
        uses: gradle/gradle-build-action@v2

      - name: Run tests
        run: ./gradlew test

I suspected gradle-build-action in cancelling daemons of neighbour runs, in README they mention:

"By default, the action will stop all running Gradle daemons in the post-action step"


And I found an issue in their bug tracker. But we use Github-hosted runners.

When the job begins, GitHub automatically provisions a new VM for that job.

They must be independent and isolated. And it shouldn't be a memory issue, according to profiling resilts, we have more than enough resources. So, I'm out of ideas.

[fadhyyusuf]

Same issue here.
Is there a way to fix this?

[ChumachenkoRoman]

Hey, it's me again. Found a solution which works in our case.

I decided to profile our build locally using Visual VM. I found out that each build spins up 2 daemons: 1 for gradle and 1 for kotlin compiler. Try limiting the memory for kotlin compiler. We are using this memory flags for gradle in gradle.properties:

# Dkotlin.daemon.jvm.options specifies memory limits for kotlin compiler
org.gradle.jvmargs=-Xmx4G -Dkotlin.daemon.jvm.options=-Xmx2G -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParallelGC -Dfile.encoding=UTF-8

You must find a balance between gradle daemon heap and kotlin compiler daemon heap. gradle + kotlin < 7GB or else Linux will kill gradle daemon. It is important to leave some RAM for Linux or else it'll kill gradle daemon.

If you give not enough heap to gradle/kotlin daemons build will fail with OurOfMemoryError. Different technologies and projects may require different heap sizes for gradle and compiler. Unit tests/UI tests require additional kotlin daemon. Our project is relatively big and memory-hungry so we are only able to build artifacts with configs I showed above.

So in some cases default build agents may be insufficient for your project. In that case consider paying for stronger agents
or setup local build machine in your office to cover your needs.