Renaming test file causing test class to be deleted

[mathjeff]

Current Behavior

I'm finding that if I write a test with the right class name and wrong filename, and then build+run the tests, and then rename the test to have the write filename, that the test stops running because its class no longer gets created

Expected Behavior

It would be nice if the test would build and run even if it had previously existed with a different filename

Context (optional)

I was trying to write tests for Gradle itself and found that my test wasn't running, and I suspect that this might be why.

Steps to Reproduce

To reproduce, run this script inside a checkout of Gradle itself:

#!/bin/bash
set -e

separator="************************************************************************"
testname="$1"
if [ "$testname" == "" ]; then
  testname="SampleTest"
fi

function echoAndDo() {
  echo "$*"
  eval "$*"
}

function writeTest() {
  testname="$1"
  filepath="$2"

  echo "$separator"
  echo "Writing failing test at $filepath"

  echo " 
  package org.gradle.internal.execution.history.changes
  
  import spock.lang.Specification
  
  class $testname extends Specification {
      def simpleCheck() {
          when:
          println('running simple check')
          then:
          assert false
      }
  }
  " > "$filepath"
}

function runTest() {
  echo "$separator"
  echoAndDo ./gradlew -Dorg.gradle.ignoreBuildJavaVersionCheck=true :execution:quickTest
}

testdir="platforms/core-execution/execution/src/test/groovy/org/gradle/internal/execution/history/changes"

testfile1="${testdir}/${testname}-old.groovy"
testfile2="${testdir}/${testname}.groovy"

writeTest "$testname" "$testfile1"
if runTest; then
  echo "Failing test did not cause a build failure. That's weird"
  exit 1
fi
echo "$separator"
echo "Searching for the test class:"
echoAndDo find platforms/core-execution/execution/build -name "$testname*"
echo "$separator"
echo "Deleting old test"
echoAndDo rm "$testfile1"

writeTest "$testname" "$testfile2"
if runTest; then
  echo "$separator"
  echo "Notice that renaming the test file caused the build failure to go away!"
  echo "$separator"
  echo "Searching for the test class:"
  echoAndDo find platforms/core-execution/execution/build -name "$testname*"
  echo "Is the test missing?"
  exit 1
else
  echo "$separator"
  echo "Failed to reproduce the problem. Nice!"
fi

When I run this script inside a checkout of Gradle at tag v8.7.0, I get this output:

$ /usr/local/google/workspace/util/gradletest.sh
************************************************************************
Writing failing test at platforms/core-execution/execution/src/test/groovy/org/gradle/internal/execution/history/changes/SampleTest-old.groovy
************************************************************************
./gradlew -Dorg.gradle.ignoreBuildJavaVersionCheck=true :execution:quickTest
Reusing configuration cache.

> Task :execution:compileTestGroovy
Groovy compilation avoidance is an incubating feature.

> Task :execution:test

SampleTest > simpleCheck FAILED
    org.spockframework.runtime.ConditionNotSatisfiedError at SampleTest-old.groovy:11

273 tests completed, 1 failed

> Task :execution:test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':execution:test'.
> There were failing tests. See the report at: file:///usr/local/google/workspace/gradle/platforms/core-execution/execution/build/reports/tests/test/index.html

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.7-rc-4/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD FAILED in 6s
420 actionable tasks: 7 executed, 413 up-to-date
Configuration cache entry reused.
************************************************************************
Searching for the test class:
find platforms/core-execution/execution/build -name SampleTest*
platforms/core-execution/execution/build/classes/groovy/test/org/gradle/internal/execution/history/changes/SampleTest.class
************************************************************************
Deleting old test
rm platforms/core-execution/execution/src/test/groovy/org/gradle/internal/execution/history/changes/SampleTest-old.groovy
************************************************************************
Writing failing test at platforms/core-execution/execution/src/test/groovy/org/gradle/internal/execution/history/changes/SampleTest.groovy
************************************************************************
./gradlew -Dorg.gradle.ignoreBuildJavaVersionCheck=true :execution:quickTest
Reusing configuration cache.

BUILD SUCCESSFUL in 897ms
420 actionable tasks: 5 executed, 2 from cache, 413 up-to-date
Configuration cache entry reused.
************************************************************************
Notice that renaming the test file caused the build failure to go away!
************************************************************************
Searching for the test class:
find platforms/core-execution/execution/build -name SampleTest*
Is the test missing?

Gradle version

v8.7.0

Build scan URL (optional)

No response

Your Environment (optional)

No response

[mathjeff]

My guess about the cause is that perhaps Gradle is caching the filepath of the .class file that is generated by each .groovy file but in this case applying the cached results in the wrong order: first compiling the new test source file (overwriting the existing test class), then seeing that the old test source file doesn't exist, and deleting the old test class file (whose contents are now the result of compiling the new test source file)

[asodja]

Yeah, this is incremental compilation bug. And I am pretty sure it can be reproduced also for Java.
It happens somewhere in AbstractRecompilationSpecProvider.java where we map from sources to classes and then back again from classes to sources. And we accidentally try to recompile the old file.

I would guess this bug is here from some 6.x version or at least 7.x version.

A workaround is to force non-incremental compilation, either via --rerun or --rerun-tasks.