Custom test reporting

[aSemy]

Expected Behavior

I would like to be able to report test results directly to Gradle from within a custom Gradle task.

I want the reported tests to be visible in the Gradle test report, correctly reported when the tests are launched in IntelliJ, and visible in CI tools (like GitHub actions).

Current Behavior (optional)

There is no official way to report test results directly to Gradle. There is no workaround.

TestResultProcessor is internal.

Because tests can't be reported to Gradle, this means that other tools that rely on Gradle's reporting can't pick up the custom tests.

• IntelliJ can't display the test results, which is normally very useful and allows for utils like 'click to show difference'.
• Custom tests can't be reported in CI tools, like GitHub.
• Custom tests are missing from reports, metrics, stats.

Context

Here are some examples of when I want to report tests to Gradle:

1. I want to Kotest to support nested JavaScript tests. Currently this is not possible, because of limitations in the JavaScript testing frameworks (they do not support dynamically created nested tests). Because Kotest cannot directly report test results to Gradle, it looks like support might not even be possible.

2. I want to write a quick verification task. I want this to be reported to Gradle as a test.

   val checkReadme by tasks.registering {
       group = "verification"
   
       val readme = file("readme.md")
   
       val projectVersion = project.version.toString()
   
       doLast {
           require("Current project version is $projectVersion" in readme.readText()) {
               "Incorrect project version in readme"
           }
       }
   }

3. I want BCV validation results to be reported as tests.

4. Because Gradle doesn't fully support JUnit features (see footnote 1), I want to run the tests manually using the JUnit console launcher, and report the results to Gradle. Example.

5. I want to use a non-JVM testing framework, like Jasmine, Web Test Runner, Cucumber.

Custom JUnit TestEngine

In theory, one could write a custom JUnit TestEngine to launch custom tests, but this is a prohibitive amount of work, especially for the simpler cases where I just want to report a single test case.

A custom TestEngine would also introduce a lot of extra complexity and layers, and potentially a new language for the non-JVM based tests. Even if using a custom TestEngine, Gradle doesn't fully support JUnit Platform (footnote 1), so this would be a non-starter.

Even if using a custom JUnit TestEngine, Gradle unnecessarily requires that the input property testClassesDirs of a Test task has .class files, even if not testing JVM tests!

Related

• JUnit 5: Allow resource-based testing #4773
• Kotlin MPP plugin broken with Gradle 7.6 due to signature change in TestResultProcessor #22973
• Extract TestExecuter and TestDescriptor to new testing API subproject #19549

---

Footnote 1: Here are the issues related to Gradle not fully supporting JUnit

• The JUnit Platform does not receive first-class support junit-team/junit5#2849
• Support for CompositeTestSource in junit platform test runner #20941
• JUnit 5 integration relies on Jupiter specifics #4912
• JUnit 5: Allow resource-based testing #4773
• Support other class-based test engines in junitPlatform #4252
• Reporting of failures of nested "suites" drops class name #18682
• The Gradle test runner's XML report lumps together unrelated @Nested tests under the same test/@classname #15075
• Reporting of failures of nested "suites" drops class name #18682
• When a test exclusion filter is set, JUnit 5 nested tests are ignored when running an entire test class #19808

[ov7a]

This feature request is in the backlog of the relevant team and is prioritized by them.