Allow filtering/choosing which variants are aggregated in a root report

[gmazzo]

What is your use-case and why do you need this feature?
Since 0.8.0-Beta2, Kover successfully aggregates Android modules.

Given the following setup at the root project:

dependencies {
    kover(":app")
    kover(":lib-android")
    kover(":lib-jvm")
}

Will produce correctly an aggregated report, taking main variant on JVM modules, and all variants (usually debug and release) on Android ones.

However, this is a problem (at least for us) in large-scale projects, with either complex variants (by using productFlavors for instance) or a large number of modules, as it will cause tall tests to run (testDebug and testRelease for instance), increasing the overall CI time exponentially.

We'd like to have granular control on which variants are aggregated.

https://scans.gradle.com/s/2nb3nedv4ntsm/timeline?details=txc6dqjkewdsm&expanded=WyIwLjEiXQ&show=predecessors

Describe the solution you'd like
Having some kind of DSL to choose which variants are aggregated.

With the current DSL approach (which I'll probably suggest improvements in another ticket) it would be something like this:

// app's build.gradle.kts
kover.currentProject {
    providedVariant("release") {
        aggregate = false
    }
}

[shanshin]

Sorry for delayed answer.

This is a shortcoming of the current documentation: it is not recommended to use the total reports for Android projects.
It is for this purpose that custom options are designed.

The expected usage algorithm is as follows:

• a custom variant with the same name is created in all measured projects (eg main)
• in each project, only those Android build variants that should be included in the merged report are added

kover {
   currentProject {
       createVariant("main") {
           add("debug") // or add("jvm") in JVM-only subproject
       }
   }
}

• choose the merging project, and add kover dependencies to all measured projects (in case of root project)

dependencies {
    kover(":app")
    kover(":lib-android")
    kover(":lib-jvm")
}

• use task :koverHtmlReportMain to generate HTML report

Custom variants allow to create different report slices by adding a variety of project sets and Android build variants to them - this way you can create different reports for different needs without changing of buildscripts

[shanshin]

kover.currentProject {
    providedVariant("release") {
        aggregate = false
    }
}

this approach is less flexible, it does not involve the simultaneous creation of alternative combinations of Android build variants for reports, however, as well as custom variant, it requires changes in each Android subproject.

[Faltenreich]

TL;DR: How to introduce custom variants to a total report?

it is not recommended to use the total reports for Android projects.

Unfortunately total reports are currently (Kover 0.8.0) the only way for us (native Android app with Android- and Kotlin Library submodules) to collect code coverage for components whose tests are located in a different module.

When not using total reports, code coverage stays zero for those components. Referencing modules explicitly like in your example works, but we need a global solution that works for all modules.

So we decided to go with total reports which leads to the issue @gmazzo mentioned: Tests are executed for both debug and release, which doubles execution time.

We want to skip tests for release and tried to do so by introducing a new Kover variant as follows:

// root build.gradle.kts
subprojects {
    apply(plugin = "org.jetbrains.kotlinx.kover")

    project.afterEvaluate {
        val isAndroid = project.plugins.hasPlugin("com.android.application") || project.plugins.hasPlugin("com.android.library")

        kover {
            currentProject {
                createVariant("main") {
                    if (isAndroid) {
                        add("debug")
                    } else {
                        add("jvm")
                    }
                }
            }
        }
    }
}

Unfortunately this does not create a total report in the root's build directory when running :koverHtmlReportMain.

Is there is a way to use custom variants with a total report?
Alternatively is there a way to decouple the Kover tasks from tests, so that we are able to execute tests on our own beforehand (like it was possible with pre-0.8.0)?

[shanshin]

TL;DR: How to introduce custom variants to a total report?

Sorry, I don't quite understand the question.
For each variant, its own reports are created, which you should to use.

Unfortunately this does not create a total report in the root's build directory when running :koverHtmlReportMain.

HTML report for main variant is located in the directory build/reports/kover/htmlMain.

When not using total reports, code coverage stays zero for those components.

Could you clarify about this. If you do not specify dependencies { kover(":some:subproject") }, then its coverage will not be shown in the general report either, what do you mean by those components?

If you allow the use of subprojects, then in 0.8.0 you can use a special short-cut for merging:

// root build.gradle.kts
kover {
    merge {
        // merge with all subprojects
        subprojects()

        // create reports variant 'main'
        createVariant("main") {
            // used plugins
            val isAndroid = project.plugins.hasPlugin("com.android.application") || project.plugins.hasPlugin("com.android.library")

            if (isAndroid) {
                add("debug")
            } else {
                add("jvm")
            }
        }
    }
}

then, if you call the :koverHtmlReportMain command, the report will be generated in the directory build/reports/kover/htmlMain

[Faltenreich]

Thank you for your response, it is much appreciated!
I will try to answer your questions one-by-one:

Sorry, I don't quite understand the question. For each variant, its own reports are created, which you should to use. [...] HTML report for main variant is located in the directory build/reports/kover/htmlMain.

In my example, when configuring a variant "main" for every subproject and then executing koverHtmlReportMain, no total report is being generated at root's build folder. Reports at every subprojects on the other hand will be generated but the directory you mention ("build/reports/kover/htmlMain") does not exist.

Could you clarify about this. If you do not specify dependencies { kover(":some:subproject") }, then its coverage will not be shown in the general report either, what do you mean by those components?

By "those components" I mean classes that are being tested in another module. Without using Kover's total reports, I was not able to generate code coverage for those classes.

If you allow the use of subprojects, then in 0.8.0 you can use a special short-cut for merging: [...] then, if you call the :koverHtmlReportMain command, the report will be generated in the directory build/reports/kover/htmlMain

This leads to the following error on Gradle Sync:

Could not find the provided variant 'jvm' to create a custom variant 'main'.
Specify an existing 'jvm' variant or Android build variant name, or delete the merge.

This is the same error I receive when trying to create a Kover variant for the root project which, in our case, is not a JVM library module.

[Faltenreich]

We managed to fix the redundant execution of testReleaseUnitTest by using Kover's exclusion of test tasks:

kover {
    currentProject {
        instrumentation {
            disabledForTestTasks.add("testReleaseUnitTest")
        }
    }
}

This skips all tests for the release build variant and in our case cuts execution time in half.

@gmazzo Maybe this will solve your problem as well?

[shanshin]

In my example, when configuring a variant "main" for every subproject and then executing koverHtmlReportMain, no total report is being generated at root's build folder. Reports at every subprojects on the other hand will be generated but the directory you mention ("build/reports/kover/htmlMain") does not exist.

Let's clarify a little bit

• total report - are reports that are always generated for absolutely all classes and tests in the application, and they are generated only by tasks named koverXmlReport, koverHtmlReport, koverVerify etc. If we are talking about an variant named main, then this is a custom reports (with tasks koverXmlReportMain, koverHtmlReportMain, koverVerifyMain etc). Perhaps here by total report you meant merged report.
• in order for the merged report to appear in the build directory of the root project, the main variant must also be created in the root project. https://kotlin.github.io/kotlinx-kover/gradle-plugin/#generating-reports-5.
  To always call only report in the root project, please use the full path in the name of the task being started, as indicated in the examples :koverHtmlReport, :koverXmlReport etc

By "those components" I mean classes that are being tested in another module. Without using Kover's total reports, I was not able to generate code coverage for those classes.

please, add kover dependencies to the config of root project

// root build.gradle.kts

dependencies { 
    kover(project(":some:subproject")) 
    // ... all necessary subprojects 
}

This leads to the following error on Gradle Sync:

You can add an analysis to understand whether there is a JVM in the subproject or not, or use the optional parameter like this:

// root build.gradle.kts
kover {
    merge {
        // merge with all subprojects
        subprojects()

        // create reports variant 'main'
        createVariant("main") {
            // used plugins
            val isAndroid = project.plugins.hasPlugin("com.android.application") || project.plugins.hasPlugin("com.android.library")

            if (isAndroid) {
                add("debug")
            } else {
                // add classes and tests from JVM target if it exists
                add("jvm", optional = true)
            }
        }
    }
}

[Faltenreich]

@shanshin I agree that I might have confused myself in the process and mixed up total- with merged reports. Thanks for clearing it up!

Our current solution - using total reports, if I got you right - looks (dumbed down) like this:

// root/build.gradle.kts

plugins {
    alias(libs.plugins.kover)
}

subprojects {
    apply(plugin = "org.jetbrains.kotlinx.kover")

    kover {
        currentProject {
            instrumentation {
                // (1) Disabling tests for release build variant to reduce execution time
                disabledForTestTasks.add("testReleaseUnitTest")
            }
        }
   }
}

// (2) Cross-referencing modules to support components that are being tested in other modules
dependencies {
    allprojects {
        kover(this)
    }
}

This way tests only run for the debug build variant (1) and reports/kover/report.xml or reports/kover/html/index.html at the root build directory contain code coverage for every (sub-)module (2).

[gmazzo]

As for my current understanding, the whole concept of dealing with tasks is an antipattern on Gradle.
Projects
Tasks are meant to be artifact's providers and external projects should not be accessing (direct or by name) directly, but trough outgoing variants and the dependency resolution.

kover is going in that direction, but considering manually a custom variant for Android projects is a bit complex for our needs, and it's even more complex when you introduce flavors.

I was expecting more a convention over configuration approach. So, we park the migration for now.

I was looking into it for a possible replacement of JaCoCo but we decided to wait a bit while it's gets more mature.

[shanshin]

@Faltenreich, using disabledForTestTasks.add("testReleaseUnitTest") is an inappropriate use, because in this case, classes from the source set for the release variant still get into the report

[shanshin]

@gmazzo,
I'll close this issue for Kover (project) Gradle Plugin.

We will try to take these wishes into account when designing #608.

[Faltenreich]

I agree that dealing with Gradle tasks is a sub-par solution, but the best we have so far and in our case fitting to our requirements (we do not have relevant code at the specific source set for release).

[shanshin]

@Faltenreich
specifying

// root build.gradle.kts
kover {
    merge {
        // merge with all subprojects
        subprojects()

        // create reports variant 'main'
        createVariant("main") {
            add("debug", optional = true)
            add("jvm", optional = true)
        }
    }
}

in one place only in the root project requires much fewer configuration changes, and is also more reliable by excluding all unspecified sources and tests