numShards, shardIndex support

[jerry-jeon]

The sharding feature that's available in AndroidJunitRunner is not working in AndroidJunit5Runner.
Is there any way to achieve it? I tried to do it through the ExecutionCondition extension, but it makes the test "ignored". I want to make the runner doesn't recognize tests that aren't included in the specific sharding.

I briefly changed the code by refering the AndroidJunitRunner code and it's working well. But I am not sure it's the right way to do it.

internal fun createRunnerParams(testClass: Class<*>): AndroidJUnit5RunnerParams {
    val instrumentation = InstrumentationRegistry.getInstrumentation()
    val arguments = InstrumentationRegistry.getArguments()

    // Parse environment variables & pass them to the JVM
    val environmentVariables = arguments.getString(ARG_ENVIRONMENT_VARIABLES)
        ?.run { PropertiesParser.fromString(this) }
        ?: emptyMap()

    // Parse system properties & pass them to the JVM
    val systemProperties = arguments.getString(ARG_SYSTEM_PROPERTIES)
        ?.run { PropertiesParser.fromString(this) }
        ?: emptyMap()

    // Parse configuration parameters
    val configurationParameters = arguments.getString(ARG_CONFIGURATION_PARAMETERS)
        ?.run { PropertiesParser.fromString(this) }
        ?: emptyMap()

    // Parse the selectors to use from what's handed to the runner.
    val selectors = ParsedSelectors.fromBundle(testClass, arguments)

    // The user may apply test filters to their instrumentation tests through the Gradle plugin's DSL,
    // which aren't subject to the filtering imposed through adb.
    // A special resource file may be looked up at runtime, containing
    // the filters to apply by the AndroidJUnit5 runner.
    val filters = mutableListOf<Filter<*>>()
    val contextFilters = GeneratedFilters.fromContext(instrumentation.context)
    filters.addAll(contextFilters)

    class ShardingFilter(val numShards: Int, val shardIndex: Int) : PostDiscoveryFilter {
        override fun apply(descriptor: TestDescriptor): FilterResult {
            if (descriptor.isTest) {
                val remainder = abs(descriptor.hashCode()) % numShards
                val runningShard = remainder == shardIndex
                return if (runningShard) {
                    FilterResult.included(null)
                } else {
                    FilterResult.excluded("excluded")
                }
            }

            for (each in descriptor.children) {
                return if (apply(each).included()) {
                    FilterResult.included(null)
                } else {
                    FilterResult.excluded("excluded")
                }
            }

            return FilterResult.excluded("excluded")
        }
    }

    val numShards = arguments.getString("numShards")?.toInt() ?: -1
    val shardIndex = arguments.getString("shardIndex")?.toInt() ?: -1
    if (numShards > 0 && shardIndex >= 0 && shardIndex < numShards) {
        filters.add(ShardingFilter(numShards, shardIndex))
    }

    return AndroidJUnit5RunnerParams(
        selectors,
        filters,
        environmentVariables,
        systemProperties,
        configurationParameters
    )
}

[mannodermaus]

I deployed a new 1.3.1-SNAPSHOT version of the instrumentation libraries where support for numShards and shardIndex are added. Would you try it out and let me know if this works for you?

dependencies {
  androidTestImplementation("de.mannodermaus.junit5:android-test-core:1.3.1-SNAPSHOT")
  androidTestRuntimeOnly("de.mannodermaus.junit5:android-test-runner:1.3.1-SNAPSHOT")
}

[jerry-jeon]

Hello @mannodermaus . Thanks for working on it.
It’s working well, but there’s a problem related to the test result.

I have tests both of junit4 and junit5. For the shard that only contains junit4 tests, the test result considered as failed even if all tests are passed. (As you can see the screen shot)
When it happens the suspicious log appears - I/TestRunner: finished: JUnit Jupiter If the shard contains at least one junit5 test, it doesn’t appear.

Here's the part of XML file that exported from the result. Each tests are passed but in the testSuite tag, the result is failed. I guess it’s why the result shown as failed in the android studio.

I created the sample project that can reproduce the issue. In the project problematic numShards and shardIndex value is assigned already, so if you run the whole tests then you will see the issue.

Because of it, my project always fails on the Firebase Testlab. I would really appreciate it if you could help me.
If you need more information, please let me know.
Thank you.

[mannodermaus]

Hey, apologies for only coming back to this now. Thank you for the sample project! I executed the instrumentation tests in it and it ran the same tests I can see in your screenshots, but my XML file has a different format with less information. I wonder if there have been changes to AS itself that resolved an internal issue here? (For context, I'm running from AS Iguana Canary 3)