ParameterizedTests crash on Android 8 with Kotlin code

[paulo-raca]

I've run in an annoying crash when using @ParameterizedTests on Android 8.0/8.1 devices:

After some digging, turns out that inspecting parameter annotations (Which happens in ParameterizedTestMethodContext.isAggregator(Parameter) crashes consistently if the method has any annotation present.

However I'm using Kotlin to write my tests, and it automatically adds either @Nullable or @NotNull to the parameters

Is that a known problem? Any workarounds?

Here is the (Java) code reproducing the issue:

public class ParameterAnnotationTest {
    @Test
    public void test() {
    }

    // Works!
    @ParameterizedTest
    @ValueSource(strings = {"a", "b"})
    void testParameterized(String value) {
    }

    // Works
    @Test
    public void testGetParameterAnnotationEmpty() throws Exception {
        for (Parameter parameter : getClass().getDeclaredMethod("testParameterized", String.class).getParameters()) {
            parameter.getDeclaredAnnotation(NotNull.class);
        }
    }

    // Crashes
    @ParameterizedTest
    @ValueSource(strings = {"a", "b"})
    void testParameterizedWithAnnotation(@NotNull String value) {
    }


    // Crashes
    @Test
    public void testGetParameterAnnotationNotEmpty() throws Exception {
        for (Parameter parameter : getClass().getDeclaredMethod("testParameterizedWithAnnotation", String.class).getParameters()) {
            parameter.getDeclaredAnnotation(NotNull.class);
        }
    }
}

And this is the crash:

  2021-05-05 11:16:23.283 18739-18739/? A/DEBUG: Build fingerprint: 'google/walleye/walleye:8.0.0/OPD1.170816.025/4424668:user/release-keys'
    2021-05-05 11:16:23.283 18739-18739/? A/DEBUG: Revision: 'MP1'
    2021-05-05 11:16:23.283 18739-18739/? A/DEBUG: ABI: 'arm64'
    2021-05-05 11:16:23.283 18739-18739/? A/DEBUG: pid: 18710, tid: 18732, name: roidJUnitRunner  >>> android.junit5.parameterized_test <<<
    2021-05-05 11:16:23.283 18739-18739/? A/DEBUG: signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
    2021-05-05 11:16:23.283 18739-18739/? A/DEBUG: Cause: null pointer dereference
    2021-05-05 11:16:23.283 18739-18739/? A/DEBUG:     x0   0000007f5b0be9a8  x1   0000000000000000  x2   0000000000000001  x3   0000007f5b0bea74
    2021-05-05 11:16:23.284 18739-18739/? A/DEBUG:     x4   0000000000000000  x5   0000007f71c2f8cd  x6   0000000070417bcc  x7   0000000000000000
    2021-05-05 11:16:23.284 18739-18739/? A/DEBUG:     x8   81f3c05b56ee51bc  x9   0000000000000006  x10  0000000000080011  x11  000000000000000d
    2021-05-05 11:16:23.284 18739-18739/? A/DEBUG:     x12  0000007f5ba80f28  x13  000000000000000d  x14  0000007f722f67d4  x15  0000000000000000
    2021-05-05 11:16:23.284 18739-18739/? A/DEBUG:     x16  0000007f5b0beae8  x17  0000000000000000  x18  0000000000000008  x19  0000007f5b0bea74
    2021-05-05 11:16:23.284 18739-18739/? A/DEBUG:     x20  0000007f733587c0  x21  0000000000000001  x22  0000000000000000  x23  0000007f5b0be9a8
    2021-05-05 11:16:23.284 18739-18739/? A/DEBUG:     x24  0000007f724a56c0  x25  0000000000000000  x26  0000007f6768e4a0  x27  0000000000000003
    2021-05-05 11:16:23.284 18739-18739/? A/DEBUG:     x28  0000000000000000  x29  0000007f5b0be990  x30  0000007f71f72bc4
    2021-05-05 11:16:23.284 18739-18739/? A/DEBUG:     sp   0000007f5b0be890  pc   0000007f71f6fc98  pstate 0000000060000000
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG: backtrace:
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #00 pc 0000000000193c98  /system/lib64/libart.so (art::(anonymous namespace)::GetAnnotationItemFromAnnotationSet(art::(anonymous namespace)::ClassData const&, art::DexFile::AnnotationSetItem const*, unsigned int, art::Handle<art::mirror::Class>, bool)+68)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #01 pc 0000000000196bc0  /system/lib64/libart.so (art::annotations::GetAnnotationForMethodParameter(art::ArtMethod*, unsigned int, art::Handle<art::mirror::Class>)+480)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #02 pc 00000000003c3f54  /system/lib64/libart.so (art::Parameter_getAnnotationNative(_JNIEnv*, _jclass*, _jobject*, int, _jclass*)+228)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #03 pc 0000000000267140  /system/framework/arm64/boot.oat (offset 0x1db000) (java.lang.Class.classForName [DEDUPED]+208)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #04 pc 000000000051a838  /system/lib64/libart.so (art_quick_invoke_static_stub+600)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #05 pc 00000000000d8ba4  /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+264)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #06 pc 0000000000289a98  /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+340)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #07 pc 0000000000284180  /system/lib64/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+708)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #08 pc 000000000050358c  /system/lib64/libart.so (MterpInvokeStatic+476)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #09 pc 000000000050c394  /system/lib64/libart.so (ExecuteMterpImpl+14612)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #10 pc 000000000026568c  /system/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+444)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #11 pc 000000000026b214  /system/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame*, art::JValue*)+216)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #12 pc 0000000000284160  /system/lib64/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+676)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #13 pc 0000000000501ec4  /system/lib64/libart.so (MterpInvokeVirtual+800)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #14 pc 000000000050c214  /system/lib64/libart.so (ExecuteMterpImpl+14228)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #15 pc 000000000026568c  /system/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+444)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #16 pc 00000000004f59ec  /system/lib64/libart.so (artQuickToInterpreterBridge+1052)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #17 pc 0000000000523a1c  /system/lib64/libart.so (art_quick_to_interpreter_bridge+92)
    2021-05-05 11:16:23.344 18739-18739/? A/DEBUG:     #18 pc 000000000000974c  /dev/ashmem/dalvik-jit-code-cache (deleted)

[paulo-raca]

I have also added this issue on Android's tracker: https://issuetracker.google.com/issues/187242751

[mannodermaus]

Thanks for reporting this and sorry for the late response. I've been playing around with this and was able to reproduce it quite easily on an Android 8 emulator. I'm not sure that there's a way to work around this at the moment, as it doesn't seem to be a problem specific to JUnit 5. Translating your block from above to JUnit 4 and Kotlin, I'm able to get the same results and a stack trace very close to the one posted above:

import org.jetbrains.annotations.NotNull
import org.junit.Test

class ParameterAnnotationTest {

  @Test
  fun test() {
    javaClass.getDeclaredMethod("testParameterizedWithAnnotation", String::class.java).parameters.forEach {
      println("param: $it")
      val a = it.getDeclaredAnnotation(NotNull::class.java)
      println("annotation $a")
    }
  }

  fun testParameterizedWithAnnotation(value: String) {
  }
}

Some other observations I made:

• This happens with any introspected parameter, not just the parameterized methods in use by JUnit 5
• While java.lang.ref.Parameter was introduced in API 26, parameter introspection seems to work in the actual app (i.e. copy-pasting the above into an Activity will work)

The only possible way I could see this being solvable from this end is to restrict or bypass the ParameterizedTest code completely:

• Disable these affected tests when running on Android 8.0 and 8.1
• Alternatively, add a Lint check to flag these tests
• Rewrite the detection of parameterized tests to use a custom version of ParameterizedTestMethodContext that doesn't use parameter introspection. Not really an option, especially with the history of this plugin following alongside the mainline JUnit 5 codebase for years now

Happy to hear your thoughts or new insight since raising the issue. 🙏

[mannodermaus]

@paulo-raca Hello again. I'm revisiting this ticket and having a hard time reproducing this with the latest versions of the plugin+instrumentation library enivronment. Just in case you're still invested in JUnit 5 for instrumentation tests, I wanted to reach out and ask if this is something you still encounter on Android 8.x? From my end, it kinda looks like some update along the line has fixed the issue, either through a revision of ART or by means of a change inside JUnit 5 or by accident in something I changed on the instrumentation lib side. Thanks!