ExpectedToFail doesn't work with parametrized tests

[NewbieProger]

Hi. I even don't know what to show.
When I put annotation on single test - is ok, but when I use on Parametrized test it doesn't work :(

@DisplayName("")
@Description("")
@ParameterizedTest
@MethodSource("com#getAccountByAccountWithErrorPayload")
@ExpectedToFail
void gettingAccountByAccountIdError(String messagePayload, String expectedError) {
    ErrorModelRabbitDto errorModelRabbitDto = rabbitmqSteps.getAccountByAccountIdWithError(messagePayload);

    rabbitmqSteps.checkErrorRabbitMQMessage(expectedError, errorModelRabbitDto);
}

[beatngu13]

I wrote this little example:

class ExpectedToFailVsDisabledTest {

	@Nested
	class WithExpectedToFail {

		@ExpectedToFail
		@Test
		void testSingleFail() {
			fail();
		}

		@ExpectedToFail
		@Test
		void testSingleSuccess() {
			assertTrue(true);
		}

		@ExpectedToFail
		@ParameterizedTest
		@ValueSource(booleans = {true, false})
		void testParameterizedFail(boolean b) {
			fail();
		}

		@ExpectedToFail
		@ParameterizedTest
		@ValueSource(booleans = {true, false})
		void testParameterizedSuccess(boolean b) {
			assertTrue(true);
		}

	}

	@Nested
	class WithDisabled {

		@Disabled
		@Test
		void testSingleFail() {
			fail();
		}

		@Disabled
		@Test
		void testSingleSuccess() {
			assertTrue(true);
		}

		@Disabled
		@ParameterizedTest
		@ValueSource(booleans = {true, false})
		void testParameterizedFail(boolean b) {
			fail();
		}

		@Disabled
		@ParameterizedTest
		@ValueSource(booleans = {true, false})
		void testParameterizedSuccess(boolean b) {
			assertTrue(true);
		}

	}

}

It indeed feels strange @ExpectedToFail (and similar extensions like @DisableUntil or @DisableIfTestFails) doesn't support parameterized tests. Especially, given it explicitly mentions @Disabled:

The big difference compared to JUnit's @Disabled annotation is that […]

Thanks for bringing this to our attention. At the moment, I can't tell if the behaviour is intended. We will discuss this.

[Michael1993]

This is an interesting question because I don't think it's straightforward what the annotation is supposed to do if you put it on a @ParametrizedTest - is it supposed to fail if any pass or fail only if all pass? Should there be a 'threshold', like with @RepeatedTest?

[NewbieProger]

I think it should:

• If at least one of them doesn't fail - do nothing with passed tests, with failed - just disable
• If all passed - alert 'delete annotation'
• if all broken - just disable tests

[Michael1993]

I don't agree. I think when you put @ExpectedToFail on a test it communicates that the test as a whole is expected to fail - meaning all possible cases should fail. Your interpretation would "cover up" cases where builds should fail because some of the tests start passing (even though they were expected to fail).

I also think it's just a tad too convenient to have @ExpectedToFail partially disable your test.