Skip to content

Latest commit

 

History

History
68 lines (49 loc) · 4.01 KB

expected-to-fail-tests.adoc

File metadata and controls

68 lines (49 loc) · 4.01 KB
Raw

Often tests fail due to a bug in the tested application or in dependencies. Traditionally such a test method would be annotated with JUnit’s @Disabled. However, this has disadvantages when the bug that causes the test failure is fixed:

  • the developer might not notice the existing test method and create a new one

  • the existing test method might not be noticed and remains disabled for a long time after the bug has been fixed, adding no value for the project

@ExpectedToFail solves these issues. Unlike @Disabled it still executes the annotated test method but aborts the test if a test failure or error occurs. However, if the test is executed successfully it will cause a test failure because the test is working. This lets the developer know that they have fixed the bug (possibly by accident) and that they can now remove the @ExpectedToFail annotation from the test method.

The annotation can only be used on methods and as meta-annotation on other annotation types. Similar to @Disabled, it has to be used in addition to a "testable" annotation, such as @Test. Otherwise the annotation has no effect.

Important

This annotation is not intended as a way to mark test methods which intentionally cause exceptions. Such test methods should use JUnit’s assertThrows or similar means to explicitly test for a specific exception class being thrown by a specific action.

Basic Use

The test is aborted because the tested method brokenMethod() returns an incorrect result.

link:../src/demo/java/org/junitpioneer/jupiter/ExpectedToFailExtensionDemo.java[role=include]

An aborted test is no failure and so the test suite passes (if all other tests pass, of course). Should brokenMethod() start returning the correct value, the test invocation passes, but @ExpectedToFail marks the test as failed to draw attention to that change in behavior.

A custom message can be provided, explaining why the tested code is not working as intended at the moment.

link:../src/demo/java/org/junitpioneer/jupiter/ExpectedToFailExtensionDemo.java[role=include]

Only Abort on Expected Exceptions

A test that is @ExpectedToFail will change its behavior by starting to actually fail, once the code under test behaves correctly. If the underlying failure changes, though, for example from an assertion error to an exception or from an UnsupportedOperationException of a formerly missing implementation to a runtime exception of buggy implementation, the @ExpectedToFail-test will keep passing.

To better react to such changes, @ExpectedToFail has an attribute withExceptions that can be used to enumerate the exceptions which when thrown will result in an aborted (and thus passing) test. Any other exception thrown by the code under test will result in a failing test.

In the following example a test case for productionFeature() has been implemented. While the test is fully implemented, the production code has been stubbed and throws an UnsupportedOperationException to indicate this.

link:../src/demo/java/org/junitpioneer/jupiter/ExpectedToFailExtensionDemo.java[role=include]

Once productionFeature() is implemented, @ExpectedToFail will fail the test, as no UnsupportedOperationException is thrown anymore. By using withExceptions you can thus prevent "masking" a faulty implementation (e.g. when a value other rather than 10 is returned) with an aborted test (which would be the result when no withExceptions is set).

Thread-Safety

This extension is safe to use during parallel test execution.