I am still looking for a solution to retry tests which fit all my requirements

[pandoras-toolbox]

I did not find any good retry solution for JUnit 5 tests, no offense intended.

Maybe I have high expectations. When automating E2E tests in a professional way then you need certain features for retries and not only the bare minimum. For retrying certain steps or group of steps there is a great Java library called Failsafe. It has everything I need and more.

But when I need to repeat a whole test I did find only half-baked solutions.

What do I need? Basically all what can be done with Failsafe's retry policy, including, maximum number of retries, a delay between retries, supporting a jitter factor, backoffs and being able to handle specific exceptions only. A retry policy should be configurable per test and should not have to be a general one for all the tests in an execution. Beyond that it should be possible to set a maximum total number of retries for all the tests combined. It should output flaky re-run information in test report XML so that they can be properly displayed in tools like Allure Report. Ideally it would also be possible to specify if a test is considered as failed if it needed to be retried.

None of the tools I know fit to these requirements, neither Maven Surefire plugin, nor test-retry-gradle-plugin for Gradle. Also junit-pioneer does not support many of these requirements, for instance in the Allure report the retried tests are counted as ignored tests and not identified as retried tests, because the report XML does not output re-run information but marks retryed tests as disabled.

So while I am super happy with Failsafe for retrying test steps, I did not find anything of that quality for retrying tests itself. Again, no offense, it is just how it is. Obviously no one created a solution for all the requirements I mentioned.

But why not? Is it not possible? Or not many have such high expectations like me for it? Or why?

[Michael1993]

First and foremost, JUnit is a unit test library, which makes retrying tests an anti-pattern. If you know the FIRST acronym, it means unit tests are supposed to be reliable (or, alternatively, repeatable), meaning they should always run exactly the same way and give the same result (for the same build). If you have to introduce retry logic, that means your test is either not reliable or not a unit test.

But hey, what if it is not a unit test? The idea of having to retry test is a bit less clear in this case, but it could be pointing to underlying problems in the code that you are testing.

• Are you having to retry tests because you wait for an answer from a remote service? One could argue that you are testing your connection speed or the remote service in this case (which you should not do either of those).
• Do you have a timeout on an operation that sometimes exceeds this limit? This could be either a performance problem indicator or a multithreading issue.

These examples don't mean that you should 100% never ever retry your tests. Sometimes retrying your tests is the best solution or it is simply unavoidable. But this is the edge-case, meaning there is not a lot of demand for a retry mechanic.

When there is demand, it's usually covered by what Pioneer does - just slap the annotation on your test and you are good to go: your test will run again if it failed (up to a certain number of retries). Would these people benefit from a more robust retrying extension? Almost certainly, but for most use-cases I think simply having a way to restart a test is "good enough".

Another thing you have to consider is that your example Failsafe is a library entirely centered around failure handling. Pioneer is a more generic library with a variety of extensions to JUnit. It is a concern that if we were to support something like Failsafe as an extension it would be a big demand on the project and the maintainers. I'm a little wary of introducing such a thing because of this.

There are other, more practical reasons why this is not in Pioneer. One of it is the way we register and parameterize extensions, mainly annotations. The more input a certain extension needs, the more property the annotation has. It goes without saying that after a certain number this just looks bad:

@RetryingTest(maxRetries = 8, minSucces = 4, delayForMs = 6, exponentialDelay = true, jitterFactor = 0.3,
exceptionHandling = { "UnsupportedOperationException", "handleUOE", "NullPointerException", "handleNPE" })

I hate just even looking at something like this, to be quite honest. 😕

Okay, it could also look a bit more reasonable:

@RetryingTest(maxRetries = 8, minSuccess = 4, exponentialDelayMs = 6, jitter = 0.3)
@HandleRetryException(value = UnsupportedOperationException.class, method = "handleUOE")
@HandleRetryException(value = NullPointerException.class, method = "handleNPE")

But this is still a lot. We would need a way to parameterize the extension without all this clutter. Which could look like this:

@RetryingTest(policy = "retryPolicy")
// tests and other stuff...
static RetryPolicy retryPolicy() {
 // ...
}

This is an okay solution - but it's a mindset shift from how the extension is currently set up.

But ultimately, the simple reason we don't have something like this is because no one thought about it. I believe you are a first person who wants to have something like this in Pioneer.

[pandoras-toolbox]

Thank you for your detailed answer.

The JUnit team is seeing that framework only intended for unit tests. That is why they always refused to support defining dependencies between tests (which would prevent tests to be run which will fail of another test did not run first and succeeded, something on my wish list). I use it for E2E tests because no other tool I know has such a flexibility. I think many people use JUnit also for integration and E2E tests.

In my view retries are needed if the test itself is not reliable as you wrote, but also if the object under test is not reliable. For unit tests the object under test is much more reliable, nearly 100% if it is a real unit test.

For what are retries needed in E2E tests?

• temporary hickups of the system which we do not want to check when running functional tests
• rare issues of unknown origin, too hard to reproduce or too unimportant to spend effort to investigate
• rare issues of known origin, too hard to reproduce or a too minor problem for the end user

The more E2E test are executed, the more likely it becoms that tests fail. And the more tests are flaky, the more time is lost with analysing just to realize after a hour or so: "Oh it was this thing again where we in practice cannot do anything about it."

Retrying a whole test is the last resort for me, even for E2E tests. I would rather retry certain steps. Right now I have a situation where retrying whole tests helps.

Actually I could use Failsafe to retry everying in the test method body and have all the features I potentionally need. This possibility came right now into my mind while writing this. The advantage would be that the retried tests are not counted wrongly as additional (but disabled) tests.

Correct me if I am wrong, but the only benefit of the RetryingTest annotation is that when you run it in your IDE you can easily see that the test is retried. If you do not need this and want a correct test count in the result XML then you can use Failsafe for the entire test method body and have even more options for the retry policy.

Maybe there is no need at all for a JUnit extension to do the retry because one can simply use Failsafe? Especially because this would ensure that the test count in the result XML is correct.

I understand why you are reluctant to support all this in your extension. I think I would not bother to write such a JUnit extension if it cannot output information about retries correctly in the result XML, like Maven Surefire does (but this retry solution has other problems of course).

Maybe just using Failsafe to retry the whole test is the best solution? Failsafe code is relatively easy to read, while annotations can be hard to read and are more limited.

So I might just use Failsafe to retry an entire test body. But I will try to avoid it, because actually it should never be needed, even for E2E tests.

Thank you for the discussion. What you make of it is of course completely up to you. If it does not make sense to add more features to the RetryingTest annotation, it can very well be.

[Bukama]

Hey you 👋,
I've read the discussion with high interest and can't add much to what you to have already written (esp. what @Michael1993 has written) as you two have nearly showed every aspects for both of your requirements/concerns, but want I wanted to add three other things, just as information:

• JUnit Pioneer does not write an own test report, as all the extensions are "just" additional features for JUnit execution, but not for a / the report engine. The only extension we provide stuff to registered ServiceLoader is the IssueExtension, so the user can write his/her own SPI, grab those information and do whatever he/she wants to do with them, e.g. writing an additional report. History behind this is a) that the JUnit team declined any changes in their (still, but plan to change) static report and b) that we in JUnit Pioneer decided that creating and maintaining an own (additional) test report would be out of the focus of what the project is. Why do I write this? Because I think creating such a service to provide information about how often a retried test was executed is the only way to collect and present this information at the moment.
• The RetryingTest extension was formerly created by @nipafx more ore less as a "nobody will ever need or even use such an extension" as he wanted to have a simple example to demonstrate the JUnit Pioneer extension model. And as it was already there we kept it. Over the years we recognized that users really use it 😲 and so we polished it to the state as it is today. But not with the focus / feedback from users who use it extensively for E2E tests. I'm open to improve it further, but as @Michael1993 has already pointed out what it would mean to further enhance it and pointed our problematic situations, e.g. the annotation clutter, I see the need for a proper concept first. And - this would still not fulfill your needs for additional information for a test report.
• You, @pandoras-toolbox, wrote that you might get what you need when running whole tests again. I can't tell from my view if it's possible if you can write state full test with ordered (see @Order in JUnit Guide: https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-execution-order-methods ) test steps which are then annotated with a retry to just redo the particular test. As said, not sure if possible and only a spontaneous idea.

[pandoras-toolbox]

Sorry for the late reply.

When a retry is done then the produced XML should look like below:

<testcase name=".." classname=".." time="0.1">
  <flakyFailure message="exception message" type="assertion exception">
    <stackTrace>flaky failure stack trace</stackTrace>
    <system-out> flaky failure System.out log message </system-out>
  </flakyFailure>
  <system-out> success </system-out>
</testcase>

This is what JUnit calls the legacy XML format. But many tools support this convention to identify flaky tests, for example the Allure report.

JUnit advocates for a new test reporting format, but it obviously does not support to contain information about retries.

So I asked on their Github project about it now: ota4j-team/open-test-reporting#62

I understand that you do not want to be dragged into creating/maintaining a test report.

Now I have your extension for retrying and also implemented to use Failsafe for the same purpose, side by side for a while to see which one to keep.

My impression until now is that using Failsafe is overall better for me. Retries are not so transparent with Failsafe in the Allure test report, but the number of tests is correct and I can do more with its RetryPolicy.

I am not sure what you meant with your idea, ordering tests is not really possible for me because I do want to run them in full parallel mode.

Currently I am not sure if there is a perfect solution at all regarding the test report side of things. Maybe if the open-test-reporting Github project will support retries then there can be a chance to craft a good solution for the future.