Support targeting different JUnit TestEngines using PIT

[RobertDiebels]

The JUnit platform has the concept of a TestEngine, the engines most people are familiar with are vintage and jupiter.
There are other engines such as jqwik and this engine in particular is one I am interested in running along side PIT.

I'm not familiar with the PIT code-base, but I did skim through the discovery mechanism for test-units.
In any case, if I make an incorrect claim in any of the following then please correct me.

I noticed that the TestUnitFinder implementations all directly depend on certain annotations from each JUnit engine in order to discover TestUnits. I believe that this could be circumvented by depending on the overarching TestEngine, calling it's discovery mechanism, and retrieving the tests that are supported by that specific engine using the returned TestDescriptor.

Doing this should create a situation where adding support for other test engines such as jqwik would require less work.

I could not find an ADR or any other kind of record as to why the TestEngine interface is currently not used hence I may be suggesting a nonsensical approach due to not understanding the current approach.

[hcoles]

Pitest predates 'TestEngine' and junit 5. It was originally written against Junit 3 which follows a very different execution model, then updated for junit 4.

Support for JUnit 5 lives in the plugin

https://github.com/pitest/pitest-junit5-plugin

The open ended nature of junit 5 tests engines is problematic for mutation testing. Pitest needs to (ideally) run individual test cases against each mutant, rather than whole test classes or suite. Dynamic test discovery, test ordering, and other behaviours that each implementation might exhibit make this difficult to achieve.

If you have any suggestions for how the code in the Junit 5 plugin can be improved I'd be happy to look at a pull request.

[RobertDiebels]

@hcoles

Pitest predates 'TestEngine' and junit 5. It was originally written against Junit 3 which follows a very different execution model, then updated for junit 4.

Okay, that makes a lot of sense, thanks for the explanation.

The open ended nature of junit 5 tests engines is problematic for mutation testing.

Just to clarify, by open-ended you mean that the TestEngine test discovery contract does not guarantee that it discovers tests that adhere to the expectations PIT has regarding TestUnits (such as ordering, etc.)?
And what are the exact expectations PIT has regarding discovery?

If you have any suggestions for how the code in the Junit 5 plugin can be improved I'd be happy to look at a pull request.

Of course, I'd be happy to dig deeper into this.

Regarding such a PR.

Do you have any upfront requirements that it should fulfill or preserve?
For instance, is the JUnit 3 execution model still supported and would dropping support for that execution model be permissible?
I'm not familiar with JUnit 3 hence using it as an example requirement.

Another question that comes to mind is one regarding documentation, is there any documentation I could read that describes the current model regarding discovery phase, execution phase, etc. in more detail than the hackers-guide?
I'm not opposed to delve into the code and derive a mental model from there, but I find it's faster to read documentation where available.

Thanks for the reply, looking forward to contribute.

[hcoles]

Hi Robert,

By "open ended" I mean there is an unbounded number of things that a junit5 engine might do. It could define tests dynamically by reading text files, calling out to a network, etc. It's not possible to determine what and how many tests it will produce except by executing it. Contrast this to junit 3 and (to a lesser extent) 4, where you could determine the tests produced by a class by examining it statically.

I'm afraid there are no developer docs, just the code.

Suggesting a PR was probably the wrong approach however. If you'd like to tackle this, the most sensible way forward would be to produce a new plugin. This would allow end users to opt in or out of whatever compromises the solution might need to make.

The existing junit 5 plugin provides a guide of how to produce one.