PIT support

[eis]

Support for PIT mutation coverage tool, http://pitest.org, would be nice!

Coveralls-gradle-plugin (https://github.com/kt3k/coveralls-gradle-plugin) just got it, so it would be great if maven plugin could do the same.

[andrioli]

PIT is a great tool. But I'm not sure if, in Coveralls, mutation coverage is useful.

The problems that I noticed are: (1) lines that can't be mutated will never be shown as covered. (2) PIT can use more than one mutator per line. What should happen for example if only one of three mutators were killed? In this case the gradle plugin reports the line as covered. IMHO the line should be not covered (or at least partially covered). (3) The information about which mutator were killed or not is completely lost.

These issues make me think about the usefulness of a mutation report in Coveralls.

Coveralls was simply not made for mutation coverage.

[andrioli]

On the other hand, I think that add support for PIT line coverage will be a contribution.

But there is a difficulty of consuming the line coverage XML report 😢 The report is dependent of the current implementation. ie., the XML have only info about which blocks were covered. There no data about which lines belongs to a block.

[trautonen]

Good discussion. As far as I can see the implementation of the Gradle plugin can be replicated to this plugin. Does it make sense? That I don't know, because I'm not familiar with PIT.

About the technical implementation. This plugin relies on CoverageParser interface which can do pretty much anything it likes. Basically a parser is initialized with a found report file (in this case a mutation report) and a source loading strategy which tries to find the correct file, see CoverageParsersFactory.

Then what the parser must do is report back with Source definitions, that map to the model what Coveralls uses. The coverage model is just simple Integer array which maps to each line of the source file with a value of null or the number of hits to that line. null means it's not relevant to the coverage report and not counted as missed line.

[andrioli]

Does it make sense?

@trautonen: this is exactly my point!. Maybe @hcoles can share your opinion with us 😄

[hcoles]

@andrioli Not familiar with coveralls so can't comment on if it makes sense from that point of view, but to answer some of the points made further up.

In its own reports PIT reports a line as uncovered if it contains one or more surviving mutants regardless of how many are killed.

If it is not possible to display the details of the mutants then the information is not actionable - I personally wouldn't find it useful.

The line coverage format the PIT spits out is a mess - it is intended as a debugging aid rather than for consumption. Even if it was cleaned up (which would be a good idea regardless of coveralls support) I'm not sure why anyone would want to use it as their coverage system. JaCoco is better at line coverage - PIT's system is optimised for the specific purpose of optimising mutation coverage.

The only advantage it has is that it reports per test coverage. At the point I wrote it this was either impossible or difficult with JaCoCo, but I think that might have changed. Depending on whether I'm right on that last point it might make sense to support PIT's line coverage system if coveralls itself is able to display per test coverage.

[andrioli]

@hcoles I agree with you.

Looks like Coveralls do not support line annotation. So we can't add descriptions for mutator nor test names. All we have is hit count.

So, IMO, mutation coverage and per-test line coverage is pretty useless in this context