How to Generate and Analyze Pitest Reports
PIT is a state of the art mutation testing system, providing gold standard test coverage for Java and the jvm. It's fast, scalable and integrates with modern test and build tooling. Visit https://pitest.org/ and https://pitest.org/quickstart/mutators/ for more information. Good details on how to deal with pitest mutations in checkstyle is described at https://github.com/checkstyle/checkstyle/issues/12341
There is way to export mutation in bytecode - https://github.com/pitest/export-plugin (export hack from our team)
How to get list of pitest profiles
Decompiler usage to see bytecode
How to find java code that can kill mutation
Experimental Argument Propagation
To see the values can take, run:
groovy .ci/pitest-survival-check-xml.groovy --listoutput something like:
Supported profiles:
pitest-common-2
pitest-annotation
pitest-naming
pitest-sizes
....
you can use shell variable (example)export PITEST_PROFILE=pitest-common-2
to reuse command further down in this wiki page.
mvn -e --no-transfer-progress -P"$PITEST_PROFILE" clean test-compile org.pitest:pitest-maven:mutationCoverage
This will generate the pit report in the /target folder. We use a custom made script to analyze this report.
groovy .ci/pitest-survival-check-xml.groovy "$PITEST_PROFILE"The default format for reporting violations by the report is:
New surviving mutation(s) found:
Source File: "XMLLogger.java"
Class: "com.puppycrawl.tools.checkstyle.XMLLogger$FileMessages"
Method: "getExceptions"
Line Contents: "return Collections.unmodifiableList(exceptions);"
Mutator: "org.pitest.mutationtest.engine.gregor.mutators.experimental.ArgumentPropagationMutator"
Description: "replaced call to java/util/Collections::unmodifiableList with argument"
Unnecessary suppressed mutation(s) found and should be removed:
Source File: "XMLLogger.java"
Class: "com.puppycrawl.tools.checkstyle.XMLLogger$FileMessages"
Method: "getExceptions2"
Line Contents: "return Collections.unmodifiableList(exceptions);"
Mutator: "org.pitest.mutationtest.engine.gregor.mutators.experimental.ArgumentPropagationMutator"
Description: "replaced call to java/util/Collections::unmodifiableList with argument"
The report can also be generated in XML format using the -g or
--generate-suppression flag.
groovy .ci/pitest-survival-check-xml.groovy "$PITEST_PROFILE" -gOutput (Hypothetical):
New surviving mutation(s) found:
<mutation unstable="false">
<sourceFile>XMLLogger.java</sourceFile>
<mutatedClass>com.puppycrawl.tools.checkstyle.XMLLogger$FileMessages</mutatedClass>
<mutatedMethod>getExceptions</mutatedMethod>
<mutator>org.pitest.mutationtest.engine.gregor.mutators.experimental.ArgumentPropagationMutator</mutator>
<description>replaced call to java/util/Collections::unmodifiableList with argument</description>
<lineContent>return Collections.unmodifiableList(exceptions);</lineContent>
</mutation>
Unnecessary suppressed mutation(s) found and should be removed:
<mutation unstable="false">
<sourceFile>XMLLogger.java</sourceFile>
<mutatedClass>com.puppycrawl.tools.checkstyle.XMLLogger$FileMessages</mutatedClass>
<mutatedMethod>getExceptions2</mutatedMethod>
<mutator>org.pitest.mutationtest.engine.gregor.mutators.experimental.ArgumentPropagationMutator</mutator>
<description>replaced call to java/util/Collections::unmodifiableList with argument</description>
<lineContent>return Collections.unmodifiableList(exceptions);</lineContent>
</mutation>Note: Always make sure you have the correct profile pit report present in \target folder to avoid unexpected results.
We have ability to generate behavior diff report based on long list of real code of opensource projects.
If you create branch and PullRequest where you make mutation as it done by Pitest, CI can help to highlight this code.
We have special branch in repository that contain commit that disable all CI executions and let you to run only diff report. Commit to cherry pick in your branch https://github.com/checkstyle/checkstyle/commit/998c65dc977d457eeb645c91834018655280ef63.
Example of PR with disablement of CIs and hardcoded mutation and config(in PR description) for diff report: https://github.com/checkstyle/checkstyle/pull/11938
Trigger of diff report is done by comment - https://github.com/checkstyle/checkstyle/pull/11938#issuecomment-1190325464
All details on how to use CI diff reeport generator please read at https://github.com/checkstyle/contribution/tree/master/checkstyle-tester#report-generation
Note: Pit operates on bytecode, so if changes in source code don't allign with pit report then see the bytecode to figure out stuff. You can use any decompiler if you are not familiar with bytecode.
From the javadoc of the mutator
Remove switch statements. We get an array of labels to jump to, plus a
default label. We change each label to the default label, thus removing it.
Example: Source Code:
void method(int a) {
switch (a) {
case 1:
foo1();
break;
case 2:
foo2();
break;
case 3:
foo3();
break;
case 4:
foo4();
break;
case 5:
case 6:
break;
default:
throw new IllegalArgumentException("illegal");
}
} Decompiled bytecode:
void method(int a) {
switch (a) {
case 1:
this.foo1();
break;
case 2:
this.foo2();
break;
case 3:
this.foo3();
break;
case 4:
this.foo4();
case 5:
case 6:
break;
default:
throw new IllegalArgumentException("illegal");
}
} Mutation: RemoveSwitch 0 mutation
This mutation will change the 0th index case label and put it into the default label list.
Here is the 0th index case label is case 1:
Mutated bytecode (decompiled):
void method(int a) {
switch (a) {
case 1:
default:
throw new IllegalArgumentException("illegal");
case 2:
this.foo2();
break;
case 3:
this.foo3();
break;
case 4:
this.foo4();
case 5:
case 6:
}
} Similarily mutations like RemoveSwitch 3 mutation,
RemoveSwitch 10 mutation can be tackled.
Attention: index of case in message might be not matching to index in Java code. Index in message is index of case in byte code. See example at https://github.com/checkstyle/checkstyle/pull/12318#issuecomment-1287030155 . So you can analyze byte code or simply try to remove case one by one (but one at the time) and run tests.
From the javadoc of the mutator:
Mutator for non-void methods that have a parameter that matches the return
type: it replaces the result of the method call with a parameter.
E. g. the method call
public int originalMethod() {
int someInt = 3;
return someOtherMethod(someInt);
}
private int someOtherMethod(int parameter) {
return parameter + 1;
} is mutated to
public int mutatedMethod() {
int someInt = 3;
return someInt;
} Issue reported with this mutator:
From the javadoc of the mutator:
Mutator for non-void methods whos return type matches
the receiver's type that replaces the method call with the receiver.
E. g. the method call
public int originalMethod() {
String someString = "pit";
return someString.toUpperCase();
}is mutated to:
public int mutatedMethod() {
String someString = "pit";
return someString;
}Example of resolving: commit1
Different decompilers may optimize the bytecode (eg- remove default branch in some cases). In case there is some problem, javap command should be utilized to view the bytecode.
Using the javap command:
$ javap -c -p -l SomeClass.classNote: For nested classes, add quotes around the class name.
$ javap -c -p -l 'SomeClass$NestedClass.class'