Fewer mutations generated with version 1.15.4

[Stephan202]

While reviewing PicnicSupermarket/error-prone-support#984, I noticed that version 1.15.4 of Pitest reports fewer mutations than version 1.15.3. On cursory inspection it appears that version 1.15.4 no longer mutates deferred code that is referenced by static initialization expressions, such as:

• Lambda expressions directly or indirectly referenced by enum and static fields initialization expressions, such as here and here.
• Methods that are referenced only by a method reference passed to a static fields initialization expression, such as here.

The following reproduction steps show the issue (the above are just some examples; quite a lot of other classes are impacted as well):

# Clone the repo.
git clone git@github.com:PicnicSupermarket/error-prone-support.git
cd error-prone-support

# Collect mutation coverage before the upgrade.
./run-mutation-tests.sh
mv error-prone-contrib/target/pit-reports /tmp/pit-reports-before

# Collect mutation coverage after the upgrade.
git checkout origin/renovate/pitest-maven-plugin-1.x
./run-mutation-tests.sh
mv error-prone-contrib/target/pit-reports /tmp/pit-reports-after

# Compare the reports.
firefox /tmp/pit-reports-before/index.html
firefox /tmp/pit-reports-after/index.html

An example of a dramatic differences for the class tech.picnic.errorprone.bugpatterns.CanonicalAnnotationSyntax:

• Before:
• After:

[hcoles]

Thanks for the report.

I've just taken a quick look and I suspect the issue is that the release notes missed out the inclusion of #1274

This change expands pitest's search for code that is only called from static initializers. Although we'd ideally like to mutate this, because the code is only executed once within a JVM, it is only possible to kill these mutants if pitest happens to have launched a fresh JVM just before the mutant is inserted.

From a very quick scan of your example class, this looks to be working as intended since dropRedundantParentheses etc are called only when initializing a static field.

[Stephan202]

@hcoles thanks for the quick response!

Perhaps I misunderstand, but since execution of the impacted code is deferred, I would expect it to be excluded from the search. After all, the version 1.15.3 report shows that the mutants could indeed be killed prior to this change (without restarting the JVM).

Put another way, given a method foo that is referenced only by a static field, I would would expect it to be excluded when used as

private static final String CONST = foo();

but not when used as

private static final Supplier<String> CONST = () -> foo();
private static final Supplier<String> CONST_2 = ThisClass::foo;

Because in the latter cases any mutation of foo will impact use of the static Suppliers.

[hcoles]

The mutants are not completely unkillable, but they are not reliably killable.

If they happen to be the first mutant in one of the JVMs that pitest launches, they will be killed by an effective test. Unfortunately, they may also cause other mutants to be falsely killed as any side effects from their execution (e.g bad state stored in a static variable) will endure in the JVM from that point on.

However, I think you are correct that there is an issue here. The change has not considered delayed execution from Suppliers etc.

I'll most likely re-relase 1.15.5 tomorrow with the change rolled back until this is reexamined.

Thanks again for the report.