enforce msi

[mschop]

Is it somehow possible to enforce an MSI of 100% in my build pipeline?
I didn't found a solution. There is nothing mentioned in the docs. There is also no command line parameter, that would do what I'm thinking of.

[padraic]

You may need to expand on what you'd need Humbug to do. Also, remember that MSI is an "Indicator". It's perfectly valid to see a lower MSI since it's an approximate gauge dependent on factors from false positive count all the way to Humbug's mutation scope (which will vary over time as we add more mutation support).

[theofidry]

I still think we should be able to enforce a score, provided we have a nice way to handle false positives.

May be out of scope for the 1.0 though

[padraic]

I would say it's out of scope for 1.0.

[mschop]

I was able to bypass all false positives. For me the goal is to reach an MSI of 100% in some libraries. No build should succeed, if the line coverage or msi is below 100%.
For me it would be perfect, if you would provide an option that enables me to define a minimal msi for the mutation test to succeed. E.g. --min-msi 95

I think this is essential for a mutation testing tool. Without such option humbug is a nice tool to see how good my semantic coverage is. With such option, I can enforce collaborators to write good tests.

[theofidry]

I agree this would be nice, but only with a viable way to register false positives, which is not the case yet.

[mbj]

As an author of a mutation testing tool and likely one of its biggest users I can only recommend against setups that use some kind of minimal threshold. The problems I've identified in years of usage include:

• If you set a threshold of 95% and pass it, you'll not notice automatically if your coverage increased, and you could from now on use a higher bound like 96% and keep it. You'd have to constantly "check" if your program is now eligible for a higher coverage bar and adjust it by hand.
• If you set a threshold of 95% and pass it now, and your next build also passes 95% you cannot be sure if you passed that threshold because you did keep the coverage for your subjects or you reduced it and increased it for others.

Summary: that pattern can easily lead to unaccounted "flux" in coverage.

Instead on systems that cannot yet go for hard 100% coverage I recommend the following: Set the coverage expectation to 100% and whitelist all uncoverage, like excluding some scopes from analysis or exclude certain applied mutatoins that fail to be killed. Keep the allowed uncoverage in a central place and threat it like a TODO list. In that whitelist you can than explicitly also keep the usually few equivalent mutations.

This scheme does not suffer from all the problems I described above and improves the value of having mutation testing on the CI. Its also to easier onboard new mutation testing users to it, as its mirroring the default modus operandi of various smell detection tools.