Metrics explanation possible need of improvement

[SelrahcD]

Hi all,

First, thanks for the great project.

I've been reading the metrics explanation several times, and I have difficulty understanding the part related to coverage.

Can someone help me clarify some points?

After the discussion, if we feel that we can improve this part once done, I'll be glad to create a PR.

About the MSI, the documentation says:

Given the code coverage of 65%, there is a 18% difference so Code Coverage was a terrible quality measurement in this example.

I don't get where the 65% comes from. I guess it's just used as an example here. The "the" in the sentence feels like we've seen it before.

I don't understand the relation between the 18% difference and the conclusion of code coverage being a terrible quality measurement.
I can feel that there might be some relation between escaping mutants and code coverage, but I'm unable to formulate it properly in my head. Can someone help me with this?

Not really important for now, just to keep it in mind: 18% should probably be 18 points here, like later in the documentation.

I also have trouble with Mutation Code Coverage, mostly because I'm not sure what a not covered mutant is. My bet is that it's a mutant which mutation is on a line that tests do not cover. Is it correct?

Thanks in advance for your help.

Have a great day

[sanmai]

You have a point. The math there is the following:

• A project has code coverage of 67% (we take this number from MCC, which is assumed to be about same the usual coverage).
• And MSI of 47% means that tests verifiably test only 47% of the code.
• This means there's a 20% difference between "estimated" and "proven" code coverage.
  If you can make this and other sections more clear, you're very much welcome.

My bet is that it's a mutant which mutation is on a line that tests do not cover. Is it correct?

That's correct.

[SelrahcD]

Hi Alexey,

Thank you for your reply. Some of this starts to make more sense to me, but I still have things that I can feel but not grasp clearly.

A project has code coverage of 67% (we take this number from MCC, which is assumed to be about same the usual coverage).

MCC is the % of mutants covered + if a mutant was covered, the line was covered => MCC is about the same as coverage.

I still don't get why about the same, and not equal to coverage? Do you have an example where the two might not be equal?

EDIT: Ah, Is it because some lines can't have mutants? => MCC is equal or slightly lower than Code Coverage?

And MSI of 47% means that tests verifiably test only 47% of the code.

This is the thing I can feel but not yet put an explanation on it. I can make a clear link between MSI and some sort of coverage.

I'm taking notes for two hours now to try tying it all together. I'll take a break a see if it clears itself. 😅

If you can expand on that part, maybe it'll keep me rolling.

Anyway, thank you for your time.