Mutation testing

[jnyrup]

I've been playing around with mutation testing of Fluent Assertions using Stryker.NET.
From their docs

What is mutation testing?
Bugs, or mutants, are automatically inserted into your production code. Your tests are ran for each mutant. If your tests fail then the mutant is killed. If your tests passed, the mutant survived. The higher the percentage of mutants killed, the more effective your tests are.

It gives a nice html report of all found mutations making it easy to see what mutations survived our test suite.
mutation-report.zip

I've used this to identify and add some missing unit tests.
I didn't find any bugs in the library, but found some missing extension methods for DateTimeOffset.

If anyone wants to chime on improving the test suite, this is a great tool.

[jnyrup]

Ran the mutation tests again since Stryker has evolved over the past two years and our test suite should be completely free of timing dependencies now.

All mutators enabled
complete.zip

All mutators minus string and linq
Complete_minus_string_and_linq.zip

After a quick glimpse Assertion classes in Data/ have methods that are completely untested.
cc @logiclrd

[dennisdoomen]

Nice.

[logiclrd]

@jnyrup #1814

[jnyrup]

Did a new run of mutation testing
mutation-report.zip
Again I chose all mutations minus string+linq mutations and using the net5.0 TFM.

The new report is not 1:1 comparable with previous reports as the new version of Stryker generates more mutations, e.g. remove statements and entire blocks of code.

The general mutation score decreased from 87.46% to 86.86%

• Data/ increased from 56.25% to 63.70%
• Data related types in Equivalency/Steps decreased from 73.42% to 69.77%

[logiclrd]

What does this mean exactly? Mutation testing is a thing that makes random changes to the code and then sees whether they are detected in the form of test failures? What do the numbers you quote mean for Data? PR #1814 results in fewer mutations being caught by tests? Or, you say that this run was with a new version of the tool, which generates different mutations, and some of the new mutations aren't being detected? So #1814 probably actually increases the rate at which mutations are caught but the baseline is lower now??

Does a Stryker run take a long time?? I would be curious to know what the percentage is without #1814 but with the new Stryker version, so that the numbers are 1:1 compatible, but not if that's a lot of effort. :-)

[jnyrup]

Since I pointed Data out earlier, I wanted to show how #1814 improved the case, which it did.
I used a newer version of Stryker that has a different baseline, that's the reason of the decrease.

A complete run takes ~1 hour on my machine, but it can be configured to skip parts of code and tests.

Below I've run the mutation testing only for Data, which took 3.5 minutes.

• Stryker version 1.4.2
• Only mutate files matching Data/*.cs or Equivalency/Steps/(Constraint|Data)*.cs
• Annotated Data test classes with [Trait("Category", "System.Data")], so it only runs those.

before: 60.65%
before1814_a60899055a40e5b6aef88a580004a108ffee4f85.zip

after: 68.61%
after1814-418a405031b71caf93023ba0d39562525e937f13.zip

stryker-config.json

{
    "stryker-config":
    {
        "project": "Src\\FluentAssertions\\FluentAssertions.csproj",
        "test-projects": [
            "Tests\\FluentAssertions.Equivalency.Specs\\FluentAssertions.Equivalency.Specs.csproj"
        ],
        "reporters": [
            "progress",
            "html"
        ],
        "verbosity": "info",
        "concurrency": 6,
        "mutation-level": "Complete",
        "ignore-mutations": [
            "string",
            "linq"
        ],
        "mutate": [
            "**/Data/*.cs",
            "**/Equivalency/Steps/Constraint*.cs",
            "**/Equivalency/Steps/Data*.cs"
        ],
        "test-case-filter": "Category=System.Data"
    }
}

[logiclrd]

That's pretty cool :-)

[eNeRGy164]

stryker-config.json

Add "target-framework": "net6.0", for more coverage :)

[eNeRGy164]

After a fix in Stryker.NET, so VSTest does not time-out, I finally could do a full run (~1h 20m)

mutation-report.zip

Total score: 85.43%

[jnyrup]

Total score: 85.89%

mutation-report.zip