Statistics reporter

[Lakitna]

Is your feature request related to a problem? Please describe.

In #2434 we've been running into some data gathering issues. It would be great if we can fix these once and for all with a new reporter.

Describe the solution you'd like

Introduce a new reporter, maybe called json-statistics, statistics, or statistics-json. As the name implies, this is a JSON reporter that outputs as much data as possible relating to statistical analysis, performance measurement, etc. It could output something like this:

{
    // Optional.
    "timestamp": "2020-12-07T11:56Z",

    // Final score of the run.
    // Percentage
    "score": 75.7,

    // Average number of tests executed per mutant.
    "testPerMutant": 2.3,

    // Number of tests executed during the initial run.
    "tests": 2,

    // How long things take.
    // Seconds
    "duration": {
        "total": 1234,
        "initialRun": 1,
        "mutation": 1233
    },

    // Basic mutant numbers of entire run
    "mutants": {
        "total": 12000,
        "killed": 500,
        "survived": 500,
        "error": 100,
        "timeout": 100,
        "noCoverage": 0,
        "ignored": 0
    },

    // Data for each mutated file.
    "files": [
        {
            // Initially the file path, replaced with a hash to anonymise the results.
            // This allows real-world users to submit their statistics without having to sanitise
            // paths of their closed-source project. Hashing the path allows for run-to-run
            // comparisons. There are no immediate plans for doing something like that.
            "id": "d5886a034ba69e2ebdccb36e71cd6416",

            // Score of the file.
            // Percentage
            "score": 12.32,

            // I don't think we have this data point?
            "testPerMutant": 2.3,

            // I don't think we have this data point?
            "duration": 123,

            // Basic mutant numbers of the file.
            "mutants": {
                "total": 12,
                "killed": 5,
                "survived": 5,
                "error": 1,
                "timeout": 1,
                "noCoverage": 0,
                "ignored": 0
            }
        },
        // {...}
    ],

    // Details regarding how Stryker was ran
    "stryker": {
        // Might have to be replaced with an object containing all Stryker packages.
        "version": "0.0.1",

        // The parsed Stryker config. Includes all CLI flag things.
        "config": {
            "testRunner": "mocha",
            "coverageAnalysis": "perTest",
            "concurrency": 15,
            // ...
        }
    },

    // Details on the machine that ran the tests
    "system": {
        // `true` if the run was performed on a CI like Github Actions.
        "ci": false,

        "os": {
            "version": "Windows 10 Pro",
            "platform": "win32",
            "release": "10.0.19041"
        },

        "cpu": {
            "model": "Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz",
            "logicalCores": 16,
            "baseClock": 2304
        },

        // RAM can have an impact on performance when there isn't enough.
        "ram": {
            "size": 34.07,
            // ...
        }
    }
}

Describe alternatives you've considered

So far we've copy-pasted from other reporters. @kmdrGroch even wrote a little Python script to help with that. This should be a better solution.

Additional context

Initial discussion on this: #2434 (comment)

[Lakitna]

I might pick this up myself as this sounds like a good Christmas holiday project. Though I don't know what's the best place for this to live. A separate package?

[bartekleon]

welp for now we could make this as separate package I believe :) We could integrate it once @nicojs accepts it :)

• keep in mind that integrating it makes we have to maintain it ;p

[Lakitna]

Just to get the logistics out of the way:

A separate repo inside https://github.com/stryker-mutator or should I just use my own namespace https://github.com/lakitna. It should probably be called @stryker-mutator/statistics-reporter on NPM.

[bartekleon]

For now let's use your own namespace. Am i not sure about the name tho. I think we should ask on slack so Nico is more likely to answer :)

[nicojs]

Thanks for opening this issue.

I see a lot of overlap with the current "json" reporter. Maybe we can add performance statics to that JSON file? If so, we might want to solidify them in the JSON schema at https://github.com/stryker-mutator/mutation-testing-elements.

See also stryker-mutator/mutation-testing-elements#720

A separate repo inside https://github.com/stryker-mutator or should I just use my own namespace https://github.com/lakitna. It

It depends. Do you want us to maintain it? Or do you want to maintain it? If you want to maintain it, do you want to become part of the Stryker-mutator contributors?

If we put it in Stryker, then I would prefer this to land in @stryker-mutator/core, since creating a new reporter under @stryker-mutator/xxx comes with maintenance overhead that isn't needed. This is why we ended up moving @stryker-mutator/html-reporter to @stryker-mutator/core.

Side note: I would argue that this overhead is needed for the test-runner plugins because they integrate with packages in the outside world.

[rouke-broersma]

Keep in mind that merging this with the schema means that you will need to use generic terms.

There is overlap with some other feature requests such as: stryker-mutator/mutation-testing-elements#720

[Lakitna]

One of the downsides of using the json reporter for this, is that it exposes the source code of every file. A more anonymous approach might result in more people sharing statistics with us. Specifically when we're talking about closed-source software. We could use those statistics for analysing real-world usage (and potentially also performance issues).

[nicojs]

@Lakitna we've created stryker-mutator/mutation-testing-elements#786. It should be 100% compatible with your suggestions in this issue (all data should be there).

[nicojs]

Update, we've closed stryker-mutator/mutation-testing-elements#786. With the updated schema, we can now simply add these fields here:

https://github.com/stryker-mutator/stryker/blob/a2f6107ff0d2ae570a0d1ef80e1c921379fe14da/packages/core/src/reporters/mutation-test-report-helper.ts#L136-L142

The mutation-testing-elements-schema also comes with the typescript interfaces, so that should help a lot.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.