Skip to content

Commit

Permalink
feat(resporter): add json reporter (#2582)
Browse files Browse the repository at this point in the history
Add support for `--reporters json`. It writes the mutation testing report to a JSON file, which validates against the [mutation-testing-report-schema](https://github.com/stryker-mutator/mutation-testing-elements/tree/master/packages/mutation-testing-report-schema). 

The default location for this file is `reports/mutation/mutation.json`, but you can change this location, see README.md for more details.
  • Loading branch information
Tummerhore committed Nov 6, 2020
1 parent 494e821 commit d18c4aa
Show file tree
Hide file tree
Showing 8 changed files with 147 additions and 18 deletions.
17 changes: 17 additions & 0 deletions packages/api/schema/stryker-core.json
Expand Up @@ -133,6 +133,18 @@
}
}
},
"jsonReporterOptions": {
"title": "JsonReporterOptions",
"additionalProperties": false,
"type": "object",
"properties": {
"fileName": {
"description": "The relative filename for the json report.",
"type": "string",
"default": "reports/mutation/mutation.json"
}
}
},
"mutationScoreThresholds": {
"title": "MutationScoreThresholds",
"additionalProperties": false,
Expand Down Expand Up @@ -349,6 +361,11 @@
"description": "The options for the html reporter",
"$ref": "#/definitions/htmlReporterOptions"
},
"jsonReporter": {
"description": "The options for the json reporter",
"$ref": "#/definitions/jsonReporterOptions",
"default": {}
},
"disableTypeChecks": {
"description": "Configure a pattern that matches the files of which type checking has to be disabled. This is needed because Stryker will create (typescript) type errors when inserting the mutants in your code. Stryker disables type checking by inserting `// @ts-nocheck` atop those files and removing other `// @ts-xxx` directives (so they won't interfere with `@ts-nocheck`). The default setting allows these directives to be stripped from all JavaScript and friend files in `lib`, `src` and `test` directories. You can specify a different glob expression or set it to `false` to completely disable this behavior.",
"anyOf": [
Expand Down
10 changes: 6 additions & 4 deletions packages/core/README.md
Expand Up @@ -311,17 +311,19 @@ you can consult [npm](https://www.npmjs.com/search?q=stryker-plugin) or
### `reporters` [`string[]`]

Default: `['clear-text', 'progress', 'html']`
Command line: `--reporters clear-text,progress,dots,dashboard,html`
Config file: `reporters: ['clear-text', 'progress', 'dots', 'dashboard', 'html']`
Command line: `--reporters clear-text,progress,dots,dashboard,html,json`
Config file: `reporters: ['clear-text', 'progress', 'dots', 'dashboard', 'html', 'json']`

With `reporters`, you can set the reporters for stryker to use.
These reporters can be used out of the box: `html`, `progress`, `clear-text`, `dots`, `dashboard` and `event-recorder`.
These reporters can be used out of the box: `html`, `json`, `progress`, `clear-text`, `dots`, `dashboard` and `event-recorder`.
By default, `clear-text`, `progress`, `html` are active if no reporters are configured.
You can load additional plugins to get more reporters. See [stryker-mutator.io](https://stryker-mutator.io)
for an up-to-date list of supported reporter plugins and a description on each reporter.

The `html` reporter allows you to specify an output folder. This defaults to `reports/mutation/html`. The config for your config file is: `htmlReporter: { baseDir: 'mypath/reports/stryker' }`

The `json` reporter allows specifying an output file name (may also contain a path). The config for your config file is: `jsonReporter: { fileName: 'mypath/reports/mutation.json' }`

The `clear-text` reporter supports three additional config options:
* `allowColor` to use cyan and yellow in printing source file names and positions. This defaults to `true`, so specify as `clearTextReporter: { allowColor: false },` to disable if you must.
* `logTests` to log the names of unit tests that were run to allow mutants. By default, only the first three are logged. The config for your config file is: `clearTextReporter: { logTests: true },`
Expand Down Expand Up @@ -480,4 +482,4 @@ async function main() {
}
```

Stryker is written in TypeScript, so it is recommended to use Stryker as well to get the best developer experience.
Stryker is written in TypeScript, so it is recommended to use Stryker as well to get the best developer experience.
15 changes: 8 additions & 7 deletions packages/core/src/reporters/html/html-reporter.ts
Expand Up @@ -7,8 +7,9 @@ import { mutationTestReportSchema, Reporter } from '@stryker-mutator/api/report'

import fileUrl = require('file-url');

import * as ReporterUtil from '../reporter-util';

import { bindMutationTestReport } from './templates/bind-mutation-test-report';
import * as HtmlReporterUtil from './html-reporter-util';

const DEFAULT_BASE_FOLDER = path.normalize('reports/mutation/html');
export const RESOURCES_DIR_NAME = 'strykerResources';
Expand All @@ -33,13 +34,13 @@ export default class HtmlReporter implements Reporter {
const indexFileName = path.resolve(this.baseDir, 'index.html');
await this.cleanBaseFolder();
await Promise.all([
HtmlReporterUtil.copyFile(
ReporterUtil.copyFile(
require.resolve('mutation-testing-elements/dist/mutation-test-elements.js'),
path.resolve(this.baseDir, 'mutation-test-elements.js')
),
HtmlReporterUtil.copyFile(path.resolve(__dirname, 'templates', 'stryker-80x80.png'), path.resolve(this.baseDir, 'stryker-80x80.png')),
HtmlReporterUtil.copyFile(path.resolve(__dirname, 'templates', 'index.html'), path.resolve(this.baseDir, 'index.html')),
HtmlReporterUtil.writeFile(path.resolve(this.baseDir, 'bind-mutation-test-report.js'), bindMutationTestReport(report)),
ReporterUtil.copyFile(path.resolve(__dirname, 'templates', 'stryker-80x80.png'), path.resolve(this.baseDir, 'stryker-80x80.png')),
ReporterUtil.copyFile(path.resolve(__dirname, 'templates', 'index.html'), path.resolve(this.baseDir, 'index.html')),
ReporterUtil.writeFile(path.resolve(this.baseDir, 'bind-mutation-test-report.js'), bindMutationTestReport(report)),
]);
this.log.info(`Your report can be found at: ${fileUrl(indexFileName)}`);
}
Expand All @@ -60,7 +61,7 @@ export default class HtmlReporter implements Reporter {
}

private async cleanBaseFolder(): Promise<void> {
await HtmlReporterUtil.deleteDir(this.baseDir);
await HtmlReporterUtil.mkdir(this.baseDir);
await ReporterUtil.deleteDir(this.baseDir);
await ReporterUtil.mkdir(this.baseDir);
}
}
2 changes: 2 additions & 0 deletions packages/core/src/reporters/index.ts
Expand Up @@ -7,6 +7,7 @@ import EventRecorderReporter from './event-recorder-reporter';
import ProgressAppendOnlyReporter from './progress-append-only-reporter';
import ProgressReporter from './progress-reporter';
import HtmlReporter from './html/html-reporter';
import JsonReporter from './json-reporter';

export const strykerPlugins = [
declareClassPlugin(PluginKind.Reporter, 'clear-text', ClearTextReporter),
Expand All @@ -15,5 +16,6 @@ export const strykerPlugins = [
declareClassPlugin(PluginKind.Reporter, 'dots', DotsReporter),
declareClassPlugin(PluginKind.Reporter, 'event-recorder', EventRecorderReporter),
declareClassPlugin(PluginKind.Reporter, 'html', HtmlReporter),
declareClassPlugin(PluginKind.Reporter, 'json', JsonReporter),
declareFactoryPlugin(PluginKind.Reporter, 'dashboard', dashboardReporterFactory),
];
36 changes: 36 additions & 0 deletions packages/core/src/reporters/json-reporter.ts
@@ -0,0 +1,36 @@
import * as path from 'path';

import { StrykerOptions } from '@stryker-mutator/api/core';
import { Logger } from '@stryker-mutator/api/logging';
import { commonTokens, tokens } from '@stryker-mutator/api/plugin';
import { mutationTestReportSchema, Reporter } from '@stryker-mutator/api/report';

import fileUrl = require('file-url');

import * as ReporterUtil from './reporter-util';

const INDENTION_LEVEL = 0;
export const RESOURCES_DIR_NAME = 'strykerResources';

export default class JsonReporter implements Reporter {
private mainPromise: Promise<void> | undefined;

constructor(private readonly options: StrykerOptions, private readonly log: Logger) {}

public static readonly inject = tokens(commonTokens.options, commonTokens.logger);

public onMutationTestReportReady(report: mutationTestReportSchema.MutationTestResult) {
this.mainPromise = this.generateReport(report);
}

public wrapUp() {
return this.mainPromise;
}

private async generateReport(report: mutationTestReportSchema.MutationTestResult) {
const filePath = path.normalize(this.options.jsonReporter.fileName);
this.log.debug(`Using relative path ${filePath}`);
await ReporterUtil.writeFile(path.resolve(filePath), JSON.stringify(report, null, INDENTION_LEVEL));
this.log.info(`Your report can be found at: ${fileUrl(filePath)}`);
}
}
@@ -1,4 +1,4 @@
import * as path from 'path';
import * as ReporterUtil from 'path';
import { promisify } from 'util';
import { createReadStream, createWriteStream, promises as fs } from 'fs';

Expand All @@ -20,6 +20,6 @@ export const deleteDir = promisify(rimraf);
export const mkdir = mkdirp;

export async function writeFile(fileName: string, content: string) {
await mkdirp(path.dirname(fileName));
await mkdirp(ReporterUtil.dirname(fileName));
await fs.writeFile(fileName, content, 'utf8');
}
10 changes: 5 additions & 5 deletions packages/core/test/unit/reporters/html/html-reporter.spec.ts
Expand Up @@ -6,7 +6,7 @@ import { expect } from 'chai';
import * as sinon from 'sinon';

import HtmlReporter from '../../../../src/reporters/html/html-reporter';
import * as HtmlReporterUtil from '../../../../src/reporters/html/html-reporter-util';
import * as ReporterUtil from '../../../../src/reporters/reporter-util';
import { bindMutationTestReport } from '../../../../src/reporters/html/templates/bind-mutation-test-report';

describe(HtmlReporter.name, () => {
Expand All @@ -17,10 +17,10 @@ describe(HtmlReporter.name, () => {
let sut: HtmlReporter;

beforeEach(() => {
copyFileStub = sinon.stub(HtmlReporterUtil, 'copyFile');
writeFileStub = sinon.stub(HtmlReporterUtil, 'writeFile');
deleteDirStub = sinon.stub(HtmlReporterUtil, 'deleteDir');
mkdirStub = sinon.stub(HtmlReporterUtil, 'mkdir');
copyFileStub = sinon.stub(ReporterUtil, 'copyFile');
writeFileStub = sinon.stub(ReporterUtil, 'writeFile');
deleteDirStub = sinon.stub(ReporterUtil, 'deleteDir');
mkdirStub = sinon.stub(ReporterUtil, 'mkdir');
sut = testInjector.injector.injectClass(HtmlReporter);
});

Expand Down
71 changes: 71 additions & 0 deletions packages/core/test/unit/reporters/json-reporter.spec.ts
@@ -0,0 +1,71 @@
import * as path from 'path';

import { mutationTestReportSchema } from '@stryker-mutator/api/report';
import { testInjector } from '@stryker-mutator/test-helpers';
import { expect } from 'chai';
import * as sinon from 'sinon';

import JsonReporter from '../../../src/reporters/json-reporter';
import * as JsonReporterUtil from '../../../src/reporters/reporter-util';

describe(JsonReporter.name, () => {
let writeFileStub: sinon.SinonStub;
let sut: JsonReporter;

beforeEach(() => {
writeFileStub = sinon.stub(JsonReporterUtil, 'writeFile');
sut = testInjector.injector.injectClass(JsonReporter);
});

describe('onMutationTestReportReady', () => {
it('should use configured file path', async () => {
const fileName = 'foo/bar/myReport.json';
const expectedPath = path.normalize(fileName);
testInjector.options.jsonReporter = {
fileName,
};
actReportReady();
await sut.wrapUp();
expect(testInjector.logger.debug).calledWith(`Using relative path ${expectedPath}`);
});

it('should use default base directory when no override is configured', async () => {
const expectedPath = path.normalize('reports/mutation/mutation.json');
actReportReady();
await sut.wrapUp();
expect(testInjector.logger.debug).calledWith(`Using relative path ${expectedPath}`);
});

it('should write the mutation report to disk', async () => {
const report: mutationTestReportSchema.MutationTestResult = {
files: {},
schemaVersion: '1.0',
thresholds: {
high: 80,
low: 60,
},
};
sut.onMutationTestReportReady(report);
await sut.wrapUp();
expect(writeFileStub).calledWith(path.resolve('reports', 'mutation', 'mutation.json'), JSON.stringify(report));
});
});

describe('wrapUp', () => {
it('should resolve when everything is OK', () => {
actReportReady();
return expect(sut.wrapUp()).eventually.undefined;
});

it('should reject when "writeFile" rejects', () => {
const expectedError = new Error('writeFile');
writeFileStub.rejects(expectedError);
actReportReady();
return expect(sut.wrapUp()).rejectedWith(expectedError);
});
});

function actReportReady() {
sut.onMutationTestReportReady({ files: {}, schemaVersion: '', thresholds: { high: 0, low: 0 } });
}
});

0 comments on commit d18c4aa

Please sign in to comment.