Adding a 'checkCoverage' option to compare coverage with a defined threshold

[sebarmeli]

At the moment it's not possible to run the 'check-coverage' command available in Istanbul. It would be cool to have a way of specifying thresholds for statement coverage, function coverage and branch coverage.

I was thinking something like

checkCoverage: {
   statement : 90,
   branch: 80,
   function: 100
}

to be added to the Karma conf file.

Thoughts?

[xzyfer]

+1

[roman01la]

👍

[rpocklin]

👍 and bump instanbul to 0.2.7 so I can generate clover reports :)

[andersekdahl]

👍

[aymericbeaumet]

@rpocklin I bumped Istanbul to 0.2.10 (#78), you will have it on the next release of this plugin.

[bfowle]

+1

[andrewconnell]

+1 bump

[aymericbeaumet]

I'm not that much involved in this plugin development anymore, but feel free to PR guys. I'll review it.

[stramel]

+1

[stramel]

This will accomplish what you guys are looking for. Would be nice to add this to the documentation. First number is the threshold for Red to Yellow, second number is the threshold for Yellow to Green

// @File: karma.config.js

coverageReporter: {
  type : 'html',
  dir: 'reports',
  subdir: 'coverage',
  watermarks: {
    statements: [50,75],
    lines: [50,75],
    functions: [50,75],
    branches:[50,75]
  }
},

[peter-mouland]

@stramel if the thresholds are red, is there a way for the task to fail?

[stramel]

@peter-mouland Good question! I would have to look into it.

[peter-mouland]

thanks for the interest, i've done it quite crudely with gulp tasks. unfortunately i dont have the time to go through the coverage plug and see how to do it properly. hopefully one day....

gulp.task('test:single-run', function (done) {
    karma.start({
        configFile: findup(paths.test.config),//path to  karma.conf.js
        singleRun: true
    }, done);
});
gulp.task('test', ['test:single-run'], function(cb){
    var results = require(findup(paths.test.summary)); //path to the json-summary.json set in karma.conf.js
    var config = require(findup(paths.test.config));//path to  karma.conf.js
    var coverage = config({set: function(conf){return conf;}}).coverageReporter;
    var thresholds = coverage.reporters[0].watermarks; //path to watermarks within for type: html
    var err = false;
    for (var file in results){
        for (var threshold in thresholds){
            if (results[file][threshold].pct < thresholds[threshold][0]){
                handleError(file + ' : ' + threshold + ' Coverage is too low (<' + thresholds[threshold][0] + '%)');
                err = true;
            }
        }
    }
    if (err) process.exit(1); //exit once all errors have been gathered
    cb();
});

[andrew-cunliffe]

The watermarks are just settings for the report output colour breakpoints it seems (which is nice to see how to control this), the request looks more like a way to send information to istanbul check-coverage

check-coverage
          checks overall coverage against thresholds from coverage JSON
          files. Exits 1 if thresholds are not met, 0 otherwise

This would be very useful I use istanbul with mocha on backend node.js projects with check-coverage there is almost essential. Much like how the OP has stated for configuration.

At the moment it appears looking at the code only the report option is being implemented, so implementation of check-coverage would be great

👍

[stramel]

@andrew-cunliffe @peter-mouland I think just installing istanbul and requiring child_process's spawn should result in an easy implementation for now.

$ npm install istanbul

// File: Gulpfile.js
// NOTE: This is an untested snippet of code

var spawn = require('child_process').spawn;

gulp.task('coverage:check', function () {
var child = spawn('istanbul', [
  '--statements=100',
  '--functions=100',
  '--branches=100',
  '--lines=100'
], {cwd: process.cwd()});

child.on('close', function (errorCode) {
  // Handle Error
});

[nmalaguti]

I'm working on a PR for this issue. The implementation mostly comes from https://github.com/gotwarlost/istanbul/blob/master/lib/command/check-coverage.js and is based on the istanbul configuration file.

It will cause the build to fail by modifying the results.exitCode value if the coverage checks fail.

Info from the README:

check

Type: Object

Description: This will be used to configure minimum threshold enforcement for coverage results. If the thresholds are not met, karma will return failure. Thresholds, when specified as a positive number are taken to be the minimum percentage required. When a threshold is specified as a negative number it represents the maximum number of uncovered entities allowed.

For example, statements: 90 implies minimum statement coverage is 90%. statements: -10 implies that no more than 10 uncovered statements are allowed.

global applies to all files together and each on a per-file basis. A list of files or patterns can be excluded from enforcement via the exclude property. On a per-file or pattern basis, per-file thresholds can be overridden via the overrides property.

coverageReporter: {
  check: {
    global: {
      statements: 50,
      branches: 50,
      functions: 50,
      lines: 50,
      excludes: [
        'foo/bar/**/*.js'
      ]
    },
    each: {
      statements: 50,
      branches: 50,
      functions: 50,
      lines: 50,
      excludes: [
        'other/directory/**/*.js'
      ],
      overrides: {
        'baz/component/**/*.js': {
          statements: 98
        }
      }
    }
  }
}

[ashwinr]

ping :) can someone look at PR #141 please?