Asserting that assertions were made

[mstade]

Is there a way to assert whether any chai assertions have been made?

We've been bitten a number of times by tests passing gloriously in Mocha, only to realize that it's because no assertions were made for whatever reason (a common one is the misspelling of assertions, such as calledwith instead of calledWith. This would easily be caught by having a test hook that runs after each test to make sure that assertions were indeed made. It won't help with situations where some assertions are made and others are broken by misspelling or such, but in our experience it's been less common (if occurring at all.)

If not possible, I'm happy to contribute, just let me know!

[logicalparadox]

This seems a bit tricky as a single line of an assertion can actually have multiple "assertions" (calls to this.assert) nested within it; counting could be misleading. However, open to suggestions/contributions if you have thoughts on how to implement...

[aeosynth]

QUnit has expect:

Specify how many assertions are expected to run within a test.

[mstade]

I'd have to dig into the code to familiarize myself with it before suggesting implementations. As for the actual assertion count, I'd say it's less important than knowing whether any assertions were made at all. The former is useful for stats geeks I guess, but hardly useful for any type of code quality measurements. However, tests that don't have any assertions what so ever need to be singled out, as they are by definition pretty useless in that case :)

[logicalparadox]

I like the idea of counts, just cause I'm a stats nerd. But the issue we run into is that QUnit's counting works because (1) the assertions and the test runner are integrated (2) the assertions aren't chain-able.

A cheap and dirty way to do this as a plugin would have this interface...

var chai = require('chai')
  , chaiCount = require('chai-counter')
  , should = chai.should();

chai.use(chaiCount);

var c = chai.counter();

something.should.equal('hello universe').count(c);
otherThing.should.have.property('hello')
  .and.be.a('string').count(c);

c.assert(2);

The count has to be the last in the chain in order for it to count properly. If it isn't last, there is no guarantee that the full thread finished.

I'm sure there might be a better way to do this...

[mstade]

:)

My use case is a little bit different. I want to do this on a very global level, where before I run any test, the assertion counter (or flag) is reset, and after the test has run the counter/flag will be checked to see if it was hit at all. I use chai with mocha, and it's very easy to hook into before and after each test, but well in there I have no idea what to do with chai. I suppose solving that problem will inherently give you a correct counter as well, since you'd have to know whenever any type of assertion were made. Am I making any sort of sense? I feel like I might be confusing things.

Edit: The benefit of having a global counter or flag is of course that I can then make sure to fail any tests that didn't make any assertions at all. It happens a little bit too often that I misspell the code and mocha happily reports these tests as A-OK, which is clearly not right.

[logicalparadox]

The only way I can think of a global counter/flag is to hook into Assertion.prototype.assert. The issue with this is that a flag/count can be misleading. Some assertions include multiple calls to this.assert. .length(n) is one example. It, internally, calls Assertion.prototype.assert to assert that the subject of the assertions has the property .length before actually checking the value of length. Your count for expect(arr).to.have.length(4) would be 2.

Alternatively, if you were doing something like expect(bln).to.be.a('boolean').and.treu (intentionally misspelled), your global flag would trigger true even though not all of your assertions were invoked. Not that efficient either.

If you know all of chai's internals completely, the counting solution isn't too bad. But I don't think that should be a requisite.

So, rock/hard place...

[mstade]

Rock/hard place indeed. I think though, for my current use case, that hooking into Assertion.prototype.assert will be good enough. All I need is to know whether the test made any assertions at all and if hooking into that method will give me that, then I'm all game, even if it may be called more than once per true (for lack of a better word) assertion. I'll give it a go with my local set up and see if works out, thanks for that!

[logicalparadox]

There should be sufficient information on writing helper on the docs site to get you started, but feel free to post code for review here if you need further assistance. Also, when you have a solution you like, please post it as a Gist and include your information on the wiki page for helpers. Thank you kindly!

[domenic]

You could hook into should and expect and assert. Each of them should only be called once per conceptual "assertion".

[nzakas]

This thread seems to have died, but I have a need for this. I'm more interested in if any assertions occurred than the exact number. If there's any interest, I can take a look and make a PR for discussion. Shall I go ahead?

[logicalparadox]

Sure, be interested to see what you come up with!

[tstirrat]

I'm definitely interested in this. I've fallen victim to the problem outlined above where no assertions were made but the test passes.

[lo1tuma]

+1

[mattfysh]

The issue may be the hidden usage of property getters. Does anyone know of a library that rewrites these into actual functions, so the invocation of these properties will throw an error if misspelled? E.g. expect(foo).to.be.treu() should rightly err.

[mstade]

tpyo is a proof-of-concept to show how proxy can be used to allow misspellings, perhaps there's inspiration to be had there.

[radkodinev]

By the way - to count assertions (or rather groups of assertions) you can use checkmark - the tiny library I created to overcome problems of that kind. Not the most elegant solution -- a kind of a workaround -- but it's simple, works for me and might also work for you.

[sirlancelot]

I like @radkodinev's way if it were to be written as a Chai plugin. For instance:

var arr = ["array"];

it("should make an assertion", function(done) {
    // After one assertion is made, run `done`
    expect(1).check(done);

    // `.mark` will increment the internal counter above.
    arr.should.have.length(1).mark;
})

[mstade]

FWIW I've since switched to must which solves all of this nonsense quite elegantly.

[sirlancelot]

I made a gist implementing checkmark for chai if anyone's interested. It should accomplish most of what OP wants when using it with the done() callback structure of test frameworks.

[keithamus]

I'm going to keep this open. I think this is a reasonable enough add-on to the core, if someone comes up with a sensible solution. In the meantime, @sirlancelot if you fancy making your gist an proper plugin (with an npm/bower package), you could add it to our plugins list (just edit the plugins.js file in chai/chai-docs)

[sirlancelot]

I got around to making a full-fledged repository with linting: https://github.com/sirlancelot/chai-checkmark

[doublerebel]

@sirlancelot chai-checkmark is useful, thanks. I've implemented it in najax tests to make sure all different HTTP request types are covered.

My usage case is somewhat different than the described case. I would be interested to see more usage examples here, to show how test implementations could look.

[Ciantic]

Just got hit by this, this should be fixed.

I can't rely on test framework which can't guarantee all assertions have been run. I'm now trying this must thing.

P.S I just opened an issue in Mocha to implement pluggable assertion counting in Mocha

[aseemk]

Is this still an issue? I think you can call assertions as methods now, which prevents typos, etc.

E.g. https://github.com/thingdom/node-neo4j/blob/495e42f224181ec91e91245873d62b9b91b25245/test-new/constructor._coffee#L61

Edit: https://github.com/chaijs/chai/blob/master/ReleaseNotes.md#noop-function-for-terminating-assertion-properties

[keithamus]

I think I jumped the gun a bit earlier by saying a PR is wanted. It would be better to flesh out a design here before moving to making PRs

[Turbo87]

I've written down a proposal of how it could look like in #670. Feel free to comment.

It would be awesome if this could also work with the ava test runner, which run tests concurrently, but in that case we would need to store the planned/executed variables on some sort of test context.

[dtracers]

I have a quick question.

how does it know what the last link in the chain is?

Because if you just do it by the last link in the chain and that every link in the chain is chainable or is valid then it would solve most of the issues

example:

expect(foo).to.be.true

this will add the number to the assertions when true is called bc its the last in the chain.

This may not fix the typo issue like true but it does get much closer to the problem.

It's like the mark library but built into the last assertion.

[meeber]

@dtracers I don't think it's possible for a chain to know what its last link is, or for a link to know that it's the last link in a chain. At least not without some extreme AST parsing.

A consequence of enabling expressive chainable assertions is that there's nothing stopping someone from doing some pretty whacky things, such as splitting each link of a chain onto a separate line and then branching into multiple assertions.

This is a valid test:

it("hurts my soul", function () {                                               
  var link1 = expect(42);                                                       
  var link2 = link1.to;                                                         
  var link3a = link2.equal(42);                                                 
  var link3b = link2.not;                                                       
  var link4b = link3b.equal(43);                                                
});  

[dtracers]

so basically the only way to really know is after each test you build a tree of all of the calls for that specific test then check leaf nodes of that built tree.

But the problem is that chai isnt really integrated on the test level.

[meeber]

The biggest issue here (with typos and and non-existent property assertions) is resolved via #721.
Discussion regarding possible future measures (e.g., validating assertions were run, or replacing property assertions with method assertions) can be discussed in #726.

[martin-g]

It seems this feature was not actually implemented. At least I didn't find anything neither in the documentation nor in the tests showing how to do it.
In my application we use the assert interface, so here is how I implemented it:

in src/test/util/chai-extensions.js:

'use strict';

/**
 * An extension/plugin for ChaiJS that collects the number of executed assertions and
 * compares them against the expected ones in MochaJS's afterEach hook
 */
module.exports = {

  /**
   * Overrides Chai's 'assert' method to intercept any calls to it and to count them.
   *
   * 'this' is chai['test name'] to isolate the extra added properties from overwriting
   * in concurrently executed tests.
   *
   * @param _chai The instance of Chai
   * @param utils The Chai utilities used to override the 'assert' method
   */
  assertionCountingPlugin: function (_chai, utils) {
    const self = this;
    self.expectedAssertions = 0;
    self.countedAssertions = 0;
    self.expect = function(num) {
      self.expectedAssertions = num;
    };

    utils.overwriteMethod(_chai.Assertion.prototype, 'assert', function (_super) {
      return function () {
        self.countedAssertions++;
        _super.apply(this, arguments);
      };
    });
  },

  /**
   * MochaJS hook that is used to verify that the number of expected assertions matches the number
   * of executed ones
   */
  afterEach: function() {
    if (this.expectedAssertions !== 0 && this.expectedAssertions !== this.countedAssertions) {
      throw new Error('Expected ' + this.expectedAssertions + ' but ' + this.countedAssertions + ' have been recorded!');
    }
    // reset
    this.expectedAssertions = 0;
    this.countedAssertions = 0;
  }
};

In the tests:

const chai = require('chai');
const chaiExtensions = require('./util/chai-extensions');
const assert = chai.assert;

const SpecName = 'MySpecialTest';
const chaiContext = chai[SpecName] = {};
chai.use(chaiExtensions.assertionCountingPlugin.bind(chaiContext));
afterEach(chaiExtensions.afterEach.bind(chaiContext));

describe(SpecName, function() {

  it('testing', async () => {
    chaiContext.expect(3);

    assert.equal(1, 1);
    assert.ok('truthy');
    assert.isFalse(false);
  });

[sevillaarvin]

Hi, jest has an expect.assertions method that verifies that a certain number of assertions are called during a test. I was hoping there was a chai equivalent to this. I've been searching in google and couldn't find anything so far.

[martin-g]

In the meantime I've improved a bit my extension (in TypeScript):

'use strict';

import HookFunction = Mocha.HookFunction;

/**
 * An extension/plugin for ChaiJS that collects the number of executed assertions and
 * compares them against the expected ones in MochaJS's afterEach hook
 */
export class ChaiExtension {

  private expectedAssertions: number = 0;
  private countedAssertions: number = 0;

  constructor(specName: string,
              chaiUse: (fn: (chai: any, utils: any) => any) => void,
              mochaAfterEach: HookFunction) {

    chaiUse(this.countAssertions.bind(this));

    mochaAfterEach(this.afterEach);
  }

  expect (num: number) {
    this.expectedAssertions = num;
  }

  /**
   * Overrides Chai's 'assert' method to intercept any calls to it and to count them.
   *
   * @param _chai The instance of Chai
   * @param utils The Chai utilities used to override the 'assert' method
   */
  private countAssertions(_chai, utils) {
    const self = this;
    utils.overwriteMethod(_chai.Assertion.prototype, 'assert', function (_super) {
      return function () {
        self.countedAssertions++;
        _super.apply(this, arguments);
      };
    });
  }


  /**
   * MochaJS hook that is used to verify that the number of expected assertions matches the number
   * of executed ones
   */
  private afterEach() {
    if (this.expectedAssertions !== 0 && this.expectedAssertions !== this.countedAssertions) {
      throw new Error('Expected ' + this.expectedAssertions + ' but ' + this.countedAssertions + ' have been recorded!');
    }
    // reset
    this.expectedAssertions = 0;
    this.countedAssertions = 0;
  }
};

and the usage:

const chai = require('chai');
import { ChaiExtension } from './util/chai-extensions';
const assert = chai.assert;

const SpecName = 'Some Test';
const chaiExtension = new ChaiExtension(SpecName, chai.use, afterEach);

describe(SpecName, function() {


  it('blah', async function () {
    chaiExtension.expect(14);
    ....
  });

[aryehb]

@martin-g This won't have the expected behavior for any of the members methods (e.g. assert.includeMembers()).

For those methods, Chai also runs an additional 2 assertions internally to assert that actual and expected are both arrays (see here), so the recorded number of assertions will always be 2 more than expected.

[bompus]

I've made a quick helper to assist with this, that doesn't involve hooking into beforeEach/afterEach. I only use it where I need it.

In the example, it is wired up to expect() in the outer scope. It's pretty basic in that it would only work with the single expect() function from either npm expect or chai's expect, but it works for my case so I'd figure it was worth sharing.

class ExpectCounter {
  constructor(expected) {
    this.actual = 0;
    this.expected = expected;
  }

  expect(...args) {
    this.actual++;
    return expect(...args);
  }
}

it('should handle reject', async function () {
  const count = new ExpectCounter(2);

  await promiseCbAsync(badFnAsync(), (err, rtn) => {
    count.expect(err).toBeInstanceOf(Error);
    count.expect(err.message).toBe('bad');
  });

  expect(count.actual).toStrictEqual(count.expected);
});