Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🚀 Feature: Re-run a test multiple times: this.repeat #2332

Open
guyb7 opened this issue Jun 26, 2016 · 23 comments · May be fixed by #5011
Open

🚀 Feature: Re-run a test multiple times: this.repeat #2332

guyb7 opened this issue Jun 26, 2016 · 23 comments · May be fixed by #5011
Labels
semver-major implementation requires increase of "major" version number; "breaking changes" status: accepting prs Mocha can use your help with this one! type: feature enhancement proposal

Comments

@guyb7
Copy link

guyb7 commented Jun 26, 2016

this.retries() allows to run a test multiple times, but the test will pass on the first successful run.

I'm suggesting to add this.rerun() for tests that might randomly succeed (false positives). It will run the test n times and will pass only if all of the reruns have passed successfully. It's useful when testing behavior that is probabilistic.

At the moment I'm running the test in a loop:

for (var i = 1; i <= 100; i++) {
  it('tests something #' + i, function(done) {
    //
  });
}

The problem with this approach is that it outputs 100 different tests and messes up the test counting.

@boneskull boneskull added the type: feature enhancement proposal label Jun 27, 2016
@boneskull
Copy link
Member

Hmm. What's the problem with putting the loop inside your test?

@guyb7
Copy link
Author

guyb7 commented Jun 27, 2016

It's just not as convenient - I have to manually call beforeEach() on each iteration and call the next iteration (and call done() on the last one).

Compare this: (can be improved)

describe('#multiple-runs', function() {
  var beforeEachFunction = function() {
    // Code
  };

  beforeEach(beforeEachFunction);

  it('runs 15 times', function(done) {
    var recursive_test = function(n) {
      if (n > 15) {
        done();
        return;
      }
      beforeEachFunction();
      // Code
      recursive_test(n + 1);
    }
    recursive_test(1);
  });
});

With this:

describe('#multiple-runs', function() {
  beforeEach(function(){
    // Code
  });

  it('runs 15 times', function(done) {
    this.rerun(15);
    // Code
  });
});

@ghost
Copy link

ghost commented Aug 23, 2017

i think a better sintax could be something like this

it.rerun(15, 'runs 15 times', function(done) {
   //Code
});

@fantasyRqg
Copy link

did this function realize ?

@Bamieh
Copy link
Contributor

Bamieh commented Sep 11, 2018

@fantasyRqg no.

@plroebuck
Copy link
Contributor

plroebuck commented Sep 12, 2018

No. But in case someone wants to mess with it, here's some code that does testing manually.
Tests if a random integer is divisible by subset of prime numbers over a series of 5 runs.

Save the following as "multirun.spec.js".

var debuglog = require('util').debuglog('multirun');

describe('#multiple-runs', function () {
  var NRUNS = 5;

  // Returns vector of prime numbers (up to max) [Sieve of Eratosthenes]
  function getPrimes(max) {
    var primes = [], sieve = [];

    for (var i = 2; i <= max; ++i) {
      if (!sieve[i]) {
        // i has not been marked -- it is prime
        primes.push(i);
        for (var j = i << 1; j <= max; j += i) {
          sieve[j] = true;
        }
      }
    }
    return primes;
  }

  // Returns random integer (up to max)
  function getRandomInt(max) {
    return Math.floor(Math.random() * Math.floor(max));
  }

  // Randomize array element order in-place. [Durstenfeld]
  function shuffle(array) {
    for (var i = array.length - 1; i > 0; i--) {
      // Pick a remaining element
      var j = Math.floor(Math.random() * (i + 1));

      // Swap it with current element
      var temp = array[i];
      array[i] = array[j];
      array[j] = temp;
    }
    return;
  }

  //=========================================================
  it('should succeed ' + NRUNS + ' times', function (done) {
    (function _suite(nruns) {
      var candidate, primes, somePrimes;

      function _suiteSetup() {
        var MAXPRIME = 31;
        primes = getPrimes(MAXPRIME);
        debuglog('primes[' + primes.length + '] =', primes, '(orig)');

        // :NOTE: Remove '2' from list to improve odds of test success
        primes = primes.slice(1);
      }

      function _setup() {
        var MAX = 100;
        candidate = getRandomInt(MAX);

        // (Re)shuffle primes
        shuffle(primes);
        debuglog('primes[' + primes.length + '] =', primes, '\n');

        // Use first NKEEP prime #s for test
        var NKEEP = 3;
        somePrimes = primes.slice(0, NKEEP);
        debuglog('somePrimes[' + somePrimes.length + '] =', somePrimes);
      }

      // Code being tested [throw to indicate failure]
      function _test(run) {
        somePrimes.forEach(function (prime) {
          debuglog('candidate:', candidate + ', prime:', prime);
          if (candidate % prime === 0) {
            throw new Error('' + candidate + ' is divisible by ' + prime);
          }
        });
      }

      function _teardown() {
        candidate = undefined;
        somePrimes = undefined;
      }

      try {
        _suiteSetup();
        for (var n = 0; n < nruns; n++) {
          debuglog('==========');
          _setup();
          _test(n);
          _teardown();
        }
      }
      catch (err) {
        return done(new Error('run ' + (n+1) + ' failed: ' + err.message));
      }

      return done();
    })(NRUNS);
  });
});

Execute it, expecting both failure/success:

$ mocha multirun.spec.js

Same as above, with verbose output.

$ NODE_DEBUG=multirun mocha multirun.spec.js

@AidanWelch
Copy link

This is also useful for testing that the first run didn't break any future usages of whatever your testing without requiring a restart.

@sriharikapu
Copy link

I am facing the similar issues with tests as well cant run the same test multiple times

@shunjizhan
Copy link

shunjizhan commented Oct 17, 2022

one scenario this can be useful is that when I suspect a test to be flaky, instead of modifying the code to re-run it multiple times, I can just use this command to rerun a test many times to test it's flakiness

@TierK
Copy link

TierK commented Nov 8, 2022

Any date prognosis of realization?

@kberg
Copy link

kberg commented Jan 16, 2023

one scenario this can be useful is that when I suspect a test to be flaky, instead of modifying the code to re-run it multiple times, I can just use this command to rerun a test many times to test it's flakiness

This is what I'm facing. I'll just wrap it in a shell script and run it a couple hundred times.

@mmomtchev
Copy link

Does anyone know if there is any reason why this hasn't been implemented besides no one actually doing it?

My use case is testing for leaks in C++ modules.

mmomtchev added a commit to mmomtchev/mocha that referenced this issue Sep 13, 2023
Refs: Re-run a test multiple times mochajs#2332
@mmomtchev mmomtchev linked a pull request Sep 13, 2023 that will close this issue
@mmomtchev
Copy link

npm i mocha@github:mmomtchev/mocha#repeats

We will see if we will be able to get this into mocha.

@JoshuaKGoldberg JoshuaKGoldberg changed the title Re-run a test multiple times 🚀 Feature: Re-run a test multiple times, always Dec 27, 2023
@JoshuaKGoldberg
Copy link
Member

I like this.repeats as implemented by #5011, personally. It has a nice symmetry with the existing this.retries. 🙂

cc @mochajs/maintenance-crew - would like to hear from another maintainer before deciding anything.

@JoshuaKGoldberg JoshuaKGoldberg added the status: in discussion Let's talk about it! label Feb 6, 2024
@JoshuaKGoldberg JoshuaKGoldberg changed the title 🚀 Feature: Re-run a test multiple times, always 🚀 Feature: Re-run a test multiple times: this.repeat Feb 6, 2024
@mmomtchev
Copy link

mmomtchev commented Feb 6, 2024

@JoshuaKGoldberg In case you are wondering, judicial corruption come first before any Github PRs.
That's how the pyramid of Maslow goes.

@kberg
Copy link

kberg commented Feb 6, 2024

@JoshuaKGoldberg In case you are wondering, judicial corruption come first before any Github PRs. That's how the pyramid of Maslow goes.

What does judicial corporation have to do with this?

@mmomtchev
Copy link

mmomtchev commented Feb 6, 2024

I have been living in a total isolation for the last 4 years since an employer covered up a huge judicial scandal involving sex and corruption. I have been working on lots of opensource projects during this time while living on social welfare. The situation has been worsening ever since, with many of my PRs initially being blocked - after which I am signaled by simultaneous comments that if I was to shut up about the affair, they will be willing to merge them. I am also offered a job - which includes a condition of a sexual nature - if I accept to not talk about the affair anymore. @JoshuaKGoldberg's employer is one of the companies involved and he posted this simultaneously with a few other pings.

PS. Just as your comment, too.

@voxpelli
Copy link
Member

voxpelli commented Feb 6, 2024

Very sorry to hear about that @mmomtchev. @JoshuaKGoldberg himself is an independent freelancer and has no employer. And we're all here to make Mocha the best possible project. Judicial corruption and similar is out of scope of this issue and while that does not at all diminish the importance of such issues, they are better dealt with in other channels 🙏

@mmomtchev
Copy link

@voxpelli it is not, since I was clearly shown that these two subjects were in fact related. It is a condition that I have been asked in a significant number of open-source projects on Github during the last few years and my only answer can be no. Should anyone have any interest in pushing this PR forward, I am willing to contribute. Should there by any other conditions, my answer is negative.

@kberg
Copy link

kberg commented Feb 6, 2024

I like this.repeats as implemented by #5011, personally. It has a nice symmetry with the existing this.retries. 🙂

cc @mochajs/maintenance-crew - would like to hear from another maintainer before deciding anything.

I'm not a maintainer but this.repeats looks good.

@JoshuaKGoldberg
Copy link
Member

Talked with @voxpelli: we think this is a good idea and worth having in. Marking as accepting PRs (i.e. #5011). ✅

Note that per #5027 our main focus is on maintenance rather than features. This particular feature goes a bit against our maintenance goals... but we think it's a straightforward + useful enough edition to justify it.

...and it doesn't hurt that #5011 is a wonderfully written PR (docs! tests! 😍!).

@JoshuaKGoldberg JoshuaKGoldberg removed the status: in discussion Let's talk about it! label Feb 13, 2024
@JoshuaKGoldberg JoshuaKGoldberg added the status: accepting prs Mocha can use your help with this one! label Feb 13, 2024
@kberg
Copy link

kberg commented Feb 13, 2024

That's great, thank you.

@JoshuaKGoldberg JoshuaKGoldberg added the semver-major implementation requires increase of "major" version number; "breaking changes" label Mar 4, 2024
@JoshuaKGoldberg
Copy link
Member

Marking as semver-major because this changes output formats and generally touches a lot of areas of code. Even if it might technically be a semver-minor change in some interpretations, we're trying to be careful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver-major implementation requires increase of "major" version number; "breaking changes" status: accepting prs Mocha can use your help with this one! type: feature enhancement proposal
Projects
None yet
Development

Successfully merging a pull request may close this issue.