Feature request: loop test run until failure for watch mode

[capaj]

Clear and concise description of the problem

sometime we have a flaky test in the suite. We think we have made a change which could fix the flaky spec, but how do we verify this?
You need to run the test like 100x in a row or something. With the current watch mode that would entail hitting enter key manually 100x.

Suggested solution

add run until failure mode to watch usage screen:

then you can just enable this, mode grab a ☕ and come back in a few minutes refreshed to see if the flakyness is fixed or not

Alternative

manually hitting enter key 100x

Additional context

No response

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[capaj]

@sheremet-va if I were to open a PR for this, would it get considered or is this feature something vitest does not even want to include?

[sheremet-va]

@sheremet-va if I were to open a PR for this, would it get considered or is this feature something vitest does not even want to include?

Feel free to add this feature 👍🏻

[obadakhalili]

@sheremet-va I could work on this if it is still desired. But to fully understand this, basically, we want to implement "rerun all tests" but n times, right? So it could be a new option before the quit one named "n" which when pressed prompts the user to enter a number which is the n times the tests will be rerun for.

UPDATE: Should this be for all tests or failed tests only?

[capaj]

I would prefer to have also option to keep rerunning until first failure. Specifying a number of reruns is nice to have but I often don't know how many reruns will be needed.

[obadakhalili]

Hmm ... So you are suggesting that pressing the option "n" keeps running n times or until failure? This should be the behavior of my suggested option, but not a separate option. Are we on the same page @capaj?

[capaj]

Almost on the same page.
I guess I can just pass arbitrary high numbers like 10000 for example to make sure it runs long enough.

[obadakhalili]

I guess what you are looking for is a different option, say i, that runs forever or until failure. But it has to be monitored because there is a chance that tests will never fail, and just take processing space forever.

[sheremet-va]

There is PR for "repeat" mode: #2652

Maybe once it's merged you can build your solution on top of it.

[obadakhalili]

@sheremet-va I would love that. I'll subscribe for changes for when it is merged, and also feel free to ping me to work on this once the repeat mode PR is merged because I might forget. Meanwhile, got anything for me to work on?

[jacknewberry]

I would also use this functionality for fixing flaky tests.

The Ginkgo test framework for Golang has a flag called --until-it-fails which runs forever until a test fails.

[Maxim-Mazurok]

I use this bash script on Ubuntu WSL2 for running cypress tests until failure, the same can be used for vitest:

#!/bin/bash

# use this script to run tests until failure

# start testing until failure
while true
do
  # replace --spec with your desired spec
  npm run test-e2e -- --spec="cypress/e2e/auth/login.spec.js"

  # or to run all tests, use:
  # npm run test-e2e

  if [ $? -ne 0 ]; then
    echo "Error found!"
    # test failed, exit process, stop testing
    exit 1
  fi
  # test didn't fail - rerun the test again
done