-
-
Notifications
You must be signed in to change notification settings - Fork 88
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature idea: Condition for retrying tests #1172
Comments
Seems reasonable. I'd prefer to gate this only on exit codes to avoid heuristics that can go wrong. Would that be good enough for you? |
Since in this case, the test gets a |
I have the execution side of this mostly working -- I started writing a parser for this but realized that it'll likely conflict with some other work going on in #1159. Hopefully we'll figure out a way to sequence this work soon. |
Oh, the mentioned PR has landed now! ^^ |
We're considering switching to
nextest
to run our Rust tests in CI over at Ruffle. (Cc: ruffle-rs/ruffle#14461) It's been great so far!The final issue we have is, that compared to vanilla
cargo test
- perhaps because of the many processes starting and stopping - we are occasionally triggering what looks like a rare bug in the Ubuntu 22.04 graphics drivers.See for example: https://github.com/torokati44/ruffle/actions/runs/7254876905/job/19764476196#step:12:1448
We wouldn't want to allow retries/flakyness in general just because of this, because we don't actually have flaky tests, it's only the environment that's not ideal. And we would like to know if one of our tests actually becomes flaky, by having it fail.
Rather, what I think would be great for this, is if there was a way to say that we want to retry tests only if they failed because of this specific reason, but not otherwise.
For example, filtering conditions for this could be the exit code, or a substring/tail of the stdout/stderr output of the testing process.
The text was updated successfully, but these errors were encountered: