Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reporting for retries #75

Open
tim-yao opened this issue Jul 26, 2021 · 4 comments
Open

reporting for retries #75

tim-yao opened this issue Jul 26, 2021 · 4 comments

Comments

@tim-yao
Copy link
Member

tim-yao commented Jul 26, 2021

Is your feature request related to a problem? Please describe.
Cucumber report(html, json formatter) doesn't include retried details when using --retry.

When we use --retry, report will mark the flaky test pass if there is one retry get pass. That is working as expected. However, if we can also add some details of retry in the report, e.g show the number of failed attempt, that will help us to find potential intermittent bugs, instead of hiding bugs.

The idea is coming from https://automationpanda.com/2021/06/14/are-automated-test-retries-good-or-bad/

Describe the solution you'd like
Something on the report to indicate there are retry attempts happened. Like the number of failed attempt. A example:
specflow-report

@luke-hill
Copy link
Contributor

@SabotageAndi how do you do this in Specflow :D Care to offer a quick win :)

@aslakhellesoy
Copy link
Contributor

We're not making any more changes to the JSON formatter, but this could be implemented in https://github.com/cucumber/common/tree/main/react/javascript#readme

Shall we move this issue to cucumber/common and tag it with react?

@aurelien-reeves aurelien-reeves transferred this issue from cucumber/cucumber-js Jul 27, 2021
@aurelien-reeves
Copy link
Contributor

aurelien-reeves commented Jul 27, 2021

I think the messages would have all the info for all messages-based formatters to be enhanced too

But first, let's focus on the html-formatter

@aslakhellesoy aslakhellesoy changed the title Reporting for retries @cucumber/react: Reporting for retries Jul 27, 2021
@davidjgoss davidjgoss transferred this issue from cucumber/common Mar 3, 2022
@davidjgoss
Copy link
Contributor

Noting here that @cucumber/query will likely need some new smarts to handle this. Currently it always gives you data from the last attempt of any given pickle or test case, but it'll need to be able to list attempts and get data for a given attempt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants