Reduced logs for successful scenarios [Console Reporter] #1654
Comments
jan-molak
added
@serenity-js/console-reporter
enhancement
|
Ha! Funny you should mention that; I was thinking about this very thing recently. I wrote the current (and fairly verbose) console reporter when Serenity/JS supported only Protractor. Protractor supported only one reporter at the time, so printing each Serenity/JS task as they were completed provided a nice real-time view of scenario progress. However, things have moved on since then. Both WebdriverIO and Playwright Test support multiple reporters, they also apply similar tactics to avoid having them write all over each other's console output - they buffer the complete log per reporter and then print it out when the whole test suite is finished. Unfortunately, this means that real-time reporting is no longer possible, and also if your test suite has hundreds or thousands of tests - when the test suite completes, you get swamped with console output. I wonder if a better way to approach this could be to provide a high-level summary of all the executed scenarios (grouped by file to make navigation easier), then followed by a detailed report of any failures that might have occurred. So something perhaps like this: What do you think? I also wonder if we could use hyperlinks to navigate from the scenario result to its file system location. CC: @viper3400 Potentially related: #18 |
this would be awesome |
|
Hi @jan-molak, what a coincidence haha. Thanks for your answer. Some thoughts:
Maybe to have the hyperlink to the scenario in the Also, if errors in the |
|
I had a little play with the idea of using hyperlinks last night, but it doesn't seem to be supported consistently across terminals... However, So, thinking aloud, given I have a project checked out locally at: And the code is stored in a repository that's available via a web UI: I could point to a specific file and line:
Now, since absolute paths are much longer than just the few digits of the line number, perhaps the console reporter should print something like this instead: We could probably have the One case we haven't discussed yet is when you've just started working on a test scenario and actually do want to see all the details. It almost feels like the console reporter should be printing two reports:
Which perhaps could be configured like this 🤔: crew: [
[ '@serenity-js/console-reporter', {
theme: 'auto',
locationPattern: `${process.cwd()}/{path}:{line_number}`,
reports: [
[ 'summary' ]
[ 'details', { threshold: 'failure' } ], // maybe a configurable outcome threshold for the detailed report?
]
} ]
] |
|
{ threshold: 'failure' } 💯 |
|
What about having different levels for the details? Also, distinguish between the failure and all. For example, using cucumber:
Examples:interactions tasks steps scenarios For the solutions that do not use cucumber the what do you think? |
|
@damonpam Internally, Serenity/JS transforms Cucumber "steps" into "tasks", so that reporting is consistent across the test runners. So in your proposed model we could have either all the activities, so tasks and interactions: or tasks only: and if there's a failure, we print the whole thing: Something like this? 🤔 |
|
Now, this makes me wonder. Perhaps all we need is for the developer to specify what "resolution" / "report level" they want:
When there's a failure, we print the whole thing anyway. |
|
Yes, that's it! I like this 3 report level But I'd keep the failures configurable as well. And also the option to display in the |
|
Another concern I have regarding cucumber is the following. As you said, steps are transformed into tasks. So, having the following scenario: we'll have the following report As you can see, there are "duplicated" logs: is there a way to avoid this behavior? 🤔 I have many cases where the steps only perform a single task with almost the same text and I find it unnecessary. |
|
Wow, this topic is really complex. Particularly for me it would make sense that we could configure the reporter in the following ways: // type LogLevel = "scenarios" | "tasks" | "interactions" | "none";
crew: [
["@serenity-js/console-reporter", {
theme: "auto",
locationPattern: `${process.cwd()}/{path}:{line_number}`,
reports: [
["details", {
reportLevelOnFailure: "interactions", // will print interactions, task and scenarios
reportLevelOnSuccess: "tasks" // will print tasks and scenarios
}],
["summary"]
]
}]]If I want only the details for the failures: crew: [
["@serenity-js/console-reporter", {
theme: "auto",
locationPattern: `${process.cwd()}/{path}:{line_number}`,
reports: [
["details", {
reportLevelOnFailure: "interactions", // will print interactions, task and scenarios
reportLevelOnSuccess: "none" // wont print anything
}]
]
}]]
}; |
for me this is quite too specific, because in my project I have loads of custom interactions and tasks in between, like interaction 1, interaction 2, task, interaction 3. In that configuration it wouldn't make sense to print the task.
I really like the format @jan-molak proposed, though I'd like to address retries in here. Currently retries count as separate tests and each test on CI gets 3 of them, so f. ex. two tests running 3 times (1 normal, 2 retries) will count as 6 tests in the current summary. That's why It would be nice to have in a little bit better in the report, f. ex. |
It would be nice to have a configuration property to hide steps logs in successful scenarios:
Inside the ConsoleReporter class. We tried to implement it but most of the stuff is private.
However, it works fine in runtime:
The text was updated successfully, but these errors were encountered: