Add soft assert support

[mgrybyk]

No description provided.

[christian-bromann]

What does soft assert support mean?

[mgrybyk]

A way to do multiple assertions within test then throw all the failed assertions in when the test ends.
By test I mean, for example, it block for mocha.

I'll add proper description of course, but a bit later

[alfonso-presa]

@mgrybyk, I just made this https://github.com/alfonso-presa/soft-assert (because we need something similar for long and expensive cucumber tests) and I though it might cover your needs. It's assertion and test-harness library agnostic, so it might fit well. A little bit more verbose than having it built in the assertion library though. Sorry for the advertising....

edit: Also the focus of this issue if I understood correctly is for assertion failure to be kept until the end of the test unit, which in common frameworks is the it block, but in case of cucumber is the step. We needed to retain the failures along the whole scenario until the end. I don't see how that could be done only by the assertion library without it being coupled to the testing frameworking.... That's why I created the library as an standalone tool....

[darkartswizard]

Adding another request for a SoftAssert feature in webdriverio-expect.

What a soft assert does is let the test continue even when the assertion fails.

This way you get a very telling metric: How many assertions passed and how many failed. Not that 1 of 10 tests failed - which seems insignificant - while hiding the fact that the next 30 assertions in that test failed as well - which seems rather significant.

Likewise if 1 assertion fails and 30 passed, it's likely going to be quick work to resolve.

[christian-bromann]

Given expect-webdriverio enhances the expect package with wdio related matchers it seems to me that if expect supports soft assertion this package would support it too. Maybe worth raising an issue there?

[darkartswizard]

Fair enough.

[circa1741]

NUnit has Multiple Asserts. See https://docs.nunit.org/articles/nunit/writing-tests/assertions/multiple-asserts.html.

It would be nice if Webdrive.IO has the same capability.

"This is particularly useful for testing things like object initialization and UI appearance as well as certain kinds of integration testing."

[christian-bromann]

@circa1741 I think we all agreed that soft assertions would be a nice option. Please get involved and help make it reality 😉

[DivyangRaval]

Hi All,

It is been 2 years since this request was raised.
This is very basic functionality as it() block represents a Test and one Test will have multiple Test Steps which need to be asserted if any step fails we should continue to the next step to make sure get the complete execution result of all steps.

Is there any specific reason why this request never gets prioritized?
Or we already have some feature/solution for this?

[christian-bromann]

@DivyangRaval as I mentioned already in my previous posts: get involved and help us implement this feature.

[PipulPant]

Any update on soft assertion or alternative approach to achieve this?

[christian-bromann]

@PipulPant did you read the thread here?

[darkartswizard]

Big fan of multiple validation.

[christian-bromann]

It seems like there is some discussion going on in the expect package, see jestjs/jest#13205.

[anandmdas]

Adding this post on the same for developers webdriverio-community/wdio-cucumberjs-json-reporter#255

[christian-bromann]

Inspiration for a possible implementation can be found in vitest-dev/vitest#3509

[pavankumardmg]

example
https://playwright.dev/docs/test-assertions#soft-assertions

[christian-bromann]

@pavankumardmg interesting, it seems like vitest did the same and created an expect.soft. This makes it easier for us to do the same without being dependent on expect. Any contributions are welcome!