Split pending and skipped tests

[just-boris]

This is a first step to solve webdriverio-boneyard/wdio-allure-reporter#26

Now skipped tests will be reported with skipped status. The pending status is still there for really pending tests (tests with empty implementation).

[just-boris]

But the pending tests are still broken. They are firing test:start event that still breaks Allure.
I don't know how widely pending feature is used, but we need to find a solution for this as well.
Fixed that, now they are all also are saved as skipped for sake of consistency

[just-boris]

⚠️ Breaking change: after that fix, all reporters, like spec will stop report skipped steps as pending. To bring the previous behavior back, they should listen to new test:skipped event.

[just-boris]

ping @christian-bromann

[christian-bromann]

To bring the previous behavior back, they should listen to new test:skipped event.

Let's make sure first to have all reporters updated before landing this change.

[christian-bromann]

@just-boris I still don't know why we change the event name here, can't we handle this in the allure reporter only?

[christian-bromann]

The pending status is still there for really pending tests (tests with empty implementation)

According to this change, pending, skipped and ambiguous tests are still treated equally.

[just-boris]

The point of this change is normalizing behavior of Cucumber with other frameworks.

Neither Mocha nor Jasmine doesn't emit test:start for pending tests.
I tried to cancel test:start event for pending test, but Cucumber API is designed in a way that doesn't let us know about pending test in advance.

This is why I came up with the current solution

[christian-bromann]

The problem is that the base reporter does not trigger if the event name is skipped, see https://github.com/webdriverio/webdriverio/blob/master/lib/utils/BaseReporter.js#L114

[christian-bromann]

if the allure reporter is the only reporter that has problems with it why not just fixing it there?

[just-boris]

It is a valid point, but then here is a different problem.
Mocha framework doesn't fire test:start event for pending tests, but Cucumber does.
Allure is the only reporter that suffers from it just because it is the only reporter from the standard set, that uses this event, but there is a general problem in inconsistency, that I am trying to mitigate.

[esclapes]

I just bumped into this issue. Allure is not able to generate the reports from our cucumber features and this patch seems to fix the issue. As I understand the malformed XML files are missing stop and status attributes. Is there any way we can get those from current events?

How could I help?

[christian-bromann]

@esclapes @just-boris we need to mitigate the inconsistencies between all frameworks and make sure that they all fire the same events

[esclapes]

I have had a look at the different framework adapters to check how/when are they emitting events. There are indeed some inconsistencies. I might have missed something, but here are my two cents:

• there seems to be no test:end event sent in the cucumber adapter. Is that right?
• cucumber seems to have many more status so the bundling makes the abstraction more leaky. Would it hurt to add new states on the wdio side even if they do not map to all frameworks?
• in terms of implementation a more explicit mapping of framework events like the one in MochaAdapter::EVENTS would make it easier to standardize and spot the differences. It is specially difficult to know what is emitted when doing things like var e = 'test:' + test.status.replace(/ed/, '')

This is the overview.

	baseReporter	cucumber	mocha	jasmine	qunit
suite:start	x	Before feature and scenario	suite	suiteStarted	suiteStart
suite:end	x	After feature and scenario	suite end		testStart
test:start	x	Before step	test	specStarted	testStart
test:end	x	-	test end	specDone: any	testEnd
test:fail	x	Handle: FAILED & UNDEFINED	fail	specDone: failed	testEnd: “other”
test:pass	x	Handle: PASSED	pass	specDone: passed	testEnd: passed
test:pending	x	Handle: PENDING, SKIPPED & AMBIGUOUS	pending	specDone: pending	testEnd: pending
test:disabled?	-			specDone: disabled
hook:start	x	-	hook		?
hook:start	x	-	hook end		?