Subsequent Tests Do Not Run If beforeEach ever fails

[alexmi256]

Environment (please complete the following information):

• WebdriverIO version: 7.16.0
• Additional wdio packages used (if applicable): @wdio/allure-reporter, @wdio/junit-reporter, @wdio/mocha-framework

Describe the bug
If the beforeEach of a suite ever fails, then the suite will stop executing any further tests.
I believe this behavior was originally intended but ultimately results in a poor user experience especially related to wdio reporters

Given the suite

describe('Example Suite', () => {
  let testNum = 0;
  beforeEach(async () => {
    console.log('Example BeforeEach');
    if (testNum % 2 !== 0) {
      await browser.$('%%%').getText();
    }
  });
  afterEach(function() {
    console.log('Example AfterEach');
    testNum++;
  });
  it(`Test 1`, () => {});
  it(`Test 2`, () => {});
  it(`Test 3`, () => {});
});

Only Test 1 & 2 will be run and thus Allure/JUnit test reporters will not report on Test 3.
This results in inconsistent test counts and may not be what the user desires.
This is especially true if the beforeEach setup requires network calls which can occasionally fail when you have many suites with many tests.
Similarly, a user may wish to ensure that the page they're loading is in a testable state in order to fail early. Currently this is ill-advised not all tests are guaranteed to run.

To Reproduce

• Run test suite/file from above

Expected behavior

• All 3 tests in suite run
• Reporter shows state of all 3 tests in suite

Additional context

• The whole test suite is failed when beforeEach hook is failed webdriverio-boneyard/wdio-mocha-framework#154
• https://gitter.im/webdriverio/webdriverio?at=5bcf7a3b069fca52a58229d2
• Report skipped tests upon beforeEach hook failure mochajs/mocha#4233
• Hook error handling (#534) -- continue testing even if some hooks are failing mochajs/mocha#1043

[christian-bromann]

@alexmi256 thanks for filing the issue.

I don't believe this is something that WebdriverIO controls. This is Mocha specific. Can you verify by checking the above test suite with Jasmine?

[praveendvd]

mochajs/mocha#1014

could you try these recommendations of using retry ?

[christian-bromann]

I checked both Jasmine and Mocha using this test suite:

describe('Example Suite', () => {
    let testNum = 0
    beforeEach(async () => {
        console.log('Example BeforeEach')
        if (testNum % 2 !== 0) {
            throw new Error('ups')
        }
    })
    afterEach(function() {
        console.log('Example AfterEach')
        testNum++
    })

    it('Test 1', () => {})
    it('Test 2', () => {})
    it('Test 3', () => {})
})

The spec reporter output is:

------------------------------------------------------------------
[chrome 95.0.4638.54 mac os x #0-0] Running: chrome (v95.0.4638.54) on mac os x
[chrome 95.0.4638.54 mac os x #0-0] Session ID: cc5c93d46e83ea1f732c11040267bcc8
[chrome 95.0.4638.54 mac os x #0-0]
[chrome 95.0.4638.54 mac os x #0-0] » /mocha/mocha.test.js
[chrome 95.0.4638.54 mac os x #0-0] Example Suite
[chrome 95.0.4638.54 mac os x #0-0]    ✓ Test 1
[chrome 95.0.4638.54 mac os x #0-0]    ? Test 2
[chrome 95.0.4638.54 mac os x #0-0]    ✖ "before each" hook for Example Suite
[chrome 95.0.4638.54 mac os x #0-0]
[chrome 95.0.4638.54 mac os x #0-0] 1 passing (126ms
[chrome 95.0.4638.54 mac os x #0-0] 1 failing
[chrome 95.0.4638.54 mac os x #0-0]
[chrome 95.0.4638.54 mac os x #0-0] 1) Example Suite "before each" hook for Example Suite
[chrome 95.0.4638.54 mac os x #0-0] ups
[chrome 95.0.4638.54 mac os x #0-0] Error: ups
[chrome 95.0.4638.54 mac os x #0-0]     at Context.<anonymous> (/path/to/webdriverio/examples/wdio/mocha/mocha.test.js:8:19
[chrome 95.0.4638.54 mac os x #0-0]     at Context.executeAsync (/path/to/webdriverio/packages/wdio-utils/src/shim.ts:398:27
[chrome 95.0.4638.54 mac os x #0-0]     at Context.testFrameworkFnWrapper (/path/to/webdriverio/packages/wdio-utils/src/test-framework/testFnWrapper.ts:71:32
[chrome 95.0.4638.54 mac os x #0-0]     at processTicksAndRejections (node:internal/process/task_queues:96:5)

@alexmi256 can you explain which output you would expect?

[alexmi256]

I would expect something being shown for Test 3, either pass/fail/skip.
Unfortunately with Mocha, Test 3 is not even run so the reporters don't even know about it.
I'm not sure what Jasmine would do, I'll have to set it up and try it out.
I assume @wdio/mocha-framework is mostly just a mocha wrapper and can't really change deep inner workings of Mocha itself.
It might be that this ends up being a know limitation with WDIO reporters and mocha framework.

[christian-bromann]

I assume @wdio/mocha-framework is mostly just a mocha wrapper and can't really change deep inner workings of Mocha itself.

That is correct.

It might be that this ends up being a know limitation with WDIO reporters and mocha framework.

More a limitation of Mocha which WebdriverIO uses. I recommend to open an issue there and we can follow up.

[alexmi256]

I've left additional comments in mochajs/mocha#4392, which seems to be a thing for quite a while