Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In concurrent mode, subsequent runners call reporter.reportTestStart before the first runner completes reporter.reportTaskStart #7062

Closed
VasilyStrelyaev opened this issue May 31, 2022 · 1 comment · Fixed by #7102
Assignees
Labels
AREA: server SYSTEM: reporting TYPE: bug The described behavior is considered as wrong (bug).

Comments

@VasilyStrelyaev
Copy link
Collaborator

VasilyStrelyaev commented May 31, 2022

What is your Scenario?

I run tests concurrently with a Dashboard reporter (i.e. a reporter that relays the events like reportTaskStart, reportTestStart, etc., to a remote URL) on a slow CI machine.

What is the Current behavior?

TestCafe does not awaits for the first runner to complete reporter.reportTaskStart before other concurrent runners send their reporter.reportTestStart. So, my reporter sends reporter.reportTestStart just after reporter.reportTaskStart, before the latter is finished. Sometimes reporter.reportTestStart reaches the server before reporter.reportTaskStart, which breaks server-side logic.

On the attached screenshot (I use a huge concurrency factor to reproduce the issue locally, on CI -c 4 is enough), you can see 11 concurrent runners sending their reportTestStart before the first runner receives a successful response for reportTaskStart. Three of these reportTestStart requests reach the server before reportTaskStart - which causes the errors.

What is the Expected behavior?

TestCafe awaits for the first runner to complete reporter.reportTaskStart before other concurrent runners send their reporter.reportTestStart.

What is your public website URL? (or attach your complete example)

about:blank

What is your TestCafe test code?

const TEST_COUNT    = 30;
const TEST_DURATION = 1;

fixture `Many tests`;

for(let i = 0; i < TEST_COUNT; i++) {
    test(`Test No. ${i}`, async t => {
        await t
            .wait(TEST_DURATION)
            .expect(true).ok();
    });
}

Your complete configuration file

No response

Your complete test report

On the screenshot

Screenshots

image

Steps to Reproduce

  1. Run tests concurrently with a Dashboard reporter. Use the arguments provided below. I specify a huge concurrency factor to reproduce the issue locally. On CI, it's reproduced with -c 4.

To print debug info, as shown on the screenshot, I add console.log to the Dashboard reporter to reportTaskStart (as indicated) and reportTestStart (the first line).

TestCafe version

1.19.0

Node.js version

No response

Command-line arguments

testcafe chrome:headless many-tests.js -c 12 -r dashboard

Browser name(s) and version(s)

No response

Platform(s) and version(s)

No response

Other

No response

@github-actions
Copy link

github-actions bot commented Jul 8, 2022

Release v1.20.0-rc.1 addresses this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AREA: server SYSTEM: reporting TYPE: bug The described behavior is considered as wrong (bug).
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants