Quarantine mode does not work anymore when configured via commandline option after uptdate to testcafe version 1.15.0

[wombatka]

After update tescafe from 1.14.0 to 1.15.0 quarantine mode does not rerun test after failure
I used a new quarantine mode options: attemptLimit=3,successThreshold=1
I run tescafe via npm script :
testcafe -c 5 chrome src/test/tests -q attemptLimit=3,successThreshold=1 --assertion-timeout 10000 -r spec,jenkins:report.xml -s path=SCREENSHOTS_PATH/screenshots,takeOnFails=true,fullPage=false
In case of a test failure, the test is not rerun anymore. From what I understand from the docs, it should be run 3 times in case of a failure, and be marked as passed in case it passes at least once out od three attempts

[simplyageek]

I also faced this issue, but if you use the full flag: --quarantineMode attemptLimit=3,successThreshold=1 then it works. Odd, but yeah...

[wombatka]

It does not work for me with full flag either, btw full flag is --quarantine-mode, not --quarantineMode.
However it works when configured in .testcaferc.json , like this
"quarantineMode": {
"successThreshold": 1,
"attemptLimit": 3
},

I cannot use this configuration though as I do not want to switch on quarantine mode for all tests

[VasilyStrelyaev]

I tried to reproduce the behavior you described with v1.15.0 and a simple example but to no luck.

This is my setup:

package.json

"scripts": {
    "e2e": "testcafe -c 5 chrome test.js -q attemptLimit=3,successThreshold=1 --assertion-timeout 10000 -r spec,jenkins:report.xml -s path=artifacts/screenshots,takeOnFails=true,fullPage=false"
}

test.js

fixture `My fixture`;

let attempt = 0;

test('Test', async t => {
    attempt++;

    console.log(`Run attempt: ${attempt}`);

    await t.expect(attempt).eql(3);
});

Output

npm run e2e

> my-test-project@0.1.0 e2e /Users/vasily.strelyaev/projects/test
> testcafe -c 5 chrome test.js -q attemptLimit=3,successThreshold=1 --assertion-timeout 10000 -r spec,jenkins:report.xml -s path=artifacts/screenshots,takeOnFails=true,fullPage=false

 Running tests in:
 - Chrome 92.0.4515.107 / macOS 10.15.7

 My fixture
Run attempt: 1
Run attempt: 2
Run attempt: 3
 ✓ Test (unstable) (screenshots: /Users/vasily.strelyaev/projects/test/artifacts/screenshots/2021-07-28_15-04-10/test-1)

 1 passed (5s)

This output indicates that the quarantine mode works correctly.

Please send a complete example that we can run on our side and reproduce the behavior you encountered. This example should include the test code, test pages, and configuration files (if used).

[wombatka]

You need to add .testcaferc.json with the following
{
"screenshots": {
"path": "testoutput/screenshots",
"takeOnFails": true,
"fullPage": true
},
"reporter":
{
"name": "spec"
}

}

now run the test, it will fail, it failed for me:

all the rest I used was same as in your example, except the scipt, i removed jenkins reporter so the exact script was:
"scripts": {
"e2e": "testcafe -c 5 chrome test.js -q attemptLimit=3,successThreshold=1 --assertion-timeout 10000 -r spec path=artifacts/screenshots,takeOnFails=true,fullPage=false"
}

[rob4629]

I'm seeing the same issue as the OP. CLI args don't appear to be overriding my .testcaferc settings:

.testcaferc

{
  "quarantineMode": false,
}

testcafe -q attemptLimit=3,successThreshold=1
testcafe --quarantine-mode attemptLimit=3,successThreshold=1

Test run stops after the first failure

[deeptigetfeedback]

Should I create a different bug for the Test Runner? We are also seeing this issue of test stopping after first failure with the configuration in our Test Runner

.run({
quarantineMode: { attemptLimit: 3, successThreshold: 2 }
}

[VasilyStrelyaev]

@wombatka Thank you for the clarification! I have reproduced this behavior. Any configuration file, even an empty one, seems to prevent quarantine mode options from being applied.

We will look into this issue and update this thread once we have any news.

@deeptigetfeedback no, the issue doesn't seem to depend on the way options are passed, neither will the fix.