protractor-flake doesn't run the remaining tests after retrying the failed test

[fubha]

• Operating system and version = MacOS 10.14.3
• Node.js version = 8.9.0
• Protractor version = 5.1.2
• Protractor flake version = 3.3.0

Flake file:

var protractorFlake = require('protractor-flake');
var suites = process.argv.splice(2);

protractorFlake({
	protractorPath: './node_modules/.bin/protractor',
	maxAttempts: 3,
	protractorArgs: ['./config.js', ...suites]
}, function(status) {
	process.exit(status);
});

Config file:

var failFast = require('protractor-fail-fast');
var AllureReporter = require('jasmine-allure-reporter');
exports.config = {
	seleniumAddress: 'http://localhost:4444/wd/hub',
	suites: {
		suite1: [
			'../suite1/spec1.js',
			'../suite1/spec2.js',
			'../suite1/spec3.js'
		],
		suite2: [
			
			'../suite2/spec4.js'		]
	},
	multiCapabilities: [
		{
			directConnect: true,
			browserName: chrome
		}
	],
	maxSessions: 1,

	onPrepare: function() {
		jasmine.getEnv().addReporter(failFast.init());
		browser.manage().timeouts().implicitlyWait(40000);
		jasmine.getEnv().addReporter(new AllureReporter({
			allureReport: {
				resultsDir: '../allure-results'
			}
		}));
	}
};

Command:
node config/run-protractor.js

Suppose I've started running all the tests

• spec1.js passed and spec2.js fails randomly
• It retries spec2.js and it's passed as a result of retry but then it exits
• It doesn't run the remaining test i.e. spec3.js and spec4.js

Is this the correct behavior or am I missing something here? I want to continue running the remaining tests.

[NickTomlin]

What my expectation would be is:

1. initial run: all tests targeted are run (since no suites are provided by default)
   • spec1, spec3 and spec 4 pass
   • spec 2 fails and is passed for retry
2. Re-run of failed specs
   • spec 2 passes
3. Tests exit successfully

But it sounds like something else is happening. Can you provide some sanitized output, or an example repository?

Also as a side note, is there a reason why you are using JS to run flake? I think you could get the equivalent by adding protractor-flake -- config.js --suites suite1 in npm scripts

[fubha]

@NickTomlin in my case it's not running all the targeted tests. It retries the failed spec file right away after it's failed. And then after doing retries it exits.

These are the logs I am getting in terminal;

(node:15001) [DEP0022] DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead.
[10:52:17] I/launcher - Running 1 instances of WebDriver
[10:52:17] I/hosted - Using the selenium server at http://localhost:4444/wd/hub
Started
.Login
Clicking
Assert
..Login
Clicking
Clicking
F

Failures:
1) spec2
  Message:
    Failed: no such window: window was already closed
      (Session info: chrome=72.0.3626.109)
      (Driver info: chromedriver=2.46.628411 (3324f4c8be9ff2f70a05a30ebc72ffb013e1a71e),platform=Mac OS X 10.14.3 x86_64)
  Stack:
    NoSuchWindowError: no such window: window was already closed
      (Session info: chrome=72.0.3626.109)
      (Driver info: chromedriver=2.46.628411 (3324f4c8be9ff2f70a05a30ebc72ffb013e1a71e),platform=Mac OS X 10.14.3 x86_64)

Ran 4 of 725 specs
4 specs, 1 failure
Finished in 27.927 seconds

[10:52:47] I/launcher - 0 instance(s) of WebDriver still running
[10:52:47] I/launcher - chrome #01 failed 1 test(s)
[10:52:47] I/launcher - overall: 1 failed spec(s)
[10:52:47] E/launcher - Process exited with error code 1

Using standard to parse output
Re-running tests: test attempt 2
Re-running the following test files:
/users/app/suite1/spec2.js

(node:15290) [DEP0022] DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead.
[10:52:47] I/launcher - Running 1 instances of WebDriver
[10:52:47] I/hosted - Using the selenium server at http://localhost:4444/wd/hub
Started
.Login
Clicking
Clicking
Assert
.


2 specs, 0 failures
Finished in 23.434 seconds

[10:53:13] I/launcher - 0 instance(s) of WebDriver still running
[10:53:13] I/launcher - chrome #01 passed

There is no particular reason for using JS to run flake. The command you shared also giving the same output as above.

[NickTomlin]

I think the issue here is actually jasmine fail fast. Protractor flake assumes that your test run is going to run all tests and only pass it the failures. If fail-fast is failing on the first spec, flake is only going to re-run that failed test.

I'd typically solve this by only running fail fast locally, and then run all tests in CI.

Does that make sense? If not, we can talk about how to productize this into a feature, or if there's a way we can work around it for your use case.

[fubha]

@NickTomlin I want to retry the failed tests right away and don't want to run all the failed tests at the end, because in some cases my tests have dependency on each other so running them at the end won't do any good.

Currently it's retrying the failed test right away, which is good for me, but it doesn't continue to run the remaining tests. Is this possible to achieve?

[fires1989]

I ran into the very same issue .Today to put in simple terms we have a login spec and a logout spec .If the login fails the retry happens only for the login spec and the logout spec isn't run .So i would suggest you add this as an important feature at the earliest since every app out their will definitely have dependencies .Willing to discuss with you regarding this ASAP.@fubha Burney have you overcome this