How to organize a collection of Test Specs/Suites where majority can run in parallel and a subset should run in 1 instance only

[troyyer]

Hi
I would appreciate some ideas on how to setup my WDIO config files and package.json script commands to execute my E2E tests so that 3 out of 4 of the scripts will run in parallel and the last script should run in only one instance after the others have completed.

At a high level I use scripts in package.json to tell what wdio conf to run. So each script line would specify the env to run against and the wdio config to run. Within the wdio config i have the test specs to run and the configs have the capabilities defined in them
I have 2 different conf files (it's actually 3 cause i use a base.conf.js and merge with the others 2 separately). 1 conf file is for one capability (chrome) and I have another config file, that runs the test on different devices (1 firefox desktop, 1 safari desktop, 1 Android phone, 1 ios phone), so in essence 1 test can run x4 times for each of the capabilities. I have also set maxinstance to 3 for parallel execution

rough idea of the execution for both configs

In general this has worked for us however I have hit a bit of a hitch, and cant think how to update this pattern to make it easy for others to run on their end with the new spec file being added in

So I have 4 spec files
We have 3 specs (say roughly 10 tests in each) that I can run in parallel and 1 spec (this might also be split to 2 files) that I need to either run on it's own at the start or at the end of the test run.
Now this 1 "special" spec is for checking if a registration page is offline with a message, if the global offline mode is enabled. So if I set the offline flag on, then my other specs would fail as they expect the page to be online. I have a check in my first 3 specs, in their respective before and beforeeach hooks to check that the offline flag is false (and if it's enabled to disabled it), and vice versa, in the one spec checking that offline mode is enabled (and if disabled to enabled it). The obvious issue with this, is if any of tests spec 1-3 are run in parallel with spec 4 at the same time, then either one of those scripts can fail with the reason that the other spec is 'flipping' the offline mode flag
I was trying to see if grouping the tests for sequential execution would work as per https://webdriver.io/docs/organizingsuites/#grouping-test-specs-to-run-sequentially

 specs: [
    './test/specs/Registration/*.spec.js',
        /*special section but not really working out well... */
    [
      /*this set of test will run sequentially. */
      './test/specs/Registration_outage/*.spec.js',
      
   ],
   ]

so when I run with 1 capability with my existing maxinstance of 3 for parallel execution, the first 3 specs are in their own browsers with the 4th being queued up, and once one of them completes, then the offline test spec will now be run on that free instance, but since the others are running, some of the last few tests of the other 2 spec files will start to fail, or the start of the offline spec file will start to fail.
so the issue to resolve on this side is, how can I run the grouped tests that I want in sequence, only to run after all the other tests have completed their execution (would like the others to run in parallel)?

Now the next aspect is when I run with the multi device capability, again how would i allows the other specs to run with the maxinstance of 3, to run on all the devices and then once i need to run spec4, reduce the maxinstance 1 so that only one of the capabilities browsers runs at a time?

[christian-bromann]

@troyyer hey there 👋

so that 3 out of 4 of the scripts will run in parallel and the last script should run in only one instance after the others have completed.

I like to use npm-run-all as it allows you to easily define NPM commands to run in parallel or sequentially, e.g.: given I have the following package.json, e.g.:

{
    "scripts": {
        "test": "npm-run-all -p test:single:* -s test:multi:*",
        "test:single:sit": "ENV=SIT npx wdio run wdio.conf.js",
        "test:single:sig": "ENV=SIG npx wdio run wdio.conf.js",
        "test:multi:sit": "ENV=SIT npx wdio run wdio_multi.conf.js",
        "test:multi:sig": "ENV=SIG npx wdio run wdio_multi.conf.js",
    }
}

When running npm test it will run test:single:sit and test:single:sig in parallel (see -p flag), and after both finish, it kicks of test:multi:sit and after that finishes it runs test:multi:sig. It's a great tool to help you manage NPM tasks efficiently. We use it all over in the WebdriverIO project.

so in essence 1 test can run x4 times for each of the capabilities.

I am not sure if I understand. If you have WDIO config A with Chrome defined and WDIO config B with Firefox, Safari and Android defined, these are 4 capabilities that can run 12 sessions in parallel if each capability is suppose to run spec1.spec.js, spec2.spec.js and spec3.spec.js (4 caps x 3 spec files = 12 sessions).

and cant think how to update this pattern to make it easy for others to run on their end with the new spec file being added in

It is always possible for users to run e.g. the config file with Chrome defined and a --spec parameter to point to a specific test file they would like to run tests in , e.g. ENV=SIT npx wdio run wdio.conf.js --spec ./specFoobar.spec.js.

if the global offline mode is enabled.

This seems to me that this requires a special set up for your application and it might be a good idea to create a custom wdio.conf.js file that encapsulates this. If you can change these modes dynamically e.g. through a API request, you might get away with doing this in a hook within the test file and organise the spec files through grouping.

I was trying to see if grouping the tests for sequential execution would work as per

Based on this config I am expecting that the testrunner would run './test/specs/Registration/*.spec.js' in parallel first and after that it would run './test/specs/Registration_outage/*.spec.js' sequentially. Note that due to the nature of parallel execution, chances are that one capability is still running './test/specs/Registration/*.spec.js' while a different already finished that part and moved on with './test/specs/Registration_outage/*.spec.js'. If this is a problem I recommend to split the testruns into "online" vs. "offline" mode, maybe by specifying the mode through an environment variable and defining the spec option accordingly. Then you run all online mode tests first and run all "offline" mode tests once all online ones are finish.

I am not sure if I understood everything here but I hope this provides some information!

[troyyer]

@christian-bromann
Thanks so much for your suggestion and sample idea. I have implemented something that should work for my requirements

1. as per your suggestion I installed the npm-run-all package
2. in my base.conf.js file separate to the specs section, I created a suites specifically for outage and moved those tests into the suite.
3. in my other conf files, I put in at the top of the file const isSequential = process.env.SEQUENTIAL === 'true';
   and also maxInstances: isSequential ? 1 : 3,
4. in my package.json file here is a sample for SIT env that will be copied for the STG env

"scripts": {
    "junitreportmerge": "node .\\test\\support\\junitreportmerge.js",
    "browserstackall:sit" : "npm-run-all -p browserstack:sit -s browserstack_sequentialsuite:sit",
    "browserstack:sit": "set ENV=sit&& set BROWSERSTACK_BUILD_RUN_IDENTIFIER=%USERNAME%_%DATE:/=%&& npx wdio run conf/BS_multi_mobile.conf.js  || npm run junitreportmerge",
    "browserstack_sequentialsuite:sit": "set ENV=sit&& set SEQUENTIAL=true&& set BROWSERSTACK_BUILD_RUN_IDENTIFIER=%USERNAME%_%DATE:/=%&& npx wdio run conf/BS_multi_mobile.conf.js --suite outage  || npm run junitreportmerge"
  },

so the sequential script line has SEQUENTIAL=true and running browserstackall:sit will execute the specs thru browserstack:sit and
browserstack_sequentialsuite:sit will run just the suite specified and it will be sequentially

obvious one negative is this pattern makes the scripts section quite wordy and another is that I am limitting the usage of the suites capability.

Thanks again for helping out

[umer-ali-khan]

while running tests in sequence, how the browser session is managed. Like in scenarios where first sequential test failed, and second sequential spec kicks in, would it start from login page, what will be the browser state? or we need to manage it in our wdio hooks separately or wdio takes care of these tasks?

[troyyer]

@umer-ali-khan
to be clear, when I'm saying running in sequence, I am talking about running the indivdual Test spec javascript files one after the other. With webdriverio I am setting the maxInstances configuration to just 1 https://webdriver.io/docs/configuration/#maxinstances, so the runner will run the test spec file one at a time and only run the next file after the current running one has completed. So for each spec file when run, it will open 1 browser session related to that spec and once the spec file completes (can be either pass or fail) the browser session closes and then when the new spec is picked up by the runner , a new browser session is opened up.

what I am assuming you are talking about is the management of a test block within the test spec file. All the tests it blocks within one spec file will run sequentially so you will have to put in hooks for that test to manage the state of the site and the browser. here is a pseudo example with the mocha BDD framework.

describe("Login Suite Feature ", () => {
  before("Login to Portal", async() => {   
 //code to login once to your portal
  });

 beforeEach("setting user back to logged in homepage", async() => {
 //code to have user on their home page  
});
 
 after("Logout of portal ", async() => {
 //code to logout once to your portal 
});
 
it('Logged Test 1 ', async () => {
  // code for Test 1
})


  it('Logged in Test 2', async () => {
// code for Test 2
 })

it('Logged in Test 3', async () => {
 // code for Test 3
})
});

you should not have 2 separate Spec file dependent on each other. They should be able to run independently. so if both Test Spec files 1 and 2 are login tests, then both files should have the before and beforeeach hooks to do the login and save the state for that spec file