Scenario-Level Parallelization

[Code-Grump]

Scenario-level parallelization is something that's long been mentioned for SpecFlow and for Reqnroll has come up at least twice recently:

• Parallel execution using Mstest+reqnroll #116
• Parallel scenario tests fail #114

There's clearly a desire to run tests with scenario-level parallelization; I think this might be a solid feature to add, something with real benefits to encourage users to adopt Reqnroll.

Do we have any idea what it would actually take to enable this?

• What do we currently expect to break when trying to run scenarios in parallel?
• Are there any parts of Reqnroll's design that we know will be a problem for parallel invocation?
• For the long-time maintainers: have any attempts been made to address this in the past? Can it teach us anything?

[gasparnagy]

Here is my brain dump regarding this:

• I am not aware of any conceptual problem that would disallow scenario-level parallelism. The fact that it is not working should be considered as misbehavior. (Of course it can turn out that I was wrong.)
• It seemed in the past that the problematic part is BoDi, but earlier when that was a separate project/repo, diagnosing and experimenting with that was hard.
• I don't know if it is relevant, but we (I 😀) have made a design mistake: I assumed that within a scenario execution, there is no need for thread-safety, because that is the most atomic element of test parallelism. Therefore the ScenarioContext and the related BoDi container was not designed to be thread-safe. It turned out however, that the automation code CAN make multiple parallel threads (explicitly or implicitly via async), so there can be cases where this causes an issue. Maybe this is not the most direct problem, but this is going to be a problem as well and might be related. So in short, we have to make sure that the ScenarioContext is thread safe.
• It was also a related problem that the DI resolves the classes from the grouping containers (testrun, testthread, feature) on demand. That means that if a rarely used interface was registered to one of these global containers, they might have been only resolved when the first scenario needed it. If multiple scenarios running in parallel that need this, it might cause strange issues. As far as I know these problems were solved now, but there might have been a few cases when this is still an issue.
• Another problem was that it was hard to make an automated test to verify this. I hope the SystemTests will provide a solution for this now.

As far as I know, @tzongithub did the last attempt to fix parallel issues, maybe he can add further notes.

I would say that probably the strategy should be to pick a test runner (e.g. MsTest) and make a test in the SystemTests that reproduces the error (with a good probability). Once we see that, we can start analyzing and fixing it.

[tulavalle]

Yes, parallelizing at the cear level is very relevant. From what I researched, parallelism is limited at the feature level by the runner, in my case, xunit. Until version 5.0 of Net 5.0 I used specrun, there the parallelism was at the scenario level. Great. However, I needed to change the Net version to 6.0, at the time, as I always need to keep the latest version, but Specrun was discontinued, stopping at Net5.0 I needed to change it and opted for xUnit, however, it and other executors only have parallelism at feature level. A loss. It's been a while since I researched the subject, but I don't think it has changed.

[gasparnagy]

Back in 2019, I wrote a summary about parallel problems. I don't know how much that is relevant but here is the link: SpecFlowOSS/SpecFlow#1535

[Code-Grump]

Relevant: https://microsoft.github.io/coyote/ - a framework for concurrency-testing.

The Coyote testing tool can systematically explore a large number of interleavings of concurrent operations as well as non-deterministic choices so that it covers a large set of behaviors in a very short time.

[gasparnagy]

@obligaron started a PR related to this: #119

[gasparnagy]

Duplicate topic: https://github.com/orgs/reqnroll/discussions/53
I will keep this one as this has more recent conversations.