Hardcoded scenario (pickle) order

[hovi]

🤔 What's the problem you've observed?

The library has only 3 hardcoded orderings "lexical", "reverse" and "random" (with possible seed parameter).
The library is not suited for extending this.

✨ Do you have a proposal for making it better?

Probably the simplest way to do it with the least amount of code affected by this, is to try to interpret this string as a fully qualified class name and try to create instance of the class in PickleOrderParser.parse.

Of course it would be better to allow customizable factory, but I don't know, how much would that affect the whole ecosystem.

📚 Any additional context?

I would like to give specific tags order preference over others and then give lexical ordering within the same tag.
The relevant project is here:
https://github.com/iv4xr-project/iv4xr-se-plugin/

Happy to create PR if this makes sense to you.

[mpkorstanje]

A PR would be welcome.

You can try to interpret the ordering string as a fully qualified class name.

Though rather then using this string to load a class I would prefer a solution based on Service Provider Interfaces. It hides all problems associated with class loading and should play nicely with modules in the future. The PickleOrder class could be the service interface. For inspiration consider looking at ObjectFactoryServiceLoader.

[jkronegg]

I don't see the need for this new feature.

All test scenarios should be independent. Thus, there is no need for specific execution order.

The default execution order should be stable from one run to the other, so that reports could be compared between runs (which is the case with "lexical"). I like the idea of the "random" execution order to ease detection of dependent scenarios (#1645).

By adding scenario running order possibilities, we encourage the bad practice of having dependent test scenarios. Thus, from this point of view, we should not implement this proposal.

As side comments:

• I don't even see the need to have the "reverse" order. I would suggest to deprecate it.
• if execution order is required for reporting, we could even argue that the ordering operation is a responsibility of the plugin (not the cucumber engine responsibility).

[mpkorstanje]

All test scenarios should be independent. Thus, there is no need for specific execution order.

There are uses for ordering that are not related to scenario dependence. For example running slow scenarios first, or frequently failing ones first, recently changed first, ect.

Especially when combined with a fail-fast (which Cucumber also doesn't have yet) these help getting feedback faster.

Since we can't quite predict which of these people will want I reckon it is useful to provide an extension point.

[hovi]

We have a couple of reasons:

• Our system is very expensive to reset and sometimes non-deterministic
• Some of these tests indeed affect each other so reset is necessary afterwards, completely random execution can cause failures
• Some tests can be run repeatedly without reset
• Some tests can be run only once and then need reset
• We reflect all this with tags
• We can then optimize ordering of tests (while still randomize it to some degree). Even then our test suite takes hours.
• If we were to do "independent test scenarios", tests would take almost a day due to reset times

[jkronegg]

Thanks for clarification. Indeed, very specific needs imply custom sort capability.
@hovi For curiosity : what is your Cucumber volumetry (in terms of parameter types, step definitions, features, rules, test scenarios) ?