Ensure plugin add-on for supporting thresholds per scenario

[AlexGoja]

Hi, I've developed an enhancement to the ensure plugin so that it supports thresholds per scenario rather than having just one globally defined threshold that applies to all scenario as we have a need for that specific ability in our company. Would still need to add tests, but I read that all additional changes should be done through the discussion channel or issue tracker.

This is the fork for the changes mentioned, would be good to know whether it is feasible to be merged assuming that tests will be added.

[hassy]

sounds interesting @AlexGoja I'll take a look at the fork!

[AlexGoja]

@hassy Cheers. Added a test for the ensure and fixed one existing test as it was relying on the log output lines since a new line Scenario: <scenario name> was introduced to know for which scenario the threshold was ran for (added an example output).

 Checks:
./test/index.spec.js 1> Scenario: Scenario1
./test/index.spec.js 1> ok: vusers.failed < 5
./test/index.spec.js 1> ok: http.response_time.p99 < 1500
./test/index.spec.js 1> ok: http.response_time.p95 < 2000
./test/index.spec.js 1> ok: http.response_time.p75 < 1000
./test/index.spec.js 1> Scenario: Scenario2
./test/index.spec.js 1> ok: vusers.failed < 5
./test/index.spec.js 1> ok: http.response_time.p99 < 900
./test/index.spec.js 1> ok: http.response_time.p95 < 800
./test/index.spec.js 1> ok: http.response_time.p75 < 500

Let me know if I missed something or if anything else needs to be done to proceed to create a PR. Cheers.

[AlexGoja]

Hey @hassy any thoughts on the above?

[hassy]

Just had a look @AlexGoja, sorry it took a while. I'm not sure I understand how the change is supposed to work. In the example script here:

main...AlexGoja:artillery:main#diff-4fce84e3747d4bd52aa7990bc17538c980ceb60f6a9a39a41644672177b5558aR15

Artillery does not report per-scenario HTTP metrics, so there is no way to distinguish between http.response_time.p75 for Scenario 1, and http.response_time.p75 for Scenario 2. The only way this could work is if you run one scenario at a time (with the --scenario-name flag for example), but the example test case is running both scenarios at the same time:

main...AlexGoja:artillery:main#diff-b6fe17aaa662a2e9e48111bd49b50b7949c80756a9c08871131f1fac10096521R12

Are there some missing commits in your branch?