[Feature Request] Configurable playwright trace sampling in Artillery Cloud

[bshostak]

Wasn't sure where to put feature requests so feel free to close this and direct me elsewhere.

Story
While using Artillery Cloud with Fargate runs, it is a great feature to see playwright trace reports for failed VUsers. I chatted with Hassy when I noticed that it wasn't logging the traces for all of the failed VUsers. He let me know that Artillery Cloud only records a subset of failing VUs as failures from around the same time during a test (since they are likely the same error).

I'd like to request an option to output all traces from all failed users and/or the option to configure the sampling recording.

My reasoning is related to a more robust user flow such as a checkout:

1. go to shopping page
2. add products to cart
3. click checkout
4. enter in customer info
5. enter in shipping info
6. enter in payment info
7. place order
8. get confirmation

If we are sending a large load and we get one or two users that fail at say step 3, it would be nice to see what that experience looked like for those particular VUsers (even though the vast majority got through). There any many service and micro-service calls that could have thrown network and/or console errors at that time that could provide more in-depth analysis from developers.

thanks again for creating an amazing tool!

[hassy]

Thanks for the extra context @bshostak! we'll add a way to control how many traces get recorded. The scenario we want to avoid (at least with default settings) is when a large load test with lots of failing VUs results in hundreds or thousands of identical traces being uploaded.

[bshostak]

Thanks for the extra context @bshostak! we'll add a way to control how many traces get recorded. The scenario we want to avoid (at least with default settings) is when a large load test with lots of failing VUs results in hundreds or thousands of identical traces being uploaded.

Thanks! That is very valid and I understand if due to that scenario this feature is not added. Because you are right, someone could end up generating a whole ton of those :(

[sahmeepee]

I can see that this would be useful. Could you add a user-configurable "maxTraces" setting with a limit to how high people could set it?

[HugoImaios]

Maybe comparing ("uniquifying" in fact) playwright traces by another way than time of ending would give better experience.
Just a very naive 2 cents here : the exact playwright errors log (sometime there is more than 1) ? or even the md5 of the zip file ? just throwing ideas :)