2.0.11 Feedback - Playwright Errors in Report and Console

[bshostak]

Hello, this isn't really a "bug" per se (idk maybe it is) but I upgraded our package to the newest artillery version to checkout the new errors feature and I had some questions.

1. When an expect fails we see this locally:

Locator: getByLabel('bbb').first()
Expected: visible
Received: hidden
Call log:
  - locator._expect with timeout 5000ms
  - waiting for getByLabel('bbb').first()

But in Artillery Cloud we only see:
pw_failed_assertion.toBeVisible 1 (attached image)

This is fine but it would be awesome if there is an error like this we can see the Trace associated with it to reference what exactly failed (this run did not record a trace). If not, is there a way the errors can include a little more details like what I provided? It allows for easier troubleshooting to see what expect or what playwright action failed specifically (since the Call Log tells me exactly what locator did not appear).

I also wanted to add that the actual errors that appear in the terminal while running artillery do not actually appear in the worker logs. For example:

1. I run artillery script
2. the expect fails and I see it in the terminal
3. after the run I go to the worker logs but i do not see the error display there.

This isn't an issue if running locally, but in aws we do not have a way to go back and see the terminal output and we rely on the worker logs for all those details.

If there is something I should be doing differently please let me know. Thanks!

5.0.11

Running this command:

npx artillery run myconfig.yml

I expected to see this happen:

maybe more details and/or trace

Instead, this happened:

not very clear where the error happened

My specific code at a certain step in the script is doing this:

      await expect(this.page.getByLabel(name).first(), 'Shipping Options did not appear').toBeVisible({ timeout: this.expectTimeout });
      await this.page.getByLabel(name).first().check();
      console.log('selected shipping option');
      await this.shippingOptionNextButton.click();
      console.log('clicked next after adding shipping');
    

Which allows us to ensure we see the shipping options appear before continuing. However, if I have multiple expects that do something similar, the logs that output in cloud and summary report will not tell me which specific step or expect failed. Only that there was a failure.

Other thoughts:

Here is a snippet of the summary report:

browser.step.Select Shipping Method:
--
29-Apr-2024 11:03:00 | min: ......................................................................... 3039
29-Apr-2024 11:03:00 | max: ......................................................................... 7642
29-Apr-2024 11:03:00 | mean: ........................................................................ 4540.5
29-Apr-2024 11:03:00 | median: ...................................................................... 4492.8
29-Apr-2024 11:03:00 | p95: ......................................................................... 6439.7
29-Apr-2024 11:03:00 | p99: ......................................................................... 6569.8
29-Apr-2024 11:03:00 | errors.Place Order button is still disabled after 10 seconds: .................. 2
29-Apr-2024 11:03:00 | errors.pw_failed_assertion.toBeVisible: ........................................ 4
29-Apr-2024 11:03:00 | vusers.completed: .............................................................. 34
29-Apr-2024 11:03:00 | vusers.created: ................................................................ 40
29-Apr-2024 11:03:00 | vusers.created_by_name.Artillery Cart Checkout: ................................ 40
29-Apr-2024 11:03:00 | vusers.failed: ................................................................. 6
29-Apr-2024 11:03:00 | vusers.session_length:

You'll notice that there is a errors.Place Order button is still disabled after 10 seconds. This is a log thrown from a line in my code where i do throw new Error(). (btw, thank you for this feature, I love seeing it in the summary)
It would be cool if the errors.pw_failed_assertion.toBeVisible could include the message that is given to the expect method like so:

await expect(this.page.getByLabel(name).first(), 'Shipping Options did not appear').toBeVisible({ timeout: this.expectTimeout });

could look like: errors.pw_failed_assertion_Shipping Options did not appear.toBeVisible

[bernardobridge]

Hey @bshostak 👋

Thanks for the feedback, and glad it's being useful! It's always good for us to get this feedback, especially so early into people using the new feature.

For some context, this feature was designed intentionally like that for a couple of reasons:

• For now we wanted to have some balance between too little information (e.g. not even knowing what the assertion type was), and too much information. Playwright errors can vary quite a bit, and we felt it wasn't always obvious what we should add, and could also lead to e.g. very unpredictably long error names.
• What we recommend is to pair this with our Playwright Tracing feature, which then will allow you to dig deeper into the issue. We're already working on improving Playwright Tracing too, so updates to that should help here.

Regardless of the above, we're also actively discussing internally how to improve the error experience in Artillery and Artillery Cloud, so we'll take this feedback into account! The feature was meant to be a first small iteration toward that, so you can expect we'll be working on more. If you have any other ideas or feedback, feel free to let us know.

---

P.S. Converted to a discussion instead of a bug.

[bshostak]

@bernardobridge Thank you! You guys are the best.