Questions regarding artillery integration with Playwright and integration with AWS Fargate

[Romarionijim]

Hello - I've been diving into artillery documentation especially with it's integration with Playwright.

I wanted to ask a few general questions to understand few things regarding artillery in general, it's integration with playwright and AWS.

Question 1 (Integration with Playwright page API)

• I'm relatively new to performance and load testing, I've been using Playwright for 3 years mostly for UI end to end testing and also for API testing, I wanna make sure that I understand artillery docs and it's usage, the docs says that it has full access to playwright page APIs, so the recommendation is to perform the load via the UI in order to perform and hit different APIs on the backend instead of only testing out the APIs? i.e does the integration hit both frontend and backend if performing the tests via the UI?

Question 2:

• If i wanna test the load on the backend only, in this case I should use a different engine such as HTTP?

Question 3:

• how can I generate reports after running artillery tests both using Playwright or other engines? I did not quite see how to generate report locally, I only saw how to generate report when running a test via CI e.g github actions but I wanna know how can I generate a report locally as well especially with playwright and also without. (I'm not sure if the example of generating a report via github actions used playwright though) - I would appreciate adding it to the documentation.

Question 4:

• regarding running the tests on AWS Fargate, it says in the docs that it generates a report from the individual containers but where do these report go exactly? do they get dumped into S3? or somewhere else? I was not sure how to refer to these reports after a test run.

I would really appreciate if you could address these questions, the tool looks great!

Thank you!

[bernardobridge]

👋 Hope this helps:

Playwright tests

I'm relatively new to performance and load testing, I've been using Playwright for 3 years mostly for UI end to end testing and also for API testing, I wanna make sure that I understand artillery docs and it's usage, the docs says that it has full access to playwright page APIs, so the recommendation is to perform the load via the UI in order to perform and hit different APIs on the backend instead of only testing out the APIs? i.e does the integration hit both frontend and backend if performing the tests via the UI?

The integration will run a Playwright test under the hood, so it will work the same as your UI tests - it will hit the UI pages you tell it to. Then the page(s) might be making calls to your backend, and in that sense you are testing your backend too (but without having to recreate all the backend calls that happen in a single page visit).

What Artillery does is do this at scale - so you can recreate for example 100 users arriving at your website at the same time.

Backend testing

If i wanna test the load on the backend only, in this case I should use a different engine such as HTTP?

Yes, or whatever your backend is using (we also support other protocols). The difference here as I said above is you'd have to recreate the backend flow, but perhaps you're interested in testing single endpoints or specific flows without the UI.

Reporting

how can I generate reports after running artillery tests both using Playwright or other engines? I did not quite see how to generate report locally, I only saw how to generate report when running a test via CI e.g github actions but I wanna know how can I generate a report locally as well especially with playwright and also without. (I'm not sure if the example of generating a report via github actions used playwright though) - I would appreciate adding it to the documentation.

Using the --output flag gives you a JSON report (https://www.artillery.io/docs/reference/cli/run#--output---create-a-json-report) which contains the summary (aggregate) as well as intermediate reports.

For an HTML report summary, we recommend Artillery Cloud, where you'll have a lot of enhanced features like dedicated Playwright graphs, note taking, sharing with stakeholders, etc. You can sign up at https://app.artillery.io/ and there's a generous free tier.

Reporting with Fargate

The same --output flag works for Fargate and generates the JSON report (aggregated over all Fargate containers you use) in the machine you triggered the test from.

Again, we would recommend Artillery Cloud for advanced reporting with Fargate which also works in the same way.

[Romarionijim]

@bernardobridge so the html report is only available on artillery cloud as I understand?
regarding the pricing - the tool itself is open source but the artillery cloud is commercial? because it says Artillery CLI is open source, which includes the tests and code that we discussed earlier I assume expect for Artillery cloud