Request to support Playwright Test and TypeScript

[JulieMatic]

My team is implementing all of our functional UI tests with Playwright Test written in TypeScript. We also currently have a very small subset of these tests which we would like to use for load testing via Artillery. Are there any future plans to support Playwright Test or TypeScript when running tests via Artillery?

Currently, the Playwright Test repository for our functional tests takes advantage of some of the very useful features that Playwright Test has beyond the core Playwright library. For example, we use:

• beforeEach() / beforeAll() / afterEach() / afterAll() methods to handle setup and cleanup in a given spec.ts file
• test.describe() and test() methods to organize tests in a given spec.ts file
• expect() method for assertions
• Test Fixtures to implement our Page Object Model structure and obfuscate some of the class setup from test writers
• Setting different timeouts for long-running spec.ts files

Our current workaround for the lack of Playwright Test support is to store 2 copies of any test that needs to be performed as both a functional UI test and a load test and stripping out any Playwright Test features before compiling the test to JavaScript via a build script. I've seen on other posts that a suggested workaround is to create a function that both the Artillery and Playwright Test tests can call, but this isn't ideal for my team for a few reasons:

• We likely wouldn't be able to reuse a lot of our existing framework because it heavily utilizes things like Test Fixtures and the expect() method
• We are trying to keep our functional UI tests as simple as possible so that test writers who aren't experienced yet in writing code don't need to look at the framework
• This will make it more difficult to maintain our code, as we are essentially storing multiple copies of identical locators and near-identical assertion methods and would need to make any necessary updates in multiple places

[hassy]

hi @JulieMatic 👋

Thanks so much for all the details, it's very helpful for our understanding of where the gaps are with the current Playwright support in Artillery.

We absolutely want to support both Playwright Test and TypeScript. We're still figuring out what that's going to look like exactly.

[JulieMatic]

Hi @hassy 👋

Thanks for the fast reply! Feel free to reach out if you need any more info about how we're implementing our Playwright Test framework or how we're using Artillery.

[hassy]

thank you! your email isn't public on GitHub, but I'd love to learn more - please feel free to drop me a line on h@artillery.io

[Brijendra9322]

Here in typescript for running parallel tests at least it should allow multiple testFunction that run all tests in parallel. which is not the case right now.

scenarios:

• engine: playwright
  testFunction: "test1",
• engine: playwright
  testFunction: "test2"

here if you run the yml file it randomly pic the testFunction and run once. is there any way that we can run both function in parallel?

[ArCiGo]

Following this topic. I'm in the same situation. I want to reuse my existing POM made with TS + Playwright. Is there any update on this?

[philwolstenholme]

Does anyone have any tips for compiling TS tests to JS for Artillery? I tried using tsc on the CLI but had trouble with using the combinations from my TS config file as CLI commands when trying to output to a single file.

[dyllandry]

I feel a bit disappointed. Artillery says you can use playwright, and that you can reuse your smoke tests as load tests. But if you've written your Playwright tests like Playwright recommends, using features of Playwright Test, you can't actually reuse your smoke tests as load tests and Artillery doesn't support Playwright.

Maybe this should be added to Artillery's limitations section of the Playwright engine.

[hassy]

Indeed we don't support @playwright/test yet. It's a completely different project from playwright the automation library, which is what Artillery integrates. Most of the time most of the existing Playwright code can be adapted to be used for load testing with Artillery if it's structured well, e.g using the Page Object Model approach.

[dyllandry]

My bad, I didn't know they were separate projects. I just double checked, the Playwright Getting Started docs don't point out a difference. It makes it seem like they are the same thing.

[hassy]

Yes you're right @dyllandry, @playwright/test is how most people use Playwright. It's perfect for the common use case of running end-to-end tests on some functionality. We do intend to make as much of it supported as possible, but right now only the APIs from the library are supported.

[hassy]

Initial TypeScript support was merged in #2360. It hasn't been released in a stable release of Artillery yet but is available in the canary release for anyone who wants to try it - npm install -g artillery@canary

[HugoImaios]

great news ! Looking forward for the official release!

[philwolstenholme]

This sounds really promising, thank you for your work on this :)

[HugoImaios]

One thing that seems critical to me and has not yet been mentionned here :
without the expect() provided by Playwright Test, we completely lose the auto-waiting system.
Even with Page Object Model or making functions and helpers, containing no expect() calls, in order to have "artillery runnable" scripts, it does not solve this issue, and we end up hard-coding waiting timeout, having flaky tests, and the metrics recap at the end of an artillery load charge became irrelevant

[HugoImaios]

you're welcome, Artillery is a really promising tool, don't mind the very harsh tone of my previous message, I'm just non-native speaker, we do appreciate your work here a lot !

[hassy]

No worries at all, appreciate the feedback!

[bernardobridge]

Hi @HugoImaios and others 👋 . Just to add something here:

While we don't currently officially support or include the @playwright/test library as part of Artillery's Playwright engine, it often should still be possible to use expect in tests, and I believe it correctly fails virtual users. You'd have to install and import the package directly yourself.

So, for example:

const { expect } = require('@playwright/test');

async function checkArtilleryPage(page) {
  await page.goto("https://www.artillery.io");

  //should pass
  await expect(page.getByText("The Artillery Manifesto")).toBeVisible();

  //should fail vu
  await expect(page.getByText("this doesn't exist")).toBeVisible();
}

Again, this isn't officially supported at the moment, but it should mostly work if using Javascript (may not work directly with first iteration of upcoming experimental Typescript support).

[HugoImaios]

oh ! That looks obvious now that you said it 😅
I'll definitely give it a try, mainly to try if auto-waiting is handled (but ofc failing the VUs is nice too)
No worry for the TS support, personally, I've already set a minimal tsc step

[dyllandry]

I tried it out and as far as I can tell the auto waiting works, but artillery configuration values for the playwright engine such as defaultTimeout don't work. My testing wasn't super thorough, but I tried setting expect.timeout in playwright.config.js too but that didn't work. For now I'll use .waitFor() instead of expect() in my artillery flows so that defaultTimeout works in my artillery config.

[bernardobridge]

Artillery v2.0.4 is out with experimental TypeScript support: https://github.com/artilleryio/artillery/releases/tag/artillery-2.0.4

[bernardobridge]

Hey folks 👋

The new Artillery release (v2.0.6) contains improvements to the Typescript support. You should now be able to use expect from @playwright/test within your test scripts.

Some documentation:

• https://www.artillery.io/docs/reference/test-script#preventing-bundling-of-typescript-packages
• https://www.artillery.io/docs/reference/engines/playwright#known-limitations

cc @dyllandry , @HugoImaios , @philwolstenholme , @ArCiGo , @JulieMatic

[HugoImaios]

that's great news ! Thank you folks ! Will definitely give it a new try in a week or two !

[ArCiGo]

Thanks for the notification :)

[gselsidi]

Might be off topic here, but do you have a pageObject example? Because in playwright we have a fixture.ts file which declares all the pageObject pages which extends playwright base pages...

Or do we need to go to a more simpler version, of having pageObject.ts file and importing it into our test.ts?

[ArCiGo]

Out of topic, @bernardobridge .-

I checked again the samples here, and I have some questions about the implementation.

In this sample, the URL is defined twice (in the target property and in the helloFlow() function).

• Why was it added twice?
• Is it mandatory to set a value in the target property?
  • If not, can I get rid of this property?
    • Can this also be applicable not only for Playwright feature but for API feature too?
  • If yes, can I set any URL there taking into consideration that my helloFlow() function will point to the right URL I want to use?

Thanks!

[bernardobridge]

@ArCiGo that sample needs to be updated.

You do have to set target, but Artillery will use that as baseURL since version 2.0.6. Then, you can just set paths

So for example in the sample you linked to , you could just do:
await page.goto('/');

[ArCiGo]

@bernardobridge Thanks for your answer!

I have another question: In my case, we have a library that performs all the API calls to our internal services, but we don't have exposure to the endpoints. This tool does all the calls for us internally. We build the requests in a programmatic way (so, when we execute our scripts, the tool internally performs what we want). Unfortunately, we can't have endpoints to manipulate.

I was doing a basic example of JS + Playwright + Artillery. I created this basic script.-

const { test, expect } = require('@playwright/test');

async function artilleryScript(page) {
    await page.goto('https://www.saucedemo.com/v1/index.html');

    await page.locator('#user-name').fill('standard_user');
    await page.locator('#password').fill('secret_sauce');
    await page.locator('#login-button').click();
    
    await page.waitForURL('**/v1/inventory.html', { timeout: 3000 });
    expect(await page.locator('.product_label')).toBeVisible();

    await page.locator('.bm-burger-button').click();
    await page.locator('#logout_sidebar_link').click();

    await page.waitForURL('**/v1/index.html', { timeout: 3000 });
    await page.locator('.login-box').isVisible();
    // await page.locator('.login-box').isEnabled();
    expect(await page.locator('.login-box')).toBeVisible({ timeout: 5000, visible: true });
}

module.exports = {
    artilleryScript
};

and in my *.yml file, I did this.-

config:
  target: https://www.sample.com
  engines:
    playwright:
      launchOptions:
        headless: false
  processor: "./tests/artillery/login.js"
scenarios:
  - engine: playwright
    testFunction: "artilleryScript"

When I execute the script, I get these results.-

Test run id: <some_random_ID>
Phase started: unnamed (index: 0, duration: 1s) 12:37:01(+0300)

Phase completed: unnamed (index: 0, duration: 1s) 12:37:02(+0300)

--------------------------------------
Metrics for period to: 12:37:10(+0300) (width: 2.324s)
--------------------------------------

browser.http_requests: ......................................................... 25
browser.page.codes.200: ........................................................ 25
vusers.completed: .............................................................. 1
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 2324.7
  max: ......................................................................... 2324.7
  mean: ........................................................................ 2324.7
  median: ...................................................................... 2322.1
  p95: ......................................................................... 2322.1
  p99: ......................................................................... 2322.1


All VUs finished. Total time: 4 seconds

--------------------------------
Summary report @ 12:37:06(+0300)
--------------------------------

browser.http_requests: ......................................................... 25
browser.page.codes.200: ........................................................ 25
vusers.completed: .............................................................. 1
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 2324.7
  max: ......................................................................... 2324.7
  mean: ........................................................................ 2324.7
  median: ...................................................................... 2322.1
  p95: ......................................................................... 2322.1
  p99: ......................................................................... 2322.1

But, if I set in the target the URL I have in the login.js script (https://www.saucedemo.com/v1/index.html), I get more outputs.-

Test run id: t9znw_kdkp5tm76nqgff63634d496qarxet_cpw7
Phase started: unnamed (index: 0, duration: 1s) 12:35:24(+0300)

Phase completed: unnamed (index: 0, duration: 1s) 12:35:25(+0300)

--------------------------------------
Metrics for period to: 12:35:30(+0300) (width: 2.676s)
--------------------------------------

browser.http_requests: ......................................................... 25
browser.page.CLS.https://www.saucedemo.com/v1/index.html:
  min: ......................................................................... 0
  max: ......................................................................... 0
  mean: ........................................................................ 0
  median: ...................................................................... 0
  p95: ......................................................................... 0
  p99: ......................................................................... 0
browser.page.FCP.https://www.saucedemo.com/v1/index.html:
  min: ......................................................................... 112.4
  max: ......................................................................... 112.4
  mean: ........................................................................ 112.4
  median: ...................................................................... 113.3
  p95: ......................................................................... 113.3
  p99: ......................................................................... 113.3
browser.page.LCP.https://www.saucedemo.com/v1/index.html:
  min: ......................................................................... 150.6
  max: ......................................................................... 150.6
  mean: ........................................................................ 150.6
  median: ...................................................................... 149.9
  p95: ......................................................................... 149.9
  p99: ......................................................................... 149.9
browser.page.TTFB.https://www.saucedemo.com/v1/index.html:
  min: ......................................................................... 16.3
  max: ......................................................................... 65
  mean: ........................................................................ 40.7
  median: ...................................................................... 16.3
  p95: ......................................................................... 16.3
  p99: ......................................................................... 16.3
browser.page.codes.200: ........................................................ 25
vusers.completed: .............................................................. 1
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 2675.4
  max: ......................................................................... 2675.4
  mean: ........................................................................ 2675.4
  median: ...................................................................... 2671
  p95: ......................................................................... 2671
  p99: ......................................................................... 2671


All VUs finished. Total time: 4 seconds

--------------------------------
Summary report @ 12:35:30(+0300)
--------------------------------

browser.http_requests: ......................................................... 25
browser.page.CLS.https://www.saucedemo.com/v1/index.html:
  min: ......................................................................... 0
  max: ......................................................................... 0
  mean: ........................................................................ 0
  median: ...................................................................... 0
  p95: ......................................................................... 0
  p99: ......................................................................... 0
browser.page.FCP.https://www.saucedemo.com/v1/index.html:
  min: ......................................................................... 112.4
  max: ......................................................................... 112.4
  mean: ........................................................................ 112.4
  median: ...................................................................... 113.3
  p95: ......................................................................... 113.3
  p99: ......................................................................... 113.3
browser.page.LCP.https://www.saucedemo.com/v1/index.html:
  min: ......................................................................... 150.6
  max: ......................................................................... 150.6
  mean: ........................................................................ 150.6
  median: ...................................................................... 149.9
  p95: ......................................................................... 149.9
  p99: ......................................................................... 149.9
browser.page.TTFB.https://www.saucedemo.com/v1/index.html:
  min: ......................................................................... 16.3
  max: ......................................................................... 65
  mean: ........................................................................ 40.7
  median: ...................................................................... 16.3
  p95: ......................................................................... 16.3
  p99: ......................................................................... 16.3
browser.page.codes.200: ........................................................ 25
vusers.completed: .............................................................. 1
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 2675.4
  max: ......................................................................... 2675.4
  mean: ........................................................................ 2675.4
  median: ...................................................................... 2671
  p95: ......................................................................... 2671
  p99: ......................................................................... 2671

• Can you explain to me the differences between these two outputs?
• Am I missing something about how Artillery works?

For my purposes, I found useful (I'm probably mistaken) to set the target property with any random URL since my internal library performs all the calls to our internal API (with the URL that should be used) and since we can't get endpoints exposed.-

• Can I use this approach?
• Are there any inconveniences of using this approach?

TIA! :D

[bernardobridge]

Hello @ArCiGo can I ask that you please open a separate discussion for this? I'm afraid it's getting very off-topic, which doesn't help people find the right support when looking! Thanks!

[ArCiGo]

Done, @bernardobridge : #2713 :D