Playwright Engine flakiness compared to Playwright Library

[bgratini]

I am attempting to run some playwright test using artillery and whenever I run in Artillery the tests become quite flaky and fail often; but pass consistently when using Playwright library and running in parallel. I am really hoping for some suggestions to either mimic with Playwright standalone, to better debug with trace and or some suggestions to increase stability.

I am using the same version of Playwright/test which the artillery version is using. The test run fine when it is just 2 vusers, but as as soon as I increase to 3, it starts to fail. I have run in CI/CD with Jenkins to limit issues as such as resources, but do not see any difference in result.

I have attempted additional visual waits; slowMo configuration, and adding it expects, and yet it remains inconsistent.

My company really hopes to adopt this for UI focused load testing.
Yaml:

config:
    payload:
      - path: "payloads/{{ $environment }}-logins.csv"
        skipHeader: true
        fields:
          - "email"
          - "password"
    phases:
      - name: Warm up
        duration: 30
        arrivalCount: 3
    environments:
      prod:
        target: "https://example-site.com"
        log: "***************** Target url is is set to: {{ env.target }}"
      int:
        target: "https://example-integration-site.com"
        log: "***************** Target url is is set to: {{ env.target }}"
    variables:
      context: null
      pre-call-topic: null
      post-call-topic: null
      complianceProfile:
      script: null
      question: null
      NR_ACCOUNT_ID: #####
      NR_LICENSE_KEY: #########
      NODE_ENV: "{{ $environment }}"
    plugins:
      metrics-by-endpoint:
        # Group metrics by request name rather than URL:
        useOnlyRequestNames: true
    engines:
      playwright:
        launchOptions:
          headless: true
          permissions: [ 'microphone' ]
        defaultTimeout: 120
    processor: "./e2e-flows.js"
  before:
    flow:
      - log: "====================== STARTING TEST ======================"
      - log: "Current artillery environment is set to: {{ $environment }}"
      - log: "Current node environment is set to: {{ NODE_ENV}}"
      - log: "==========================================================="
  scenarios:
    - name: e2e
      engine: playwright
      testFunction: "e2e"

TEST SCRIPT

const hasCompliance = false;

async function e2e(page, vuContext, events, test) {
  dotenv.config({
    path: path.
        resolve(`${__dirname}/config`,
            `.env.${vuContext.vars.NODE_ENV}`),
  });
  global.page = page;

  const {step} = test;
  const loginPage = new LoginPage();
  const landingPage = new LandingPage();
  const cardContextDescription = null;
  const topic = null;
  const callType = null;
  const profile = null;`


  await step('navigate_to_login_page', async () => {
    await loginPage.navigateTo();
  });

  await step('submit_login', async () => {
    const userid = vuContext.vars.$uuid;
    const email = vuContext.vars.email;
    const password = vuContext.vars.password;

    console.log(`vUser is: ${userid} and email is: ${email} `);
    await loginPage.login(email, password);
    await expect(await landingPage.isLoaded()).toBeTruthy();
  });

  await step('configure_call', async () => {
    await landingPage.selectCallType(callType);
    await landingPage.selectCallContext(cardContextDescription);
    await landingPage.selectPreCallTopic(topic);
    if (hasCompliance) {
      await landingPage.selectComplianceProfile(profile);
    }
  });

[hassy]

Hi @bgratini 👋 When you say "fail" what do you mean exactly? Your config.phases configuration creates only 3 VUs in 30s, which I wouldn't expect to fail due to lack of resources, so it's probably something else.

There's a couple of things you can try:

1. Run the Artillery test locally in headed mode. You can set headless: false under launchOptions to see what each VU is doing:

config:
  engines:
    playwright:
      launchOptions:
        headless: false

2. Sign up for Artillery Cloud (https://app.artillery.io) and try the experimental Playwright tracing support there. Set trace: true in your configuration to make Artillery automatically record traces for failed Playwright VUs:

config:
  engines:
    playwright:
      trace: true

[Beliefg]

Hello, thanks for the comment. Let me clarify my statements on failing; when I run with Playwright Library, I run up to ten parallel instances and they all pass consistently. When I run the same script with Artillery, using shared code repository for Playwright methods, the test regularly encounters issues with clicking on an element, target page closed, or timeouts with a locator. The same ones that running directly with Playwright work consistently.

I will try the trace with Artillery, I was not aware of that feature and update with results. Thank you

[Beliefg]

@hassy
I've updated my yaml file but am not receiving traces:

engines: playwright: trace: true launchOptions: headless: true permissions: [ 'microphone' ] useSeparateBrowserPerVU: true defaultTimeout: 120

[vikramjb]

@Beliefg The trace flag works when you push the results to artillery's cloud service. Here's some documentation on the cloud service

https://www.artillery.io/docs/cloud/get-started