Understanding Artillery's target property configuration and metrics (Case: Custom library for internal API calls).

[ArCiGo]

Following this previous conversation with @bernardobridge , but in a proper place.-

We have a library that performs all the API calls to our internal services, but we don't have exposure to the endpoints. This tool does all the calls for us internally. We build the requests in a programmatic way (so, when we execute our scripts, the tool internally performs what we want). Unfortunately, we can't have endpoints to manipulate.

I was doing a basic example of JS + Playwright + Artillery. I created this basic script.-

const { test, expect } = require('@playwright/test');

async function artilleryScript(page) {
    await page.goto('https://www.saucedemo.com/v1/index.html');

    await page.locator('#user-name').fill('standard_user');
    await page.locator('#password').fill('secret_sauce');
    await page.locator('#login-button').click();
    
    await page.waitForURL('**/v1/inventory.html', { timeout: 3000 });
    expect(await page.locator('.product_label')).toBeVisible();

    await page.locator('.bm-burger-button').click();
    await page.locator('#logout_sidebar_link').click();

    await page.waitForURL('**/v1/index.html', { timeout: 3000 });
    await page.locator('.login-box').isVisible();
    // await page.locator('.login-box').isEnabled();
    expect(await page.locator('.login-box')).toBeVisible({ timeout: 5000, visible: true });
}

module.exports = {
    artilleryScript
};

and in my *.yml file, I did this.-

config:
  target: https://www.sample.com
  engines:
    playwright:
      launchOptions:
        headless: false
  processor: "./tests/artillery/login.js"
scenarios:
  - engine: playwright
    testFunction: "artilleryScript"

When I execute the script, I get these results.-

Test run id: <some_random_ID>
Phase started: unnamed (index: 0, duration: 1s) 12:37:01(+0300)

Phase completed: unnamed (index: 0, duration: 1s) 12:37:02(+0300)

--------------------------------------
Metrics for period to: 12:37:10(+0300) (width: 2.324s)
--------------------------------------

browser.http_requests: ......................................................... 25
browser.page.codes.200: ........................................................ 25
vusers.completed: .............................................................. 1
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 2324.7
  max: ......................................................................... 2324.7
  mean: ........................................................................ 2324.7
  median: ...................................................................... 2322.1
  p95: ......................................................................... 2322.1
  p99: ......................................................................... 2322.1


All VUs finished. Total time: 4 seconds

--------------------------------
Summary report @ 12:37:06(+0300)
--------------------------------

browser.http_requests: ......................................................... 25
browser.page.codes.200: ........................................................ 25
vusers.completed: .............................................................. 1
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 2324.7
  max: ......................................................................... 2324.7
  mean: ........................................................................ 2324.7
  median: ...................................................................... 2322.1
  p95: ......................................................................... 2322.1
  p99: ......................................................................... 2322.1

But, if I set in the target the URL I have in the login.js script (https://www.saucedemo.com/v1/index.html), I get more outputs.-

Test run id: t9znw_kdkp5tm76nqgff63634d496qarxet_cpw7
Phase started: unnamed (index: 0, duration: 1s) 12:35:24(+0300)

Phase completed: unnamed (index: 0, duration: 1s) 12:35:25(+0300)

--------------------------------------
Metrics for period to: 12:35:30(+0300) (width: 2.676s)
--------------------------------------

browser.http_requests: ......................................................... 25
browser.page.CLS.https://www.saucedemo.com/v1/index.html:
  min: ......................................................................... 0
  max: ......................................................................... 0
  mean: ........................................................................ 0
  median: ...................................................................... 0
  p95: ......................................................................... 0
  p99: ......................................................................... 0
browser.page.FCP.https://www.saucedemo.com/v1/index.html:
  min: ......................................................................... 112.4
  max: ......................................................................... 112.4
  mean: ........................................................................ 112.4
  median: ...................................................................... 113.3
  p95: ......................................................................... 113.3
  p99: ......................................................................... 113.3
browser.page.LCP.https://www.saucedemo.com/v1/index.html:
  min: ......................................................................... 150.6
  max: ......................................................................... 150.6
  mean: ........................................................................ 150.6
  median: ...................................................................... 149.9
  p95: ......................................................................... 149.9
  p99: ......................................................................... 149.9
browser.page.TTFB.https://www.saucedemo.com/v1/index.html:
  min: ......................................................................... 16.3
  max: ......................................................................... 65
  mean: ........................................................................ 40.7
  median: ...................................................................... 16.3
  p95: ......................................................................... 16.3
  p99: ......................................................................... 16.3
browser.page.codes.200: ........................................................ 25
vusers.completed: .............................................................. 1
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 2675.4
  max: ......................................................................... 2675.4
  mean: ........................................................................ 2675.4
  median: ...................................................................... 2671
  p95: ......................................................................... 2671
  p99: ......................................................................... 2671


All VUs finished. Total time: 4 seconds

--------------------------------
Summary report @ 12:35:30(+0300)
--------------------------------

browser.http_requests: ......................................................... 25
browser.page.CLS.https://www.saucedemo.com/v1/index.html:
  min: ......................................................................... 0
  max: ......................................................................... 0
  mean: ........................................................................ 0
  median: ...................................................................... 0
  p95: ......................................................................... 0
  p99: ......................................................................... 0
browser.page.FCP.https://www.saucedemo.com/v1/index.html:
  min: ......................................................................... 112.4
  max: ......................................................................... 112.4
  mean: ........................................................................ 112.4
  median: ...................................................................... 113.3
  p95: ......................................................................... 113.3
  p99: ......................................................................... 113.3
browser.page.LCP.https://www.saucedemo.com/v1/index.html:
  min: ......................................................................... 150.6
  max: ......................................................................... 150.6
  mean: ........................................................................ 150.6
  median: ...................................................................... 149.9
  p95: ......................................................................... 149.9
  p99: ......................................................................... 149.9
browser.page.TTFB.https://www.saucedemo.com/v1/index.html:
  min: ......................................................................... 16.3
  max: ......................................................................... 65
  mean: ........................................................................ 40.7
  median: ...................................................................... 16.3
  p95: ......................................................................... 16.3
  p99: ......................................................................... 16.3
browser.page.codes.200: ........................................................ 25
vusers.completed: .............................................................. 1
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 2675.4
  max: ......................................................................... 2675.4
  mean: ........................................................................ 2675.4
  median: ...................................................................... 2671
  p95: ......................................................................... 2671
  p99: ......................................................................... 2671

• Can somebody explain to me the differences between these two outputs?
• Am I missing something about how Artillery works?

For my purposes, I found useful (I'm probably mistaken) to set the target property with any random URL since my internal library performs all the calls to our internal API (with the URL that should be used) and since we can't get endpoints exposed.-

• Can I use this approach?
• Are there any inconveniences of using this approach?

TIA! :D

[bernardobridge]

Hi @ArCiGo 👋 ,

Thanks for opening this in its own issue and apologies it took me a bit to get back to you.

But, if I set in the target the URL I have in the login.js script (https://www.saucedemo.com/v1/index.html), I get more outputs.-
Can somebody explain to me the differences between these two outputs?

So, the reason this happens is that target is needed in order to emit web vital metrics (metrics are listed here in our docs). So the additional metrics are web vital metrics from hitting https://www.saucedemo.com/v1/index.html. For example, the mean of browser.page. LCP.https://www.saucedemo.com/v1/index.html is telling you that on average it took 150.6 ms for the largest image/text block to be visible in that page.

If there's no target, Artillery won't know in reference to what page to report these metrics.

For my purposes, I found useful (I'm probably mistaken) to set the target property with any random URL since my internal library performs all the calls to our internal API (with the URL that should be used) and since we can't get endpoints exposed.

If you aren't interested in the web vital metrics, it's fine to do that, but there's also no harm in them being reported?

If you're interested in measuring backend timings, while there isn't a direct way to do that, you can break down your flow into steps and measure the individual parts of the user journey that might hit your backend. For example login -> search. We have an example here: https://www.artillery.io/docs/reference/engines/playwright#teststep-argument

---

Let me know if the above makes sense and if you have any other questions!