Skip to content

E2E Testing (Fabric)

Jump to bottom
Chiara Mooney edited this page Dec 18, 2023 · 5 revisions

End-to-End Testing (Fabric)

Here is the documentation for the E2E test app for the new Fabric architecture.

For information on our testing progress, see E2E Testing Progress

e2e-test-app-fabric project structure

A test app, test library and test cases are in /packages/e2e-test-app-fabric/, and are organized as below.

  • test – includes Jest tests using webdriverio
  • test\__snapshots__ - holds the snapshot files for the Jest Snapshots tests and tree dump tests
  • windows – the native win32 app running on the new Fabric architecture
  • jest.config.js – configuration for Jest and WebDriverIO
  • jest.setup.js - setup file for Jest

Running Tests

Install WinAppDriver v1.2.1

This will be automatically done for you if you use the RNW dependency script with rnwDev as an argument.

The E2E tests assume it is installed in the default location under C:\Program Files (x86)\Windows Application Driver\WinAppDriver.exe. If you choose a different path, you will need to specify it in /packages/e2e-test-app-fabric/jest.config.js.

module.exports = {
...
  testEnvironmentOptions: {
    ...
    winAppDriverBin: 'D:\\Program Files (x86)\\Windows Application Driver\\WinAppDriver.exe',
  },
};

Build the native test app

C:\repo\react-native-windows> cd packages\e2e-test-app-fabric

C:\repo\react-native-windows\packages\e2e-test-app-fabric> yarn windows --no-launch --no-deploy

If you want to run the E2E test app, either run via Visual Studio or launch the executable for the app in the `windows[x86|x64][Release|Debug]\RNTesterApp-Fabric directory.

Running all tests

⚠ Set your monitor's display scale to 100%. Windows machines often default to 150%. This is needed in order for the component size and offset data match CI.

C:\repo\react-native-windows\packages\e2e-test-app-fabric> yarn e2etest

Running a specific test

⚠ Only the test filename (without the rest of the path) should be included.

C:\repo\react-native-windows\packages\e2e-test-app-fabric> yarn e2etest visitAllPages.test.ts

Break on app start

C:\repo\react-native-windows\packages\e2e-test-app-fabric> yarn e2etest:debug visitAllPages.test.ts

Update Snapshots

C:\repo\react-native-windows\packages\e2e-test-app-fabric> yarn e2etest:updateSnapshots

or

Copy the "snapshots" artifact uploaded from a failing PR into the test\__snapshots__ directory.

⚠ Note if you are seeing unexpected changes in the size/offset data within the snapshots, make sure your monitor's display scale to 100%. Windows machines often default to 150%. This is needed in order for the component size and offset data match CI.

Debugging E2E Tests in CI

Increasing verbosity

By default the only messages printed during tests are related to errors returned by WinAppDriver or assertion failures. It is possible to increase verbosity to show individual WebDriver wire commands by editing /packages/e2e-test-app-fabric/jest.config.js.

module.exports = {
...
  testEnvironmentOptions: {
    ...
    webdriverOptions: {
      // Level of logging verbosity: trace | debug | info | warn | error
      logLevel: 'error',
      ...
    },
  },
};

Test Artifacts

If you have access to the AzureDevOps pipeline you'll be able to see test failures and debug crashes. Here are the artifacts that are produced during the build:

  • error screenshots of the app when a test failed
  • test run XML - this contains some information like the name of the wdio test that failed and the JS stack
  • tree dump outputs - you can compare these to the output of the main branch to see if there is a the difference responsible for the test failing.
  • crash dumps of the e2e test app (RNTesterApp)

You can access these by going to the AzureDevOps run for your PR and clicking on the artifacts link:

image

Architecture

WinAppDriver

WinAppDriver uses UIA to inspect and manipulate a test application, allowing clients to connect to it using the w3c WebDriver protocol.

WebDriverIO

WebDriverIO is a JavaScript library which connects to WinAppDriver using the WebDriver protocol. It provides global APIs for querying the application UI tree such as selectors (see below) and a global browser object to manipulate the test application.

Jest

Jest is the test runner used for end-to-end testing, including assertsion libraries, test selection, etc. WebDriverIO setup is provided by a custom environment @react-native-windows/automation.

Authoring Tests

Tests are written using Jest and WebDriverIO (see more below) against a test app running RNTester. RNTester example pages are used as Test UI, which is examined via Jest tests. Tests should attempt to target an existing RNTester page, but "Legacy" tests exist as custom pages only used by e2e-test-app

Writing a test against an RNTester example page

// LegacyControlStyleTestPage.test.ts

describe('FancyWidget', () => {

  beforeAll(async () => {
    await goToComponentExample('FancyWidget');
  });

  test('FancyWidget is populated with placeholder', async () => {
    // Query for an element with accessibilityId of "foo" (see "locators" below)
    const field = await app.findElementByTestID('foo');
    expect(await field.getText()).toBe('placeholder');
  });

});

Adding a Custom RNTester Page or Sample

New examples may be integrated into the Windows fork of RNTester, @react-native-windows/tester. If you are looking to add a sample to a RNTester page that doesn't have a *.windows.js fork in the @react-native-windows/tester directory. You can create a newly forked page using react-native-platform-override.

When adding a new sample, make sure the component you are looking to test has its testID property set. This property will be used by WinAppDriver to identify the component. To test that your new sample has its testID set correctly:

  1. Launch the E2E test app and the Accessibility Insights app.
  2. Click on the "Live Inspect" tab and hover over your new sample.
  3. You should see the sample's testID as the component's automationId property.

⚠ View components are not accessible by default. If you are looking to add a test for a View component, it must have it's accessible property set to true. Verify the control can be accessed by WinAppDriver using the same steps as above.

In many cases, it may be useful to upstream the new sample. Feel free to submit a PR to React Native to have your sample in react-native and reduce forking.

If you are looking to author a new test page. Below is a code sample of how to add a page. Hooks are recommended to author the test page. (see https://reactjs.org/docs/hooks-intro.html and this Pluralsight course to learn more about Hooks)

// ControlStyleTestPage.tsx
export const title = 'LegacyControlStyleTest';
export const description = 'Legacy e2e test for Control Styles';
export const examples = [
  {
    render: function(): JSX.Element {
      return <ControlStyleTestPage />;
    },
  },
];
  // RNTesterList.windows.js
  ...
  {
    key: 'LegacyControlStyleTest',
    module: require('../examples-win/LegacyTests/ControlStyleTestPage'),
  },
  ...

Note after adding a new RNTester page or making an edit to an existing RNTester page you must rerun yarn build from the root of the repository and rebuild the e2e-test-app-fabric in order for your changes to be applied to the app and your next test run.

Writing Snapshot tests

E2ETests uses snapshot tests to detect unintended visual changes. Instead of testing against the react tree, e2e-test-app-fabric compares the Windows Composition visual tree and the UIA tree. This allows testing the correctness of ComponentViews.

The dumpVisualTree command dumps both the Windows Composition tree for the control and its UIA tree. See E2E Testing Progress for information on which properties are included within the tree dumps. Note: The tree dumps do not include the full tree of properties.

import {dumpVisualTree} from '@react-native-windows/automation-commands';

test('Example test', async () => {
  const dump = await dumpVisualTree('test-id-here');
  expect(dump).toMatchSnapshot();
});

Snapshots can be updated by either:

  1. Locally running yarn e2etest:updateSnapshots.
  2. Copying from the "snapshots" artifact uploaded from a failing PR into the test\__snapshots__ directory.

⚠ Note if you are seeing unexpected changes in the size/offset data within the snapshots, make sure your monitor's display scale to 100%. Windows machines often default to 150%. This is needed in order for the component size and offset data match CI.

Writing Functional Tests

To write functional tests we will make use of WebDriverIO/WinAppDriver API's. To see the full list of API's supported in the E2E test app see [AutomationElement APIs].

Use setValue()/getText() for TextInput Testing

The WebDriverIO APIs of setValue() and getText() can be used to test out the functionality of TextInput components. Here is a sample:

const component = await app.findElementByTestID('test-id');
await component.waitForDisplayed({timeout: 5000});
const dump = await dumpVisualTree('test-id');
expect(dump).toMatchSnapshot();
await app.waitUntil(
  async () => {
  await component.setValue('Hello World');
  return (await component.getText()) === 'Hello World';
},
{
  interval: 1500,
  timeout: 5000,
  timeoutMsg: `Unable to enter correct text.`,
},
);
expect(await component.getText()).toBe('Hello World');

Note WebDriverIO can sometimes input keyboard input too quickly leading to mistypes. For example, the TextInput in the above sample would read "Helol World" instead of "Hello World". To avoid test failures from this case, we have Jest retry the input until a timeout is reached. If the TextInput still does not display the correct text at the end of the timeout we can confidently say there is an issue with the TextInput control.

Use click() and dumpVisualTree() for Visual State Change Testing on Components

The click() and dumpVisualTree() APIs can be used to test how a component visually changes following an onPress event. Here is a sample:

 const searchBox = await app.findElementByTestID('example_search');
    await app.waitUntil(
      async () => {
        await searchBox.setValue("Cha");
        return (await searchBox.getText()) === "Cha";
      },
      {
        interval: 1500,
        timeout: 5000,
        timeoutMsg: `Unable to enter correct search text into test searchbox.`,
      },
    );
const component = await app.findElementByTestID('test-id');
await component.waitForDisplayed({timeout: 5000});
const dump = await dumpVisualTree('test-id');
expect(dump).toMatchSnapshot();
component.click();
const dump2 = await dumpVisualTree('test-id');
expect(dump2).toMatchSnapshot();
await app.waitUntil(
  async () => {
  await searchBox.setValue(['Backspace', 'Backspace', 'Backspace']);
    return (await searchBox.getText()) === "";
  },
  {
    interval: 1500,
    timeout: 5000,
    timeoutMsg: `Unable to enter correct search text into test searchbox.`,
  },
);

⚠ When using the click() API you must make sure that the component you are clicking on can be seen within the app window. If the component is currently offscreen (for example the component lives inside of a scrollview and is further down), WebDriverIO will try to click on the component at its offscreen location and end up un-focusing the app window. This causes a bunch of testing errors. Make sure to search for the example you are testing when the sample set is longer than the height of the app window.

Clone this wiki locally