id | title |
---|---|
ci |
Continuous Integration |
Playwright tests can be executed in CI environments. We have created sample configurations for common CI providers.
3 steps to get your tests running on CI:
-
Ensure CI agent can run browsers: Use our Docker image in Linux agents or install your dependencies using the CLI.
-
Install Playwright:
# Install NPM packages npm ci # or npm install # Install Playwright browsers and dependencies npx playwright install --with-deps
pip install playwright playwright install --with-deps
mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="install --with-deps"
dotnet build pwsh bin/Debug/netX/playwright.ps1 install --with-deps
-
Run your tests:
npx playwright test
pytest
mvn test
dotnet test
- langs: js
We recommend setting workers to "1" in CI environments to prioritize stability and reproducibility. Running tests sequentially ensures each test gets the full system resources, avoiding potential conflicts. However, if you have a powerful self-hosted CI system, you may enable parallel tests. For wider parallelization, consider sharding - distributing tests across multiple CI jobs.
import { defineConfig, devices } from '@playwright/test';
export default defineConfig({
// Opt out of parallel tests on CI.
workers: process.env.CI ? 1 : undefined,
});
The Command line tools can be used to install all operating system dependencies on GitHub Actions.
name: Playwright Tests
on:
push:
branches: [ main, master ]
pull_request:
branches: [ main, master ]
jobs:
test:
timeout-minutes: 60
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: 18
- name: Install dependencies
run: npm ci
- name: Install Playwright Browsers
run: npx playwright install --with-deps
- name: Run Playwright tests
run: npx playwright test
- uses: actions/upload-artifact@v3
if: always()
with:
name: playwright-report
path: playwright-report/
retention-days: 30
name: Playwright Tests
on:
push:
branches: [ main, master ]
pull_request:
branches: [ main, master ]
jobs:
test:
timeout-minutes: 60
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r local-requirements.txt
pip install -e .
- name: Ensure browsers are installed
run: python -m playwright install --with-deps
- name: Run your tests
run: pytest
name: Playwright Tests
on:
push:
branches: [ main, master ]
pull_request:
branches: [ main, master ]
jobs:
test:
timeout-minutes: 60
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: '17'
- name: Build & Install
run: mvn -B install -D skipTests --no-transfer-progress
- name: Install Playwright
run: mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="install --with-deps"
- name: Run tests
run: mvn test
name: Playwright Tests
on:
push:
branches: [ main, master ]
pull_request:
branches: [ main, master ]
jobs:
test:
timeout-minutes: 60
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup dotnet
uses: actions/setup-dotnet@v3
with:
dotnet-version: 6.0.x
- run: dotnet build
- name: Ensure browsers are installed
run: pwsh bin/Debug/netX/playwright.ps1 install --with-deps
- name: Run your tests
run: dotnet test
- langs: js
GitHub Actions supports sharding tests between multiple jobs using the jobs.<job_id>.strategy.matrix
option. The matrix
option will run a separate job for every possible combination of the provided options. In the example below, we have 2 project
values, 10 shardIndex
values and 1 shardTotal
value, resulting in a total of 20 jobs to be run. So it will split up the tests between 20 jobs, each running a different browser and a different subset of tests, see here for more details.
name: Playwright Tests
on:
push:
branches: [ main, master ]
pull_request:
branches: [ main, master ]
jobs:
playwright:
name: 'Playwright Tests - ${{ matrix.project }} - Shard ${{ matrix.shardIndex }} of ${{ matrix.shardTotal }}'
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
project: [chromium, webkit]
shardIndex: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
shardTotal: [10]
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: 18
- name: Install dependencies
run: npm ci
- name: Install browsers
run: npx playwright install --with-deps
- name: Run your tests
run: npx playwright test --project=${{ matrix.project }} --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
Note: The
${{ <expression> }}
is the expression syntax that allows accessing the current context. In this example, we are using thematrix
context to set the job variants.
GitHub Actions support running jobs in a container by using the jobs.<job_id>.container
option. This is useful to not pollute the host environment with dependencies and to have a consistent environment for e.g. screenshots/visual regression testing across different operating systems.
name: Playwright Tests
on:
push:
branches: [ main, master ]
pull_request:
branches: [ main, master ]
jobs:
playwright:
name: 'Playwright Tests'
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/playwright:v1.34.2-jammy
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: 18
- name: Install dependencies
run: npm ci
- name: Run your tests
run: npx playwright test
name: Playwright Tests
on:
push:
branches: [ main, master ]
pull_request:
branches: [ main, master ]
jobs:
playwright:
name: 'Playwright Tests'
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/playwright/python:v1.34.2-jammy
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r local-requirements.txt
pip install -e .
- name: Run your tests
run: pytest
name: Playwright Tests
on:
push:
branches: [ main, master ]
pull_request:
branches: [ main, master ]
jobs:
playwright:
name: 'Playwright Tests'
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/playwright/java:v1.34.2-jammy
steps:
- uses: actions/checkout@v3
- uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: '17'
- name: Build & Install
run: mvn -B install -D skipTests --no-transfer-progress
- name: Run tests
run: mvn test
name: Playwright Tests
on:
push:
branches: [ main, master ]
pull_request:
branches: [ main, master ]
jobs:
playwright:
name: 'Playwright Tests'
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/playwright/dotnet:v1.34.2-jammy
steps:
- uses: actions/checkout@v3
- name: Setup dotnet
uses: actions/setup-dotnet@v3
with:
dotnet-version: 6.0.x
- run: dotnet build
- name: Run your tests
run: dotnet test
- langs: js
GitHub Actions supports sharding tests between multiple jobs using the jobs.<job_id>.strategy.matrix
option. The matrix
option will run a separate job for every possible combination of the provided options. In the example below, we have 2 project
values, 10 shardIndex
values and 1 shardTotal
value, resulting in a total of 20 jobs to be run. So it will split up the tests between 20 jobs, each running a different browser and a different subset of tests, see here for more details.
name: Playwright Tests
on:
push:
branches: [ main, master ]
pull_request:
branches: [ main, master ]
jobs:
playwright:
name: 'Playwright Tests - ${{ matrix.project }} - Shard ${{ matrix.shardIndex }} of ${{ matrix.shardTotal }}'
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/playwright:v1.34.2-jammy
strategy:
fail-fast: false
matrix:
project: [chromium, webkit]
shardIndex: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
shardTotal: [10]
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: 18
- name: Install dependencies
run: npm ci
- name: Run your tests
run: npx playwright test --project=${{ matrix.project }} --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
This will start the tests after a GitHub Deployment went into the success
state.
Services like Vercel use this pattern so you can run your end-to-end tests on their deployed environment.
name: Playwright Tests
on:
deployment_status:
jobs:
test:
timeout-minutes: 60
runs-on: ubuntu-latest
if: github.event.deployment_status.state == 'success'
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: 18
- name: Install dependencies
run: npm ci
- name: Install Playwright
run: npx playwright install --with-deps
- name: Run Playwright tests
run: npx playwright test
env:
# This might depend on your test-runner/language binding
PLAYWRIGHT_TEST_BASE_URL: ${{ github.event.deployment_status.target_url }}
We have a pre-built Docker image which can either be used directly, or as a reference to update your existing Docker definitions.
Suggested configuration
- Using
--ipc=host
is also recommended when using Chromium. Without it Chromium can run out of memory and crash. Learn more about this option in Docker docs. - Seeing other weird errors when launching Chromium? Try running your container
with
docker run --cap-add=SYS_ADMIN
when developing locally. - Using
--init
Docker flag or dumb-init is recommended to avoid special treatment for processes with PID=1. This is a common reason for zombie processes.
For Windows or macOS agents, no additional configuration required, just install Playwright and run your tests.
For Linux agents, you can use our Docker container with Azure Pipelines support running containerized jobs. Alternatively, you can use Command line tools to install all necessary dependencies.
For running the Playwright tests use this pipeline task:
jobs:
- deployment: Run_E2E_Tests
pool:
vmImage: ubuntu-22.04
container: mcr.microsoft.com/playwright:v1.34.2-jammy
environment: testing
strategy:
runOnce:
deploy:
steps:
- checkout: self
- task: Bash@3
displayName: 'Run Playwright tests'
inputs:
workingDirectory: 'my-e2e-tests'
targetType: 'inline'
failOnStderr: true
env:
CI: true
script: |
npm ci
npx playwright test
This will make the pipeline run fail if any of the playwright tests fails.
If you also want to integrate the test results with Azure DevOps, use the task PublishTestResults
task like so:
jobs:
- deployment: Run_E2E_Tests
pool:
vmImage: ubuntu-22.04
container: mcr.microsoft.com/playwright:v1.34.2-jammy
environment: testing
strategy:
runOnce:
deploy:
steps:
- checkout: self
- task: Bash@3
displayName: 'Run Playwright tests'
inputs:
workingDirectory: 'my-e2e-tests'
targetType: 'inline'
failOnStderr: true
env:
CI: true
script: |
npm ci
npx playwright test
- task: PublishTestResults@2
displayName: 'Publish test results'
inputs:
searchFolder: 'my-e2e-tests/test-results'
testResultsFormat: 'JUnit'
testResultsFiles: 'e2e-junit-results.xml'
mergeTestResults: true
failTaskOnFailedTests: true
testRunTitle: 'My End-To-End Tests'
condition: succeededOrFailed()
Note: The JUnit reporter needs to be configured accordingly via
["junit", { outputFile: "test-results/e2e-junit-results.xml" }]
in playwright.config.ts
.
Running Playwright on CircleCI is very similar to running on GitHub Actions. In order to specify the pre-built Playwright Docker image, simply modify the agent definition with docker:
in your config like so:
executors:
pw-jammy-development:
docker:
- image: mcr.microsoft.com/playwright:v1.34.2-jammy
Note: When using the docker agent definition, you are specifying the resource class of where playwright runs to the 'medium' tier here. The default behavior of Playwright is to set the number of workers to the detected core count (2 in the case of the medium tier). Overriding the number of workers to greater than this number will cause unnecessary timeouts and failures.
Sharding in CircleCI is indexed with 0 which means that you will need to override the default parallelism ENV VARS. The following example demonstrates how to run Playwright with a CircleCI Parallelism of 4 by adding 1 to the CIRCLE_NODE_INDEX
to pass into the --shard
cli arg.
playwright-job-name:
executor: pw-jammy-development
parallelism: 4
steps:
- run: SHARD="$((${CIRCLE_NODE_INDEX}+1))"; npx playwright test -- --shard=${SHARD}/${CIRCLE_NODE_TOTAL}
Jenkins supports Docker agents for pipelines. Use the Playwright Docker image to run tests on Jenkins.
pipeline {
agent { docker { image 'mcr.microsoft.com/playwright:v1.34.2-jammy' } }
stages {
stage('e2e-tests') {
steps {
// Depends on your language / test framework
sh 'npm install'
sh 'npx playwright test'
}
}
}
}
Bitbucket Pipelines can use public Docker images as build environments. To run Playwright tests on Bitbucket, use our public Docker image (see Dockerfile).
image: mcr.microsoft.com/playwright:v1.34.2-jammy
To run Playwright tests on GitLab, use our public Docker image (see Dockerfile).
stages:
- test
tests:
stage: test
image: mcr.microsoft.com/playwright:v1.34.2-jammy
script:
...
- langs: js
GitLab CI supports sharding tests between multiple jobs using the parallel keyword. The test job will be split into multiple smaller jobs that run in parallel. Parallel jobs are named sequentially from job_name 1/N
to job_name N/N
.
stages:
- test
tests:
stage: test
image: mcr.microsoft.com/playwright:v1.34.2-jammy
parallel: 7
script:
- npm ci
- npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
GitLab CI also supports sharding tests between multiple jobs using the parallel:matrix option. The test job will run multiple times in parallel in a single pipeline, but with different variable values for each instance of the job. In the example below, we have 2 PROJECT
values, 10 SHARD_INDEX
values and 1 SHARD_TOTAL
value, resulting in a total of 20 jobs to be run.
stages:
- test
tests:
stage: test
image: mcr.microsoft.com/playwright:v1.34.2-jammy
parallel:
matrix:
- PROJECT: ['chromium', 'webkit']
SHARD_INDEX: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
SHARD_TOTAL: 10
script:
- npm ci
- npx playwright test --project=$PROJECT --shard=$SHARD_INDEX/$SHARD_TOTAL
With the default behavior, Playwright downloads the browser binaries in the following directories:
%USERPROFILE%\AppData\Local\ms-playwright
on Windows~/Library/Caches/ms-playwright
on MacOS~/.cache/ms-playwright
on Linux
To cache the browser downloads between CI runs, cache this location in your CI configuration, against a hash of the Playwright version.
Playwright supports the DEBUG
environment variable to output debug logs during execution. Setting it to pw:browser*
is helpful while debugging Error: Failed to launch browser
errors.
DEBUG=pw:browser* npx playwright test
DEBUG=pw:browser* pytest
DEBUG=pw:browser* mvn test
DEBUG=pw:browser* dotnet test
By default, Playwright launches browsers in headless mode. This can be changed by passing a flag when the browser is launched.
// Works across chromium, firefox and webkit
const { chromium } = require('playwright');
const browser = await chromium.launch({ headless: false });
// Works across chromium, firefox and webkit
import com.microsoft.playwright.*;
public class Example {
public static void main(String[] args) {
try (Playwright playwright = Playwright.create()) {
BrowserType chromium = playwright.chromium();
Browser browser = chromium.launch(new BrowserType.LaunchOptions().setHeadless(false));
}
}
}
import asyncio
from playwright.async_api import async_playwright
async def main():
async with async_playwright() as p:
# Works across chromium, firefox and webkit
browser = await p.chromium.launch(headless=False)
asyncio.run(main())
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
# Works across chromium, firefox and webkit
browser = p.chromium.launch(headless=False)
using Microsoft.Playwright;
using var playwright = await Playwright.CreateAsync();
await playwright.Chromium.LaunchAsync(new()
{
Headless = false
});
On Linux agents, headed execution requires Xvfb to be installed. Our Docker image and GitHub Action have Xvfb pre-installed. To run browsers in headed mode with Xvfb, add xvfb-run
before the Node.js command.
xvfb-run node index.js
xvfb-run python test.py
xvfb-run mvn test
xvfb-run dotnet test