Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(ui): refactor e2e tests to use playwright cli #4721

Merged
merged 18 commits into from Dec 17, 2023

Conversation

hi-ogawa
Copy link
Contributor

@hi-ogawa hi-ogawa commented Dec 11, 2023

Description

While thinking about expanding e2e tests for coverage preview fix PR #4717, I noticed that current e2e test in test/ui uses playwright-chromium package directly (introduced in #2710) and it doesn't benefit from usual playwright test authoring goodies (e.g. debug mode, trace viewer, multiple browsers, etc...) which are provided only for playwright test CLI usage and @playwright/test package.

Here, I'm proposing to rewrite current e2e tests into standard playwright test for the ease of maintenance.
I feel this is worth it (and overall change is very simple at this point), but I would like to know what maintainers think about this direction.

I also noticed that people from cypress was suggesting to use cypress at some point #590, so I'm not sure what is the overall opinion on vitest ui testing.


here is a quick rundown of potential testing workflow:

# run normal vitest to test fixtures
pnpm -C test/ui test-fixtures

# run playwright test in headless mode (this uses above vitest fixtures on --ui mode and html report mode)
pnpm -C test/ui test-e2e

# run playwright test with debug mode (this allows step debugging playwright test + recording new tests directly on browser)
pnpm -C test/ui test-e2e -- --debug

Previously the test was run during pnpm test:ci, but I think it makes sense to move it as a separate category, so I added a new CI job test-ui-e2e to just run pnpm -C test/ui test-e2e. I'll continue to think about what's the ideal structure to manage this on CI.

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

  • It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
  • Ideally, include a test that fails without this PR but passes with it.
  • Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

  • Run the tests with pnpm test:ci.

Documentation

  • If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

  • Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

Copy link

netlify bot commented Dec 11, 2023

Deploy Preview for fastidious-cascaron-4ded94 canceled.

Name Link
🔨 Latest commit 8078600
🔍 Latest deploy log https://app.netlify.com/sites/fastidious-cascaron-4ded94/deploys/657e75c8f3eac10008a5c5ee

@hi-ogawa hi-ogawa changed the title test: refactor UI e2e tests test(ui): refactor e2e tests to use playwright cli Dec 11, 2023
@@ -109,6 +109,32 @@ jobs:
- name: Test UI
run: pnpm run ui:test

test-ui-e2e:
runs-on: ubuntu-latest
Copy link
Contributor Author

@hi-ogawa hi-ogawa Dec 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this run on multiple os/browsers (like test and test-browser jobs)?


I just tested on macos and windows 02c77f7

Macos is okay but Windows seems to have problems when killing processes.
https://github.com/vitest-dev/vitest/actions/runs/7162660207/job/19499945258?pr=4721#step:7:24

I think this is a typical Windows issue, so probably I can find tips somewhere (runVitestCli helper?).


Windows seems to be fine after using the same kill/exit await technique as runVitestCli 5f58331.

@hi-ogawa hi-ogawa marked this pull request as ready for review December 11, 2023 03:21
Copy link
Member

@AriPerkkio AriPerkkio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. This simplifies the testing setup a lot. Also happy to see execa not used as it used to randomly cause issues in these tests.

test/ui/package.json Show resolved Hide resolved
@AriPerkkio AriPerkkio merged commit 217910b into vitest-dev:main Dec 17, 2023
18 of 19 checks passed
LorenzoBloedow pushed a commit to LorenzoBloedow/vitest that referenced this pull request Dec 19, 2023
@hi-ogawa hi-ogawa deleted the test-refactor-ui-e2e branch December 19, 2023 23:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants