Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Chromatic visual testing to our site/E2E tests #12806

Open
7 tasks
Kira-Pilot opened this issue Mar 28, 2024 · 3 comments · May be fixed by #13103
Open
7 tasks

Add Chromatic visual testing to our site/E2E tests #12806

Kira-Pilot opened this issue Mar 28, 2024 · 3 comments · May be fixed by #13103
Assignees
@aslilac
Labels
chore Non-customer facing refactors, cleanup, or technical debt. site Area: frontend dashboard
Milestone
Code Stability

Comments

@Kira-Pilot
Copy link
Member

Kira-Pilot commented Mar 28, 2024 •
edited

Chromatic has introduced visual testing for Playwright. We should explore adding this additional layer of user flow bug detection to our current E2E tests within site/e2e.

Requirements:

  • Add manual snapshots to each of our E2E tests as appropriate - these can be component snapshots or full page snapshots
  • Remove any redundant storybook component tests (if there are any)
  • Add a NPM script to our package.json for running Chromatic with the playwright flag
  • Update our CI workflow's Chromatic setup such that it also supports Playwright
  • Update our FE README to make mention of this new tooling
  • Establish baseline speed of our CI run and then compare after we have added this layer - it is important we don't significantly slow down the job considering these tests are run on every PR.
  • Understand monthly cost of these additional snapshots as Chromatic's pricing depends on # of snapshots - is this addition going to move the needle significantly?
@Kira-Pilot Kira-Pilot added this to the Code Stability milestone Mar 28, 2024
@coder-labeler coder-labeler bot added ☂️ epic An issue of issues chore Non-customer facing refactors, cleanup, or technical debt. site Area: frontend dashboard labels Mar 28, 2024
@matifali matifali removed the ☂️ epic An issue of issues label Mar 28, 2024
@aslilac
Copy link
Member

aslilac commented Apr 29, 2024

good news:

Image

bad news:

it seems like it makes some existing flakes fail almost 100% of the time, so I'm gonna have to spend some time fixing more of them.

@aslilac
Copy link
Member

aslilac commented Apr 29, 2024

and I still need to comb through and see where adding additional snapshots would provide value

@aslilac aslilac linked a pull request Apr 30, 2024 that will close this issue
Draft
@Kira-Pilot
Copy link
Member Author

@Emyrk can help take a look during the buffer week.
@aslilac to see if we can isolate this one test (or does it affect all 5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aslilac aslilac

Labels
chore Non-customer facing refactors, cleanup, or technical debt. site Area: frontend dashboard
Projects
None yet
Milestone
Code Stability
Development

Successfully merging a pull request may close this issue.

test: use chromatic playwright wrapper coder/coder
3 participants
@aslilac @matifali @Kira-Pilot