Epic: Testing improvements

[ecraig12345]

This is a belated "final" version of a planning document that was started back in January/Feburary. Some of the work is already done (✅ sections, done by a mix of Elizabeth, Ling, Tristan, and Martin) but various longer-term items remain.

Overview

We have various layers of testing in Fluent UI, but a lot of the tools are out of date, and there are ways we could make our tests easier to write, faster to run, and generally more effective. These issues reduce development velocity and quality/quantity of tests.

This project's goal is to "clean house," making changes that will facilitate writing great tests to help take our library's quality to the next level as we continue to build out more converged components.

General areas:

• Getting up to date: Upgrade Jest, Cypress, and other testing-related deps to the latest versions (and help unblock upgrades of other deps such as React)

• Screener: Make our visual tests faster and more reliable, update outdated Storybook usage, and investigate whether a different visual testing framework would serve us better in the long run

• Migrate away from Enzyme: Enzyme is effectively dead, was delaying our repo's internal React 17 upgrade, and will entirely block an in-repo React 18 upgrade. We need to work on switching to Testing Library and Cypress, which provide better-supported, less buggy, and more realistic testing.

• Improve conformance tests: Conformance tests enable us to test both current and future components, and maintain API/behavior coherency as we build more components. The @fluentui/react-conformance package had ambitious goals, but its current structure is holding us back, so it needs to be rewritten (and then we need to add more tests).

• Documentation: We need up-to-date documentation to help people write great tests.

• Address "paper cuts": Eliminate or improve small things that are detracting from the developer experience (with respect to testing)

• Future ideas: Some additional types of tests we'd like to add at some point in the future

Getting up to date

Our testing-related dependencies had gotten pretty badly out of date, and new versions continue to be released.

✅ Done

• Updated Jest from 24 to 26
• Updated testing-library to latest as of January 2022
• Updated Cypress to latest

Medium/long-term

• Update @testing-library/user-event to 14 (bug fixes and API changes) - Upgrade to user-event v14 #22260
• Update other testing-library deps to latest (again)
• Update Jest and ts-jest to 27 (de-prioritized b/c it's not immediately blocking anything else) - Update jest and ts-jest to 27 #18442

Screener (vr-tests)

✅ Done

For various reasons (outlined in #21070), our screener runs were becoming less reliable as we added more v9 tests. Also, having v8 and v9 in the same package was slowing down builds (due to needing to build both + run tests for both when either changed).

• Made custom screener proxy support multiple projects
• Moved v9 screener tests to a separate package from v8 screener tests, and run them via custom proxy
• Only run screener for affected projects

Medium/long-term

The storybooks in our vr-tests packages are using an outdated story format, which will need to be fixed before we can migrate to storybook 7 (once it comes out). #21779

We also haven't been impressed with various aspects of screener, so longer-term we'd like to investigate other solutions (possibly chromatic if we can find an interaction testing solution). #22369

Migrate away from Enzyme

Enzyme is no longer in active development (no official React 17 adapter after 1.5 years) and won't work with React 18 without a substantial rewrite of the adapter code/helpers. It's also just kind of difficult to work with and buggy.

A better solution is to use a combination of React Testing Library and Cypress.

• Testing Library provides a more user-centric way of testing which often makes it easier to write tests, plus it's less buggy (a bit more closely resembles the browser environment) and makes cleanup easier.
• Cypress allows testing in a real browser, for scenarios such as focus which are just not simulated realistically enough by jsdom.

Prior to this work, all the tests for v8 and v0 used Enzyme, and so did the react-conformance and a11y-testing packages shared between versions. v9 did not use Enzyme aside from conformance/a11y.

✅ Done

Goal: do enough to unblock migration to React 17 (#20145). This involves a hybrid approach using the "community" adapter @wojtekmaj/enzyme-adapter-react-17 and porting any tests that don't work to either testing-library or cypress.

• Got rid of Enzyme in our testing helpers (react-conformance, a11y-testing)
• Enabled Cypress testing for v8 - this provides an "out" for especially tricky tests like FocusTrapZone, or if we upgrade react and/or jest and any other tests stop working even in testing-library
• Started porting component tests to testing-library (see [Epic]: Migrate from enzyme to testing-library #21663 for progress)
• Tried out React 17 with the community adapter and fixed the handful of broken tests (either made small fixes or ported to testing-library or cypress)
  • At this point the React 17 upgrade should be unblocked from a testing standpoint

We also communicated to the Redmond team that all new v8 tests should be written in testing-library (and strongly encouraged that if you have to fix an enzyme test, go ahead and port it). Ideally this would be enforced by tooling, but this is more challenging since a lint rule can't easily be applied to new code only.

Medium/long-term

Completely get rid of Enzyme by porting all tests (v8 and v0) to either testing-library or cypress. As mentioned above, this will be necessary before we can adopt React 18 in-repo. #21663

An alternative might be finding a way to run only the Enzyme tests against React 17, but that would be challenging to set up.

Conformance tests

Component conformance tests are important because they enable us to test both current and future components, and help us ensure that we have a coherent API surface and behaviors as we continue to build more components.

Short-term

@fluentui/react-conformance was written with the goal of being reusable and extensible, but it's not a very good design. It also has the issue of including tests in the core package that are actually specific to v8/v9/v0 (which can cause dependency graph issues and is generally unhelpful), and it introduced enzyme+node types into the global namespace anywhere it's imported.

• Removed Enzyme, so the package no longer causes global type pollution
• Split out v8/9/0 tests
  • Moved v0-only tests back to v0
  • Split v8/9 tests
• Update package architecture to be properly extensible/reusable - Rewrite react-conformance API to be more extensible/modular #21665

Medium/long-term

• Improve performance for tests that need type information - Improve perf for typing-related conformance tests #19697
• Add more conformance tests for converged patterns

Documentation

Our testing documentation was very out of date and didn't cover current recommended tools/patterns.

✅ The main testing page has been revamped and new pages added with specific info about Jest, Cypress, and Screener.

• Overview
• Testing with Jest
• E2E testing (Cypress)
• Visual testing (Screener)

For the future, it would be good to continue expanding those docs with more information about best practices, "gotchas," etc.

Address "paper cuts"

There are various things creating small slowdowns or inconveniences which detract from the developer experience.

When we encounter one of these things, we should consider "is this thing still providing value? (if not, how hard is it to remove?) or "is there an easy way to make this a little better?"

✅ Short-term

• Eliminated example snapshots in v7/8: nobody on the Redmond team could think of a specific instance where an example snapshot caught a real issue, so the snapshots were mostly just an annoyance and a source of confusion for contributors
• Made assorted improvements to error messages

Deferred/ideas

• Improve reliability of Cypress tests: currently the cypress tests (especially in v0 but sometimes other places) are the most frequent cause of random PR build failures, which is annoying
• Fix ID churn issues in v8 snapshots, possibly by calling resetIds() in a global afterEach

Future ideas

Longer-term, what are some ways we can take our tests to the next level?

• Add memory tests
• Perf test improvements (needs a new owner)
  • Use the same tool for v8/9/0
• Bundle size test improvements (needs a new owner)
  • Use the same tool for v8/9/0
  • Ability to have a report link to share out
• Partner integration tests

[msft-fluent-ui-bot]

Because this issue has not had activity for over 150 days, we're automatically closing it for house-keeping purposes.

Still require assistance? Please, create a new issue with up-to date details.