chore: use playwright/test for integration tests

[kettanaito]

• Closes Migrate to "@playwright/test" #1342

Changes

• Adds and configures @playwright/test for integration tests instead of page-with.
• Improves the stability of the integration tests.

Roadmap

• Fix failing src/utils/handleRequest.test.ts:251:30
• Migrate the rest of the tests
  • rest-api
  • graphql-api
  • msw-api
• Add .use() support to @open-draft/test-server for runtime middleware
• Rewrite existing tests to always use HttpServer from test-server
• Consider if spawning API server per test would be a better idea
• Adjust the test:integration script to use playwright
• Fix the failing 9 tests
• Split the browser and node integration tests (incl. CI)
• Account for new tests from fix: remove duplicate response logging in the browser console #1418
• Resolve leaking state in tests
• Update CONTRIBUTING.md guidelines on how to run --debug mode in PW
• Fix all test.skip() and test.fixme()
• Configure CI to upload Playwright test output artifacts for better debugging (although videos and screenshots are of little use, given most of the things happen in Network/Console). It'd be much better to export each failed test's Network into an HAR file and attach that to the failed run.

Resources

• webpack-http-server. As a part of this change, I'm decoupling the runtime compilation capabilities of page-with into a standalone package.

[codesandbox-ci]

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

Latest deployment of this branch, based on commit a71acf7:

Sandbox	Source
MSW React	Configuration

[kettanaito]

Discovery: page.request vs page.evaluate(fetch)

PW comes with a built-in page.request utility but requests it makes bypass the network. I believe that's intentional in order for PW to spy on requests you make. Since we need requests to actually happen to hit the service worker, I'm adding a custom fetch fixture.

[kettanaito]

@shwarcu, so I've tried spawning the webpack server once but the issue is that the tests need a reference to that server. It's used to perform compilations (server.compile()) and also to modify compilation's behavior on the test's runtime, such as appending new routes, overriding what gets served as the worker script, etc.

There doesn't seem to be a way to create server once and then expose it to tests. Playwright's configs run in a separate process, and I've come to believe that even tests are isolated as I cannot use a singleton pattern—the value of persisted server is always undefined across tests.

I think I can technically split the compilation server and the preview server so that we use a single compilation server to save up resources but spawn a new preview server for each test but that's an overkill given that the compilation server already does a great job at scoping compilations and serving them like a preview (its entire point).

Do you happen to know if we can achieve what I've described above?

[kettanaito]

What I decided to do in the end is to spawn 1 compilation server per 1 worker. I approached it this way: tests are completely isolated within the worker process so there's no way to have a shared state between workers. Within a single worker, however, I will establish a single compilation server to save resources.

I've given this a quick test and can confirm that this setup works. Now testing the flakiness...

[kettanaito]

Sadly, even after rewriting the Worker console logic and ensuring the webpack server is scoped per worker, the tests are still flaky, timing out on the "context" error. Huh.

@shwarcu, what do you think we could look at as the next step?

[shwarcu]

Hi @kettanaito 👋 I need more time for 'decompression' after work and I won't be looking at the code today and tomorrow. But I will continue my engagement in this topic and I will get back to you once I have something valuable to say (probably closer to Saturday/Sunday)

[kettanaito]

Absolutely no worries! Please make sure to reset properly. Work can be daunting but a good chunk of rest does wonders. See you around!

[kettanaito]

I went and improved a bit on how we handle runtime compilations. I've added the ability to dispose of the compilation once the test settles so they don't take memory anymore:

• Kill the underlying webpack compiler;
• Remove any in-memory records of the compilation that webpack-http-server keeps internally.

I don't know if that helped or not but I've got the first CI passing since forever. Will retry it to confirm...

The tests do seem more reliable now. I had 4 runs, 3 succeeded, but the 4th one failed with:

  1) [chromium] › msw-api/setup-worker/scenarios/iframe/iframe.test.ts:40:5 › intercepts a request from a deeply nested iframe 
error Command failed with exit code 1.

    Test timeout of 10000ms exceeded.

    frame.waitForSelector: Target closed
    =========================== logs ===========================
    waiting for locator('#first-name') to be visible
    ============================================================

      57 |
      58 |   await deepFrame.evaluate(() => window.request())
    > 59 |   const firstNameElement = await deepFrame.waitForSelector('#first-name')
         |                                            ^
      60 |   const firstName = await firstNameElement.evaluate((node) => node.textContent)
      61 |
      62 |   expect(firstName).toBe('John')

        at /Users/runner/work/msw/msw/test/browser/msw-api/setup-worker/scenarios/iframe/iframe.test.ts:59:44

These iframe tests are dependent on file system, so I suspect there's an issue where two tests try to write to the same directory (although it should be impossible now given the dir name is based on the worker id + directory name).

Well, at least it seems that disposing of webpack instances helped with the other tests.

[kettanaito]

As the next step, I've skipped the iframe tests to verify whether they are the only ones left failing. Will re-run the CI multiple times to confirm that.

[kettanaito]

Alas, the tests are still flaky so I moved on with adding some additional debugging to them. Now each failed test will flush its runtime console state to a log file and upload it as an artifact to the CI. This should help us debug flaky tests since screenshots and videos are pretty much useless given the way we write things in the console primarily.

[kettanaito]

So based on https://github.com/mswjs/msw/actions/runs/4148868358, which has failing tests with no logs whatsoever, I suspect the issue is in asynchronicity/servers/etc. Quite a pickle then, I was hoping for some browser-side reporting to help but it looks like I need to dive into why PW/webpack server has trouble doing what it's designed to do.

[kettanaito]

Released: v1.1.0 🎉

This has been released in v1.1.0!

• 📄 Release notes
• 📦 npm package

Make sure to always update to the latest version (npm i msw@latest) to get the newest features and bug fixes.

---

Predictable release automation by @ossjs/release.