[bug] Test runner crashes with a11y tests on Storybook 8

[AmirTugi]

Describe the bug
A11y tests crash in the test runner when upgrading Storybook to v8.
I followed the upgrade manual and everything passes but the a11t tests in the test runner.
It's saying

page.evaluate: Target crashed 
    at waitForPageReady (/mnt/ramdisk/repo/node_modules/@storybook/test-runner/dist/index.js:15314:14)
    at preVisit (/mnt/ramdisk/repo/devtools/storybook/.storybook/test-runner.ts:15:5)

To Reproduce
Steps to reproduce the behavior:

Upgrade Storybook to 8.0.8 and use @storybook/test-runner@0.17.0

Expected behavior
The a11y tests should pass as before

Screenshots

System

Storybook Environment Info:

  System:
    OS: macOS 14.4.1
    CPU: (12) arm64 Apple M2 Pro
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 20.10.0 - ~/.volta/tools/image/node/20.10.0/bin/node
    Yarn: 1.22.19 - ~/.volta/tools/image/yarn/1.22.19/bin/yarn <----- active
    npm: 10.2.3 - ~/.volta/tools/image/node/20.10.0/bin/npm
  Browsers:
    Safari: 17.4.1

Additional Context
Here's the test-runner.ts file I have

import type { TestRunnerConfig } from '@storybook/test-runner';
import { getStoryContext } from '@storybook/test-runner';

import { checkA11y, injectAxe, configureAxe } from 'axe-playwright';

/*
 * See https://storybook.js.org/docs/writing-tests/test-runner#test-hook-api
 * to learn more about the test-runner hooks API.
 */
const config: TestRunnerConfig = {
  tags: {
    include: ['a11y-include'],
  },
  async preVisit(page) {
    await injectAxe(page);
  },
  async postVisit(page, context) {
    // Get the entire context of a story, including parameters, args, argTypes, etc.
    const storyContext = await getStoryContext(page, context);

    // Apply story-level a11y rules
    await configureAxe(page, {
      rules: storyContext.parameters?.['a11y']?.config?.rules,
    });

    await checkA11y(
      page,
      '#storybook-root',
      {
        detailedReport: true,
        axeOptions: {
          // Limit the result types to only violations that the test managed to find
          resultTypes: ['violations'],
        },
        detailedReportOptions: {
          html: true,
        },
      },
      false,
      'v2'
    );
  },
};

export default config;

[ethriel3695]

Hi @AmirTugi, we will take a look and let you know what we find.

[yannbf]

Hey @AmirTugi thanks a lot for elaborating, but could you please provide a reproduction repo? You can use https://storybook.new for a minimal repro setting.
I tried the test-runner with Storybook 8.0.8 with your test-runner.ts file and the a11y tests worked without issues.

I actually find it weird that your tests failed with waitForPageReady while in your snippet such function is not used. That function is not used internally by the test-runner, only if you add it yourself. I think you might have it in your preVisit hook. Maybe what's going on is you are using Webpack and somehow the waitForPageReady function is hanging because of some Webpack request that won't resolve, but I am not sure as I don't know the configuration of your project.