Skip or Exclude not working on play-tests

[AmirTugi]

Describe the bug
I've added the skip tag (also tried with the exclude tag) to skip currently failing a11y tests, and I still see that some (or maybe all) the play-tests are running, regardless of the "skip" tag I added to them.

I also noticed that this is happening only to play tests where the play is async.

I noticed that the smoke-tests are skipped and that works fine. It's just the play-test.

Expected behavior
The play-test should also be skipped.

Screenshots

System

Storybook Environment Info:

System:
OS: macOS 14.2.1
CPU: (12) x64 Apple M2 Max
Shell: 5.9 - /bin/zsh
Binaries:
Node: 20.10.0 - ~/.volta/tools/image/node/20.10.0/bin/node
Yarn: 1.22.19 - ~/.volta/tools/image/yarn/1.22.19/bin/yarn <----- active
npm: 10.2.3 - ~/.volta/tools/image/node/20.10.0/bin/npm
Browsers:
Chrome: 120.0.6099.234
Safari: 17.2.1

[AmirTugi]

I managed to find a workaround for now, and to add tags: ['a11y-skip'] directly to the story, and not in the component meta.
Seems like I was wrong about the async aspect.
It seems like that when tags is in the component-level - it works only for smoke-tests, and when it's for a specific story - it's also for its play-test.

I still think this is an issue, and at least should be documented.

[renatewr]

We had the exact same issue, and have spent hours debugging.

The link to the erroring story/play test is working, but running test-storybook the "play test" logged out "message: null".

I can confirm that this workaround works - the story is then excluded.

However, I would like to figure out why it fails, but I cannot get any debug information, since the built story is working fine.
I suspect that the reason is that the story is actually loading a video player.