Fix the VRT tests that are failing in CI.

[techanvil]

Feature Description

Following up from #6151, there will be quite a few VRT tests failing in CI.

• Most of them are failing correctly and simply need to have their reference image updated.
• Two tests have been identified as needing work to fix the related Storybook stories:
  • WPDashboardWidgets > Ready with Activate Analytics CTA
  • SearchConsole > Ready with Activate Analytics CTA.
  • It appears that some sibling stories to the above also need attention although they do not have VRT scenarios.

We should apply an appropriate fix to all of these failing tests.

---

Do not alter or remove anything below. The following sections will be managed by moderators only.

Acceptance criteria

• All of the VRT tests which are failing correctly in CI should have their reference images updated.
• The VRT tests which need Storybook work to fix should be fixed and any related stories that may be broken should also be fixed. These tests are:
  • WPDashboardWidgets > Ready with Activate Analytics CTA - this has a VRT scenario
  • WPDashboardWidgets > Ready with Activate Module CTA
  • WPDashboardWidgets > Ready with Complete Analytics Activation CTA
  • SearchConsole > Ready with Activate Analytics CTA - this has a VRT scenario
  • SearchConsole > Ready with Complete Analytics Activation CTA
• All of the tests should pass in CI. Discrepancies between CI and local test runs can be addressed separately via Resolve VRT issues for local runs #5824

Implementation Brief

Due to stability problems running the suite locally, we can fix the test suite using reference images from a CI run before moving on to address the local run issues in #5824.

• Download the report from a CI run of the VRT workflow.
• Check through the test failures.
• For the tests which are failing correctly, copy the reference images from the report into tests/backstop/reference/.
• Fix those tests which need Storybook work.
  • Refer to the AC for the list of stories. Confirm these are valid at the time of implementation.
  • Apply fixes and either run locally to generate the new reference images, or if necessary push to CI and download/update reference images via the report as above.
• Remove the code that allows overriding misMatchThreshold configuration for individual scenarios, as this is no longer needed and is currently the cause of some false positives (see below for more info).
  • Ensure the related Progress Bar reference images are correctly updated along with the others.

Test Coverage

• No new tests needed.

QA Brief

QA:Eng

• Create a sample PR and ensure the VRT passes in the CI.
• Alternatively, if you have an existing PR, pull the latest develop and push it and ensure the VRT is passing in the CI.
• Ensure there shouldn't any failures.

Changelog entry

• N/A.

[tofumatt]

ACs and IB look good, but the IB here needs an estimate so assigned to @techanvil to assign one.

After that feel free to move right to the Execution Backlog, as otherwise this is IB ✅ from me 🙂

[techanvil]

Thanks @tofumatt! Have added an estimate of 7, moving on over to the EB 👍

[aaemnnosttv]

@techanvil I just noticed this in a failing VRT run (where most are still failing) but in this situation there seem to be tests which should have failed that passed.

E.g.

Any idea how this could happen?

[aaemnnosttv]

I've pulled this forward into this sprint – let's try to resolve this ahead of the release if possible.

[techanvil]

@techanvil I just noticed this in a failing VRT run (where most are still failing) but in this situation there seem to be tests which should have failed that passed.

...

Any idea how this could happen?

Hey @aaemnnosttv - good spot there!

It turns out this is a bit of a historical issue with a simple fix.

To diagnose I first reverted to v1.81.0, prior to the ARM/M1 related fix, and found the issue was still there. I dug into it further and discovered the cause: We are overriding the misMatchThreshold value for this scenario. Looks like this must have been put in place before animations were stopped for VRT tests.

site-kit-wp/stories/progress-bars.stories.js

Line 43 in 1aa46bc

misMatchThreshold: 10, // Handle animation differences.

site-kit-wp/.storybook/storybook-data.js

Line 171 in d00a821

misMatchThreshold: 10,

I think it would be good to include a fix for it in this issue and would propose adding a line to the IB:

• Remove the misMatchThreshold: 10 configuration from the Progress Bars scenario (see comment below) in order to prevent false positives for this test.

Does that sound OK to you, or would you rather address it in a separate issue?

[aaemnnosttv]

Thanks for digging into this @techanvil – let's remove misMatchThreshold overrides as part of this as well as those shouldn't be needed or used 👍

[techanvil]

Thanks @aaemnnosttv, good shout. I have updated the IB accordingly.

[aaemnnosttv]

Great, thanks @techanvil 👍

IB ✅

[asvinb]

QA: ✔️

Checked a recent PR and the VRT action is no longer failing:

[aaemnnosttv]

This is mostly good to go although there are still some tests failing since this was merged.

These will be fixed in #6324.