New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Resolve VRT issues for local runs #5824
Comments
Update after spending some time on the ticket:
|
@FlicHollis I think we can remove this from Sprint 85 as it looks like it needs more work on the IB :-) |
This has been removed, thanks @eclarke1 |
Latest updates:
|
I am also seeing failures on my Linux workstation(Ubuntu 22.04, AMD64), similar to the description but more, ~30 failures here. |
@aaemnnosttv Not sure if am missing something here and how the GH actions are all passing, but if you look at one of the reference image on develop and you compare it with what we currently have in Storybook for that image, they are different, with the GA4 CTA present in Storybook and not in the reference image. Weird how it VRT fails locally on |
Wow, that is indeed strange. Perhaps @techanvil or @nfmohit could investigate this further? |
Having discovered that the VRT suite is not actually running as expected in CI, I am parking this issue until that has been resolved: #6151 |
Based on the work on #6151, it looks like this issue will still be relevant, with a reduction in the number of test failures but still a couple of cases that need fixing and also a more general issue of stability with unpredictable timeouts occurring locally. |
@asvinb @aaemnnosttv |
I think it could be worth keeping to see about fixing the timeouts that occur when running locally... |
@aaemnnosttv what machine do you use? For me on mac M1 the VRT tests do not fail. |
@aaemnnosttv @derweili I just checked locally (Mac with Intel chip) and there are no failures: I see images have been updated last night, can you try again @aaemnnosttv ? |
@asvinb the updates were only about removing unused reference images. This should not change anything for this issue. |
That's odd @asvinb – I'm also on a Intel based Mac. I ran it again on develop just now and got the same results 😞 |
@aaemnnosttv is your VRT docker image up to date? Maybe you could try rebuilding it. |
@derweili I deleted the images I had and ran it again and everything's passing now 🤷♂️ 🎉 I'll close this out then but do you know what changed in the image? It's built based on the specific version of backstop but something must have changed 🤔 |
@aaemnnosttv actually I don't really know. With #6324 only images were updated that had visual changes, mostly due to font rendering errors. The VRT image from your example was updated 1 month ago but also without visual changes. https://github.com/google/site-kit-wp/pull/6303/files However, I experienced the error you had (errors without visual change), when I ran backstopjs locally without docker. I did this to debug font loading. So backstop used a local instance of chromium. When running the VRT tests this way, almost all the tests failed even though there were no visual changes. So I though it might be related to an outdated VRT docker image with different settings. But if deleting the test images helped for you, should be delete the test images before the test? Why do we need to keep the test images for all test runs? |
I didn't delete the test images, I deleted the local Docker image! When running again, it rebuilt the image which seemed to do the trick. It must have been something in the
I think this is just how Backstop works, but I could be wrong. It references the generated test files in the report it creates but I'm really not sure why it accumulates them the way it does. |
Bug Description
We recently updated our VRT infrastructure to support running on ARM/M1 machines which are becoming more common rather quickly.
While the build runs on these machines where it didn't before, it seems to have introduced some degree of instability in local runs on Intel machines (and M1s?).
At current, I see ~10 scenario failures when running locally, most of which seem to be very small differences in font rendering although some appear to have timeouts as well, albeit inconsistently.
I do see some amount of failures consistently though. See #4619 (comment)
Steps to reproduce
npm run test:visualtest
locallyAdditional Context
Do not alter or remove anything below. The following sections will be managed by moderators only.
Acceptance criteria
Implementation Brief
Test Coverage
QA Brief
Changelog entry
The text was updated successfully, but these errors were encountered: