New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
VRT tests cannot be run on ARM Macs/M1 Macs #4619
Comments
This comment has been minimized.
This comment has been minimized.
Update: looks like garris/BackstopJS#1300 (comment) might fix it for us, I'll try that later today as I'm working on an issue that'll require VRT updates 😅 |
Another update: looks like Puppeteer will soon have an ARM build, there's a PR here: puppeteer/puppeteer#7546 and the issue is here: puppeteer/puppeteer#6622. Waiting a bit longer on this one as the workarounds aren't working for me… |
Hi @tofumatt just checking you are still planning on working on this IB? If not feel free to unassign yourself. |
Hi @FlicHollis indeed, it's mostly waiting on the upstream fixes mentioned in #4619 (comment). But I'm keeping myself assigned to write up the IB once that's ready. |
Looks like Puppeteer 14 includes a fix. I'll test that and write up an IB soon if that works 🙂 |
Just another update here: even using Backstop 6.1.0 this still isn't fixed, I think we're waiting on more from garris/BackstopJS#1300. I might try making our own |
Creating a custom Docker image using this
And then building it and using that image, along with ensuring Docker for Mac could use at least 4 CPUs let me run the VRTs. Without upping the CPU limit to at least 4, it would hang: |
This is mostly ready, but any issues from CI images and the involvement of multiple developers to test on both Intel and ARM Macs means I'm putting this one at a 7. |
IB ✅ |
@tofumatt could you please review my PR #5692 and try it on your end? I have added a custom image for backtopjs uses chromium browser to render stories. Unfortunately, sometimes it works unstable on my end and fails due to some small fonts changes. Can you try to troubleshoot it on your end? Also as to the registry, I think we won't be able to use GitHub's container registry because it saves images on the organisation level, not on the repository one. So it means that if we want to publish an image, it will be available for the whole Google organisation which is undesirable. |
Approval
|
Hmm, I'm not currently able to run the tests on Intel on a Mac, just on a Windows PC I have. And even then, I usually run them in a Linux VM. Even when I'd run the VRTs on my M1 in "Intel" it'd be x86 emulation running a Linux VM, and it was so slow that I'd often get weird failures. I think we'll need someone on Intel to test, but I think @eugene-manuilov mentioned the tests were a bit flakey for him as well after this change? I wonder if there's anything we should be doing to improve that? Or just have longer timeouts? 🤷🏻♂️ That said, I see failures locally too: I saw similar issues when I ran the tests locally from the original PR, but then updated the VRTs and saw they still passed on CI (which isn't ARM) so I figured it was all good. But it seems there are super-subtle rendering differences and timeout issues with the new image 🤔 |
Closing this as done as a first step. Will open a new issue to see about addressing the failures we're seeing locally. |
@FlicHollis I've opened #5824 as the follow up for this sprint which is already assigned 👍 |
Bug Description
Using an ARM/M1 Mac, running
npm run test:visualtest
will fail with an error because the Docker image for Backstop is not configured to run an ARM64 image. It will segfault and fail. If you are on a machine that you migrated from an Intel Mac, it will then open whatever previous results you ran in your project folder.Until BackstopJS provides a fully-integrated solution (see: garris/BackstopJS#1300), we will need to update our codebase to allow the VRTs to run on ARM Macs, as several core team members are using these machines and can't run (and thus can't update) the VRTs locally.
Do not alter or remove anything below. The following sections will be managed by moderators only.
Acceptance criteria
npm run test:visualtest
locally on both Intel and ARM machines. It would be preferable if these could be run on ARM without needing to create any custom images/etc. locally.Implementation Brief
googlesitekit/vrt:VERSION
image instead ofbackstopjs/backstopjs:VERSION
Test Coverage
QA Brief
Changelog entry
The text was updated successfully, but these errors were encountered: