Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[tests] bot that runs tests for Ubuntu arm64 is unhealthy #11112

Closed
aslushnikov opened this issue Dec 27, 2021 · 2 comments
Closed

[tests] bot that runs tests for Ubuntu arm64 is unhealthy #11112

aslushnikov opened this issue Dec 27, 2021 · 2 comments
Assignees
@aslushnikov

Comments

@aslushnikov
Copy link
Collaborator

The bot is using docker + qemu to run tests for aarch64 Ubuntu on the x86_64 host machine and fails with the `disk ran out of space" issue.

Turns out Chromium is crashing all the time, so we have lots of artifacts that fill the disk. And Chromium is crashing due to Qemu emulation - it fails with CHECK:

 [1227/190634.136081:FATAL:zygote_main_linux.cc(161)] Check failed: sandbox::ThreadHelpers::IsSingleThreaded().

And indeed, in Qemu environment there are two threads instead of a single one. It might be that Qemu is doing not-good-enough job in remapping these or it might be by design, and Chromium is just overly strict in its assumptions.

Other folks around the internet also reported this, e.g. here and here.

Despite some claims around internet, the --single-process flag doesn't help.

The interim solution would be to disable chromium tests for ubuntu arm64.

Longterm, it looks like this check is not required when chromium is run without sandbox, so it could be moved out of the execution path.
@aslushnikov aslushnikov self-assigned this Dec 27, 2021
aslushnikov added a commit to aslushnikov/playwright that referenced this issue Dec 27, 2021
@aslushnikov
devops: fix docker tests
ed3e1fa 
This patch:
- introduces "#smoke" tests - a subset of tests that makes sure some
  basic Playwright functionality works.
- starts running only smoke tests inside our docker containers.
- stops running Chromium tests inside aarch64 docker container

References microsoft#11112
aslushnikov added a commit to aslushnikov/playwright that referenced this issue Dec 27, 2021
@aslushnikov
devops: fix docker tests
6f5b43e 
This patch:
- introduces "#smoke" tests - a subset of tests that makes sure that
  basic Playwright functionality works. This set is loosely defined;
  feel free to add/remove tests to the set. The only goal is to keep
  this set minimal & fast to run.
- starts running only smoke tests inside our docker containers. This
  should reduce docker tests run from hours to ~15-20 minutes, with most
  time spent building containers.
- stops running Chromium tests inside aarch64 docker container due to
  Chrome disliking QEMU emulation.

References microsoft#11112
aslushnikov added a commit to aslushnikov/playwright that referenced this issue Dec 27, 2021
@aslushnikov
devops: fix docker tests
8258c88 
This patch:
- introduces "#smoke" tests - a subset of tests that makes sure that
  basic Playwright functionality works. This set is loosely defined;
  feel free to add/remove tests to the set. The only goal is to keep
  this set minimal & fast to run.
- starts running only smoke tests inside our docker containers. This
  should reduce docker tests run from hours to ~15-20 minutes, with most
  time spent building containers.
- stops running Chromium tests inside aarch64 docker container due to
  Chrome disliking QEMU emulation.

References microsoft#11112
@aslushnikov aslushnikov mentioned this issue Dec 27, 2021
Closed
@aslushnikov
Copy link
Collaborator Author

My experiments concluded that it's currently impossible to run our browser tests on x86_64 hosts using the qemu emulation of aarch64. These tests simply take too long to complete.

The plan forward now is to use our M1 hardware to run docker ubuntu arm64 tests. I do want to land the #smoke hashtag for tests so that we don't spend too much time running tests.

@aslushnikov
Copy link
Collaborator Author

These are moved to internal repository now and run #smoke tests; this is fixed now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aslushnikov aslushnikov

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@aslushnikov