GitHub Action: New M1 runner available to all plans, including open source 🚀

[Steve-Glass]

Breaking changes

The new M1 runner is available for all plans, free in public repositories, and eligible to consume included free plan minutes in private repositories. The new runner operates exclusively on macOS 14 and to use it, simply update the runs-on key in your YAML workflow file to macos-14.

Change log

Target date

Available as of 1/30/2023

The motivation for the changes

N/A

Possible impact

N/A

Platforms affected

• Azure DevOps
• GitHub Actions

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Mitigation ways

N/A

[fwcd]

There's something strange with the Xcode installation in these runners, running xcrun --show-sdk-version outputs

xcodebuild: error: SDK "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" cannot be located.
xcrun: error: unable to lookup item 'SDKVersion' in SDK '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk'

See e.g. fwcd/m1xxx#63

I can't reproduce this locally, even with xcode-select -s /Library/Developer/CommandLineTools.

[darthmaim]

The new macos-14 runners contain a link at the top of the log to included software:

Included Software: https://github.com/actions/runner-images/blob/macOS-14/20240116.1/images/macos/macos-14-arm64-Readme.md

This link is a 404 and should instead link to https://github.com/actions/runner-images/blob/main/images/macos/macos-14-arm64-Readme.md

[Vyazovoy]

@darena-patrick, looks like this ticket says exactly that. If you switch to macos-14 your jobs will be running on the M1 (aka arm64) runner.

[darena-patrick]

@darena-patrick, looks like this ticket says exactly that. If you switch to macos-14 your jobs will be running on the M1 (aka arm64) runner.

yea, I deleted my comment once I realized that :)

[darena-patrick]

@Steve-Glass any plans to support Docker on these images? Otherwise, any word on when linux/arm64 will be available?

[darthmaim]

@Steve-Glass any plans to support Docker on these images? Otherwise, any word on when linux/arm64 will be available?

No docker for the M1 images:
https://docs.github.com/en/actions/using-github-hosted-runners/about-larger-runners/about-larger-runners#limitations-for-macos-larger-runners:

Due to a limitation of Apple's Virtualization Framework, which our hypervisor uses, nested-virtualization is not supported by arm64 runners.

And for linux arm images there is this open (but locked) issue: #5631

[mikhailkoliada]

The new macos-14 runners contain a link at the top of the log to included software:

Included Software: https://github.com/actions/runner-images/blob/macOS-14/20240116.1/images/macos/macos-14-arm64-Readme.md

This link is a 404 and should instead link to https://github.com/actions/runner-images/blob/main/images/macos/macos-14-arm64-Readme.md

This is expected as no full public releases happened yet, it should be fixed with the next release :)

[Korijn]

When is this coming to Azure Pipelines?

[douglascamata]

@Steve-Glass there's a huge breaking change here: the M1 processor does not support nested virtualization. This will break every workflow out there using Colima/Lima or any other VM-based solution for running Docker in macOS.

Neither QEMU nor Virtualization.framework can be used to start VMs inside the runners, because they are already VMs.

[douglascamata]

QEMU will report {"level":"debug","msg":"qemu[stderr]: qemu-system-aarch64: Error: HV_UNSUPPORTED","time":"2024-02-01T19:34:57Z"}

Trying to use Virtualization.framework will report {"level":"fatal","msg":"Error Domain=VZErrorDomain Code=2 Description=\"Invalid virtual machine configuration. Virtualization is not available on this hardware.\" UserInfo={\n NSLocalizedFailure = \"Invalid virtual machine configuration.\";\n NSLocalizedFailureReason = \"Virtualization is not available on this hardware.\";\n}","time":"2024-02-01T19:36:49Z"}

[Vyazovoy]

@douglascamata, I'm not sure that it should be called a breaking change since it was explicitly mentioned as a limitation.

[douglascamata]

@Vyazovoy being a documented change doesn’t change the fact that it is a breaking change. They are two different things.

[cgrindel]

I would like to upvote @fwcd comment. I too am seeing an error running xcrun --show-sdk-version.

[Luthaf]

Is the GPU available on these runners? Trying to use them with PyTorch and the MPS (i.e. Metal) backend result in error like this (See https://github.com/lab-cosmo/metatensor/actions/runs/7847356344/job/21416128283)

RuntimeError: MPS backend out of memory (MPS allocated: 0 bytes, other allocations: 0 bytes, max allowed: 7.93 GB). Tried to allocate 256 bytes on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).

Setting PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 in the env changes the error message to (see https://github.com/lab-cosmo/metatensor/actions/runs/7847470210/job/21416465675)

RuntimeError: MPS backend out of memory (MPS allocated: 0 bytes, other allocations: 0 bytes). Tried to allocate 256 bytes on shared pool.

---

There are other reports of the same issue on the PyTorch forums: https://discuss.pytorch.org/t/mps-back-end-out-of-memory-on-github-action/189773

[wtdcode]

"isb" instructions are recognized as EXC_BAD_INSTRUCTION in the new M1 runners.

[hovancik]

@fwcd @cgrindel were you able to fix errors with xcrun --show-sdk-version?

[cgrindel]

Unfortunately, no. I still see the following:

++ xcrun --show-sdk-version
2024-04-28 15:28:55.452 xcodebuild[11203:62473] Writing error result bundle to /var/folders/3m/p59k4qdj0f17st0gn2cmj3640000gn/T/ResultBundle_2024-28-04_15-28-0055.xcresult
xcodebuild: error: SDK "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" cannot be located.
xcrun: error: unable to lookup item 'SDKVersion' in SDK '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk'

[hovancik]

I've created issue #9811

[joffrey-bion]

What's the point using mac runners for Docker tasks if you can use Linux runners?

Sometimes Docker is just here to provide a side service while running native tests that require a specific system. I want to test my web socket client implementation on macOS, not on linux, but I still want to run my docker-based test server during the build.

[marciniwanicki]

It's now possible to build apps relying on Virtualization.framework 🥳, I love it, thank you so much!