Arm wheels missing from latest release

[westonpace]

These files are present in the 3.10 release but the equivalent files are missing from the 3.11 release:

h5py-3.10.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
h5py-3.10.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
h5py-3.10.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
h5py-3.10.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

[westonpace]

Further context: #2405 (comment)

[glenn-jocher]

We've seen the same problem at https://github.com/ultralytics/ultralytics. Missing aarch64 wheels have crashed the Arm64 Docker deployments that install tensorflow (with h5py as a required tensorflow dependency).

We've temporarily resolved this by avoiding h5py==3.11.0 installs on aarch64 in ultralytics/ultralytics#9956

[lpsinger]

cibuildwheel provides instructions for how to do aarch64 builds under qemu emulation on the free tier of GitHub Actions.

[tacaswell]

I would be happy to review a PR moving us to cibuildwheel on GHA.

This issue highlights some of the inherent problems in the way wheels get built (see https://pypackaging-native.github.io for more details).

[lpsinger]

I would be happy to review a PR moving us to cibuildwheel on GHA.

I would find that a little daunting for h5py because of the many different old CI workflows this would have to replace: Travis, Azure Pipelines, and the existing nightly GH Actions wheel job.

But all you really have to do is add this to your pyproject.toml file, and cibuildwheel will do the rest for you:

[tool.cibuildwheel.linux]
# Build wheels for aarch64 under emulation.
archs = ["auto", "aarch64"]

[aragilar]

My concern with using GHA with QEMU via cibuildwheel is that from what I can see it doesn't actually test the wheel, whereas our current setup across the different provided does do that (that was the reason to use travis, to test on native hardware). Also, given we've previously had issues specific to non-x86 systems, I'm really wary of publishing something that is untested on its native platform (or just straight up untested).

I think if someone either provides a PR which replaces travis with a CI provider who provides native ARM64 (e.g. what's the state of ARM MacOS on GHA, and can we run a linux VM on there?), or works out how to run the entire flow inside QEMU (so we do the build+test on ARM64), I would be inclined to review it (not speaking for any of the other maintainers, but I'd rather a native platform, just so we're not bitten by any emulation bugs).

[lpsinger]

My concern with using GHA with QEMU via cibuildwheel is that from what I can see it doesn't actually test the wheel

Yes, it does test the wheels built under emulation. Here are three packages that I work on that build Linux aarch64 wheels under QEMU emulation on GitHub actions; in all cases cibuildwheel tests the wheels built under emulation.

• https://github.com/healpy/healpy
• https://github.com/nasa/radbelt
• https://github.com/astropy/astropy-healpix

[aragilar]

pypa/cibuildwheel#1771 suggests this won't work for h5py, as we also have custom manylinux wheels 😞

[lpsinger]

pypa/cibuildwheel#1771 suggests this won't work for h5py, as we also have custom manylinux wheels 😞

Do you mean to say, it won't work for h5py because you have a custom manylinux Docker image?

[aragilar]

Yep, images, not wheels (and now I'm going to bed, obviously too tired to write logical sentences).

[lpsinger]

It's odd that they are having a problem with pypa/cibuildwheel#1771 given that this works with the default manylinux Docker images. Perhaps there is a problem with the metadata for their particular Docker images?

[Edward-Knight]

Not immediately useful or relevant, but perhaps a maintainer could sign up for the private beta of native arm runners: https://resources.github.com/devops/accelerate-your-cicd-with-arm-and-gpu-runners-in-github-actions/
Also apparently this may leave beta later this year?

[tacaswell]

but perhaps a maintainer could sign up for the private beta of native arm runners:

Done, but from the language on the form they are looking for paid early adopters 🤣

[njzjz]

For anyone who encounters this issue in the CI, setting

export PIP_PREFER_BINARY=1

will download a wheel from an old version.

[DDxPlague]

Looks like Github is looking to add more folks in in Jun/July.

https://github.com/orgs/community/discussions/19197#discussioncomment-9167725

In the meantime if @AWSjswinney and I can swing some credits for an AWS Graviton self-hosted runner, would you be willing to get the build working on that for now? And then when the managed runners are released, move to a self managed runner?

[joerick]

pypa/cibuildwheel#1771 suggests this won't work for h5py, as we also have custom manylinux wheels 😞

Hello there, I spotted the back-link from that issue. Curious if you have actually hit this issue or just assuming you will? Many of cibuildwheel's users use custom manylinux images in emulation without problems, but I'd like to know if it's a wider issue.

[AWSjswinney]

From #2414:

If you are at AWS @AWSjswinney does this mean aarch64 h5py is critical to something at AWS? If so, you may be able to help us solve the issue with building aarch64 wheels by sponsoring some CI cycles on travis CI.

@tacaswell We reached out to TravisCI and they let us know that you should have credits again. Can you check please?

[hoechenberger]

Hello, sorry for asking, but I, too, ran into the problem of missing wheels for aarch64 (running a Docker container on my Apple Silicon machine). I'm not sure I fully understand the above conversation – are there plans to provide aarch64 wheels in the future? What is the expected timeframe? Thank you very much for your time and hard work!

[larsoner]

Over in #2434, using emulation at least builds nightly wheels for aarch64. Maybe that could be used somehow, but it's at least a proof of concept.