Crosscompile python aarch64 wheels with dockcross #25418

jtattermusch · 2021-02-11T13:08:19Z

Enables building of aarch64 wheels via crosscompilation.
Uses similar tricks (overriding plat-name and EXT_SUFFIX) as protocolbuffers/protobuf#8280.

Some limitations:

currently twine check and auditwheel repair is being skipped
still need to doublecheck whether the wheels work well

The nice thing is that crosscompilation isn't slower than the regular build (unlike building under emulation).

FTR a very similar technique could be used to crosscompile the armv7 and armv6 (currently under "linux_extra" and being built under an emulator which takes ages).

jtattermusch · 2021-02-12T13:59:57Z

@lidizheng @gnossen this is not ready for merge yet, but good enough for first review pass.

jtattermusch · 2021-02-12T14:00:19Z

@janaknat please review as well.

lidizheng · 2021-02-12T18:25:09Z

tools/run_tests/artifacts/build_artifact_python.sh

+  "${PYTHON}" -m pip install virtualenv
+  "${PYTHON}" -m virtualenv venv || { "${PYTHON}" -m pip install virtualenv==16.7.9 && "${PYTHON}" -m virtualenv venv; }
+  venv/bin/python -m pip install "twine<=2.0"
+  venv/bin/python -m twine check dist/* tools/distrib/python/grpcio_tools/dist/*


Is there a reason for skipping twine checks? We are following manylinux2014, should be supported by PyPI official packages?

We need to skip auditwheel and twine check for now, since when crosscompiling we're running on x64 image, but we got aarch64 wheel. To perform checks, you'd likely need to run the check under qemu (available in the docker image), but also on aarch64 version of python etc.
The plan is to first test the aarch64 wheels manually. It should be possible to add back twine and auditwheel checks, but more work is needed.

src/python/grpcio/commands.py

tools/run_tests/artifacts/artifact_targets.py

jtattermusch · 2021-02-13T05:48:20Z

Artifact build results (for aarch64)

passing for python3.8 and 3.9
failure for python3.7 (tools/run_tests/artifacts/build_artifact_python.sh: line 26: /opt/python/cp37-cp37/bin/pip: No such file or directory) https://source.cloud.google.com/results/invocations/1bcb1668-6aa9-4916-9748-f291ec40d08e/targets/github%2Fgrpc/tests;query=manylinux2014;passed=true;group=tasks;test=build_artifact.python_manylinux2014_aarch64_cp37-cp37;row=6

built wheels can be downloaded here:
https://console.cloud.google.com/storage/browser/grpc-testing-kokoro-prod/test_result_public/prod/grpc/core/pull_request/linux/grpc_build_artifacts/22486/20210212-055319

jtattermusch · 2021-02-13T06:01:34Z

Manual run of auditwheel show on arm64 machine:

python -m auditwheel show grpcio-1.37.0.dev0-cp38-cp38-manylinux2014_aarch64.whl

grpcio-1.37.0.dev0-cp38-cp38-manylinux2014_aarch64.whl is consistent
with the following platform tag: "manylinux_2_24_aarch64".

The wheel references external versioned symbols in these system-
provided shared libraries: libpthread.so.0 with versions
{'GLIBC_2.17'}, libm.so.6 with versions {'GLIBC_2.17'}, libc.so.6 with
versions {'GLIBC_2.17'}, libstdc++.so.6 with versions
{'GLIBCXX_3.4.17', 'GLIBCXX_3.4.15', 'GLIBCXX_3.4.14',
'GLIBCXX_3.4.18', 'CXXABI_1.3.5', 'GLIBCXX_3.4.11', 'GLIBCXX_3.4.9',
'GLIBCXX_3.4.20', 'CXXABI_1.3', 'GLIBCXX_3.4'}

This constrains the platform tag to "manylinux_2_24_aarch64". In order
to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.

So looks like some work is still needed. Not sure if the problem could be the too-new version of gcc being used (we do use 4.9.4 instead of the default 4.8 because gRPC doesn't compile under gcc 4.8 - see the dockerfile)

jtattermusch · 2021-02-13T06:03:00Z

Manual run of auditwheel show on arm64 machine:

python -m auditwheel show grpcio-1.37.0.dev0-cp38-cp38-manylinux2014_aarch64.whl

grpcio-1.37.0.dev0-cp38-cp38-manylinux2014_aarch64.whl is consistent
with the following platform tag: "manylinux_2_24_aarch64".

The wheel references external versioned symbols in these system-
provided shared libraries: libpthread.so.0 with versions
{'GLIBC_2.17'}, libm.so.6 with versions {'GLIBC_2.17'}, libc.so.6 with
versions {'GLIBC_2.17'}, libstdc++.so.6 with versions
{'GLIBCXX_3.4.17', 'GLIBCXX_3.4.15', 'GLIBCXX_3.4.14',
'GLIBCXX_3.4.18', 'CXXABI_1.3.5', 'GLIBCXX_3.4.11', 'GLIBCXX_3.4.9',
'GLIBCXX_3.4.20', 'CXXABI_1.3', 'GLIBCXX_3.4'}

This constrains the platform tag to "manylinux_2_24_aarch64". In order
to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.

So looks like some work is still needed. Not sure if the problem could be the too-new version of gcc being used (we do use 4.9.4 instead of the default 4.8 because gRPC doesn't compile under gcc 4.8 - see the dockerfile)

An attempt to auditwheel repair:

python -m auditwheel repair grpcio-1.37.0.dev0-cp38-cp38-manylinux2014_aarch64.whl || echo FAILED
INFO:auditwheel.main_repair:Repairing grpcio-1.37.0.dev0-cp38-cp38-manylinux2014_aarch64.whl
usage: __main__.py [-h] [-V] [-v] command ...
__main__.py: error: cannot repair "grpcio-1.37.0.dev0-cp38-cp38-manylinux2014_aarch64.whl" to "manylinux2014_aarch64" ABI because of the presence of too-recent versioned symbols. You'll need to compile the wheel on an older toolchain.

jtattermusch · 2021-02-15T09:42:08Z

Artifact build results (for aarch64)

passing for python3.8 and 3.9

failure for python3.7 (tools/run_tests/artifacts/build_artifact_python.sh: line 26: /opt/python/cp37-cp37/bin/pip: No such file or directory) https://source.cloud.google.com/results/invocations/1bcb1668-6aa9-4916-9748-f291ec40d08e/targets/github%2Fgrpc/tests;query=manylinux2014;passed=true;group=tasks;test=build_artifact.python_manylinux2014_aarch64_cp37-cp37;row=6

built wheels can be downloaded here:
https://console.cloud.google.com/storage/browser/grpc-testing-kokoro-prod/test_result_public/prod/grpc/core/pull_request/linux/grpc_build_artifacts/22486/20210212-055319

The python3.7 failure seems to be caused by a typo that's now fixed.

janaknat · 2021-02-15T19:50:22Z

tools/dockerfile/grpc_artifact_python_manylinux2014_aarch64/Dockerfile

+FROM dockcross/manylinux2014-aarch64:20200929-608e6ac
+
+# Update the package manager
+RUN yum update -y && yum install -y curl-devel expat-devel gettext-devel openssl-devel zlib-devel


Are these installing the aarch64 dev libs? When I tried it, it pulled in the x86 version.

nope, this is installing x86 libs. I guess the reason why this doesn't matter is that they are actually not strictly required for a successful build (I copied the line from x64 dockerfile and didn't investigate whether they are really needed).

janaknat · 2021-02-15T19:50:37Z

tools/dockerfile/grpc_artifact_python_manylinux2014_aarch64/Dockerfile

+RUN /opt/python/cp36-cp36m/bin/pip install --upgrade cython
+RUN /opt/python/cp37-cp37m/bin/pip install --upgrade cython
+RUN /opt/python/cp38-cp38/bin/pip install --upgrade cython
+RUN /opt/python/cp39-cp39/bin/pip install --upgrade cython


Same question as above.

it's a x64 cython, but that doesn't really matter it seems. Even the python itself is a x64 python. That seems correct as we're crosscompiling and we need to run the build on a x64 machine (and cython is what invokes the build commands).

lidizheng · 2021-02-16T18:22:23Z

tools/run_tests/artifacts/build_artifact_python.sh

@@ -57,6 +58,7 @@ ${SETARCH_CMD} "${PYTHON}" setup.py sdist

 # Wheel has a bug where directories don't get excluded.
 # https://bitbucket.org/pypa/wheel/issues/99/cannot-exclude-directory
+# shellcheck disable=SC2086
 ${SETARCH_CMD} "${PYTHON}" setup.py bdist_wheel $WHEEL_PLAT_NAME_FLAG


Double quote (SC2086) seems applicable to this shell command. Quoting variables should fix the shellcheck complain.

That wouldn't work. WHEEL_PLAT_NAME_FLAG can be empty and in that case "$WHEEL_PLAT_NAME_FLAG" becomes "" and an empty arg will be forcibly passed to the command (which does break it).

lidizheng · 2021-02-16T18:23:43Z

tools/run_tests/artifacts/build_artifact_python.sh

+
+  # Set to empty string to disable the option (see https://github.com/grpc/grpc/issues/24498)
+  # TODO: enable ASM optimizations for crosscompiled wheels
+  export GRPC_BUILD_WITH_BORING_SSL_ASM=""


Maybe link to the PR changing the BoringSSL ASM enable flag? #25444

jtattermusch · 2021-02-16T18:36:13Z

Looks like the only problematic symbol is
0000000000000000 DF *UND* 0000000000000000 GLIBCXX_3.4.20 _ZSt24__throw_out_of_range_fmtPKcz (and it seems to be used by absl)

jtattermusch · 2021-03-01T15:09:47Z

So far I haven't figured out how to make sure that the GCC 4.9 crosscompiler in the dockcross image uses symbols that are compliant with manylinux2014 (as discussed in #25418 (comment)), but I used a workaround instead:

when crosscompiling linux aarch64 wheels, use -static-libstdc++ after which no libstdc++ are referenced (because they are linked statically) and the resulting wheels are manylinux2014 compatible (confirmed with manual run auditwheel)
this approach increases the size of the binary wheels slightly (increase of size approx. 33MB to ~35MB), but that seems reasonable since we currently don't have a better way of crosscompiling for aarch64. The plan is to remove the workaround in the future, but for now having a set of aarch64 wheels is the priority.

@lidizheng @gnossen I think this PR is now ready to merge. PTAL

gnossen

Nice investigation!

lidizheng

Thanks for digging this!

jtattermusch · 2021-03-02T06:31:39Z

known failures:
#25594

jtattermusch force-pushed the crosscompile_python_wheel branch from 73d4c2e to 8d4038c Compare February 12, 2021 13:52

jtattermusch changed the title ~~DO NOT MERGE: Crosscompile python aarch64 wheels~~ Crosscompile python aarch64 wheels with dockcross Feb 12, 2021

jtattermusch assigned gnossen Feb 12, 2021

jtattermusch requested review from lidizheng and gnossen February 12, 2021 13:59

jtattermusch marked this pull request as ready for review February 12, 2021 14:00

jtattermusch requested review from apolcyn and nicolasnoble as code owners February 12, 2021 14:00

jtattermusch added this to the ARM64 support milestone Feb 12, 2021

lidizheng reviewed Feb 12, 2021

View reviewed changes

gnossen reviewed Feb 12, 2021

View reviewed changes

src/python/grpcio/commands.py Outdated Show resolved Hide resolved

tools/run_tests/artifacts/artifact_targets.py Show resolved Hide resolved

jtattermusch added release notes: yes Indicates if PR needs to be in release notes lang/Python labels Feb 13, 2021

janaknat reviewed Feb 15, 2021

View reviewed changes

lidizheng approved these changes Feb 16, 2021

View reviewed changes

jtattermusch added 5 commits March 1, 2021 09:54

build aarch64 python wheels via crosscompilation

2820f85

yapf format code

60db2a4

fix shellcheck complaints

9ccb8e0

fix python37 aarch64 wheel build

ed8f029

build python wheels on linux aarch64 with static libstdc++

5849d85

jtattermusch force-pushed the crosscompile_python_wheel branch from 8a2ba29 to 5849d85 Compare March 1, 2021 14:59

yapf format code

fdd3bee

gnossen approved these changes Mar 1, 2021

View reviewed changes

lidizheng approved these changes Mar 1, 2021

View reviewed changes

jtattermusch merged commit fcd43e9 into grpc:master Mar 2, 2021

jtattermusch mentioned this pull request Apr 3, 2021

Provide ARM64v8 wheel #21283

Closed

This was referenced Aug 18, 2022

ARM based docker image apache/pulsar#12944

Closed

Bump grpcio dep version of Python client for supporting ARM apache/bookkeeper#3449

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crosscompile python aarch64 wheels with dockcross #25418

Crosscompile python aarch64 wheels with dockcross #25418

jtattermusch commented Feb 11, 2021 •

edited

jtattermusch commented Feb 12, 2021

jtattermusch commented Feb 12, 2021

lidizheng Feb 12, 2021

jtattermusch Feb 13, 2021

jtattermusch commented Feb 13, 2021

jtattermusch commented Feb 13, 2021

jtattermusch commented Feb 13, 2021

jtattermusch commented Feb 15, 2021

janaknat Feb 15, 2021

jtattermusch Feb 16, 2021

janaknat Feb 15, 2021

jtattermusch Feb 16, 2021

lidizheng Feb 16, 2021

jtattermusch Feb 16, 2021

lidizheng Feb 16, 2021

jtattermusch commented Feb 16, 2021 •

edited

jtattermusch commented Mar 1, 2021 •

edited

gnossen left a comment

lidizheng left a comment

jtattermusch commented Mar 2, 2021

Crosscompile python aarch64 wheels with dockcross #25418

Crosscompile python aarch64 wheels with dockcross #25418

Conversation

jtattermusch commented Feb 11, 2021 • edited

jtattermusch commented Feb 12, 2021

jtattermusch commented Feb 12, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtattermusch commented Feb 13, 2021

jtattermusch commented Feb 13, 2021

jtattermusch commented Feb 13, 2021

jtattermusch commented Feb 15, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtattermusch commented Feb 16, 2021 • edited

jtattermusch commented Mar 1, 2021 • edited

gnossen left a comment

Choose a reason for hiding this comment

lidizheng left a comment

Choose a reason for hiding this comment

jtattermusch commented Mar 2, 2021

jtattermusch commented Feb 11, 2021 •

edited

jtattermusch commented Feb 16, 2021 •

edited

jtattermusch commented Mar 1, 2021 •

edited