pip grpcio/grpcio-tools 1.62.2 installation fails on linux

[jiri-cholasta-tfs]

What version of gRPC and what language are you using?

grpcio/grpcio-tools 1.62.2 and python

What operating system (Linux, Windows,...) and version?

Linux running in docker/compose:1.29.2 docker image

What runtime / compiler are you using (e.g. python version or version of gcc)

Python 3.8.10, gcc (Alpine 9.3.0) 9.3.0

What did you do?

just run grpc python module installation:
pip3 install grpcio-tools googleapis-common-protos

What did you expect to see?

Installation finished successfully. Just like with previous 1.62.1 version which is working fine on the same environment.

What did you see instead?

gcc compilation failure
test_env_1 | creating None
test_env_1 | creating None/tmp
test_env_1 | creating None/tmp/tmpnjsr5b8s
test_env_1 | gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fomit-frame-pointer -g -fno-semantic-interposition -fomit-frame-pointer -g -fno-semantic-interposition -fomit-frame-pointer -g -fno-semantic-interposition -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/python3.8 -c /tmp/tmpnjsr5b8s/a.c -o None/tmp/tmpnjsr5b8s/a.o
test_env_1 | Traceback (most recent call last):
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/unixccompiler.py", line 188, in _compile
test_env_1 | self.spawn(compiler_so + cc_args + [src, '-o', obj] + extra_postargs)
test_env_1 | File "/tmp/pip-install-j8sg0j30/grpcio/src/python/grpcio/_spawn_patch.py", line 58, in _commandfile_spawn
test_env_1 | _classic_spawn(self, command, **kwargs)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/ccompiler.py", line 1041, in spawn
test_env_1 | spawn(cmd, dry_run=self.dry_run, **kwargs)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/spawn.py", line 68, in spawn
test_env_1 | raise DistutilsExecError(f"command {cmd!r} failed with exit code {exitcode}")
test_env_1 | distutils.errors.DistutilsExecError: command '/usr/bin/gcc' failed with exit code 1
test_env_1 |
test_env_1 | During handling of the above exception, another exception occurred:
test_env_1 |
test_env_1 | Traceback (most recent call last):
test_env_1 | File "/tmp/pip-install-j8sg0j30/grpcio/src/python/grpcio/commands.py", line 313, in build_extensions
test_env_1 | build_ext.build_ext.build_extensions(self)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 479, in build_extensions
test_env_1 | self._build_extensions_serial()
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 505, in _build_extensions_serial
test_env_1 | self.build_extension(ext)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 252, in build_extension
test_env_1 | _build_ext.build_extension(self, ext)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 560, in build_extension
test_env_1 | objects = self.compiler.compile(
test_env_1 | File "/tmp/pip-install-j8sg0j30/grpcio/src/python/grpcio/_parallel_compile_patch.py", line 70, in _parallel_compile
test_env_1 | multiprocessing.pool.ThreadPool(BUILD_EXT_COMPILER_JOBS).map(
test_env_1 | File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
test_env_1 | return self._map_async(func, iterable, mapstar, chunksize).get()
test_env_1 | File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
test_env_1 | raise self._value
test_env_1 | File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
test_env_1 | result = (True, func(*args, **kwds))
test_env_1 | File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
test_env_1 | return list(map(*args))
test_env_1 | File "/tmp/pip-install-j8sg0j30/grpcio/src/python/grpcio/_parallel_compile_patch.py", line 65, in _compile_single_file
test_env_1 | self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/unixccompiler.py", line 190, in _compile
test_env_1 | raise CompileError(msg)
test_env_1 | distutils.errors.CompileError: command '/usr/bin/gcc' failed with exit code 1
test_env_1 |
test_env_1 | During handling of the above exception, another exception occurred:
test_env_1 |
test_env_1 | Traceback (most recent call last):
test_env_1 | File "", line 1, in
test_env_1 | File "/tmp/pip-install-j8sg0j30/grpcio/setup.py", line 591, in
test_env_1 | setuptools.setup(
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/init.py", line 104, in setup
test_env_1 | return distutils.core.setup(**attrs)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 184, in setup
test_env_1 | return run_commands(dist)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
test_env_1 | dist.run_commands()
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
test_env_1 | self.run_command(cmd)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/dist.py", line 967, in run_command
test_env_1 | super().run_command(command)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
test_env_1 | cmd_obj.run()
test_env_1 | File "/usr/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 223, in run
test_env_1 | self.run_command('build')
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
test_env_1 | self.distribution.run_command(command)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/dist.py", line 967, in run_command
test_env_1 | super().run_command(command)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
test_env_1 | cmd_obj.run()
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 132, in run
test_env_1 | self.run_command(cmd_name)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
test_env_1 | self.distribution.run_command(command)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/dist.py", line 967, in run_command
test_env_1 | super().run_command(command)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
test_env_1 | cmd_obj.run()
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 91, in run
test_env_1 | _build_ext.run(self)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
test_env_1 | self.build_extensions()
test_env_1 | File "/tmp/pip-install-j8sg0j30/grpcio/src/python/grpcio/commands.py", line 317, in build_extensions
test_env_1 | raise CommandError(
test_env_1 | commands.CommandError: Failed build_ext step:
test_env_1 | Traceback (most recent call last):
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/unixccompiler.py", line 188, in _compile
test_env_1 | self.spawn(compiler_so + cc_args + [src, '-o', obj] + extra_postargs)
test_env_1 | File "/tmp/pip-install-j8sg0j30/grpcio/src/python/grpcio/_spawn_patch.py", line 58, in _commandfile_spawn
test_env_1 | _classic_spawn(self, command, **kwargs)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/ccompiler.py", line 1041, in spawn
test_env_1 | spawn(cmd, dry_run=self.dry_run, **kwargs)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/spawn.py", line 68, in spawn
test_env_1 | raise DistutilsExecError(f"command {cmd!r} failed with exit code {exitcode}")
test_env_1 | distutils.errors.DistutilsExecError: command '/usr/bin/gcc' failed with exit code 1
test_env_1 |
test_env_1 | During handling of the above exception, another exception occurred:
test_env_1 |
test_env_1 | Traceback (most recent call last):
test_env_1 | File "/tmp/pip-install-j8sg0j30/grpcio/src/python/grpcio/commands.py", line 313, in build_extensions
test_env_1 | build_ext.build_ext.build_extensions(self)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 479, in build_extensions
test_env_1 | self._build_extensions_serial()
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 505, in _build_extensions_serial
test_env_1 | self.build_extension(ext)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 252, in build_extension
test_env_1 | _build_ext.build_extension(self, ext)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 560, in build_extension
test_env_1 | objects = self.compiler.compile(
test_env_1 | File "/tmp/pip-install-j8sg0j30/grpcio/src/python/grpcio/_parallel_compile_patch.py", line 70, in _parallel_compile
test_env_1 | multiprocessing.pool.ThreadPool(BUILD_EXT_COMPILER_JOBS).map(
test_env_1 | File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
test_env_1 | return self._map_async(func, iterable, mapstar, chunksize).get()
test_env_1 | File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
test_env_1 | raise self._value
test_env_1 | File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
test_env_1 | result = (True, func(*args, **kwds))
test_env_1 | File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
test_env_1 | return list(map(*args))
test_env_1 | File "/tmp/pip-install-j8sg0j30/grpcio/src/python/grpcio/_parallel_compile_patch.py", line 65, in _compile_single_file
test_env_1 | self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
test_env_1 | File "/usr/lib/python3.8/site-packages/setuptools/_distutils/unixccompiler.py", line 190, in _compile
test_env_1 | raise CompileError(msg)
test_env_1 | distutils.errors.CompileError: command '/usr/bin/gcc' failed with exit code 1

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

See TROUBLESHOOTING.md for how to diagnose problems better.

Anything else we should know about your project / environment?

[XuanWang-Amos]

Are you installing 1.62.2? We don't have a 1.42.2 version.

Assume you mean 1.62.2, we do have wheels for musllinux + Python 3.8: grpcio_tools-1.62.2-cp38-cp38-musllinux_1_1_x86_64.whl, I don't think we need build from source when you run pip install. I also verified locally and pip is installing wheels for the environment you mentioned (docker/compose:1.29.2 + Python 3.8.10).

Can you share more details about your setup? It also helps if you can share the whole error log with us (including the command you used).

[jiri-cholasta-tfs]

Yes, you are right, it's 1.62.2, sorry for confusion. I've fixed the version in issue captions as well as problem description.
I will provide requested details.

[jiri-cholasta-tfs]

Hi,
it's strange, but today it's working fine using the same docker image as on Friday when it failed.

docker-compose image is defined as:

FROM docker/compose:1.29.2
RUN apk add "python3<3.9" py3-pip py3-wheel bash curl curl-dev postgresql-bdr-dev gcc python3-dev musl-dev wget postgresql-client maven g++ linux-headers
RUN pip3 install requests jproperties pytest psycopg2 python-dotenv pytest-timestamper python-dateutil pycurl

and today it's working fine, it successfully built grpcio_tools-1.62.2-cp38-cp38-linux_x86_64.whl and grpcio-1.62.2-cp38-cp38-linux_x86_64.whl:

pip3 install --upgrade setuptools
test_env_1 | Collecting setuptools
test_env_1 | Using cached setuptools-69.5.1-py3-none-any.whl (894 kB)
test_env_1 | Installing collected packages: setuptools
test_env_1 | Attempting uninstall: setuptools
test_env_1 | Found existing installation: setuptools 47.0.0
test_env_1 | Uninstalling setuptools-47.0.0:
test_env_1 | Successfully uninstalled setuptools-47.0.0
test_env_1 | Successfully installed setuptools-69.5.1

pip3 install grpcio-tools googleapis-common-protos
test_env_1 | /usr/bin/pip3:6: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
test_env_1 | from pkg_resources import load_entry_point
test_env_1 | Collecting grpcio-tools
test_env_1 | Using cached grpcio-tools-1.62.2.tar.gz (4.5 MB)
test_env_1 | Collecting googleapis-common-protos
test_env_1 | Using cached googleapis_common_protos-1.63.0-py2.py3-none-any.whl (229 kB)
test_env_1 | Collecting protobuf<5.0dev,>=4.21.6
test_env_1 | Using cached protobuf-4.25.3-py3-none-any.whl (156 kB)
test_env_1 | Collecting grpcio>=1.62.2
test_env_1 | Using cached grpcio-1.62.2.tar.gz (26.3 MB)
test_env_1 | Requirement already satisfied: setuptools in /usr/lib/python3.8/site-packages (from grpcio-tools) (69.5.1)
test_env_1 | Building wheels for collected packages: grpcio-tools, grpcio
test_env_1 | Building wheel for grpcio-tools (setup.py): started
test_env_1 | Building wheel for grpcio-tools (setup.py): still running...
test_env_1 | Building wheel for grpcio-tools (setup.py): still running...
test_env_1 | Building wheel for grpcio-tools (setup.py): still running...
test_env_1 | Building wheel for grpcio-tools (setup.py): still running...
test_env_1 | Building wheel for grpcio-tools (setup.py): finished with status 'done'
test_env_1 | Created wheel for grpcio-tools: filename=grpcio_tools-1.62.2-cp38-cp38-linux_x86_64.whl size=75261175 sha256=91db92b328b6206cec574e6116e197d45b92b4a4f76a86242cdfce187d8630c5
test_env_1 | Stored in directory: /root/.cache/pip/wheels/e2/28/c8/e4d1f9b7d76067c548f400444d507d72dfacdb827d5765c789
test_env_1 | Building wheel for grpcio (setup.py): started
test_env_1 | Building wheel for grpcio (setup.py): still running...
test_env_1 | Building wheel for grpcio (setup.py): still running...
test_env_1 | Building wheel for grpcio (setup.py): finished with status 'done'
test_env_1 | Created wheel for grpcio: filename=grpcio-1.62.2-cp38-cp38-linux_x86_64.whl size=115854815 sha256=f893644d131ffae01809c8772dfa7a15b17452881deb387960cb8cda8a47a4ea
test_env_1 | Stored in directory: /root/.cache/pip/wheels/23/5f/65/9ccdd12215bb7d49c5346b806ed725ccb3fa6a6d2c5e8cfa5e
test_env_1 | Successfully built grpcio-tools grpcio
test_env_1 | Installing collected packages: protobuf, grpcio, grpcio-tools, googleapis-common-protos
test_env_1 | Successfully installed googleapis-common-protos-1.63.0 grpcio-1.62.2 grpcio-tools-1.62.2 protobuf-4.25.3

Regarding usage of musllinux wheel, It used to build wheels (grpcio-1.62.2-cp38-cp38-linux_x86_64.whl, grpcio_tools-1.62.2-cp38-cp38-linux_x86_64.whl) for this docker/compose:1.29.2 docker image also with earlier grpcio versions.

For example in envoyproxy docker image where we installed the same grpcio:

FROM docker.io/envoyproxy/envoy-alpine:v1.21.6

RUN apk add bash python3 py3-pip py3-wheel gcc g++ cmd:pip3 && rm -rf /var/cache/apk/*
RUN pip3 install grpcio-tools googleapis-common-protos

uses musllinux wheels you mentioned:

#9 [ 5/15] RUN pip3 install grpcio-tools googleapis-common-protos
#9 7.451 Collecting grpcio-tools
#9 7.709 Downloading grpcio_tools-1.62.2-cp39-cp39-musllinux_1_1_x86_64.whl.metadata (6.2 kB)
#9 7.900 Collecting googleapis-common-protos
#9 7.942 Downloading googleapis_common_protos-1.63.0-py2.py3-none-any.whl.metadata (1.5 kB)
#9 8.398 Collecting protobuf<5.0dev,>=4.21.6 (from grpcio-tools)
#9 8.441 Downloading protobuf-4.25.3-py3-none-any.whl.metadata (541 bytes)
#9 9.589 Collecting grpcio>=1.62.2 (from grpcio-tools)
#9 9.713 Downloading grpcio-1.62.2-cp39-cp39-musllinux_1_1_x86_64.whl.metadata (4.0 kB)
#9 9.728 Requirement already satisfied: setuptools in /usr/lib/python3.9/site-packages (from grpcio-tools) (52.0.0)
#9 10.25 Downloading grpcio_tools-1.62.2-cp39-cp39-musllinux_1_1_x86_64.whl (3.3 MB)
#9 10.73 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 7.0 MB/s eta 0:00:00
#9 10.78 Downloading googleapis_common_protos-1.63.0-py2.py3-none-any.whl (229 kB)
#9 10.80 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 229.1/229.1 kB 13.0 MB/s eta 0:00:00
#9 11.37 Downloading grpcio-1.62.2-cp39-cp39-musllinux_1_1_x86_64.whl (5.8 MB)
#9 11.94 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.8/5.8 MB 10.3 MB/s eta 0:00:00
#9 11.98 Downloading protobuf-4.25.3-py3-none-any.whl (156 kB)
#9 12.00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 156.5/156.5 kB 10.5 MB/s eta 0:00:00
#9 12.10 Installing collected packages: protobuf, grpcio, grpcio-tools, googleapis-common-protos
#9 13.49 Successfully installed googleapis-common-protos-1.63.0 grpcio-1.62.2 grpcio-tools-1.62.2 protobuf-4.25.3

so the build failure was not present there at all unlike the docker/compose:1.29.2 image.

Anyway, now it seems to be running on the same docker/compose:1.29.2 docker image setup as it used to crash on Friday, so I'm not able to reproduce that error any more.

I've also did a try with docker/compose:1.29.2 image adding just python and bash package:

FROM docker/compose:1.29.2

RUN apk add "python3<3.9" py3-pip py3-wheel bash

Then I upgraded setuptools:

bash-5.0# pip3 install --upgrade setuptools
Collecting setuptools
Using cached setuptools-69.5.1-py3-none-any.whl (894 kB)
Installing collected packages: setuptools
Attempting uninstall: setuptools
Found existing installation: setuptools 47.0.0
Uninstalling setuptools-47.0.0:
Successfully uninstalled setuptools-47.0.0
Successfully installed setuptools-69.5.1

and installs grpcio pip modules:
bash-5.0# pip3 install grpcio-tools googleapis-common-protos
This fails because it couldn't find c++ compiler - so it wants to build it's own wheel package probably rather that use that musllinux wheel.

Since I'm not able to reproduce this issue today, it's OK to close this ticket unless you would like to investigate why it's building wheel package rather that using available musllinux in docker/compose:1.29.2.

[XuanWang-Amos]

Glad it's resolved now! Closing this issue.

[gnossen]

My guess is that you fell victim to a race condition in the upload to PyPi. A release on PyPi is a collection of files (wheel files and tarballs) that share the same version (in this case 1.62.2). As soon as the first file compatible with a client's runtime (ISA, OS, Python version, etc.) is uploaded, the client will observe the "latest" version (i.e. the version resolved by just grpcio) to be the new upload.

The from-source tarball is compatible with all environments (modulo listed supported Python runtimes) and so as soon as it is uploaded, it becomes the latest version. I suspect that if we ordered our uploads such that tarballs came last, issues like this would disappear.