Inconsistent defaulting to python implementation, and errors when using cpp

[BenRKarl]

What version of protobuf and what language are you using?
Versions: 3.17.3, 3.18.0, 3.18.1, and 3.19.0

Language: Python (versions 3.7.2, 3.8.6, 3.9.0, and 3.10.0

What operating system (Linux, Windows, ...) and version? gLinux

What runtime / compiler are you using (e.g., python version or gcc version) python and/or cpp

What did you do?
Steps to reproduce the behavior:

Run this bash script. Note that it require pyenv to be installed, along with Python versions 3.7.2.

#!/bin/bash
 
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp
pyenv global 3.7.2
python -mvenv /tmp/venv
. /tmp/venv/bin/activate
python -m pip install protobuf==3.19.0
python -c "from google.protobuf import descriptor_pb2"

What did you expect to see

I expect this script to run without error.

What did you see instead?

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/venv/lib/python3.7/site-packages/google/protobuf/descriptor_pb2.py", line 5, in <module>
    from google.protobuf import descriptor as _descriptor
  File "/tmp/venv/lib/python3.7/site-packages/google/protobuf/descriptor.py", line 47, in <module>
    from google.protobuf.pyext import _message
ImportError: cannot import name '_message' from 'google.protobuf.pyext' (/tmp/venv/lib/python3.7/site-packages/google/protobuf/pyext/__init__.py)

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

This was first detected as a performance issue, as our benchmarks began showing large regressions due to the library unknowingly running the python implementation. For more context on the history see this Issue.

Anything else we should know about your project / environment

There are issues with a number of combinations of Python versions and protobuf library versions. Here are the combinations I found that have problems when running the above reproduction script:

Python version	Protobuf version	Env var status	Result
3.7.2	3.19.0	unset	defaults to python implementation incorrectly
3.7.2	3.19.0	cpp	results in above error
3.7.2	3.18.1	unset	defaults to python implementation incorrectly
3.10.0	3.18.1	unset	defaults to python implementation incorrectly
3.7.2	3.18.1	cpp	results in above error
3.10.0	3.18.1	cpp	results in above error
3.7.2	3.18.0	unset	defaults to python implementation incorrectly
3.10.0	3.18.0	unset	defaults to python implementation incorrectly
3.7.2	3.18.0	cpp	results in above error
3.10.0	3.18.0	cpp	results in above error
3.10.0	3.17.3	unset	defaults to python implementation incorrectly
3.10.0	3.17.3	cpp	results in above error

[acozzette]

I ran into a variation of this error recently and managed to fix it by adding the flag --install-option="--cpp_implementation" to pip install. I'm not very knowledgeable about Python and have no idea if this is a good fix, but maybe this workaround will help.

[efroemling]

(UPDATED; SEE END OF COMMENT)

I just started running into what appears might be a related issue.
In my case I'm running homebrew-installed python 3.9.7 on Apple-Silicon Macs (native; not Rosetta).
If I do the following:

python3 -m pip install protobuf google-cloud-datastore
...
Successfully installed google-cloud-datastore-2.3.0 protobuf-3.19.1

python3 -c 'from google.cloud import datastore'

I get:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/homebrew/lib/python3.9/site-packages/google/cloud/datastore/__init__.py", line 59, in <module>
    from google.cloud.datastore.batch import Batch
  File "/opt/homebrew/lib/python3.9/site-packages/google/cloud/datastore/batch.py", line 24, in <module>
    from google.cloud.datastore import helpers
  File "/opt/homebrew/lib/python3.9/site-packages/google/cloud/datastore/helpers.py", line 23, in <module>
    from google.protobuf import struct_pb2
  File "/opt/homebrew/lib/python3.9/site-packages/google/protobuf/struct_pb2.py", line 6, in <module>
    from google.protobuf import descriptor as _descriptor
  File "/opt/homebrew/lib/python3.9/site-packages/google/protobuf/descriptor.py", line 47, in <module>
    from google.protobuf.pyext import _message
AttributeError: module 'google.protobuf.internal.containers' has no attribute 'MutableMapping'

The odd thing:
I have a second Mac with a pretty identical setup (exact same python/pip versions of everything) and it works there. On digging a bit, this seems to be due to the second Mac defaulting to the 'python' api_implementation in protobuf while the first one is getting 'cpp'.
On further digging, I found that this is due to the protobuf package installing .so files in the first case but not the second. If I do identical clean reinstalls of protobuf via python3 -m pip install protobuf on both machines, I see the following results on Mac 1:

% find /opt/homebrew/lib/python3.9/site-packages/google/protobuf | grep -i \\.so
/opt/homebrew/lib/python3.9/site-packages/google/protobuf/internal/_api_implementation.cpython-39-darwin.so
/opt/homebrew/lib/python3.9/site-packages/google/protobuf/pyext/_message.cpython-39-darwin.so

On Mac 2 this comes up empty (though the rest of the package is there).
So this seems like possibly 2 different problems:

• Mac 2 not getting the .so files installed while Mac 1 does
• Things being broken when the .so files are present

As I mentioned, the software environments on both are pretty identical. Mac 2 is one of the new M1 Max MBPs and does not have Rosetta installed, while the first is a regular M1 with Rosetta installed. Could either of those things be a factor somehow? (the architecture is the same and this is all native code, so I wouldn't think so, but perhaps it is somehow?..)

UPDATE: After some more thorough cleaning/reinstalling, I'm no longer seeing .so files on Mac1. Its looking like these were stale files left over from a conflicting homebrew protobuf package installed as a dependency and they were sticking around despite uninstalling/reinstalling things via pip. So now I'm seeing no .so files anywhere and everything is working via the 'python' implementation everywhere. So I'm guessing this is the expected state of the world and all is well. I'll leave this comment up though in case my little journey of discovery is useful to anyone.

[BenRKarl]

@efroemling I assume you intend to use the python implementation, right? The lack of .so files would make using the cpp implementation impossible I believe.

[haberman]

I was able to reproduce this, thanks.

I think the root cause here is that 3.18.0 moved us from "manylinux1" to "manylinux2014" as our platform tag. Earlier versions of Python appear to still require older platform tags such as manylinux1.

This change was caused by #8280 (cc @jtattermusch)

I think perhaps the solution here is to revert to manylinux1 for x86-64 and only use manylinux2014 for aarch64.

More detail follows:

---

For Python 3.7.2 with Protobuf 3.17.1, it is installing the wheel protobuf-3.17.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl:

$ python -m pip install protobuf==3.17.3
Collecting protobuf==3.17.3
  Downloading https://files.pythonhosted.org/packages/4c/53/ddcef00219f2a3c863b24288e24a20c3070bd086a1e77706f22994a7f6db/protobuf-3.17.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0MB)
    100% |████████████████████████████████| 1.0MB 25.4MB/s 
Collecting six>=1.9 (from protobuf==3.17.3)
  Downloading https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl
Installing collected packages: six, protobuf
  Found existing installation: protobuf 3.19.1
    Uninstalling protobuf-3.19.1:
      Successfully uninstalled protobuf-3.19.1
Successfully installed protobuf-3.17.3 six-1.16.0

For Protobuf 3.19.1, it doesn't appear we are building a cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64 package. Here is the list of packages build for 3.19.1:

protobuf-3.19.1-cp310-cp310-macosx_10_9_universal2.whl 
protobuf-3.19.1-cp310-cp310-manylinux2014_aarch64.whl 
protobuf-3.19.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 
protobuf-3.19.1-cp36-cp36m-macosx_10_9_x86_64.whl 
protobuf-3.19.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 
protobuf-3.19.1-cp36-cp36m-win32.whl 
protobuf-3.19.1-cp36-cp36m-win_amd64.whl 
protobuf-3.19.1-cp37-cp37m-macosx_10_9_x86_64.whl 
protobuf-3.19.1-cp37-cp37m-manylinux2014_aarch64.whl 
protobuf-3.19.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 
protobuf-3.19.1-cp37-cp37m-win32.whl 
protobuf-3.19.1-cp37-cp37m-win_amd64.whl 
protobuf-3.19.1-cp38-cp38-macosx_10_9_x86_64.whl 
protobuf-3.19.1-cp38-cp38-manylinux2014_aarch64.whl 
protobuf-3.19.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 
protobuf-3.19.1-cp38-cp38-win32.whl 
protobuf-3.19.1-cp38-cp38-win_amd64.whl 
protobuf-3.19.1-cp39-cp39-macosx_10_9_x86_64.whl 
protobuf-3.19.1-cp39-cp39-manylinux2014_aarch64.whl 
protobuf-3.19.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 
protobuf-3.19.1-cp39-cp39-win32.whl 
protobuf-3.19.1-cp39-cp39-win_amd64.whl 
protobuf-3.19.1-py2.py3-none-any.whl 
protobuf-3.19.1.tar.gz

Whereas for 3.17.3 we released:

protobuf-3.17.3-cp27-cp27m-macosx_10_9_x86_64.whl
protobuf-3.17.3-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.whl
protobuf-3.17.3-cp35-cp35m-macosx_10_9_intel.whl
protobuf-3.17.3-cp35-cp35m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
protobuf-3.17.3-cp35-cp35m-win32.whl
protobuf-3.17.3-cp35-cp35m-win_amd64.whl
protobuf-3.17.3-cp36-cp36m-macosx_10_9_x86_64.whl
protobuf-3.17.3-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
protobuf-3.17.3-cp36-cp36m-win32.whl
protobuf-3.17.3-cp36-cp36m-win_amd64.whl
protobuf-3.17.3-cp37-cp37m-macosx_10_9_x86_64.whl
protobuf-3.17.3-cp37-cp37m-manylinux2014_aarch64.whl
protobuf-3.17.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
protobuf-3.17.3-cp37-cp37m-win32.whl
protobuf-3.17.3-cp37-cp37m-win_amd64.whl
protobuf-3.17.3-cp38-cp38-macosx_10_9_x86_64.whl
protobuf-3.17.3-cp38-cp38-manylinux2014_aarch64.whl
protobuf-3.17.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
protobuf-3.17.3-cp38-cp38-win32.whl
protobuf-3.17.3-cp38-cp38-win_amd64.whl
protobuf-3.17.3-cp39-cp39-macosx_10_9_x86_64.whl
protobuf-3.17.3-cp39-cp39-manylinux2014_aarch64.whl
protobuf-3.17.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl
protobuf-3.17.3-cp39-cp39-win32.whl
protobuf-3.17.3-cp39-cp39-win_amd64.whl
protobuf-3.17.3-py2.py3-none-any.whl
protobuf-3.17.3.tar.gz

Diffing these two lists we get:

-cp27-cp27m-macosx_10_9_x86_64.whl
-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.whl
-cp35-cp35m-macosx_10_9_intel.whl
-cp35-cp35m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
-cp35-cp35m-win32.whl
-cp35-cp35m-win_amd64.whl
+cp310-cp310-macosx_10_9_universal2.whl
+cp310-cp310-manylinux2014_aarch64.whl
+cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 cp36-cp36m-macosx_10_9_x86_64.whl
-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
+cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 cp36-cp36m-win32.whl
 cp36-cp36m-win_amd64.whl
 cp37-cp37m-macosx_10_9_x86_64.whl
 cp37-cp37m-manylinux2014_aarch64.whl
-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
+cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 cp37-cp37m-win32.whl
 cp37-cp37m-win_amd64.whl
 cp38-cp38-macosx_10_9_x86_64.whl
 cp38-cp38-manylinux2014_aarch64.whl
-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
+cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 cp38-cp38-win32.whl
 cp38-cp38-win_amd64.whl
 cp39-cp39-macosx_10_9_x86_64.whl
 cp39-cp39-manylinux2014_aarch64.whl
-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl
+cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 cp39-cp39-win32.whl
 cp39-cp39-win_amd64.whl
 py2.py3-none-any.whl

The two main changes I observe here are:

1. We've dropped Python 2.7 and 3.5, and added Python 3.10.
2. We've moved from manylinux1 to manylinux2014 as our ABI target.

[haberman]

cc @jtattermusch

Any idea how to revert the manylinux2014 to manylinux1 for Linux?

[jtattermusch]

I looked into this and found that the problem most likely has nothing to do with my PR #8280 (which only affects the build or aarch64 wheels and it doesn't change anything for the x86_64 wheels).

@haberman is correct that there was a sudden switch from manylinux1 to manylinux2014 for x86_64 (so for systems older than what manylinux2014 supports, only the pure python version of protobuf python will be available - thus no cpp support, which explains the behavior reported in this issue).

It seems that the unintended switch from manylinux1 to manylinux2014 wasn't actually caused by a change in this repository, but by a change in https://github.com/matthew-brett/multibuild.
To build python wheels, we are using the multibuild scripts and we always clone the HEAD of that repository so any changes in that repository will be automatically picked up in the protobuf wheel build.

protobuf/kokoro/release/python/linux/build_artifacts.sh

Line 28 in 47e0542

git clone https://github.com/matthew-brett/multibuild.git

On Aug 21, matthew-brett/multibuild has switched the default manylinux image from 1 to 2014 (see PR https://github.com/multi-build/multibuild/pull/423/files).

Some extra evidence:

• the 3.18.0-rc1 release (from Aug 20) still has the manylinux1 wheels: https://pypi.org/project/protobuf/3.18.0rc1/#files
• the 3.18.0-rc2 release (from Aug 31) only has manylinux2014 wheels for x86_64: https://pypi.org/project/protobuf/3.18.0rc2/#files (and so the change from Aug 21 has almost certainly caused the regression).

The solution is relatively simple:

• make sure we explicitly set MB_ML_VER=1 for the x86_64 wheels
• pin the clone of matthew-brett/multibuild to a specific version to avoid unexpected changes like this in the future.
• it should also be possible to retroactively generate the missing manylinux1 wheels for older releases if needed.

[BenRKarl]

Thanks @jtattermusch - do you have a rough idea of when this fix might make it into a release?

[jtattermusch]

Thanks @jtattermusch - do you have a rough idea of when this fix might make it into a release?

That is up to the protobuf team to decide. Also, they'll need to decide about possibly backporting into the existing releases.

[BenRKarl]

Thanks @jtattermusch very much appreciated. Seems like the updates on the PR have stopped, is there anything we can do to revive that thread?

[andrea-cassioli-maersk]

Same issue here. Any timeline update for the fix release?

[lambdamusic]

I was able to get rid of the AttributeError: module 'google.protobuf.internal.containers' has no attribute 'MutableMapping' by downgrading to protobuf==3.17.3 as shown in one of the comments above:

$ pip uninstall protobuf
$ pip install protobuf==3.17.3

[jtattermusch]

It doesn't seem that the fix #9216 has made it into the 3.19.x branch for the 3.19 release.

Unless it gets backported, to 3.19.x, it will be in the next release 3.20.x.

@anandolee please consider backporting to 3.19.x

[martinxsliu]

Hi, I'm curious to know if there's a timeline for the 3.20.x release? Or if it's been announced / tracked somewhere?

[anandolee]

3.20.0 which includes the fix has already been pre released. Office release will happen in these two weeks

[BenRKarl]

@anandolee thanks for the heads up. Do you know if this fix will get back ported to already released versions? If not we'll need to exclude versions >= 3.18.0, < 3.20.0

[anandolee]

We do not have plan to back ported

[taliastocks]

Hey, we were hit by this issue too. Was this fix released in 3.20.0?

[BenRKarl]

@taliastocks if it helps, we're currently using 3.20.0 without issues. We exclude 3.18.* and 3.19.*, see here

[haberman]

We are about to release version 4.21.0, which builds wheels using an entirely new Bazel-based workflow. It should address the issues raised in this bug, and it uses manylinux2014 as the platform on Linux.

More information here: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates