Support protobuf 5.26.0

[jonyscathe]

Is your feature request related to a problem? Please describe.

Currently cannot update to protobuf 5.26.0 due to grpcio-tools dependencies.

Describe the solution you'd like

grpcio-tools requirement for protobuf to be relaxed to protobuf>=4.21.3,<6.0dev
Not sure if any other changes are needed to support protobuf v26.0. The changes there don't look too drastic so I am guessing nothing (or not much).

Describe alternatives you've considered

None - am currently sitting at protobuf v25.3 because of this

Additional context

Poetry output from poetry lock:

Because grpcio-tools (1.62.1) depends on protobuf (>=4.21.6,<5.0dev)
 and test-project depends on protobuf (5.26.0), grpcio-tools is forbidden.
So, because test-project depends on grpcio-tools (1.62.1), version solving failed.

[sodul]

I suggest to remove the artificial upper bound limit on future versions of protobuf. As noted in this post:
https://discuss.python.org/t/how-to-address-formerly-unknown-incompatibilities/45873/5

It’s not considered good practice for libraries to add upper bounds.

[CodingAnarchy]

This is especially confusing when I tried to update using pip-tools. Protobuf was updated to 5.26.1, while grpcio-tools was downgraded to 1.30 to make it "work". But 1.30 does not work with the 3.11 version of Python!

[XuanWang-Amos]

Looks like the upper bound was changed from <4.0dev to <5.0dev in this PR two years ago as part of release process: #30377

[root-kidik]

+1

[Anton3]

On our macOS dev machines, core C++ protobuf package has been forcefully updated to 5.x via brew. So we need Python protobuf 5.x. But because of grpcio's incompatibility with 5.x, our builds are essentially broken on macOS, as of now. This issue blocks development for us.

[sodul]

@Anton3 my gut feeling is that the grpcio package is not really incompatible with the new protobuf package, but it just claims to not be. If you are really blocked you might be able to trick your pip installation to bypass the claimed conflicts by calling pip install incrementally. We do this with some of our poorly opinionated third party packages we use internally such as the azure and vmware packages and that has greatly reduced frustration with no ill effect so far. Give it a try if you can.

A bit off topic, but I strongly recommend to NOT use HomeBrew to manage python and your python dependencies if you do any Python development. We switched to pyenv 5 years ago because brew kept on updating its dependency tree for random reason breaking compatibility in different ways from one laptop to the other. We use brew to install some common libraries such as xz, I think deadline comes from the Xcode CLI Tools nowadays, and then just use pyenv to have a clean, recent version of python. With the VirtualEnv plugin it even allows to easily manage having different package set per development project and test various versions of python very easily.

[Anton3]

@sodul We use venv for Python packages, but we use brew for the C++ protobuf package.

C++ protobuf package has been updated to 5.x in brew, including its protoc. We use protoc to generate Python protobuf definitions. If the major versions of protoc and Python protobuf package diverge, then the incompatibility exceptions are thrown at best. So we are forced to match C++ protobuf 5.x with Python protobuf 5.x. But then we can't run tests locally, because grpcio requirement refuses to install.

Edit: One way out is to build and install C++ protobuf manually. But it takes quite some time and effort to do so. Protobuf also provides precompiled binaries, which we might be recommending in the future due to issues like the current one.

[sodul]

@Anton3, interesting.

In our case, because of brew changing versions based on other unrelated binaries that our devs might want to experiment with, we wrote our own internal installer for third party binaries such as protoc. We are currently installing protoc 26.0 from https://github.com/protocolbuffers/protobuf/releases

On the python side we install grpcio==1.62.1, grpcio-tools==1.62.1 and protobuf==4.25.3.

We are not a C++ shop but we use GoLang, Python and Protobuf extensively and we are not experiencing any issues other than having to pin to protobuf==4.25.3 due to the grpcio-tools conflict.

It is very much possible that you are using some different protobuf features that we do and get different results. Overall if things are broken with the brew installed version of protoc, as a temporary workaround you could tell your devs to install the older protoc ( don't use brew for that and brew install protoc@25 does not seem to work, so you'll need to find the right command), or download the v5 binaries from the GitHub release and set that version first in the local PATH until the grpc maintainers provide a fix.

[Anton3]

Reportedly, the new grpcio 1.63.0 does not complain about Protobuf 5.26.x

[jonyscathe]

Was fixed in another issue

[gnossen]

Hi all, thanks for chiming in on this issue. It's clear that this caused a lot of friction for a lot of people and we're dedicated to making it better. For starters, the first version that was originally planned to include Protobuf 5.X support was 1.64, around 6 weeks from now. We backported the update to 1.63 to get a solution out faster, which is now available as 1.63.0.

But that's not enough. There have now been two major version bumps of the protobuf Python generated code API since gRPC hit 1.0 and both times the jump has caused gRPC users a lot of heartache. We've been working closely with the Protobuf team to institute a new policy where any new major version of the Protobuf Python package must also support the generated code API of the previous major version. This will turn the hard installation errors or runtime errors that you're currently seeing with new major versions into deprecation warnings with a timeline of some number of months (TBD) to regenerate your _pb2.py files. We'll share more details once the new policy is finalized.