Types/Stubs generated by protoc are invalid.

[shanduur]

What version of gRPC and what language are you using?

Python:

# protobuf = "^5.26.1"
GRPC_GENERATED_VERSION = '1.63.0rc2'
GRPC_VERSION = grpc.__version__ # '1.63.0rc2'
EXPECTED_ERROR_RELEASE = '1.65.0' 
SCHEDULED_RELEASE_DATE = 'June 25, 2024'

What operating system (Linux, Windows,...) and version?

$ uname -a
Darwin Foo.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000 arm64

What runtime / compiler are you using (e.g. python version or version of gcc)

$ poetry run python3 --version                     
Python 3.11.9
$ python3 -m grpc_tools.protoc --version
libprotoc 26.1

What did you do?

Please provide either 1) A unit test for reproducing the bug or 2) Specific steps for us to follow to reproduce the bug. If there’s not enough information to debug the problem, gRPC team may close the issue at their discretion. You’re welcome to re-open the issue once you have a reproduction.

[tool.poetry]
name = "foo"
version = "0.1.0"
description = ""

[tool.poetry.scripts]
grpc_generate = "tools.grpc_generate:start"

[tool.poetry.dependencies]
python = "~3.11"
protobuf = "^5.26.1"

[tool.poetry.group.dev.dependencies]
grpcio-tools = "^1.62.2"
pytype = "^2024.4.11"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

# ./tools/grpc_generate.py
import pathlib
import shutil
import subprocess
import sys

from grpc_tools import protoc

_REPOS = {
    "https://github.com/kubernetes-sigs/container-object-storage-interface-spec.git": {
        "path": "./modules/container-object-storage-interface-spec",
        "branch": "v0.1.0",
    }
}

_PB_PATH = pathlib.Path("./cosi_pb2")


def start() -> None:
    for key, value in _REPOS.items():
        shutil.rmtree(value["path"], ignore_errors=True)

        subprocess.run(
            args=[
                "git",
                "clone",
                key,
                pathlib.Path(value["path"]),
                "--branch",
                value["branch"],
                "--single-branch",
            ],
            stdout=sys.stdout,
            stderr=sys.stderr,
        ).check_returncode()

    _PB_PATH.mkdir(parents=True, exist_ok=True)
    init = _PB_PATH / "__init__.py"
    init.touch(exist_ok=True)

    proto_path = pathlib.Path(f"./.venv/lib/python3.{sys.version_info.minor}/site-packages/grpc_tools/_proto")
    spec_proto_path = pathlib.Path("./modules/container-object-storage-interface-spec")

    protoc.main(
        (
            "",
            f"--proto_path={proto_path}",
            f"--proto_path={spec_proto_path}",
            f"--python_out={_PB_PATH}",
            f"--pyi_out={_PB_PATH}",
            f"--grpc_python_out={_PB_PATH}",
            "cosi.proto",
        )
    )

    with open(_PB_PATH / "cosi_pb2.pyi", "r+") as f:
        content = f.read()
        f.seek(0, 0)
        f.write("# type: ignore\n" + content)

    if sys.argv[len(sys.argv)-1] == "merge":
        subprocess.run(
            args=[
                "merge-pyi",
                "-i",
                _PB_PATH / "cosi_pb2.py",
                _PB_PATH / "cosi_pb2.pyi",
            ],
            stdout=sys.stdout,
            stderr=sys.stderr,
        ).check_returncode()

What did you expect to see?

No errors

What did you see instead?

If code is generated with merge argument (poetry run grpc_generate merge), then running poetry run pytype . results with:

Computing dependencies
Analyzing 15 sources with 0 local dependencies
ninja: Entering directory `.pytype'
[1/17] check cosi_pb2.cosi_pb2
FAILED: /Users/user/foo/.pytype/pyi/cosi_pb2/cosi_pb2.pyi 
/Users/user/foo/.venv/bin/python -m pytype.main --imports_info /Users/user/foo/.pytype/imports/cosi_pb2.cosi_pb2.imports --module-name cosi_pb2.cosi_pb2 --platform darwin -V 3.11 -o /Users/user/foo/.pytype/pyi/cosi_pb2/cosi_pb2.pyi --analyze-annotated --nofail --quick /Users/user/foo/cosi_pb2/cosi_pb2.py
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 22, in <module>: Name '_enum_type_wrapper' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 24, in S3SignatureVersion: Name 'S3SignatureVersion' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 25, in S3SignatureVersion: Name 'S3SignatureVersion' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 26, in S3SignatureVersion: Name 'S3SignatureVersion' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 28, in <module>: Name '_enum_type_wrapper' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 30, in AnonymousBucketAccessMode: Name 'AnonymousBucketAccessMode' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 31, in AnonymousBucketAccessMode: Name 'AnonymousBucketAccessMode' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 32, in AnonymousBucketAccessMode: Name 'AnonymousBucketAccessMode' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 33, in AnonymousBucketAccessMode: Name 'AnonymousBucketAccessMode' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 34, in AnonymousBucketAccessMode: Name 'AnonymousBucketAccessMode' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 36, in <module>: Name '_enum_type_wrapper' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 38, in AuthenticationType: Name 'AuthenticationType' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 39, in AuthenticationType: Name 'AuthenticationType' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 40, in AuthenticationType: Name 'AuthenticationType' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 78, in CredentialDetails: Name '_message' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 86, in CredentialDetails: Name '_containers' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 108, in DriverCreateBucketRequest: Name '_message' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 118, in DriverCreateBucketRequest: Name '_containers' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 148, in DriverGrantBucketAccessRequest: Name '_message' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 162, in DriverGrantBucketAccessRequest: Name '_containers' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 167, in DriverGrantBucketAccessResponse: Name '_message' is not defined [name-error]
File "/Users/user/foo/cosi_pb2/cosi_pb2.py", line 177, in DriverGrantBucketAccessResponse: Name '_containers' is not defined [name-error]

For more details, see https://google.github.io/pytype/errors.html#name-error
ninja: build stopped: subcommand failed.
Leaving directory '.pytype'

Otherwise, it results with:

Computing dependencies
Analyzing 15 sources with 0 local dependencies
ninja: Entering directory `.pytype'
[5/17] check cosi_pb2.cosi_pb2_grpc
FAILED: /Users/user/foo/.pytype/pyi/cosi_pb2/cosi_pb2_grpc.pyi 
/Users/user/foo/.venv/bin/python -m pytype.main --imports_info /Users/user/foo/.pytype/imports/cosi_pb2.cosi_pb2_grpc.imports --module-name cosi_pb2.cosi_pb2_grpc --platform darwin -V 3.11 -o /Users/user/foo/.pytype/pyi/cosi_pb2/cosi_pb2_grpc.pyi --analyze-annotated --nofail --quick /Users/user/foo/cosi_pb2/cosi_pb2_grpc.py
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 44, in __init__: No attribute 'DriverGetInfoRequest' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 45, in __init__: No attribute 'DriverGetInfoResponse' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 65, in add_IdentityServicer_to_server: No attribute 'DriverGetInfoRequest' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 66, in add_IdentityServicer_to_server: No attribute 'DriverGetInfoResponse' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 93, in DriverGetInfo: No attribute 'DriverGetInfoRequest' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 94, in DriverGetInfo: No attribute 'DriverGetInfoResponse' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 117, in __init__: No attribute 'DriverCreateBucketRequest' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 118, in __init__: No attribute 'DriverCreateBucketResponse' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 122, in __init__: No attribute 'DriverDeleteBucketRequest' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 123, in __init__: No attribute 'DriverDeleteBucketResponse' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 127, in __init__: No attribute 'DriverGrantBucketAccessRequest' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 128, in __init__: No attribute 'DriverGrantBucketAccessResponse' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 132, in __init__: No attribute 'DriverRevokeBucketAccessRequest' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 133, in __init__: No attribute 'DriverRevokeBucketAccessResponse' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 178, in add_ProvisionerServicer_to_server: No attribute 'DriverCreateBucketRequest' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 179, in add_ProvisionerServicer_to_server: No attribute 'DriverCreateBucketResponse' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 183, in add_ProvisionerServicer_to_server: No attribute 'DriverDeleteBucketRequest' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 184, in add_ProvisionerServicer_to_server: No attribute 'DriverDeleteBucketResponse' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 188, in add_ProvisionerServicer_to_server: No attribute 'DriverGrantBucketAccessRequest' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 189, in add_ProvisionerServicer_to_server: No attribute 'DriverGrantBucketAccessResponse' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 193, in add_ProvisionerServicer_to_server: No attribute 'DriverRevokeBucketAccessRequest' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 194, in add_ProvisionerServicer_to_server: No attribute 'DriverRevokeBucketAccessResponse' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 221, in DriverCreateBucket: No attribute 'DriverCreateBucketRequest' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 222, in DriverCreateBucket: No attribute 'DriverCreateBucketResponse' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 248, in DriverDeleteBucket: No attribute 'DriverDeleteBucketRequest' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 249, in DriverDeleteBucket: No attribute 'DriverDeleteBucketResponse' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 275, in DriverGrantBucketAccess: No attribute 'DriverGrantBucketAccessRequest' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 276, in DriverGrantBucketAccess: No attribute 'DriverGrantBucketAccessResponse' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 302, in DriverRevokeBucketAccess: No attribute 'DriverRevokeBucketAccessRequest' on module 'cosi_pb2' [module-attr]
File "/Users/user/foo/cosi_pb2/cosi_pb2_grpc.py", line 303, in DriverRevokeBucketAccess: No attribute 'DriverRevokeBucketAccessResponse' on module 'cosi_pb2' [module-attr]

For more details, see https://google.github.io/pytype/errors.html#module-attr
ninja: build stopped: subcommand failed.
Leaving directory '.pytype'

Anything else we should know about your project / environment?