[Bazel] Enable grpcio-reflection to be used via Bazel

[CareF]

The current py_grpc_library results in the wrong grpc proto python code path when grpc is a third-party source code in a Bazel project. This PR should fix it.

fixes #31011

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: CareF / name: Ming Lyu (ebe3a32, db508ec, 81fdc78)
• ✅ login: gnossen / name: Richard Belleville (97146d6)

[gnossen]

Can you please add a test case to this directory?

[CareF]

Can you please add a test case to this directory?

Thanks for the remainder. I realize that there are more problems with this issue than what I have already done.
This PR fixes the pb2_grpc.py file path but yet still the generated pb2.py file tries to import beyond top-level package:

Under test/distrib/bazel/python/, the generated ./bazel-bin/external/com_github_grpc_grpc/src/python/grpcio_reflection/grpc_reflection/v1alpha/reflection_pb2.py file is

from ...com_github_grpc_grpc.src.proto.grpc.reflection.v1alpha.reflection_pb2 import *

with or without the change to proto plugin argument (my original fix to #31011). This will trigger the error ValueError: attempted relative import beyond top-level package

Traceback (most recent call last):
  File "/home/user/.cache/bazel/_bazel_lm/ce2d8b67d17f2cc0fd23ca117ea353ce/execroot/python_test_repo/bazel-out/k8-fastbuild/bin/import_from_grpcio_reflection_test.runfiles/python_test_repo/import_from_grpcio_reflection.py", line 17, in <module>
    from grpc_reflection.v1alpha import reflection
  File "/home/user/.cache/bazel/_bazel_lm/ce2d8b67d17f2cc0fd23ca117ea353ce/execroot/python_test_repo/bazel-out/k8-fastbuild/bin/import_from_grpcio_reflection_test.runfiles/com_github_grpc_grpc/src/python/grpcio_reflection/grpc_reflection/v1alpha/reflection.py", line 19, in <module>
    from grpc_reflection.v1alpha import reflection_pb2 as _reflection_pb2
  File "/home/user/.cache/bazel/_bazel_lm/ce2d8b67d17f2cc0fd23ca117ea353ce/execroot/python_test_repo/bazel-out/k8-fastbuild/bin/import_from_grpcio_reflection_test.runfiles/com_github_grpc_grpc/src/python/grpcio_reflection/grpc_reflection/v1alpha/reflection_pb2.py", line 1, in <module>
    from ...com_github_grpc_grpc.src.proto.grpc.reflection.v1alpha.reflection_pb2 import *
ValueError: attempted relative import beyond top-level package

To fix I have to add the following which is a little bit beyond the original scope:

diff --git a/bazel/python_rules.bzl b/bazel/python_rules.bzl
index d6345a4762..1fe61ca2bc 100644
--- a/bazel/python_rules.bzl
+++ b/bazel/python_rules.bzl
@@ -134,7 +134,7 @@ def _generate_py_impl(context):
         for py_src in context.attr.deps[0][PyProtoInfo].generated_py_srcs:
             reimport_py_file = context.actions.declare_file(py_src.basename)
             py_sources.append(reimport_py_file)
-            import_line = "from %s import *" % py_src.short_path.replace("/", ".")[:-len(".py")]
+            import_line = "from %s import *" % py_src.short_path.replace("..", "external").replace("/", ".")[:-len(".py")]
             context.actions.write(reimport_py_file, import_line)
 
     # Collect output PyInfo provider.
···

[CareF]

@gnossen PTAL

[CareF]

[1331/1338] Linking CXX executable test_core_iomgr_timer_list_test.exe

FAILED: test_core_iomgr_timer_list_test.exe

cmd.exe /C "cd . && "C:\Program Files\CMake\bin\cmake.exe" -E vs_link_exe --intdir=CMakeFiles\test_core_iomgr_timer_list_test.dir --manifests  -- C:\PROGRA~2\MIB055~1\2017\COMMUN~1\VC\Tools\MSVC\1414~1.264\bin\Hostx86\x86\link.exe /nologo @CMakeFiles/test_core_iomgr_timer_list_test.rsp  /out:test_core_iomgr_timer_list_test.exe /implib:test_core_iomgr_timer_list_test.lib /pdb:test_core_iomgr_timer_list_test.pdb /version:0.0  /machine:X86 /debug /INCREMENTAL /subsystem:console  && cd ."

FINAL LINK: command "C:\PROGRA~2\MIB055~1\2017\COMMUN~1\VC\Tools\MSVC\1414~1.264\bin\Hostx86\x86\link.exe /nologo @CMakeFiles/test_core_iomgr_timer_list_test.rsp /out:test_core_iomgr_timer_list_test.exe /implib:test_core_iomgr_timer_list_test.lib /pdb:test_core_iomgr_timer_list_test.pdb /version:0.0 /machine:X86 /debug /INCREMENTAL /subsystem:console /MANIFEST /MANIFESTFILE:CMakeFiles\test_core_iomgr_timer_list_test.dir/intermediate.manifest CMakeFiles\test_core_iomgr_timer_list_test.dir/manifest.res" failed (exit code 1) with the following output:
LINK : the 32-bit linker (C:\PROGRA~2\MIB055~1\2017\COMMUN~1\VC\Tools\MSVC\1414~1.264\bin\Hostx86\x86\link.exe) failed to do memory mapped file I/O on `grpc.lib' and is going to restart linking with a 64-bit linker for better throughput

LINK : restarting link with 64-bit linker `C:\tools\msys64\usr\bin\link.exe'

link: unknown option -- i
Try '/c/PROGRA~2/MIB055~1/2017/COMMUN~1/VC/Tools/MSVC/1414~1.264/bin/Hostx86/x86/link --help' for more information.

seems to be completely irreverent to this PR

[CareF]

@gnossen Anything else I need to do?

[aschleck]

@gnossen is it possible to get this submitted? It's blocking our usage of grpcio-reflection so it'd be great to have this bug fixed.

[CareF]

I've been using this commit as a patch in my private projects for a while and didn't see any issues.

[gnossen]

Thanks for the contribution! Sorry for the delay.

[mering]

@CareF @gnossen The change in bazel/python_rules.bzl breaks us when trying to upgrade with the message:

output 'external/MODULE~override/path/to/MYPROTO_pb2_grpc.py' was not created

When reverting this change and always using out_dir.path compilation works fine in your code base.

Any suggestion on how to move forward?