No output when generating some googleapis libs

[thesquelched]

I'm attempting to generate some libraries from the googleapis repo (sha f47c38882a165e0c79c70e52082e801fa2bb72d8) using the latest docker image, but I'm getting inconsistent behavior between different libraries; that is, some libraries (e.g. vision/v1) generate an output, while others (e.g. aiplatform/v1) generate nothing.

This generates output in dest (from the googleapis repo root):

/path/to/gapic.sh \
    --image gcr.io/gapic-images/gapic-generator-go:latest \
    --in google/cloud/vision/v1 \
    --out $PWD/dest \
    --go-gapic-package "github.com/foo/bar;baz"

However, this does not:

/path/to/gapic.sh \
    --image gcr.io/gapic-images/gapic-generator-go:latest \
    --in google/cloud/aiplatform/v1 \
    --out $PWD/dest \
    --go-gapic-package "github.com/foo/bar;baz"

I do get a bunch of warning about Import google/api/annotations.proto is unused, but I'm not sure if that's relevant.

Am I missing some step or doing something wrong? Are some libraries just unable to be generated?

[noahdietz]

If you aren't getting an error message, I'm not sure what the problem would be. I will have to attempt it myself at some point.

Invoking it via protoc or bazel (from inside googleapis) should work though, but I understand that requires more setup on your end (downloading all of the tools, protos, etc.). If you want to try it though, I will write you a command to run. TBH the Docker image hasn't been updated in a little bit.

Can I ask why you are generating the libs on your own and not using the ones in https://pkg.go.dev/cloud.google.com/go? It is totally fine, just curious :)

[thesquelched]

If you want to try it though, I will write you a command to run

I absolutely want to try it, so that help would be much appreciated!

Can I ask why you are generating the libs on your own and not using the ones in https://pkg.go.dev/cloud.google.com/go? It is totally fine, just curious :)

Sure. We want to maintain a fork of the clients for internal users that enforce some policies by modifying requests, but otherwise working mostly the same as the regular clients. We need to do this for both python and go, and in order to maintain feature parity with the regular clients, we thought it'd be easier to just for the gapic-generator repos so that we can regenerate our clients when needed.

[noahdietz]

Assuming you have googleapis already, and from that directory run:

protoc -I. \
  --go_gapic_out=$PWD/dest \
  --go_gapic_opt="go-gapic-package=cloud.google.com/go/aiplatform/apiv1;aiplatform" \
  --go_gapic_opt=api-service-config=google/cloud/aiplatform/v1/aiplatform_v1.yaml \
  --go_gapic_opt=grpc-service-config=google/cloud/aiplatform/v1/aiplatform_grpc_service_config.json \
  google/cloud/aiplatform/v1/*.proto

Grab the generator binary from releases or install via Go. Make sure wherever you put the binary or if you install via Go, that it is on your PATH.

Alternatively, install Bazel and from googleapis run the following:

bazel build //google/cloud/aiplatform/v1:gapi-cloud-aiplatform-v1-go

Then open the archive from the Bazel output that it reports, and you will see the GAPIC and the Go protobuf/grpc bindings.

[thesquelched]

Interestingly, the bazel build succeeded, while the protoc call did not due to a ruby-related issue:

google/cloud/aiplatform/v1/accelerator_type.proto:27:8: Option "ruby_package" unknown.

I didn't notice that error when running the original gapic command, but I wonder if they're related?

[noahdietz]

That's odd, the ruby_package is a standard protobuf option. If protoc has access to the standard include directory that comes with the protobuf/protoc installation, this should "just work". I don't mean to place blame, but I think that issue is related to your local set up of protoc. I could be totally wrong, of course :) Did you install protoc via the release page? or a package manager?

[thesquelched]

Ah yep, I was using an old protoc; upgrading fixed that issue.

I'm having some trouble with the python libs though, possibly due to googleapis/rules_gapic#46. The biggest issue here is that I don't grok bazel, so I don't understand how it sets up the build environment and can't debug this weird import error:

$ bazel build //google/cloud/aiplatform/v1:aiplatform-v1-py
DEBUG: /home/admin/.cache/bazel/_bazel_admin/5c26bff219867e71dd30c131bddc4798/external/rules_python/python/pip.bzl:61:10: DEPRECATED: the pip_repositories rule has been replaced with pip_install, please see rules_python 0.1 release notes
DEBUG: /home/admin/.cache/bazel/_bazel_admin/5c26bff219867e71dd30c131bddc4798/external/io_grpc_grpc_java/java_grpc_library.bzl:195:14: Multiple values in 'deps' is deprecated in aiplatform_java_grpc
INFO: Analyzed target //google/cloud/aiplatform/v1:aiplatform-v1-py (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/admin/git/googleapis/google/cloud/aiplatform/v1/schema/predict/params/BUILD.bazel:95:17: //google/cloud/aiplatform/v1/schema/predict/params:params_py_gapic: `bazel-out/host/bin/external/com_google_protobuf/protoc --experimental_allow_proto3_optional --plugin=protoc-gen-python_gapic=bazel-out/host/bin/external/gapic_generator_python/gapic_plugin --python_gapic_out=metadata:bazel-out/k8-fastbuild/bin/google/cloud/aiplatform/v1/schema/predict/params/params_py_gapic.srcjar.zip --python_gapic_opt=python-gapic-namespace=google.cloud.aiplatform.v1.schema.predict,python-gapic-name=params -Igoogle/cloud/aiplatform/v1/schema/predict/params/image_classification.proto=google/cloud/aiplatform/v1/schema/predict/params/image_classification.proto -Igoogle/cloud/aiplatform/v1/schema/predict/params/image_object_detection.proto=google/cloud/aiplatform/v1/schema/predict/params/image_object_detection.proto -Igoogle/cloud/aiplatform/v1/schema/predict/params/image_segmentation.proto=google/cloud/aiplatform/v1/schema/predict/params/image_segmentation.proto -Igoogle/cloud/aiplatform/v1/schema/predict/params/video_action_recognition.proto=google/cloud/aiplatform/v1/schema/predict/params/video_action_recognition.proto -Igoogle/cloud/aiplatform/v1/schema/predict/params/video_classification.proto=google/cloud/aiplatform/v1/schema/predict/params/video_classification.proto -Igoogle/cloud/aiplatform/v1/schema/predict/params/video_object_tracking.proto=google/cloud/aiplatform/v1/schema/predict/params/video_object_tracking.proto -Igoogle/api/annotations.proto=google/api/annotations.proto -Igoogle/api/http.proto=google/api/http.proto -Igoogle/protobuf/descriptor.proto=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/descriptor_proto/google/protobuf/descriptor.proto google/cloud/aiplatform/v1/schema/predict/params/image_classification.proto google/cloud/aiplatform/v1/schema/predict/params/image_object_detection.proto google/cloud/aiplatform/v1/schema/predict/params/image_segmentation.proto google/cloud/aiplatform/v1/schema/predict/params/video_action_recognition.proto google/cloud/aiplatform/v1/schema/predict/params/video_classification.proto google/cloud/aiplatform/v1/schema/predict/params/video_object_tracking.proto` failed (Exit 1) protoc failed: error executing command bazel-out/host/bin/external/com_google_protobuf/protoc --experimental_allow_proto3_optional '--plugin=protoc-gen-python_gapic=bazel-out/host/bin/external/gapic_generator_python/gapic_plugin' ... (remaining 17 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
google/cloud/aiplatform/v1/schema/predict/params/image_classification.proto:19:1: warning: Import google/api/annotations.proto is unused.
google/cloud/aiplatform/v1/schema/predict/params/image_object_detection.proto:19:1: warning: Import google/api/annotations.proto is unused.
google/cloud/aiplatform/v1/schema/predict/params/image_segmentation.proto:19:1: warning: Import google/api/annotations.proto is unused.
google/cloud/aiplatform/v1/schema/predict/params/video_action_recognition.proto:19:1: warning: Import google/api/annotations.proto is unused.
google/cloud/aiplatform/v1/schema/predict/params/video_classification.proto:19:1: warning: Import google/api/annotations.proto is unused.
google/cloud/aiplatform/v1/schema/predict/params/video_object_tracking.proto:19:1: warning: Import google/api/annotations.proto is unused.
Traceback (most recent call last):
  File "/home/admin/.cache/bazel/_bazel_admin/5c26bff219867e71dd30c131bddc4798/sandbox/processwrapper-sandbox/3779/execroot/com_google_googleapis/bazel-out/host/bin/external/gapic_generator_python/gapic_plugin.runfiles/gapic_generator_python/gapic/cli/generate.py", line 21, in <module>
    from google.protobuf.compiler import plugin_pb2
ModuleNotFoundError: No module named 'google.protobuf'
--python_gapic_out: protoc-gen-python_gapic: Plugin failed with status code 1.
Target //google/cloud/aiplatform/v1:aiplatform-v1-py failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 2.730s, Critical Path: 2.47s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

[noahdietz]

Ah yep, I was using an old protoc; upgrading fixed that issue.

Ok great!

I'm having some trouble with the python libs though

Hm I'm not as familiar with the Python generator set up. I do know that there is a weird local dependency on the system's python installation (with bazel you'd expect everything to be hermetic, but this is one leak). Make sure you have python3 installed and possibly python-is-python3 as well (convenience package that directs all python calls to python3). I'm not sure we should attempt to debug that here though. Could you open an issue in gapic-generator-python?

[thesquelched]

Oh derp, I forgot what repo I opened this issue against. Let's close this one as resolved, thanks!

…

-- Scott Kruger ***@***.***

On Wed, Sep 8, 2021, at 12:14 PM, Noah Dietz wrote: > Ah yep, I was using an old protoc; upgrading fixed that issue. > Ok great! > I'm having some trouble with the python libs though > Hm I'm not as familiar with the Python generator set up. I do know that there is a weird local dependency on the system's python installation (with bazel you'd expect everything to be hermetic, but this is one leak). Make sure you have python3 installed and possibly `python-is-python3` as well (convenience package that directs all python calls to python3). I'm not sure we should attempt to debug that here though. Could you open an issue in gapic-generator-python <https://github.com/googleapis/gapic-generator-python/issues>? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#767 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGMYLGXWWPFFMYIP5DBZZDUA6KYTANCNFSM5DJHKF4Q>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[thesquelched]

FWIW @noahdietz you were right; it was an environment issue. Building against a debian bullseye docker image fixed things. Thanks for your help!