Missing Well Known Protobuf Types in Ruby gem

[blowmage]

Hi all! I'm super excited to be using grpc and protobuf in ruby! We are working on a client to connect to several Google Cloud services, and we are having an issue using the protobuf files from the google/googleapis project. Several core/well known protobuf types were removed from that project with the assumption that they would be provided by this project. And those protobuf files do now exist in this project, but they are not included in the ruby gem.

The python package looks to build these well known protobuf types as part of its build process. But in ruby those files are missing.

To get around this we've had to compile these protobuf files into ruby and host them in our project, but we would much rather have these files provided by the gem. Is that possible?

I should also note that even though google.protobuf.Descriptor is provided in the C code of the ruby gem, any proto3 files that import it will add require 'google/protobuf/descriptor' to the generated ruby code. So we've created an empty file so those requires will succeed. So even if these well known types are provided in the C code we will still need those paths available to be required in ruby.

Thanks!

[blowmage]

I would really like to see this fixed before google-protobuf 3.0 is released. If you can give me some guidance on how you want this to be addressed I'd be happy to submit a pull request for it.

[haberman]

Hi there, thanks for the report and the offer to fix!

I agree that the well-known types should be included in the gem. Feel free to submit a PR for this.

Ultimately we should include descriptor.proto in the gem also, which should provide google/protobuf/descriptor. But we can't do that until proto2 support is implemented (see: #1198)

[blowmage]

AFAICT, Google::Protobuf::Descriptor and Google::Protobuf::DescriptorPool are implemented in the C code, so the proto2 file doesn't need to be supported to use in the gem.

$ irb
irb(main):001:0> Google::Protobuf::Descriptor
NameError: uninitialized constant Google
irb(main):002:0> require "google/protobuf"
=> true
irb(main):003:0> Google::Protobuf::Descriptor
=> Google::Protobuf::Descriptor
irb(main):004:0> Google::Protobuf::DescriptorPool
=> Google::Protobuf::DescriptorPool

[blowmage]

@haberman How do you want the well known types to be added to the gem? Can I simply submit a PR to add the generated ruby files? Or should those ruby files be generated for each gem build? I'm not sure how the gem is built and released so I'm not sure if the generated ruby files need to be added to the git repo or not.

[haberman]

Ah, I should clarify something. Google::Protobuf::Descriptor is a descriptor, but the proto types representing descriptors are different (DescriptorProto instead of Descriptor).

We need both. In particular, custom options are declared as extensions of the proto classes -- see https://developers.google.com/protocol-buffers/docs/proto#customoptions

[haberman]

We should generate the well-known types in the Rakefile, so they are built each time.

[haberman]

This was fixed in the attached PR.

[blowmage]

@haberman Is there an ETA when this will be released?

[nicolasnoble]

@blowmage these should've been released by the way: 3.0.0.alpha.5.0.4 on rubygems.