Use Google.Protobuf protoc 3.12 in Grpc.Tools and expose --experimental_allow_proto3_optional switch

[kskalski]

Is your feature request related to a problem? Please describe.

I would like to use --experimental_allow_proto3_optional switch to enable optional fields in proto3 files

Describe the solution you'd like

• Grpc.Tools should use protoc from protobuf 3.12 release
• ideally it should expose a switch to enable --experimental_allow_proto3_optional flag to protoc

Describe alternatives you've considered

• Instead of using Grpc.Tools I can use Google.Protobuf.Tools
• Instead of using explicit support for --experimental_allow_proto3_optional I can play around with custom script, environment variable to use it (PROTOBUF_PROTOC), etc. not very convenient

Additional context

At some point the experimental flag should be enabled by default I guess, so apart from switching to new release it's not clear that adding support for it here is worth the effort. However alternatively I would propose adding a parameter to <Protobuf> tag that would allow passing arbitrary custom flags (or maybe limit them to some prefix / format)

[stanley-cheung]

Yes we have plan to support this. We also need to add GetSupportedFeatures() to each of the protoc plugins.

[kskalski]

Hi, when using Grpc.Tools 2.30.0-pre1 (.NET) package, is there any way to specify --experimental_allow_proto3_optional flag in directive? Right now I get error like this:

  voting_session.proto : error : This file contains proto3 optional fields, but --experimental_allow_proto3_optional was not set. [D:\rep\Kogut\src\VotingKernel\VotingKern
el.csproj]

[jtattermusch]

@kskalski I see. Adding an extra flag is not something one commonly does,
but perhaps looking at the MSBuild targets we usewill give you the answer on how to tweak it

grpc/src/csharp/Grpc.Tools/build/_protobuf/Google.Protobuf.Tools.targets

Line 273 in 80e834a

<ProtoCompile Condition=" '@(_Protobuf_OutOfDateProto)' != '' "

[JohnGalt1717]

It appears that this should be set with GrpcOutputOptions but it doesn't work.

This fails:

<Protobuf Include="../../Account/Account.Proto/**/*.proto" GrpcServices="Both" ProtoRoot="./../.." GrpcOutputOptions="--experimental_allow_proto3_optional" />

[jtattermusch]

GrpcOutputOptions won't work because it sets the '--grpc_opt=` command line argument:

grpc/src/csharp/Grpc.Tools/ProtoCompile.cs

Line 297 in 686e038

/// --grpc_opt=opt1,opt2=val (comma-separated).

grpc/src/csharp/Grpc.Tools/ProtoCompile.cs

Line 416 in 686e038

var cmd = new ProtocResponseFileBuilder();

Looks like we currently don't have a good way of adding a custom cmdline arg to protoc invocations (which is this case is bad, but otherwise it's a protection against breaking the msbuild logic in unexpected ways).

[kskalski]

It seems we would need to wait for transition of this feature to non-experimental in protobuf library, right? I think having a way to pass custom options to protoc compiler is desirable, because in future similar use-cases might pop up. But you are probably in a more informed position to decide:

• timeline on presence field feature becoming enabled by default
• usefulness of the flag passing feature for other cases now and in future
• vs the risks of opening up MSBuild options for custom stuff in a more or less constrained fashion

[JohnGalt1717]

@kskalski I would assume that in the future there would be similar to this, and this is a major blocker for us right now because this feature determines structure and obviously optional results in better nullable types in C# and other languages versus the alternatives.

[kskalski]

Any idea if this is possible to work around by using Google.Protobuf MSBuild task directly? Or does it have the same limitation and the one from Grpc.Tools is just a consequence of that?

[stanley-cheung]

There is another way to test proto3 optional when the feature is under experimental. See https://github.com/protocolbuffers/protobuf/blob/master/docs/implementing_proto3_presence.md#satisfying-the-experimental-check

2. Make your filename (or a directory name) contain the string test_proto3_optional. This indicates that the proto file is specifically for testing proto3 optional support, so the check is suppressed.

There is an example there:

# Another option:
$ cp test.proto test_proto3_optional.proto
$ ./src/protoc test_proto3_optional.proto --cpp_out=.

So you can try that with --csharp_out and the corresponding --grpc_out I suppose.

[stale]

This issue/PR has been automatically marked as stale because it has not had any update (including commits, comments, labels, milestones, etc) for 30 days. It will be closed automatically if no further update occurs in 7 day. Thank you for your contributions!

[JohnGalt1717]

bump

[R030t1]

If anyone has a bit of time for this it would be appreciated, a general configuration mechanism would prevent blocking configuration issues from happening again.

[moserware]

I made an attempt at adding support in #25374

The basic idea is that you'd have something like this in your .csproj:

<Protobuf Include="**\*.proto" CompileOutputs="true" AdditionalProtocOptions="experimental_allow_proto3_optional" />

[moserware]

The latest (3.15+) releases of protoc no longer require this --experimental_allow_proto3_optional flag which might reduce (and ultimately eliminate) the need for this feature. Although Grpc.Tools currently ships with protoc 3.12.3, you can specify an alternate location via an environment variable and/or an MSBUILD variable.

For example, you can download the latest protoc release:

PB_REL=https://github.com/protocolbuffers/protobuf/releases
mkdir -p /tmp/protos
wget -qO- $PB_REL/download/v3.15.2/protoc-3.15.2-linux-x86_64.zip > /tmp/protos/protoc.zip
unzip /tmp/protos/protoc.zip -d /tmp/protos

In this case, protoc will be at /tmp/protos/bin/protoc

Then you can set the PROTOBUF_PROTOC environment variable to this path or you can set the Protobuf_ProtocFullPath variable (or even set the environment variable in your .csproj) as in:

<PropertyGroup>  
 <Protobuf_ProtocFullPath>/tmp/protos/bin/protoc</Protobuf_ProtocFullPath>
</PropertyGroup>

or

<PropertyGroup>
  <PROTOBUF_PROTOC>/tmp/protos/bin/protoc</PROTOBUF_PROTOC>
</PropertyGroup>

Once you rebuild the project with this set up, you should be able to compile the proto3 protos that have optional in them without errors.

In theory you could also use the same technique with the earlier versions of protoc by creating a script that calls protoc with the flag specified in addition to the arguments supplied by Grpc.Tools.

[jtattermusch]

latest (3.15+) releases

The latest Grpc.Tools now has protoc 3.14.0 (see #25131) and in the next release we'll very likely have protoc 3.15.2 (once we update our third_party/protobuf).