Update Protobuf Version to 3.10.0-RC1

[rafi-kamal]

@karthikravis

[linux-foundation-easycla]

One or more committers are not authorized under a signed CLA as indicated below. Please click here to be authorized.

• ❌ Rafi Kamal (9db1a64, 7030c70)

[jtattermusch]

I resolved conflict in depedencies.props and kicked off the test.
Please note that this PR is for testing purposes only and it should not be merged (we should only merge once 3.10.0 stable is released and we create a PR for that)

Also, it seems that we didn't finish the 3.9.0 version update the last time.
here are the attempts
#19867
#19630

[rafi-kamal]

@jtattermusch Thanks! I looked at the failures and found the following root causes:

• Could not find gem 'google-protobuf (~> 3.10)', which is required by gem 'grpc': We have already uploaded 3.10.0.rc.1 to rubygem, but I think it fails to find it since 3.10.0.rc.1 is a pre-release version. Not sure if we would be able to fix it without doing a stable release, @TeBoring do you have any ideas?
• ERROR: error loading package '@com_google_protobuf//': Unable to find package for @rules_proto//proto:defs.bzl: The repository '@rules_proto' could not be resolved.: This is happening only in C++ iOS and ObjC, @rmstar could it be happening because we haven't uploaded that C++ podspec?
• npm ERR! Failed at the grpc@1.25.0-dev install script 'node-pre-gyp install --fallback-to-build --library=static_library'.
  npm ERR! Make sure you have the latest version of node.js and npm installed.
  npm ERR! If you do, this is most likely a problem with the grpc package,
  npm ERR! not with npm itself.: Not sure why it's happening

[rafi-kamal]

I investigated the second error (error loading package '@com_google_protobuf//), I think this is happening because it fails the resolve the http_archive for com_google_protobuf (3.10.0-rc1) in https://github.com/grpc/grpc/blob/master/bazel/grpc_deps.bzl. Although I can't figure out why is it failing to download that archive, it does exist: https://github.com/google/protobuf/archive/ae1bcaad6ffcd04ca5d40f21dc3fab4f965e49cb.tar.gz. @jtattermusch could it be because the rule doesn't accept pre-release archives?

[rafi-kamal]

Actually, the error seems to be related to https://github.com/bazelbuild/rules_proto. I'm not very familiar with rules_proto, @hlopko will you be able to help?

[gnossen]

@rafi-kamal @rules_proto is a transitive dependency of ours that we pull in via @com_google_protobuf. The repository isn't being found because of a lack of support for recursive workspaces in Bazel. Protobuf pulls that dependency in here in its deps() function. It was first added since our last update of protobuf, which explains why this update is the first time we've seen the problem.

The solution is to invoke protobuf_deps() in our WORKSPACE file. Unfortunately, this is going to force all of our downstream dependencies to do the same. Not ideal. Definitely Bazel's biggest wart of all.

[rafi-kamal]

@jtattermusch / @gnossen can you please re-run the Kokoro tests now that #20256 is merged?

[rafi-kamal]

@haberman can you please have a look at this error: https://source.cloud.google.com/results/invocations/ba7bb54a-43f6-472d-9945-7435fe9b8864/targets/github%2Fgrpc%2Frun_tests%2Fobjc_macos_opt_native%2Fios-cpp-test-cronet/tests

Looks like something related to ZeroCopyOutputStream.

[rafi-kamal]

@gnossen I'm seeing a similar error: ERROR: /root/.cache/bazel/_bazel_root/d2dc70c3d9da3fab488ba0dcbbd35051/external/com_google_protobuf/BUILD:874:1: no such package '@six//': The repository '@six' could not be resolved and referenced by '@com_google_protobuf//:protobuf_python' (stacktrace). The dependency is declared in protobuf_deps and the BUILD file for six exists here.

Do you know why we are getting this error even though we are calling protobuf_deps() from the gRPC workspace?

[rafi-kamal]

I talked to @gerben-s and @nicolasnoble about the C++ failure in Obj-C tests. The root cause seems to be that gRPC is referencing old runtime protoc files. I checked the changed files in this PR, to me it looks like that protobuf references and protobuf submodule did get updated. I tried to manually run the tests using tools/run_tests/run_tests.py -l objc from my Mac, but it's failing with an No module named BaseHTTPServer import error. Running with --use_docker flag also fails with execve() arg 3 contains a non-string value error.

@nicolasnoble would you be able to help debugging what's going on? I'm not quite sure how to proceed from here.

[rafi-kamal]

@jtattermusch we are currently behind our release schedule, and I'm not sure how much time is it going to take to fix the two interop issues I mentioned above. Are you okay with us moving forward with the 3.10.x release and then continue this testing in parallel?

[jtattermusch]

Summary of the problems:

1. I see ruby issue

Bundler could not find compatible versions for gem "google-protobuf":
  In Gemfile:
    grpc was resolved to 1.25.0.dev, which depends on
      google-protobuf (~> 3.10)

but that seems easily solvable - we'd need ruby to build though to get good signal from some of the tests.

2. ASAN violation at

=================================================================
==18==ERROR: AddressSanitizer: odr-violation (0x7f0c7a58a360):
  [1] size=32 'google::protobuf::_DoubleValue_default_instance_' bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/wrappers_proto/google/protobuf/wrappers.pb.cc:21:3
  [2] size=32 'google::protobuf::_DoubleValue_default_instance_' external/com_google_protobuf/src/google/protobuf/wrappers.pb.cc:21:3
These globals were registered at these points:
  [1]:
    #0 0x43ca2d in __asan_register_globals /tmp/clang-build/src/compiler-rt/lib/asan/asan_globals.cc:362:3
    #1 0x7f0c7a57031b in asan.module_ctor (/b/f/w/bazel-out/k8-fastbuild/bin/test/core/channel/channel_trace_test@poller=epoll1.runfiles/com_github_grpc_grpc/test/core/channel/../../../_solib_k8/libexternal_Scom_Ugoogle_Uprotobuf_Slibwrappers_Uproto.so+0x2b31b)
  [2]:
    #0 0x43ca2d in __asan_register_globals /tmp/clang-build/src/compiler-rt/lib/asan/asan_globals.cc:362:3
    #1 0x7f0c7a39566b in asan.module_ctor (/b/f/w/bazel-out/k8-fastbuild/bin/test/core/channel/channel_trace_test@poller=epoll1.runfiles/com_github_grpc_grpc/test/core/channel/../../../_solib_k8/libexternal_Scom_Ugoogle_Uprotobuf_Slibprotobuf.so+0x97866b)

https://source.cloud.google.com/results/invocations/2e5c9620-069f-44b7-a3f8-7076a306e760/targets/%2F%2Ftest%2Fcore%2Fchannel:channel_trace_test@poller%3Depoll1/log

3. a bunch of sanity test failures

https://source.cloud.google.com/results/invocations/1bfc1da3-8db2-480e-873a-1ab3cd50f1e9/targets
(all should be fixable quite easily)

[jtattermusch]

Also, the protobuf podspec seems to be out of date:
https://source.cloud.google.com/results/invocations/1bfc1da3-8db2-480e-873a-1ab3cd50f1e9/targets/github%2Fgrpc%2Frun_tests%2Fsanity_linux_dbg_native%2Ftools%2Fdistrib%2Fcheck_protobuf_pod_version.sh/tests

perhaps that's the issue with ObjC tests mentioned above?

+ '[' 3.10.0 '!=' 3.8.0 ']'
+ echo 'Protobuf version in src/objective-c/!ProtoCompiler-gRPCPlugin.podspec does not match protobuf version in third_party/protobuf.'
Protobuf version in src/objective-c/!ProtoCompiler-gRPCPlugin.podspec does not match protobuf version in third_party/protobuf.

[rafi-kamal]

@veblush looked at the ASAN failure and found out that libprotobuf.so is generating cc files for wrapper protos, but the cc_grpc_library rule also includes those files from proto files defined by well_known_proto_libs. The solution will be to either remove the common wrapper protos from libprotobuf.so or to stop using well_known_proto_libs from gRPC with libprotobuf.so.

I've also updated the Ruby Gem for 3.10.0, so that should fix the Ruby problem.

[jtattermusch]

@rafi-kamal thanks! I started the tests again, let's see what the results are.

[jtattermusch]

The new ruby 3.10 gems might be broken: #20471

[jtattermusch]

Superseded by #21590