google.golang.org/grpc cannot be upgraded >= v1.30.0

[entropitor]

What happened: Kubernetes is using GRPC for a lot of stuff (CSI, ...) to allow for extensibility, however, the grpc-go driver contained a bug which basically means that writing GRPC servers for kubernetes was not allowed due to a broken implementation in the go client (grpc/grpc-go#2628). So ideally, this grpc-go dependency should be upgraded across the board and maybe even backported to deliver on the promise of interoperability with other languages

What you expected to happen: Being able to e.g. write a CSI driver in typescript / rust

How to reproduce it (as minimally and precisely as possible): See grpc/grpc-go#2628 for more information, basically the grpc-go client sets the authority field to an HTTP2 spec-uncompliant value and so it's not working in rust / nodejs

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

[nikhita]

/sig architecture
/area code-organization
/area dependency

related - #91881

cc @dims

[dims]

@entropitor same comment when we wanted to do this last #91898 (comment)

Can you please update etcd to newer grpc and then we can pull that in here? grpc ends up bringing in a bunch of things that has broken us in the past and our major usage is etcd oriented, so it would make sense to update etcd master first to give us some confidence.

[nikhita]

etcd poc - etcd-io/etcd#12155

[TheEnbyperor]

Any updates on this? Currently hitting this issue myself writing some things in Rust.

[dims]

we need to update etcd, containerd and others that we depend on to the latest google.golang.org/grpc before we can update kubernetes. If someone wants to propose PRs in those repositories (At least etcd and containerd!)

grpc/grpc-go@v1.27.0...v1.31.1 (looks like a bunch of changes in there)

[nikhita]

update from etcd maintainers - etcd-io/etcd#12124 (comment)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[dashpole]

/remove-lifecycle stale

[dims]

cc @ptabor

[ptabor]

etcd support up to grpc-1.29.1
etcd 3.5 is close to get support for grpc-1.32.

grpc-1.32 is blocked (both for K8S and etcd) due to go-protobuf>1.4.x dependency that does not work well with gogo-generated messages.

[liggitt]

ran into this issue in the go1.16 bump PR. as part of golang bumps, we upgrade golang.org/x/... dependencies to latest.

golang.org/x/oauth2@latest brings in new versions of cloud.google.com/go and google.golang.org/api which depend on grpc v1.31+

The etcd requirement for < v1.30 (etcd-io/etcd#12124) and the gogo-protobuf incompatibility (#96564) means we can no longer pick up new versions of those libraries.

[ptabor]

Just yesterday released 3.5.0-alpha.0 version of etcd (etcd-io/etcd#12498) that support grpc-1.32.

[liggitt]

glad to see that... now to push on #96564

[dims]

w00t!

[dashpole]

@ptabor great to hear! Is the plan to update to 3.5.0-alpha.0 in kubernetes/kubernetes, or wait for 3.5.0 to be released?

[liggitt]

we'll wait for a non-alpha release. #96564 also has to be resolved before we can upgrade

[dashpole]

@liggitt it looked like @ptabor was able to bump grpc to 1.32.0 while leaving protobuf at 1.3.5 in etcd-io/etcd#12709. Am I missing a reason why #96564 has to be resolved before upgrading?

[ptabor]

• grpc-1.32 is still protobuf-1.3.5 compliant.
• grpc-1.33 started to have dependency on protobuf-1.4.x.

I'm not sure if etcd-3.5 will wait for gogo-mitigation. I would rather assume it will not.

One thing to notice -> etcd-3.5 is not exact drop-in replacement for etcd-3.4, as package names has changed (forced by go-modules naming rules to have 'v3' suffix). So the usage of newer client will require some 'adaptation' work on k8s side.

[liggitt]

@liggitt it looked like @ptabor was able to bump grpc to 1.32.0 while leaving protobuf at 1.3.5 in etcd-io/etcd#12709. Am I missing a reason why #96564 has to be resolved before upgrading?

maybe I'm getting my libraries mixed up. if we can update one without waiting for the other, that's great

One thing to notice -> etcd-3.5 is not exact drop-in replacement for etcd-3.4, as package names has changed (forced by go-modules naming rules to have 'v3' suffix). So the usage of newer client will require some 'adaptation' work on k8s side.

I haven't checked yet, but switching to a v3 library and making slight adjustments to pick up the new library isn't generally an issue. Is the client library still wire-compatible with existing etcd servers?

---

edit on 2022-03-08 to leave breadcrumbs for myself

looks like google.golang.org/grpc switched from github.com/golang/protobuf v1.3.x to v1.4.x in:

• 1.34.0+ in grpc/grpc-go@06c094c
• 1.33.2+ in grpc/grpc-go@c1989b5

github.com/golang/protobuf 1.4+ is implemented in terms of google.golang.org/protobuf

google.golang.org/protobuf prior to v1.26.0 is incompatible with github.com/gogo/protobuf use of the github.com/golang/protobuf's marshaling/unmarshaling internals

tl;dr:

• if you use both github.com/golang/protobuf and google.golang.org/protobuf, you must use github.com/golang/protobuf 1.4+ (according to github.com/golang/protobuf)
• if you also use github.com/gogo/protobuf, then github.com/golang/protobuf [1.4, 1.5.2) and google.golang.org/protobuf [1.20, 1.26.0) are incompatible. The minimum versions that will work are:
  • https://github.com/golang/protobuf/releases/tag/v1.5.2
  • https://github.com/protocolbuffers/protobuf-go/releases#v1.26-support-aberrant-types

[ptabor]

The wire format has not changed.

[wojtek-t]

/cc

[liggitt]

#100488 in progress (targeting Kubernetes v1.22, assuming etcd v3.5.0 is available by then)

[liggitt]

kubernetes @ HEAD now has current versions of:

• github.com/golang/protobuf
• google.golang.org/protobuf
• google.golang.org/grpc