Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clientv3: grpc-go (v1.27.0) made API changes to balancer / resolver. #11563

Closed
eddycjy opened this issue Jan 29, 2020 · 54 comments · Fixed by #11564
Closed

clientv3: grpc-go (v1.27.0) made API changes to balancer / resolver. #11563

eddycjy opened this issue Jan 29, 2020 · 54 comments · Fixed by #11564
Assignees

Comments

@eddycjy
Copy link
Contributor

eddycjy commented Jan 29, 2020

After the release of grpc-go v1.27.0, an error occurred while pulling etcd / clientv3. The steps to reproduce it are as follows:

  1. go.mod:
go 1.13

require (
	github.com/coreos/etcd v3.3.18+incompatible // indirect
	github.com/coreos/go-systemd v0.0.0-00010101000000-000000000000 // indirect
	github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f // indirect
	github.com/gogo/protobuf v1.3.1 // indirect
	github.com/google/uuid v1.1.1 // indirect
	go.etcd.io/etcd v3.3.18+incompatible // indirect
	go.uber.org/zap v1.13.0 // indirect
	google.golang.org/grpc v1.27.0 // indirect
)

replace github.com/coreos/go-systemd => github.com/coreos/go-systemd/v22 v22.0.0
  1. command:
$ go get go.etcd.io/etcd/clientv3
# github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
../../go/pkg/mod/github.com/coreos/etcd@v3.3.18+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:114:78: undefined: resolver.BuildOption
../../go/pkg/mod/github.com/coreos/etcd@v3.3.18+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:182:31: undefined: resolver.ResolveNowOption
# github.com/coreos/etcd/clientv3/balancer/picker
../../go/pkg/mod/github.com/coreos/etcd@v3.3.18+incompatible/clientv3/balancer/picker/err.go:37:44: undefined: balancer.PickOptions
../../go/pkg/mod/github.com/coreos/etcd@v3.3.18+incompatible/clientv3/balancer/picker/roundrobin_balanced.go:55:54: undefined: balancer.PickOptions
  1. PR:clientv3: Fix grpc-go(v1.27.0) incompatible changes to balancer/resolver. #11564
@eddycjy eddycjy changed the title grpc-go (v1.27.0) made API changes to balancer / resolver. clientv3: grpc-go (v1.27.0) made API changes to balancer / resolver. Jan 29, 2020
@xiang90
Copy link
Contributor

xiang90 commented Jan 29, 2020

/cc @jpbetz

@nieaowei
Copy link

nieaowei commented Jan 30, 2020

版本不兼容,指定grpc的版本为1.26即可解决问题。
google.golang.org/grpc v1.26.0

@eddycjy
Copy link
Contributor Author

eddycjy commented Jan 30, 2020

@nieaowei Yes, but we need to solve the problem fundamentally.

@g00nix
Copy link

g00nix commented Feb 9, 2020

Did you find any workaround?

@eddycjy
Copy link
Contributor Author

eddycjy commented Feb 10, 2020

@gun1x You can first downgrade grpc to v1.26.0 to temporarily avoid this problem.

@g00nix
Copy link

g00nix commented Feb 10, 2020

@gun1x You can first downgrade grpc to v1.26.0 to temporarily avoid this problem.

ok, so the actual fix is adding this to go.mod:

require google.golang.org/grpc v1.26.0
replace github.com/coreos/go-systemd => github.com/coreos/go-systemd/v22 v22.0.0

@mostfamiliar
Copy link

@gun1x can i see your full go.mod for that fix? your suggestion does not fix the issue for me and i'm wondering if i'm missing something.

replace github.com/coreos/go-systemd => github.com/coreos/go-systemd/v22 v22.0.0

require (
	github.com/coreos/etcd v3.3.18+incompatible // indirect
	github.com/go-log/log v0.2.0 // indirect
	github.com/gogo/protobuf v1.3.1 // indirect
	github.com/json-iterator/go v1.1.9 // indirect
	github.com/lucas-clemente/quic-go v0.14.3 // indirect
	github.com/marten-seemann/qtls v0.7.1 // indirect
	github.com/micro/go-micro v1.18.0
	github.com/miekg/dns v1.1.27 // indirect
	github.com/mostfamiliar/shippy-service-consignment v0.0.0-20200212041600-94c659f72b79
	github.com/nats-io/jwt v0.3.2 // indirect
	github.com/pkg/errors v0.9.1 // indirect
	go.uber.org/atomic v1.5.1 // indirect
	go.uber.org/multierr v1.4.0 // indirect
	go.uber.org/zap v1.13.0 // indirect
	golang.org/x/crypto v0.0.0-20200210222208-86ce3cb69678 // indirect
	golang.org/x/lint v0.0.0-20200130185559-910be7a94367 // indirect
	golang.org/x/tools v0.0.0-20200211205636-11eff242d136 // indirect
	google.golang.org/genproto v0.0.0-20200211111953-2dc5924e3898 // indirect
	google.golang.org/grpc v1.27.1
)

error:

../../pkg/mod/github.com/coreos/etcd@v3.3.18+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:114:78: undefined: resolver.BuildOption
../../pkg/mod/github.com/coreos/etcd@v3.3.18+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:182:31: undefined: resolver.ResolveNowOption
# github.com/coreos/etcd/clientv3/balancer/picker
../../pkg/mod/github.com/coreos/etcd@v3.3.18+incompatible/clientv3/balancer/picker/err.go:37:44: undefined: balancer.PickOptions
../../pkg/mod/github.com/coreos/etcd@v3.3.18+incompatible/clientv3/balancer/picker/roundrobin_balanced.go:55:54: undefined: balancer.PickOptions

@eddycjy
Copy link
Contributor Author

eddycjy commented Feb 12, 2020

@mostfamiliar Your google.golang.org/grpc v1.27.1 should be modified to google.golang.org/grpc v1.26.0.

@g00nix
Copy link

g00nix commented Feb 12, 2020

@gun1x can i see your full go.mod for that fix? your suggestion does not fix the issue for me and i'm wondering if i'm missing something.

module mymod

go 1.13

require (
	github.com/coreos/etcd v3.3.18+incompatible
	github.com/coreos/go-systemd v0.0.0-00010101000000-000000000000 // indirect
	github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f // indirect
	github.com/gogo/protobuf v1.3.1 // indirect
	github.com/google/uuid v1.1.1 // indirect
	go.uber.org/zap v1.13.0 // indirect
	google.golang.org/grpc v1.26.0
)

replace github.com/coreos/go-systemd => github.com/coreos/go-systemd/v22 v22.0.0

@arCzWw
Copy link

arCzWw commented Feb 13, 2020

Has this problem been solved?I have to use version 1.27.0 or I can't use micro, but I need to use etcd to get service information.

@arges
Copy link

arges commented Feb 13, 2020

I have a fix for this issue here: #11580. How do I get a review for it?

@jbhannah
Copy link

@arCzWw Micro still uses 1.26.0, so you should be able to pin to that version in your go.mod (this solved the issue for me).

jbhannah added a commit to koverto/users that referenced this issue Feb 14, 2020
@yezihack
Copy link

yezihack commented Feb 16, 2020

遇到同样的问题. meet the same problem

# github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
vendor\github.com\coreos\etcd\clientv3\balancer\resolver\endpoint\endpoint.go:114:78: undefined: resolver.BuildOption
vendor\github.com\coreos\etcd\clientv3\balancer\resolver\endpoint\endpoint.go:182:31: undefined: resolver.ResolveNowOption
# github.com/coreos/etcd/clientv3/balancer/picker
vendor\github.com\coreos\etcd\clientv3\balancer\picker\err.go:37:44: undefined: balancer.PickOptions
vendor\github.com\coreos\etcd\clientv3\balancer\picker\roundrobin_balanced.go:55:54: undefined: balancer.PickOptions

@13642212901
Copy link

if modified to google.golang.org/grpc v1.26.0...
then protobuf build pb error undefined: grpc.SupportPackageIsVersion6..

@eddycjy
Copy link
Contributor Author

eddycjy commented Feb 18, 2020

@13642212901 The consideration is caused by the high version of protoc.

@13642212901
Copy link

@13642212901 The consideration is caused by the high version of protoc.

I know. I hope etcd-io/etcd support grpc v1.27.*,it,s so easy.

@hesongkang
Copy link

I meet the same issue.

I use the latest version protobuf in https://github.com/protocolbuffers/protobuf/releases

and I go get -u github.com/golang/protobuf/protoc-gen-go

then if I modified to google.golang.org/grpc v1.26.0

protobuf build pb error undefined: grpc.SupportPackageIsVersion6.

if I modified to google.golang.org/grpc v1.27.0

then etcd will report err:

@eddycjy
Copy link
Contributor Author

eddycjy commented Feb 19, 2020

Workaround:

  1. If it is grpc> = 1.27.0, then just change grpc v1.26.0 in the go.mod file, as follows:
require (
	...
	google.golang.org/grpc v1.26.0
)
  1. If the version of protoc is too high and grpc v1.26.0 is not compatible, you can downgrade protoc.

fix needs to wait for #11564.

@yezihack
Copy link

遇到同样的问题. meet the same problem

# github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
vendor\github.com\coreos\etcd\clientv3\balancer\resolver\endpoint\endpoint.go:114:78: undefined: resolver.BuildOption
vendor\github.com\coreos\etcd\clientv3\balancer\resolver\endpoint\endpoint.go:182:31: undefined: resolver.ResolveNowOption
# github.com/coreos/etcd/clientv3/balancer/picker
vendor\github.com\coreos\etcd\clientv3\balancer\picker\err.go:37:44: undefined: balancer.PickOptions
vendor\github.com\coreos\etcd\clientv3\balancer\picker\roundrobin_balanced.go:55:54: undefined: balancer.PickOptions

resolve method:
https://segmentfault.com/q/1010000021762281

@johnbellone
Copy link

I ended up nuking my attempt to get Bazel to work as a build environment and went back to simply using go mod and go build. I was able to quickly get it working with only the following:

go 1.14

require (
	github.com/golang/protobuf v1.4.2
	github.com/micro/go-micro/v2 v2.9.0
	google.golang.org/genproto v0.0.0-20191216164720-4f79533eabd1
)

@xmlking
Copy link

xmlking commented Jun 20, 2020

wish we move etcd dependency out of go-micro, maybe move it to plugins.
this is giving bad experience for first-time go-micro users.

@PVince81
Copy link

@invidian replacements did the job for me:

replace (
	github.com/coreos/etcd => go.etcd.io/etcd v0.5.0-alpha.5.0.20200425165423-262c93980547
	go.etcd.io/etcd => go.etcd.io/etcd v0.5.0-alpha.5.0.20200425165423-262c93980547
)

which is a little odd, since that branch is 6 years old.

doesn't seem to work, I get conflicts:

go: go.etcd.io/etcd@v0.5.0-alpha.5.0.20200425165423-262c93980547 used for two different module paths (github.com/coreos/etcd and go.etcd.io/etcd)

@vtolstov
Copy link

i'm use this trick -
replace (
google.golang.org/grpc => google.golang.org/grpc v1.27.0
github.com/coreos/etcd => github.com/ozonru/etcd v3.3.20-grpc1.27-origmodule+incompatible
)

@refs
Copy link

refs commented Jul 15, 2020

hmmmm, right. Thanks for sharing! I'll be using that one.

@KyteProject
Copy link

i'm use this trick -
replace (
google.golang.org/grpc => google.golang.org/grpc v1.27.0
github.com/coreos/etcd => github.com/ozonru/etcd v3.3.20-grpc1.27-origmodule+incompatible
)

Thanks mate, seems to have worked for me!

@zxpan
Copy link

zxpan commented Sep 30, 2020

Seems that etcd maintainer did not take serious to fix the issue. I still got this issue, where I do need latest gRPC (1.31+)

github.com/coreos/etcd/clientv3/balancer/resolver/endpoint

/home/alpha/go/pkg/mod/github.com/coreos/etcd@v3.3.25+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:114:78: undefined: resolver.BuildOption
/home/alpha/go/pkg/mod/github.com/coreos/etcd@v3.3.25+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:182:31: undefined: resolver.ResolveNowOption

github.com/coreos/etcd/clientv3/balancer/picker

/home/alpha/go/pkg/mod/github.com/coreos/etcd@v3.3.25+incompatible/clientv3/balancer/picker/err.go:37:44: undefined: balancer.PickOptions
/home/alpha/go/pkg/mod/github.com/coreos/etcd@v3.3.25+incompatible/clientv3/balancer/picker/roundrobin_balanced.go:55:54: undefined: balancer.PickOptions

How I get the latest etcd golang module if there is a fix?

@ptabor
Copy link
Contributor

ptabor commented Sep 30, 2020

@zxpan

The fix is in unreleased 3.5 branch:
4258cdd#diff-c3bc35973ca604ca1c88d111806c06c4

and backported to 3.4:
https://github.com/etcd-io/etcd/blob/release-3.4/clientv3/balancer/picker/roundrobin_balanced.go

Please try using etcd 3.4

@clarkmcc
Copy link

Why is this closed? I'm still having this issue and it looks like several others are as well.

@blueridgemountains1
Copy link

Why is this closed? I'm still having this issue and it looks like several others are as well.

yes, still an issue. Please fix, thanks

@Ayanrocks
Copy link

replace (
google.golang.org/grpc => google.golang.org/grpc v1.27.0
github.com/coreos/etcd => github.com/ozonru/etcd v3.3.20-grpc1.27-origmodule+incompatible
)

This seems to be the only workaround even in 2021. This needs to be fixed as a lot of packages is dependent on it. Apparently it's a wide known bug

@pan2za
Copy link

pan2za commented Apr 4, 2021

github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
../../../pkg/mod/github.com/coreos/etcd@v3.3.18+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:114:78: undefined: resolver.BuildOption
../../../pkg/mod/github.com/coreos/etcd@v3.3.18+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:182:31: undefined: resolver.ResolveNowOption

github.com/coreos/etcd/clientv3/balancer/picker

../../../pkg/mod/github.com/coreos/etcd@v3.3.18+incompatible/clientv3/balancer/picker/err.go:37:44: undefined: balancer.PickOptions
../../../pkg/mod/github.com/coreos/etcd@v3.3.18+incompatible/clientv3/balancer/picker/roundrobin_balanced.go:55:54: undefined: balancer.PickOptions

above given issue is solved after downgrading the etcd version mentioned below github.com/coreos/etcd v3.3.18+incompatible => github.com/coreos/etcd v3.3.4+incompatible

This solved my issue

@aka-achu
Copy link

replace (
	github.com/coreos/bbolt => go.etcd.io/bbolt v1.3.5
	google.golang.org/grpc => google.golang.org/grpc v1.26.0
)

require (
	github.com/coreos/bbolt v0.0.0-00010101000000-000000000000 // indirect
	github.com/coreos/etcd v3.3.25+incompatible // indirect
	github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f // indirect
	github.com/go-playground/validator/v10 v10.5.0
	github.com/google/btree v1.0.1 // indirect
	github.com/google/uuid v1.2.0
	github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 // indirect
	github.com/grpc-ecosystem/grpc-gateway v1.16.0 // indirect
	github.com/jonboulle/clockwork v0.2.2 // indirect
	github.com/prometheus/client_golang v1.10.0 // indirect
	github.com/soheilhy/cmux v0.1.5 // indirect
	github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802 // indirect
	go.etcd.io/etcd v3.3.25+incompatible
	go.uber.org/zap v1.16.0 // indirect
	golang.org/x/time v0.0.0-20210220033141-f8bda1e9f3ba // indirect
)

This is working for me.

mdbooth added a commit to mdbooth/cloud-provider-openstack that referenced this issue May 13, 2021
This bump is required to add support for grpc.WithNoProxy().

Note that bumping to v1.30.0 or newer causes a build failure in
go.etcd.io/etcd/clientv3, described in
etcd-io/etcd#11563. It is not possible to bump
gRPC further until etcd v3.5 is released, which updates the relevant api
uses.
mdbooth added a commit to shiftstack/cloud-provider-openstack that referenced this issue May 13, 2021
This bump is required to add support for grpc.WithNoProxy().

Note that bumping to v1.30.0 or newer causes a build failure in
go.etcd.io/etcd/clientv3, described in
etcd-io/etcd#11563. It is not possible to bump
gRPC further until etcd v3.5 is released, which updates the relevant api
uses.

RHBZ: #1944655

(cherry picked from commit 451aaf9)
k8s-ci-robot pushed a commit to kubernetes/cloud-provider-openstack that referenced this issue Jun 1, 2021
…ster proxy (#1528)

* Bump gRPC to v1.29.0 (#1527)

This bump is required to add support for grpc.WithNoProxy().

Note that bumping to v1.30.0 or newer causes a build failure in
go.etcd.io/etcd/clientv3, described in
etcd-io/etcd#11563. It is not possible to bump
gRPC further until etcd v3.5 is released, which updates the relevant api
uses.

* [manila-csi-plugin] Ignore proxy connecting to CSI sockets (#1527)
mdbooth added a commit to shiftstack/cloud-provider-openstack that referenced this issue Jun 8, 2021
This bump is required to add support for grpc.WithNoProxy().

Note that bumping to v1.30.0 or newer causes a build failure in
go.etcd.io/etcd/clientv3, described in
etcd-io/etcd#11563. It is not possible to bump
gRPC further until etcd v3.5 is released, which updates the relevant api
uses.

RHBZ: #1944655

(cherry picked from commit 451aaf9)
(cherry picked from commit 1f3c1b3)
Conflicts:
  go.mod:
    Context differences

  go.sum:
    Regenerated from scratch by go mod vendor

Cherry pick also includes go mod vendor
powellchristoph pushed a commit to powellchristoph/cloud-provider-openstack that referenced this issue Jan 19, 2022
…ster proxy (kubernetes#1528)

* Bump gRPC to v1.29.0 (kubernetes#1527)

This bump is required to add support for grpc.WithNoProxy().

Note that bumping to v1.30.0 or newer causes a build failure in
go.etcd.io/etcd/clientv3, described in
etcd-io/etcd#11563. It is not possible to bump
gRPC further until etcd v3.5 is released, which updates the relevant api
uses.

* [manila-csi-plugin] Ignore proxy connecting to CSI sockets (kubernetes#1527)
ldez pushed a commit to kvtools/etcdv3 that referenced this issue Sep 8, 2022
Can't use this workaround
(etcd-io/etcd#11563 (comment))
if valkeyrie mixes go.etcd.io/etcd with github.com/coreos/etcd for some reason.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment