Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

envoy@v1.32.3/go.mod checksum mismatch #1083

Open
jarrodhroberson opened this issue Jan 10, 2025 · 41 comments
Open

envoy@v1.32.3/go.mod checksum mismatch #1083

jarrodhroberson opened this issue Jan 10, 2025 · 41 comments

Comments

@jarrodhroberson
Copy link

I am trying to build a project and when I updated dependencies I started getting this error.

ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build b1770871-d733-4430-bc45-892600527fa9 status: FAILURE
go: downloading go1.23.4 (linux/amd64)
verifying github.com/envoyproxy/go-control-plane/envoy@v1.32.3/go.mod: checksum mismatch
        downloaded: h1:F6hWupPfh75TBXGKA++MCT/CZHFq5r9/uwt/kQYkZfE=
        go.sum:     h1:c955gQjaXHsMxMjHjEZ7nwIzMJYxXpN+sJIGufsSbg4=

I can get around it locally by setting GONOSUMDB='github.com/envoyproxy/*' but this does not let me deploy it to google app engine where I can not specify that.

Is there a solution to this?

@mmorel-35
Copy link
Contributor

Hi @jarrodhroberson ,

Would you like to share your project setup ?

@emilhauk
Copy link

emilhauk commented Jan 10, 2025

Here's an example: https://github.com/emilhauk/envoy-checksum-error
This does not install the exact same version, but the problem remains the same.

@mmorel-35
Copy link
Contributor

mmorel-35 commented Jan 10, 2025

Can you try this

go get github.com/envoyproxy/go-control-plane/envoy@latest
go mod tidy
go build ./...

@LeeBrotherston
Copy link

For what it's worth, I am experiencing the same issue and so tried the suggested fix above which yields the same error:

> go get github.com/envoyproxy/go-control-plane/envoy@latest
verifying github.com/envoyproxy/go-control-plane/envoy@v1.32.3/go.mod: checksum mismatch
	downloaded: h1:c955gQjaXHsMxMjHjEZ7nwIzMJYxXpN+sJIGufsSbg4=
	go.sum:     h1:F6hWupPfh75TBXGKA++MCT/CZHFq5r9/uwt/kQYkZfE=

SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

I would note that in my case, the downloaded and go.sum values are the opposite way around to the original poster.

Looking at the changelog I see that there are changes to envoy/go.mod which references this issue: 4b91141

I suspect that this change may be the root cause of the issue, I have not tested but this line raised some suspicion for me:

replace github.com/envoyproxy/go-control-plane@v0.13.4 => ../

@emilhauk
Copy link

emilhauk commented Jan 11, 2025

I also tried your suggestion, but to no avail. I get different hashes though (same as in the example repo linked above).

Then, I tried to install this using a basic docker container, which apparently works just fine 🙈:

$ docker run --rm -it golang:alpine
Unable to find image 'golang:alpine' locally
alpine: Pulling from library/golang
41974eb6247f: Download complete 
0a2ab5391ffc: Download complete 
4f4fb700ef54: Already exists 
1f3e46996e29: Download complete 
06f05ace1117: Download complete 
Digest: sha256:c23339199a08b0e12032856908589a6d41a0dab141b8b3b21f156fc571a3f1d3
Status: Downloaded newer image for golang:alpine

/go # go version
go version go1.23.4 linux/amd64

/go # go mod init example.com/m
go: creating new go.mod: module example.com/m
go: to add module requirements and sums:
	go mod tidy

/go # go get github.com/envoyproxy/go-control-plane/envoy@latest
go: warning: ignoring go.mod in $GOPATH /go
go: downloading github.com/envoyproxy/go-control-plane v0.13.4
go: downloading github.com/envoyproxy/go-control-plane/envoy v1.32.3
go: downloading google.golang.org/protobuf v1.35.2
go: downloading github.com/cncf/xds/go v0.0.0-20240723142845-024c85f92f20
go: downloading github.com/envoyproxy/protoc-gen-validate v1.1.0
go: downloading github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10
go: downloading google.golang.org/grpc v1.67.1
go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240814211410-ddb44dafa142
go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240814211410-ddb44dafa142
go: downloading cel.dev/expr v0.16.0
go: downloading golang.org/x/net v0.28.0
go: downloading golang.org/x/sys v0.24.0
go: downloading golang.org/x/text v0.17.0
go: added cel.dev/expr v0.16.0
go: added github.com/cncf/xds/go v0.0.0-20240723142845-024c85f92f20
go: added github.com/envoyproxy/go-control-plane v0.13.4
go: added github.com/envoyproxy/go-control-plane/envoy v1.32.3
go: added github.com/envoyproxy/protoc-gen-validate v1.1.0
go: added github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10
go: added golang.org/x/net v0.28.0
go: added golang.org/x/sys v0.24.0
go: added golang.org/x/text v0.17.0
go: added google.golang.org/genproto/googleapis/api v0.0.0-20240814211410-ddb44dafa142
go: added google.golang.org/genproto/googleapis/rpc v0.0.0-20240814211410-ddb44dafa142
go: added google.golang.org/grpc v1.67.1
go: added google.golang.org/protobuf v1.35.2

The issue seems to be local or system related, but affecting several others as well it seems. I have tried the following:

  • go clean -cache -modcache
  • manually deleting content from $GOMODCACHE

I don't get how this is the only package of countless on my machine that's acting up. Any insights and/or suggestions would be greatly appreciated 😃

@jarrodhroberson
Copy link
Author

jarrodhroberson commented Jan 11, 2025

I can not share my project, it is not public.

I am not using this directly, it is being accessed as a dependency of a dependency. I think it a Google Cloud SDK dependency.

I tracked it down to "opentelemetry" using this version.

I can get it to work LOCALLY by either setting GONOSUMDB or replacing my local hash with the expected one.

But, when I try and deploy to Google App Engine Standard, I am NOT in control of the build environment and can not set GONOSUMDB (which I think is a hack work around).

I did finally get an absolutely crap workaround to get it to deploy by replacing the hash in go.sum with the one that GAE build expects. Which is "correct" I have no idea because the next time I tried to deploy, the hashes were reversed and I got the error again.

The only reason that the checksum would be different for the exact same version is someone replaced a published binary and did not bump the version number for some reason. I know this because when I first started using Go and releasing my code I did the same thing and it bit me immediately.

Deleting the contents of go.mod and deleting go.sum completely can affect the hashes that are retrieved. I have tried everything, including doing that. I have experienced the order of the hashes reversing as a previous poster says. That is why, replacing the hash in my go.sum before pushing to Google App Engine for deployment is not a solution, the hash that GAE pulls seems to be non-deterministic.

I am using GOPROXY=direct, which should pull directly from the source,
But, if the source is a proxy itself, OR you can not control GOPROX (I can not) this is useless. This is a fundamental error that only the project team can resolve deterministically.

There is no acceptable client side fix for this if you do not control the version of the library directly.

@jarrodhroberson
Copy link
Author

jarrodhroberson commented Jan 13, 2025

I created a new empty project, and the very first time I used go mod tidy this is what I got.

GOROOT= #gosetup
GOPATH=/home/jhr/go #gosetup
/usr/local/go/bin/go get -t -v cloud.google.com/go/storage/... #gosetup
go: downloading google.golang.org/api v0.214.0
go: downloading google.golang.org/grpc v1.67.3
go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20241209162323-e6fa225c2576
go: downloading go.opentelemetry.io/otel v1.29.0
go: downloading go.opentelemetry.io/otel/trace v1.29.0
go: downloading go.opentelemetry.io/otel/metric v1.29.0
github.com/envoyproxy/go-control-plane/envoy@v1.32.3: verifying module: checksum mismatch
	downloaded: h1:c1EIw4vwYCaovxRZtyycws8aX6dJ9W2p+4bCi7mcDgw=
	sum.golang.org: h1:hVEaommgvzTjTd4xCaFd+kEQ2iYBtGxP6luyLrx6uOk=

SECURITY ERROR
This download does NOT match the one reported by the checksum server.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

this is the imports

import (
	"context"
	"fmt"
	"io"
	"os"
	"time"

	"cloud.google.com/go/storage"
)

and since go mod tidy will not run this is what my go.mod looks like

module github.com/jarrodhroberson/youbroketheinternet

go 1.23.4

whatever you guys did, you broke Google Cloud Platform libraries in a very bad way.

@zmiklank
Copy link

Hello.
Running go mod tidy results in the error described in this thread also for https://github.com/istio/proxy when running the go mod tidy on my localhost.

@valerian-roche
Copy link
Contributor

Hey, it seems pkg.dev cached a very short release prior to the main one (only partially).
I will cut another release to hopefully improve on this issue, but given (other) go.mod issues it's quite complicated and long to do

@jarrodhroberson

This comment has been minimized.

@mjnowen
Copy link

mjnowen commented Jan 21, 2025

Hi, any update on this? ETA? @valerian-roche

@klauspost
Copy link

Please release a clean (new) version. My proxy is also picking this up.

@LeeBrotherston
Copy link

Not the desired fix I know, but for others who want a workaround whilst this is resolved, you can:

  • Remove the package:
    • go get github.com/envoyproxy/go-control-plane@none
  • Remove any locally cached copies of the module
  • Re-add the module forcing a SUMDB & Proxy that has the "right" version:
    • GOSUMDB="sum.golang.org" GOPROXY="https://proxy.golang.org/" go get github.com/envoyproxy/go-control-plane
    • GOSUMDB="sum.golang.org" GOPROXY="https://proxy.golang.org/" go mod tidy

@mjnowen
Copy link

mjnowen commented Jan 27, 2025

Corporate AppSec policy forbids use of any proxy, so using GOPROXY=direct where it is broken.

@jarrodhroberson
Copy link
Author

jarrodhroberson commented Jan 27, 2025 via email

@LeeBrotherston
Copy link

LeeBrotherston commented Jan 27, 2025

FWIW I am nothing to do with this project, just another person on the internet who uses this module like you.

I'm offering what worked for me in case that's useful for others, I do realize that this is not an ideal solution, and I'm not suggesting as such.

@supakeen
Copy link

Did you move a tag? Could you cut a new no-change release 1.32.4 as this is quite an annoyance to deal with?

@schuellerf
Copy link

Sounds great, a new version with the new checksum would perfectly solve my issue, with integration tests!

@klauspost
Copy link

@valerian-roche Would that be possible?

@bombsimon
Copy link

Don't want to post a "+1" but if it helps with the decision this sounds like the proper solution and also what solved this last time it happened in #512.

@jarrodhroberson
Copy link
Author

Don't want to post a "+1" but if it helps with the decision this sounds like the proper solution and also what solved this last time it happened in #512.

this does not fix those of us suffering from this because of transient dependencies out of our control. The fact that this has happened TWICE should be of GREAT concern, especially since this breaks ALL OF GOOGLE CLOUD PLATFORM client libraries right now.

This should never happen once much less twice. Where ever you are initially publishing to should never let you deploy the same version number a second time to begin with.

@valerian-roche
Copy link
Contributor

I released envoy/v1.32.4 to hopefully address this issue.
Sorry for the delay, I did not have bandwidth to work on this repo in the last weeks.

I am hoping this will end the multiple issues we had since the module split. If you see other issues please raise them, as it is hard to have visibility on the go.mod behavior or run tests on the impact.

@valerian-roche
Copy link
Contributor

Don't want to post a "+1" but if it helps with the decision this sounds like the proper solution and also what solved this last time it happened in #512.

this does not fix those of us suffering from this because of transient dependencies out of our control. The fact that this has happened TWICE should be of GREAT concern, especially since this breaks ALL OF GOOGLE CLOUD PLATFORM client libraries right now.

This should never happen once much less twice. Where ever you are initially publishing to should never let you deploy the same version number a second time to begin with.

Hi

I apologize for the issue created here, but none of us were maintainers of this repository at this time.
This repository is not maintained by a big corporation (it is on my personal effort, not my company) but is used by some, even though it is experimental (as stated by its version). We are trying to improve the situation by splitting the repository in sub-modules as most users only depend on the envoy generated protobufs, but sadly this created a few issues. I am hoping those are now solved and we can start working on the repository again without versioning issues.

@supakeen
Copy link

supakeen commented Feb 3, 2025

I released envoy/v1.32.4 to hopefully address this issue. Sorry for the delay, I did not have bandwidth to work on this repo in the last weeks.

I am hoping this will end the multiple issues we had since the module split. If you see other issues please raise them, as it is hard to have visibility on the go.mod behavior or run tests on the impact.

Thank you Valerian :)

@mjnowen
Copy link

mjnowen commented Feb 3, 2025

Thanks @valerian-roche! This is fixed for me now.

@joaoeinride
Copy link

Thanks @valerian-roche and team

@mmorel-35
Copy link
Contributor

@jarrodhroberson ,
Would you like to confirm that the release created by @valerian-roche is fixing the issue you've met ?

@jarrodhroberson
Copy link
Author

jarrodhroberson commented Feb 11, 2025 via email

@schuellerf
Copy link

@jarrodhroberson did you try to edit your go.mod and just update the version to v1.32.4 (even though it might be "indirect") and run go mod tidy?
This should be well compatible to other libraries having this dependency…

@mjnowen
Copy link

mjnowen commented Feb 12, 2025

FWIW. This simply fixed it for me. I never expected an already shipped release to be altered. That would be an anti-pattern. Time to move on and update...

go get -t -u ./...
go mod tidy

@jarrodhroberson
Copy link
Author

jarrodhroberson commented Feb 13, 2025 via email

@jarrodhroberson
Copy link
Author

jarrodhroberson commented Feb 13, 2025 via email

TheMeier added a commit to TheMeier/prometheus that referenced this issue Feb 26, 2025
See envoyproxy/go-control-plane#1083
sadly an existing tag was moved breaking module checksums. As a workaround
a new version was released
hagen1778 pushed a commit to VictoriaMetrics/VictoriaMetrics that referenced this issue Mar 11, 2025
Solves the following error:
  verifying github.com/envoyproxy/go-control-plane/envoy@v1.32.3/go.mod: checksum mismatch

See envoyproxy/go-control-plane#1083

Signed-off-by: hagen1778 <roman@victoriametrics.com>
f41gh7 pushed a commit to VictoriaMetrics/VictoriaMetrics that referenced this issue Mar 11, 2025
Solves the following error:
verifying github.com/envoyproxy/go-control-plane/envoy@v1.32.3/go.mod:
checksum mismatch

See envoyproxy/go-control-plane#1083
hagen1778 added a commit to VictoriaMetrics/VictoriaMetrics that referenced this issue Mar 11, 2025

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
Solves the following error:
verifying github.com/envoyproxy/go-control-plane/envoy@v1.32.3/go.mod:
checksum mismatch

See envoyproxy/go-control-plane#1083

(cherry picked from commit 18d6c71)
Copy link

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label Mar 15, 2025
@nickheyer
Copy link

This is not stale, nor is it resolved. This is an issue that affects a multitude of packages that depend on this

@github-actions github-actions bot removed the stale label Mar 21, 2025
@nickheyer
Copy link

Still broken, google must be independently caching their dependencies because the internet hasn't fallen apart. That being said, the fact that this still hasn't been resolved is kind of insane given the scale of the issue.

@valerian-roche
Copy link
Contributor

Can you clarify what resolution you are expecting? We released a new version and google released new versions to remove references to the corrupted one. We do not have control over the caching proxies which are creating the issue potentially, and therefore do not have a path forward that I know of. Excluding the bad module version was used by some users with success I believe, and we have not received other reports of issues since the new releases of the modules depending on the bad release.

@LeeBrotherston
Copy link

@valerian-roche I think that the problem is that the new version had the same version number as the broken one, and so everyone is getting mismatched sums depending on which one they have, and having the same version number will not cause go to update anything.

I would suggest that the new fix is releasing a new version in the go sense, i.e. updating the version number (v.1.32.4) of the release so that go mod sees this as a new version and uses that, instead of continually trying to fetch one of two versions of v.1.32.3

@richdawe-cio
Copy link

There is actually a new tag already, see envoy/v1.32.4.

For reference, we worked around this issue by adding this to go.mod:

// v1.32.3 of this module causes checksum failures. See:
// https://github.com/googleapis/google-cloud-go/pull/11591
// https://github.com/envoyproxy/go-control-plane/issues/1083
exclude github.com/envoyproxy/go-control-plane/envoy v1.32.3

When I looked, some of the Google Cloud libraries had new tags that used the new envoy/v1.32.4 tag, others not.

@LeeBrotherston
Copy link

Ah I see, I must confess I didn't notice because the latest release is still showing as v.1.32.3, and I don't think that the tag is enough for go get or go get -u to update anything, doesn't it need to be a release?

@mmorel-35 mmorel-35 pinned this issue Mar 26, 2025
@mmorel-35 mmorel-35 unpinned this issue Mar 26, 2025
@richdawe-cio
Copy link

richdawe-cio commented Mar 28, 2025

@LeeBrotherston wrote:

Ah I see, I must confess I didn't notice because the latest release is still showing as v.1.32.3, and I don't think that the tag is enough for go get or go get -u to update anything, doesn't it need to be a release?

As far as I can tell, the tag seems to be enough on my system.

go get github.com/envoyproxy/go-control-plane/envoy gets v1.32.4. I was also able to go get -u from v1.32.2 to v1.32.4.

I could do both of those with a brand new go project. go mod init github.com/richdawe-cio/issue1083 followed by go get commands.

@LeeBrotherston
Copy link

@richdawe-cio my mistake, apologies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests