New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Race condition in v1.12.2 #1058
Comments
Thanks for reporting. We will have look at it soon. |
@kakkoyun FYI, with a quick bisect on the release-1.12 branch since v1.11.0 (which was fine), this race seems to have been originally introduced by 22da949 (#955) and was then fixed by various race condition improvements that were included in v1.12.1 (which was then passing). Up until d498b3c the branch was still passing without issue, but surprisingly the race condition was occurring again from 7eb9d11 onward and was hence included in v1.12.2 again The same issue is still present on main (currently 4ad265f) and was introduced by 11ee9ad (the same commit as 7eb9d11, just on main rather than release-1.12 branch) |
@dnwe Thanks for the detailed investigation. I haven't had much time to focus on this. I'll do my best. However, help is always welcome. If anyone has a clear fix, please go for it. |
The read is of a package-local variable The previous write is at https://github.com/golang/go/blob/go1.18.2/src/runtime/metrics.go#L67 As far as I can see, a semaphore prevents both of these running concurrently, so I wonder if the Go race-detection library is throwing a false positive? |
Thanks, @bboreham and @cristiangreco verifying this. Closing it now. |
After upgrading to github.com/prometheus/client_golang v1.12.2 I can consistently observe a race condition in our tests.
This issue does not happen with the previous versions.
I am using:
go version go1.18.2 linux/amd64
Linux **** 5.4.0-113-generic #127-Ubuntu SMP Wed May 18 14:30:56 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
The errors can be reproduced by using the open source library:
https://github.com/nexmoinc/gosrvlib
And running the metrics tests after upgrading to prometheus v1.12.2:
gotest -race -failfast -v github.com/nexmoinc/gosrvlib/pkg/metrics/prometheus
The results are:
The text was updated successfully, but these errors were encountered: