client_golang occassionally complains about superfluous response.WriteHeader calls

[GiedriusS]

This is printed from time to time in Thanos Query:

Apr 26 10:40:39 foo docker[8275]: 2019/04/26 08:40:39 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.httpError (http.go:306)

And it seems like our logger isn't hooked up there. This error should be fixed (client_golang issue 👁️ ?) and our logger should be used there too.

Version: master-2019-04-25-88544fc

[transient1]

This was the first hit when I Googled for this error message so I'm going to add here; hope that's cool. I'm running kbudde/rabbitmq-exporter. Today in my Grafana dashboard I noticed 2 gaps in metrics a minute apart, and looking in the logs for the exporter I see the same error as reported above occcuring at the gap marks:

2019-07-25T14:29:50.984499897Z 2019/07/25 14:29:50 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.httpError (http.go:306)
2019-07-25T14:30:47.083116154Z time="2019-07-25T14:30:47Z" level=info msg="Metrics updated" duration=17.006736909s
2019-07-25T14:31:49.433846886Z time="2019-07-25T14:31:49Z" level=info msg="Metrics updated" duration=19.340816118s
2019-07-25T14:31:50.078055124Z 2019/07/25 14:31:50 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.httpError (http.go:306)

I am not running Thanos though, just a single Prometheus instance running version 2.7.1.

[GiedriusS]

Fixed with the 1.2.0 version of client_golang.

[GiedriusS]

Seems like the same still happens with v0.10.0-rc.0. Need to reinvestigate.

[bwplotka]

The same thing might happen here: prometheus/prometheus#6139

[chjohnst]

I am seeing these errors when running a combination of Prometheus v2.13.1 and Thanos v0.8.1. Using a query of max(node_cpu_seconds_total) I can recreate pretty quickly and no data is being returned.

{"caller":"http: superfluous response.WriteHeader call from github.com/opentracing-contrib/go-stdlib/nethttp.(*statusCodeTracker).WriteHeader (status-code-tracker.go:19","component":"web","level":"error","msg":")","ts":"2020-01-15T20:56:06.795Z"}
{"caller":"http: superfluous response.WriteHeader call from github.com/opentracing-contrib/go-stdlib/nethttp.(*statusCodeTracker).WriteHeader (status-code-tracker.go:19","component":"web","level":"error","msg":")","ts":"2020-01-15T20:56:06.880Z"}
{"caller":"http: superfluous response.WriteHeader call from github.com/opentracing-contrib/go-stdlib/nethttp.(*statusCodeTracker).WriteHeader (status-code-tracker.go:19","component":"web","level":"error","msg":")","ts":"2020-01-15T20:56:09.170Z"}
{"caller":"http: superfluous response.WriteHeader call from github.com/opentracing-contrib/go-stdlib/nethttp.(*statusCodeTracker).WriteHeader (status-code-tracker.go:19","component":"web","level":"error","msg":")","ts":"2020-01-15T20:56:09.184Z"}

[marcusteixeira]

I had the same problem . It is a case similar to @chjohnst, running the same version of prometheus and thanos. It can occur when larger consultations are carried out, with a time interval of 30 days or with many values.

[stale]

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

[GiedriusS]

Still happening on v0.11.0.

[bwplotka]

Yes, we need to update client_golang - another reason of this was fixed there.

…

On Tue, 17 Mar 2020 at 09:52, Giedrius Statkevičius < ***@***.***> wrote: Still happening on v0.11.0. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1086 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVA3OZU2JQW7JKSQ2OJFLLRH5B6TANCNFSM4HITVLIA> .

[stale]

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

[GiedriusS]

Still valid on 0.12.0.

[dinesh4747]

@bwplotka - Any fix for this, we are seeing this potentially across the versions and none of them helped, Do we need to change to specific go_client version to fix the same.. Any pointers would ideally help

[dinesh4747]

Exceptions we see as below

May 07 14:52:31 prometheusha-1-stg-v000-tt4v prometheus[3786]: level=error ts=2020-05-07T14:52:31.081Z caller=stdlib.go:89 component=web caller="http: superfluous response.WriteHeader call from github.com/opentracing-contrib/go-stdlib/nethttp.(*statusCodeTracker).WriteHeader (status-code-tracker.go:19" msg=)
May 07 14:59:49 prometheusha-1-stg-v000-tt4v prometheus[3786]: level=warn ts=2020-05-07T14:59:49.986Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="/app/discovery/kubernetes/kubernetes.go:361: watch of *v1.Endpoints ended with: too old resource version: 498268073 (498277770)"
May 07 15:06:12 prometheusha-1-stg-v000-tt4v prometheus[3786]: level=info ts=2020-05-07T15:06:12.276Z caller=compact.go:495 component=tsdb msg="write block" mint=1588852800000 maxt=1588860000000 ulid=01E7QTD1CAYQB3CPRG907BHR0E duration=6m12.265832204s
May 07 15:06:29 prometheusha-1-stg-v000-tt4v prometheus[3786]: level=info ts=2020-05-07T15:06:29.929Z caller=head.go:662 component=tsdb msg="head GC completed" duration=12.940377064s
May 07 15:09:00 prometheusha-1-stg-v000-tt4v prometheus[3786]: level=info ts=2020-05-07T15:09:00.083Z caller=head.go:732 component=tsdb msg="WAL checkpoint complete" first=1273 last=1433 duration=2m30.153959595s
May 07 15:16:14 prometheusha-1-stg-v000-tt4v prometheus[3786]: level=warn ts=2020-05-07T15:16:14.995Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="/app/discovery/kubernetes/kubernetes.go:361: watch of *v1.Endpoints ended with: too old resource version: 498310942 (498312556)"

[dinesh4747]

As part of the investigation we have forked the latest stable build of Thanos v0.12.2 and upgraded the client_golang version from v1.5.0 to v1.5.1 presuming it might be the fix but none of them helped where started observing the similar issues

From the analysis it was evident that panic errors "superfluous response" originates only when there is a query with large cardinality

• To be specific In our case as we are operating with istio-mesh component and using Kiali for service graph

• we are pointing the Thanos querier as a datasource via grafana ton kiali any queries with more namespaces and adhoc queries with more than 1 week queries resulting into this panic errors and subsequently prometheus CPU start going high till ~70% and then it dies.

• Do we have need to add specific configs/tweaks from Thanos Queries to fix this?

• IMHO Adding Memcached can easen a bit, But it cannot be a permanent fix

Any specific directions/pointers are highly appreciable!!!

cc: @bwplotka @GiedriusS @brian-brazil

[bwplotka]

Let's discuss here maybe? prometheus/prometheus#6139

[stale]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[bwplotka]

Any update in terms of symptoms with v0.13.0-rc.1 for example? (:

…

On Wed, 10 Jun 2020 at 13:36, stale[bot] ***@***.***> wrote: Hello 👋 Looks like there was no activity on this issue for last 30 days. *Do you mind updating us on the status?* Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command <https://probot.github.io/apps/reminders/> if you wish to be reminded at some point in future. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1086 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVA3OZD4BNJPS5SGMDDKVLRV546DANCNFSM4HITVLIA> .

[stale]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗