You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When thanos sidecar starts up and there isn't a running Prometheus behind it, sidecar reports as ready.
What you expected to happen:
When sidecar starts up, if there isn't a running Prometheus behind it then sidecar should report as not ready.
How to reproduce it (as minimally and precisely as possible):
Ensure Prometheus is not running
Start thanos sidecar
curl sidecar ready endpoint
Validate response
Full logs to relevant components:
Logs from newly started sidecar when Prometheus is not running
➜ thanos git:(main) ✗ ./build/thanos sidecar
ts=2024-04-11T17:15:47.939758Z caller=sidecar.go:139 level=info msg="no supported bucket was configured, uploads will be disabled"
ts=2024-04-11T17:15:47.939921Z caller=options.go:26 level=info protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
ts=2024-04-11T17:15:47.940187Z caller=sidecar.go:385 level=info msg="starting sidecar"
ts=2024-04-11T17:15:47.940434Z caller=reloader.go:262 level=info component=reloader msg="nothing to be watched"
ts=2024-04-11T17:15:47.940488Z caller=intrumentation.go:56 level=info msg="changing probe status" status=ready
ts=2024-04-11T17:15:47.940637Z caller=intrumentation.go:75 level=info msg="changing probe status" status=healthy
ts=2024-04-11T17:15:47.940647Z caller=http.go:73 level=info service=http/server component=sidecar msg="listening for requests and metrics" address=0.0.0.0:10902
ts=2024-04-11T17:15:47.941Z caller=grpc.go:131 level=info service=gRPC/server component=sidecar msg="listening for serving gRPC" address=0.0.0.0:10901
ts=2024-04-11T17:15:47.941095Z caller=tls_config.go:313 level=info service=http/server component=sidecar msg="Listening on" address=[::]:10902
ts=2024-04-11T17:15:47.941138Z caller=tls_config.go:316 level=info service=http/server component=sidecar msg="TLS is disabled." http2=false address=[::]:10902
ts=2024-04-11T17:15:47.941577Z caller=sidecar.go:188 level=warn msg="failed to fetch prometheus version. Is Prometheus running? Retrying" err="perform GET request against http://localhost:9090/api/v1/status/buildinfo: Get \"http://localhost:9090/api/v1/status/buildinfo\": dial tcp [::1]:9090: connect: connection refused"
ts=2024-04-11T17:15:49.944287Z caller=sidecar.go:188 level=warn msg="failed to fetch prometheus version. Is Prometheus running? Retrying" err="perform GET request against http://localhost:9090/api/v1/status/buildinfo: Get \"http://localhost:9090/api/v1/status/buildinfo\": dial tcp [::1]:9090: connect: connection refused"
ts=2024-04-11T17:15:51.94443Z caller=sidecar.go:188 level=warn msg="failed to fetch prometheus version. Is Prometheus running? Retrying" err="perform GET request against http://localhost:9090/api/v1/status/buildinfo: Get \"http://localhost:9090/api/v1/status/buildinfo\": dial tcp [::1]:9090: connect: connection refused"
➜ ~ curl localhost:10902/-/ready
OK%
Anything else we need to know:
It seems this behaves as expected when at one point there is a running Prometheus and sidecar can enter its heartbeat loop (transitioning between ready/not ready depending on the state of Prometheus). In the case however where there is never a ready Prometheus, sidecar never enters its heartbeat loop. Instead it continuously tries to fetch the prometheus version and doesn't update its ready state, defaulting to true/ready.
Have raised a PR with a potential fix for this: #7291
The text was updated successfully, but these errors were encountered:
Thanos, Prometheus and Golang version used:
Object Storage Provider:
What happened:
When thanos sidecar starts up and there isn't a running Prometheus behind it, sidecar reports as
ready
.What you expected to happen:
When sidecar starts up, if there isn't a running Prometheus behind it then sidecar should report as
not ready
.How to reproduce it (as minimally and precisely as possible):
Full logs to relevant components:
Logs from newly started sidecar when Prometheus is not running
Anything else we need to know:
It seems this behaves as expected when at one point there is a running Prometheus and sidecar can enter its heartbeat loop (transitioning between ready/not ready depending on the state of Prometheus). In the case however where there is never a ready Prometheus, sidecar never enters its heartbeat loop. Instead it continuously tries to fetch the prometheus version and doesn't update its ready state, defaulting to true/ready.
Have raised a PR with a potential fix for this: #7291
The text was updated successfully, but these errors were encountered: