-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Thanos Sidecar - Flush Endpoint #7295
Comments
If Thanos sidecar implements this API, how would you call it from Prometheus operator? Would you still use the |
The idea is invoke this on the scaling down event. |
I am not against adding this functionality into Thanos sidecar.
Prometheus operator has its own sidecar like the |
My only concern about having this on projects like Besides that I think might be a useful solution for people is not using Prometheus Operator as well, but I don't have any data about it to confirm it's just feeling 😅 |
As someone having to orchestrate "flushing" outside of prometheus-operator currently, I would definitely leverage a It'd also be much easier to benefit from incremental progress on this, as I'd be able to remove a lot of my custom glue/wiring in my current setup, instead of having to wait for the changes to propagate all the way to prometheus-operator. |
Once this functionality has been added to the Prometheus operator sidecar, there should be no difference in terms of UX compared to having the API natively? But I think I am okay to add the new API. PR is welcomed. |
There was some idea to run prometheus via libraries from within sidecar; that way we could build this properly. cc @SuperQ |
I've raised a PR that flushes via TSDB snapshotting, but happy to give this a try if you can point me in a general direction. IIUC we could change the implementation of the flush to what you proposed in the future without impacting end users, too. |
Is your proposal related to a problem?
On the Prometheus Operator project, we have implemented the ability to autoscale Prometheus shards when running Prometheus using the agent mode. This might be achieved using the native HPA or any tool like Keda.
For the Prometheus Server, we are still working to make this feature available and a lot of discussion is ongoing yet.
For users using Prometheus Server + Thanos Sidecar, I'd like to provide a graceful shutdown by flushing and uploading blocks before shutdown of the Prometheus Server.
I've added the ability to upload blocks to the blob storage using Thanos Tools, we can leverage this on Prometheus Operator by doing some hacks such as a preStop hook to flush tsdb and upload blocks. As we can see we already have feature requests from users implementing this on their own, as you can see here.
This works as expected but it's a bit hacky and I'd like to propose a new feature to the Thanos Sidecar.
Describe the solution you'd like
A new endpoint on the Thanos Sidecar e.g.
/flush
,/shutdown
, or/snapshot
(naming is hard 😅).This brand-new endpoint should execute the following logic when invoked.
This new endpoint is only providing an easier experience to graceful shutdown Prometheus Server when some of the following operations are required.
By having this new endpoint I envision Prometheus Operator calling it before scaling down the instance, and after this endpoint returns success we can just delete the Statefulset and its Persistent volume.
Describe alternatives you've considered
N/A
Additional context
The text was updated successfully, but these errors were encountered: