-
Notifications
You must be signed in to change notification settings - Fork 575
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Service for read-write instance #3722
Comments
Hi @Agalin
This is an interesting idea. It sounds like you have everything you need to try it out. If it works well, we can consider automating it here in the operator.
It sounds like the first step is manual, or you've automated it already. If so, in that same step (or separately), you can update the selector of the service you're proposing. Maybe the following?
Let us know in the discord how it goes! |
First step (failover to another dc) is expected to be manual. I don't think it would be safe to automate without a PGO being multicluster-aware. Yeah, it should be possible to test by creating a service in each DC which selects nothing everywhere besides the current read-write cluster (where it selects primary) and patch it together with the cluster transitioning from standby. Now that I think about it - maybe it's doable even in a single action. Create a service which selects the local primary + a custom label configured through the postgrescluster object. Only configure this label on the read-write cluster. Then it can be changed together with standby, in a single patch operation. |
Hey @Agalin, do you have any updates on this? |
Ah sorry, was busy with various things and forgot to update this topic. 😞 I've been able to create a described setup - separate service that takes only primary node from a read-write cluster with read-write label being assigned manually. I have this service deployed in each cluster and replicated between them using Linkerd with failover rules configured so if a cluster dies it's just a single PostgresCluster resource update in a single k8s cluster (set standby to false and add that custom label) to perform a failover to a standby in all k8s clusters at at once. |
Hello @Agalin. Thank you for the update. I'm curious if, with your current setup, you still believe the additional managed service would be beneficial. There is obviously a bit of overhead with each service we manage, so if this can be accomplished as needed using existing labels, etc, that may be the best option, but I want to make sure I understand the use case as well as possible. |
I understand your concerns and believe that service isn't necessary. Neither is a new label although either that or changing From my perspective all building blocks are there. Which means a sufficient solution would be to document it. If you find this scenario not particularly useful for broader user base then I won't oppose closing this issue. |
@Agalin thank you for the additional details. After thinking a bit more through your scenario, I've added an item to are backlog for consideration. |
Overview
There should be one more service created for a cluster - one pointing only to a writeable primary. It should not point to any pod if cluster is currently in the standby mode.
Use Case
I have a service mesh (linkerd) configured in a way that makes it possible to replicate (mirror) services cross-cluster and use them as local. It's an alternative to a loadbalancer or nodeport. But in case of a primary cluster failure failover is still a multistep process.
standby
).Linkerd (and probably other services) makes it possible to automatically failover a service in case of the main one being unreachable. It can route all traffic to the new mirrored service. But it works in such a way that traffic is split equally between all secondary services. Which means with current setup you can only put a single failover cluster in the secondary list. Otherwise when primary fails traffic gets split equally to all clusters even if nodes are read-only.
With a service pointing only to a writeable node failover would look like this:
standby
).Desired Behavior
To the list of services managed by the operator:
add a new one (
db-ha-rw
,db-primary-writeable
, whatever) - or even just attach a label to the current writeable primary that it's writeable - then user can create such a service manually.Environment
Tell us about your environment:
Please provide the following details:
Kubernetes
The text was updated successfully, but these errors were encountered: