Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle housekeeping with sharded cluster #505

Open
hackerwins opened this issue Apr 5, 2023 · 2 comments · May be fixed by #529
Open

Handle housekeeping with sharded cluster #505

hackerwins opened this issue Apr 5, 2023 · 2 comments · May be fixed by #529
Labels
enhancement 🌟 New feature or request

Comments

@hackerwins
Copy link
Member

hackerwins commented Apr 5, 2023

What would you like to be added:

We recently introduced cluster mode based on the shard key. This shard key is used to assign APIs requested by clients to servers in the cluster based on project or document.

Housekeeping operations should also apply this sharded cluster structure. We can think of two options:

  1. Only the master server executes housekeeping in the cluster
  2. Split housekeeping tasks per project and execute them on servers mapped by project

Why is this needed:

  • Bugs may occur in housekeeping of sharded cluster.
@hackerwins hackerwins added the enhancement 🌟 New feature or request label Apr 5, 2023
@krapie
Copy link
Member

krapie commented Apr 5, 2023

The options that you've suggested look good to me.
I have listed some of the considerations for these options below.

  1. Only the master server executes housekeeping in the cluster.
  1. Split housekeeping tasks per project and execute them on servers mapped by project.

These options seem like opposite options to me, but I think they both can be implemented by using etcd.
I'll keep searching for more information on these options.

@krapie
Copy link
Member

krapie commented May 2, 2023

I have implemented leader election with K8s lease object, but this will make yorkie k8s dependent.

Therefore, I will change this implementation with MongoDB, where yorkie already have dependency on.

To be specific about the implementation, I will use the way of nodes try to update on the same document(same _id) by checking lease_expire_at field.

Also, to ensure that only one node becomes the leader when a lease has expired, I will use TTL indexes to remove document on expiry, preventing nodes to update the same document simultaneously.

@hackerwins hackerwins changed the title Handle housekeeping with shared clusters Handle housekeeping with sharded cluster May 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement 🌟 New feature or request
Projects
Status: Todo
2 participants