Add Leader Election for Housekeeping in Sharded Cluster using MongoDB

[krapie]

What this PR does / why we need it:

Add leader election for housekeeping in sharded cluster using mongodb.

• server/backend/election package is added for leader election, with database implementation.
• database implementation uses mongoDB and TTL indexes to ensure that only one node can acquire housekeeping leader lease by preventing other nodes try to update the same document simultaneously on document expiry.
• yorkie-cluster Helm Chart is configured to enable --housekeeping-leader-election to perform leader-only housekeeping in sharded cluster.
• Also, small error in hostname value in server/backend/backend.go is fixed. For more information, follow: Add user agent metrics PR

Which issue(s) this PR fixes:

Fixes #505

Special notes for your reviewer:

I have tested on K8s environment with yorkie-cluster Helm chart with minikube, and it worked as expected.

Does this PR introduce a user-facing change?:

Additional documentation:

Checklist:

• Added relevant tests or not required
• Didn't break anything

[codecov]

Codecov Report

Merging #529 (23e9f1d) into main (2b15ac5) will increase coverage by 0.20%.
The diff coverage is 62.18%.

@@            Coverage Diff             @@
##             main     #529      +/-   ##
==========================================
+ Coverage   51.50%   51.70%   +0.20%     
==========================================
  Files          67       68       +1     
  Lines        6932     7048     +116     
==========================================
+ Hits         3570     3644      +74     
- Misses       2891     2930      +39     
- Partials      471      474       +3     

Impacted Files	Coverage Δ
server/backend/backend.go	0.00% <0.00%> (ø)
server/backend/config.go	55.55% <ø> (ø)
server/backend/database/memory/database.go	52.61% <0.00%> (-0.55%)	⬇️
server/backend/database/mongo/indexes.go	50.00% <ø> (ø)
server/backend/database/mongo/client.go	40.81% <62.26%> (+1.45%)	⬆️
server/backend/election/mongo/election.go	82.97% <82.97%> (ø)
server/config.go	45.23% <100.00%> (+1.33%)	⬆️

[hackerwins]

Thanks for your contribution.
I left a simple question. 🙏

[hackerwins]

Thank you for the explanation.

Could you please add the tests for the two situations I asked about?

1. Lease renewal while handling a long task.
2. Handling background routines when shutting down the server.

[krapie]

I personally think we can also use K8s CronJob Object to run housekeeping in cluster mode without using leader election for housekeeping.

Using K8s CronJob will ease overhead for the server cluster to process leader elections. This might have a big impact when you have a huge cluster of servers.

[krapie]

I forgot to re-request review on this PR 😄 Requesting it now.

[hackerwins]

Thanks for applying my requests.

I personally think we can also use K8s CronJob Object to run housekeeping in cluster mode without using leader election for housekeeping.

Using K8s CronJob will ease overhead for the server cluster to process leader elections. This might have a big impact when you have a huge cluster of servers.

Using CronJob seems to be a good and straightforward approach. If we choose to use it, we can run the cluster without a leader election.

However, we should consider how to provide private APIs that are meant to be exclusively called by internal components and should not be exposed publicly. The proposed idea doesn't seem to depend on k8s, as we only need to expose the private APIs.

https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#concurrency-policy

How would you prefer to proceed?

[krapie]

@hackerwins At first glance, I thought we could introduce some sort of housekeeping-only mode on the server or CLI and run it periodically using K8s CronJob (or maybe just run a housekeeping-only server pod) since housekeeping does not require any context to receive or provide (housekeeping fetches all projects and clients from the database, and results for housekeeping only get applied to the database).

I will search for more context and information to get more ideas.