Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't lose ETCD watch updates during cluster inaccessibility #11

Open
fit51 opened this issue Jan 29, 2020 · 0 comments
Open

Don't lose ETCD watch updates during cluster inaccessibility #11

fit51 opened this issue Jan 29, 2020 · 0 comments
Labels
enhancement New feature or request

Comments

@fit51
Copy link
Owner

fit51 commented Jan 29, 2020

Current implementation - loads all key-value pairs during initialization, subscribes on changes and this is it.
For better fault tolerance I propose to add storage update activity.
This activity would periodically load all key-value data since last revision, update storage TrieMap send changes events if it finds stale data.

Possible scenario

  • Node loses network connection with ETCD cluser. WatchService starts to reconnect.
  • User update keys 1 and 2 on ETCD.
  • Network connection is restored. Node succeeds to resubscribe on Watch. Now it would receive new changes, but it has lost 1 and 2 keys update.

Problems to solve in this proposal

  • Possible update duplicates. One could come from watch stream, another - from storage update activity.
@fit51 fit51 added the enhancement New feature or request label Jan 29, 2020
@fit51 fit51 changed the title Add periodic storage overload in EtcdStorage Add periodic storage reload in EtcdStorage Jan 29, 2020
@fit51 fit51 changed the title Add periodic storage reload in EtcdStorage Don't lose ETCD watch updates during cluster inaccessibility Jan 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant