Add dynamodb multikey kv #5026
Merged
Add dynamodb multikey kv #5026
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
danielblando
force-pushed
the
ddb-kv
branch
2 times, most recently
from
7261cbe
to
5be318f
Compare
friedrichg
reviewed
Dec 13, 2022
wenxu1024
reviewed
Dec 17, 2022
wenxu1024
reviewed
Dec 17, 2022
wenxu1024
reviewed
Dec 17, 2022
wenxu1024
reviewed
Dec 19, 2022
danielblando
force-pushed
the
ddb-kv
branch
3 times, most recently
from
5e50626
to
d5d7b83
Compare
yeya24
reviewed
Dec 21, 2022
danielblando
force-pushed
the
ddb-kv
branch
5 times, most recently
from
5ea682f
to
57deeb7
Compare
friedrichg
reviewed
Dec 27, 2022
|
@friedrichg |
yeya24
reviewed
Jan 11, 2023
| level.Error(c.logger).Log("msg", "error CASing", "key", key, "err", err) | ||
| continue | ||
| } | ||
| } |
There was a problem hiding this comment.
I am a little bit worried about the multiple put and delete calls to DDB here. Can we use https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html API here to reduce this to only 1 API call?
There was a problem hiding this comment.
that makes sense. I am not sure why i didnt see this api from the beginning. I will work on changing this. Thanks
yeya24
reviewed
Jan 11, 2023
alanprot
approved these changes
Jan 12, 2023
danielblando
force-pushed
the
ddb-kv
branch
2 times, most recently
from
690176d
to
f7d4550
Compare
danielblando
force-pushed
the
ddb-kv
branch
2 times, most recently
from
01f29f4
to
256a6cc
Compare
Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com>
Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com>
Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com>
danielblando
force-pushed
the
ddb-kv
branch
3 times, most recently
from
77bae08
to
79be362
Compare
yeya24
reviewed
Jan 24, 2023
Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com>
danielblando
force-pushed
the
ddb-kv
branch
2 times, most recently
from
2e5e480
to
96d2448
Compare
Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com>
yeya24
approved these changes
Jan 24, 2023
Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com>
alexqyle
pushed a commit
to alexqyle/cortex
that referenced
this pull request
May 2, 2023
* Add dynamodb multikey kv Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com> * Reorganize import Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com> * Rework test Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com> * Change ddb CAS to use batch Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com> * Use slice to batch writeRequests Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com> * Change slice usage Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com> Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Daniel Deluiggi ddeluigg@amazon.com
What this PR does:
This CR introduce the idea of multikey for kv on the ring. The proposal was made at #4832
It also add a new kv store using dynamodb which uses the multikey implementation. The new kv split the ring by InstanceDesc making each instance be a key in the ring. The new kv also introduce the concept of stale data and ttl.
TTL is an optional time to live of any data not update for x amount of time. This helps cleaning the ring when a pod leaves without gracefully shutdown.
Stale data is used when we have a problem communicating to the kv store, in this case dynamodb. The stale is returned and we use the last time we could sync the data to check if a instance is unhealthy or not.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]