New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Shard placement table persistence #18283
Conversation
/ci-repeat |
new failures in https://buildkite.com/redpanda/redpanda/builds/48784#018f536f-edc0-402f-9e52-2ca50e32f1a1:
new failures in https://buildkite.com/redpanda/redpanda/builds/48784#018f536f-edc7-45f1-a473-165a98b13ce2:
new failures in https://buildkite.com/redpanda/redpanda/builds/48784#018f536f-edc2-43ef-9139-a303bdda8783:
new failures in https://buildkite.com/redpanda/redpanda/builds/48784#018f536f-edc5-40f5-ad6c-624808f78f3c:
new failures in https://buildkite.com/redpanda/redpanda/builds/48784#018f5377-280d-4c80-bb7a-d2bb9e5ddb04:
new failures in https://buildkite.com/redpanda/redpanda/builds/48784#018f5377-2807-4819-b098-183bc81b69b5:
new failures in https://buildkite.com/redpanda/redpanda/builds/48784#018f5377-2804-42a2-9dde-2ffdbf6632a8:
new failures in https://buildkite.com/redpanda/redpanda/builds/48784#018f5377-280a-4e25-9bbd-469cdfc10350:
new failures in https://buildkite.com/redpanda/redpanda/builds/48797#018f5531-11b1-411c-b5f0-fa728febde49:
new failures in https://buildkite.com/redpanda/redpanda/builds/48797#018f5531-11b4-4a59-a548-e8a22eabd741:
new failures in https://buildkite.com/redpanda/redpanda/builds/48797#018f5538-91ba-481f-a721-ad4d2a0382d9:
new failures in https://buildkite.com/redpanda/redpanda/builds/48797#018f5538-91b8-463c-b80f-c0ab8e17b0a1:
|
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/48784#018f5377-2807-4819-b098-183bc81b69b5 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/49101#018f777c-3dc7-4eaa-a252-0b3d846c7341 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/49101#018f777c-3dc4-4737-9422-9838579fbad7 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/49101#018f780d-974c-47ff-afa0-905c06a31a3d |
5341afa
to
88610d9
Compare
/ci-repeat |
5dcf117
to
ce4a1d8
Compare
ce4a1d8
to
611e589
Compare
611e589
to
9479497
Compare
We track group ids in partition assignments and current state because we want to use them as kvstore keys - group id is just a number so it is better in this regard than the ntp and it uniquely identifies the current incarnation of the ntp (i.e. each group id is uniquely mapped to ntp, but ntp can be mapped to several group ids, though only one will be current).
Make different components log with different prefixes in test to more easily disambiguate their messages.
This flag will signal nodes that they can switch to persisting their shard_placement_table locally in the kvstore.
e1c8fa2
to
faa76fd
Compare
Had to rebase due to kvstore changes in dev, actual changes are |
Add persistence to
shard_placement_table
. Now all shard placement updates are persisted in kvstore and restored at startup. Also implement migrating from old topic table-determined placement and update the stress test with migrations, restarts and post-restart checks.At this point shard assignments are still ultimately determined by
topic_table
, but we'll already be able to better track ongoing x-shard transfers after the restart. Also, everything is set for transitioning to node-local placement decisions.Backports Required
Release Notes