Segment Load Queue does not refresh for individual historicals untill all loadqueues have finished loading segments

[aruraghuwanshi]

Affected Version

Version 28.0.1
(Unsure if it exists in earlier versions)

Description

Coordinator leader does not assign new Segment Load Queues until all historicals have finished ingesting the segments from their previous assignments.

Scenario:

• In a cluster size of M historicals, a large volume of segments need to be decommissioned off of n historicals, by updating the decommission list in the coordinator config.
• After the decommission list is updated with the host:port numbers, the coordinator then assigns load queues to all the other remaining M-n historicals.
• The first historical finishes loading its segment queue in time t, while the historical that finishes last finishes loading in time T.
• All historicals (including the ones that have already finished their segment loading) do not get new segment load queue assignments until the last historical has finished loading the segments in its queue.
• The issue comes up when the segment load queues are so large on the pending historicals that the historicals that have completed their segment loading will have to wait for a considerably long time ( T-t ), before getting new segment load queues.

In the following example, this time exceeded 2+ hours, for the last remaining historical finished loading all segments in its loading queue. Only after that's completed, did the coordinator start assigning new segment load queues to the the waiting historicals.

Services tab:

decommission list:

Coordinator dynamic config: