Need a way to distribute ingestion task replicas across tiers

[m-ghazanfar]

Description

Using the tieredReplicants field in the loading rules I can place one replica of a segment on one tier and another replica of the segment on another tier. Then, I can have my tiers on different CSP regions or AZs to get better fault tolerance.

However, this feature is only available for historicals. AFAIK, there isn't a way to spread replicas across tiers for ingestion tasks - all replicas have to be on the same tier.

See the replicas field under the ioConfig section of the supervisor spec. If we could have something like tieredReplicants from the loading rules in the ioConfig to allow placing ingestion task replicas on different tiers, that would be awesome.

This will help replicated kafka ingestion tasks for use-cases which are very sensitive to realtime data.