Investigate next gen architacture for ecchronos

[tommystendahl]

The current architecture with one ecchronos node for each cassandra node in a cluster doesn't scale well for large clusters. It would be very good if this could be changed so we could use one or two ecchronos nodes for each DC or maybe a small number of ecchronos nodes that handles the the whole cluster.

The goal of this issue would be to propose how a new architecture could work and a plan for how it could be implemented.

Expected outcome:

• a proposal for a new architecture
• a plan with tasks or milestones to implement the new atchitecture

[patrickclee0207]

really curious about this myself. I started trying ecchronos recently and so far having better success than the way I've been using Cassandra-reaper but I've wondered how this would work on larger clusters that I have. Largest I've tested so far is 60 node cluster.

[tommystendahl]

@patrickclee0207 Thanks for showing interest and I'm happy to here that you had some success trying out ecchronos. It works fine for smaller cluster and from our experience I think it still scales ok beyond 60 nodes. But when the clusters grow in size the number of connections each ecchronos node has to the cluster becomes a bottleneck. I hope we can find some good way to redesign ecchronos to address this.