Add rack aware round-robin host policy

[horkhe]

Summary

When Cassandra is provisioned in a cloud environment, in such a way that nodes are spread across multiple availability zones, it is more const effective to query nodes in the same availability zone. Currently gocql provides DC-aware round-robin host policy, but it does not respect availability zone. This PR introduces rack-aware round-robin policy that prefers nodes in the same rack, if there are no nodes in the same rack then nodes from the same datacenter are preferred.

Usage Guidelines

• When Cassandra is provisioned in AWS with Ec2Snitch, it converts availability zone to datacenter/rack pair breaking it in half on the last dash, so availability zone eu-central-1c becomes datacenter eu-central and rack 1c;
• When Cassandra is provisioned in GCP with GoogleCloudSnitch, it converts availability zone to datacenter/rack pair breaking it in half on the last dash, so availability zone us-central1-a becomes datacenter us-central1 and rack a;

For maximum efficiency we recommend using this policy as a fallback for token aware host policy as follows:

cluster := gocql.NewCluster(servers)
cluster.PoolConfig.HostSelectionPolicy = gocql.TokenAwareHostPolicy(
    gocql.RackAwareRRHostPolicy(datacenter, rack), gocql.NonLocalReplicasFallback())

Note that the token-aware policy uses the rack-aware policy to determine if a host is local. Only hosts from the same rack are considered to be local by the rack-aware policy. Non-local replica fallback is enabled to ensure that the token-aware policy still connects to a host that has a replica, even when there are no replicas in the same rack, otherwise decision making would have been handed over to the rack-aware policy that would connect to an arbitrary host from the same datacenter.

[martin-sucha]

Thank you for the pull request!

I'm not sure whether we should add every possible policy to gocql. That said, it's not really possible to test policies that live out of tree, so we could probably add this one. But even better would be to change gocql so that external policies could easily be testable.

With the RackAwareRRHostPolicy used as a fallback in TokenAwareHostPolicy, we first select replica from local rack and then fall back to replica from other nodes regardless of whether the replica is in local or remote datacenter. Shouldn't we prefer replicas in order local rack, local DC, remote DC similar to when the policy is used without the TokenAwareHostPolicy?

[martin-sucha]

Simple q.spec = NonSpeculativeExecution{} should work here, right?

[horkhe]

It does work here, I just wanted to avoid possible allocation. Not sure if the compiler optimises it out.

[horkhe]

With the RackAwareRRHostPolicy used as a fallback in TokenAwareHostPolicy, we first select replica from local rack and then fall back to replica from other nodes regardless of whether the replica is in local or remote datacenter. Shouldn't we prefer replicas in order local rack, local DC, remote DC similar to when the policy is used without the TokenAwareHostPolicy?

That would be preferable, however interface HostSelectionPolicy defines only IsLocal() bool method that does not allow splitting hosts into 3 categories. It would be better to replace IsLocal with two methods IsRackLocal and IsDatacenterLocal, but that would be an API breaking change that I did not want to do.

[martin-sucha]

That would be preferable, however interface HostSelectionPolicy defines only IsLocal() bool method that does not allow splitting hosts into 3 categories. It would be better to replace IsLocal with two methods IsRackLocal and IsDatacenterLocal, but that would be an API breaking change that I did not want to do.

I'm still thinking about this. Technically, we could add an interface like

type IsRackLocaler interface {
	// IsRackLocal returns true if the host is rack local.
	IsRackLocal(host *HostInfo) bool
}

that the RackAwareRRHostPolicy could implement (and the token aware policy could use a type assertion to check for the method). But this looks like too specific to me to have in the token aware policy. Perhaps we should have an interface like

type HostDistancer interface {
	// HostDistance returns an integer specifying how far a host is from the client.
	// The value is used to prioritize closer hosts during host selection.
	// For example this could be:
	// 0 - local rack, 1 - local DC, 2 - remote DC
	// or:
	// 0 - local DC, 2 - remote DC
	HostDistance(host *HostInfo) uint
}

and the token aware policy could group by the distance?