Speculative execution returning error to user when all replicas are taken by other executions

[racevedoo]

What version of Cassandra are you using?

Scylla 5.1.12

What version of Gocql are you using?

Latest master

What version of Go are you using?

1.20

What did you do?

In a 2 node cluster on a keyspace with replication factor = 2, I'm running queries with speculative execution and consistency = one while one of the nodes goes down.

What did you expect to see?

The query execution should finish successfully, since there is a node available and consistency is set to one.

What did you see instead?

When the main query execution takes a little longer and speculative retry is triggered, the speculative retry flow returns a ErrNoConnections and this error is returned to the app.

This is probably because the driver code does not wait for the main execution to return a response (either error or results) and returns the speculative execution's error. The speculative execution cannot get a replica to query because there is only a single one available and it is being used by the main execution.

Being more specific, in this snippet the speculative execution publishes an Iter with err = ErrNoConnections in the results channel.

One possible solution is to wait for the main execution to finish when the speculative execution returns an error (specially something like ErrNoConnections, for other errors it might make sense to forward to the app).

A possibly related issue is #1530

This is easily reproducible by setting TimeoutDelay: 1 in the speculative execution config