Redis Cluster performance not increasing linearly with Lettuce

[bsmanit]

Redis Cluster performance not increasing linearly:

Ran the perf on GCP Cloud:
Machine Type: c3-highmem-4 (vCPU: 4 Memory: 29GB)

Standalone Redis:
SET ops/sec: 243,858 (243K) {benchmarking using memtier_benchmark tool}

5 Node Cluster:
Machine Type: c3-highmem-4 (vCPU: 4 Memory: 29GB)
All nodes are master, no replica.
SET ops/sec: 1,011,661 (1.01M) {benchmarking using memtier_benchmark tool}

Running memtier_benchmark from Two 44 vCPU Machine:
SET ops/sec: 567940*2 = 11,35,880 (1.13M)

On 5 Node cluster expected ops/sec according to linear scale:
243858(standalone Redis SET ops/sec) * 5 = 1,219,290 = 1.2M ops

Here expected ops: 1.2M, Achieved ops: 1.13M (Proves that Redis scales linearly with shards)

When ran perf with lettuce client, received the similar ops with standalone Redis with same latency.
But with 5 node Redis cluster, got max 900K ops/sec with p99: 3ms and p99.9: fluctuating b.w 3ms to 100ms.

Code used to run SET command on 5 node cluster:

public static RedisClusterClient getRedisClusterClient(RedisClusterProperties properties) {
    List<String> nodes = new ArrayList<>(Arrays.asList(properties.getNodes().split(",")));
    List<RedisURI> redisURIList = nodes
            .stream()
            .map(node -> RedisURI.create(node, 6379))
            .collect(Collectors.toList());

    RedisClusterClient clusterClient = RedisClusterClient.create(redisURIList);

    ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
            .enablePeriodicRefresh(1, TimeUnit.HOURS)
            .build();
    clusterClient.setOptions(ClusterClientOptions.builder()
            .maxRedirects(0)
            .topologyRefreshOptions(topologyRefreshOptions)
            .build());
    return clusterClient;
}

    private final StatefulRedisClusterConnection<String, String> connection;

    public Repository(RedisClusterProperties properties) {
        RedisClusterClient redisClusterClient = ConnectionFactory.getRedisClusterClient(properties);
        this.connection = redisClusterClient.connect();
        this.connection.setTimeout(properties.getCommandTimeout());
    }
    
    public void set(String key, String value) {
        connection.sync().set(key, value);
    }

How to achieve the linear performance increase with Redis cluster using the lettuce java client??

[mp911de]

Have you investigated the GC and CPU pressure that the client produces?

Generally, the idea is to make use of concurrency, while your example code uses the synchronous API that makes the benchmark wait for command completion on one node while the other nodes are idling around.

[bsmanit]

Yes, GC and CPU are within limit.
I am running the above code with managed instance group and making sure each service instance doesn't cross 40% CPU uses.

Perf Setup:

Jmeter (load generation tool) -> LB  -> Application(using lettuce client to interact with Redis Cluster) -> Redis Cluster

Generally, the idea is to make use of concurrency, while your example code uses the synchronous API that makes the benchmark wait for command completion on one node while the other nodes are idling around.

Didn't fully get it, Here we have multiple application instance so even if it's waiting for command completion, other nodes can serve the generated load and achieve the expected load.