Redis Cluster Support

[evantahler]

Right now the Grouparoo application doesn’t support redis cluster. The crux is that the redis library we are using (ioredis) has some pretty different syntax to connect to a cluster, specifically around how to provide multiple hosts (see the link above for details).

On Grouparoo’s end, we still want to use only environment variables for configuration to make Dockerize the application simple, and would need a way to:

1. collect multiple redis hosts
2. be sure that we can tell that you mean to use the cluster redis client and not the normal one

Some ideas:

1. Multiple REDIS_URL# , like REDIS_URL_1 , REDIS_URL_2 , etc. Having these set in your environment (with the number) and not the normal REDIS_URL would indicate that you want to use cluster redis connection and not not normal redis connection.
2. A single REDIS_URL like we have now, but with a ?cluster=true appended at the end of the URL string. This /may/ be less resilient because there’s only one cluster IP in the mix, but that’s technically all you need to connect
3. A single REDIS_URL , but using some delimiter (; or | maybe) to separate multiple URLs

Any suggestions? Ideally, referencing some other application conventions would be helpful.

[evantahler]

For reference, here is where the Grouparoo app parses the REDIS_URL and builds the redis connection(s) https://github.com/grouparoo/grouparoo/blob/main/core/src/config/redis.ts

[bleonard]

If we did something like #3, maybe something like REDIS_CLUSTER_URLS.

[evantahler]

Confirming a Hypothesis:

You cannot use a redis cluster client on a regular redis cluster. This means that we can't just use the Redis.Cluster-type client in all cases

2021-12-15T00:25:04.286Z - info: Redis connection `client` reconnecting
2021-12-15T00:25:04.563Z - alert: Redis connection `client` error Failed to refresh slots cache. lastNodeError=ReplyError: ERR This instance has cluster support disabled stack=ClusterAllFailedError: Failed to refresh slots cache.

[nholmes3]

Hey, just to chime in for our particular use case in AWS:
According to the AWS Documentation for Redis (Cluster Mode Enabled) endpoints,

A Redis (cluster mode enabled) cluster has a single configuration endpoint. By connecting to the configuration endpoint, your application is able to discover the primary and read endpoints for each shard in the cluster.

It seems like Option 2 would be all that is necessary for this type of connection, and AWS provides a solution for "there’s only one cluster IP in the mix" under the hood of their configuration endpoint. The ?cluster=true option would flag for the use of the Cluster Client instead of the normal one...

[evantahler]

Reading up on some other frameworks (eg rails), it does seem like it's common when using environment variables to configure redis cluster to often only provide one node. We'll go with option 2 for now

• https://www.reddit.com/r/rails/comments/nbgid8/provide_redis_cluster_endpoint_in_rails_cache/

[evantahler]

After digging into what it would take to support redis-cluster, I ran into quite a few snags.

It looks like the implementation details of how some of the commands work are rather different in redis-cluster than normal redis. Mainly, a lot of the atomic operations cannot be as atomic if they span shards 😕. - pipelining, mutli-gets, and lua commands specifically The underlying background job library we use makes heavy use of these “multi” commands to do things like “pop this job and move it over here” all at once. I’m worried about loosing some of the atomicity we get from ‘regular’ redis in exchange for cluster support.

At this time, I cannot recommend the use of redis-cluster for Grouparoo