'Keys()' and 'Scan()' not returning all matching keys after Redis Sentinel failover.

[mxcoppell]

We have the following setup:

• Redis Sentinel deployment in EKS. (Redis version 7.2.4).
• Service application running in the same EKS namespace as the Redis Sentinel.
• Go-redis(v9.5.1) failover client used in the service application.

		failoverOpts := &redis.FailoverOptions{
			MasterName:    config.RedisSentinelMasterSet,
			SentinelAddrs: []string{config.RedisConnectionString},
			DialTimeout:   time.Duration(config.RedisDialTimeout) * time.Millisecond,
			ReadTimeout:   time.Duration(config.RedisReadTimeout) * time.Millisecond,
			WriteTimeout:  time.Duration(config.RedisWriteTimeout) * time.Millisecond,
		}
		client = redis.NewFailoverClient(failoverOpts)

There are some keys with names like "node:{node-1}:data", "node:{node-2}:data", ... "node:{node-N}:data". The number of keys are very small (Let's use 10 keys in the following discussion)

We are using the following code to fetch all the keys matching the above pattern:

		keys, err := rc.db.Keys(ctx, "node:{*}:data").Result()
		if err != nil {
			log.Errorf("error finding keys in redis, err: %v", err)
			return nil, err // Return on other errors
		}

After the initial connection to the sentinel, 10 keys are returned constantly when we use the Keys() to fetch them.

In one of the edge scenario, we manually restarted the Redis Sentinel statefulset to enforce the failover test. All pods reported that the failovers were completed and connected to the new master instance. Then we started notice that random number of keys are turned (<10) from the Keys() call - no errors reported by go-redis.

At this point, if we restart the problem pods to force recreating the initial connection (without failover), everything back to normal. All 10 keys would be returned from the Keys() call.

We tested with "Scan()" and iterator and got the same result.

Anybody experienced this before? Thanks!

[mxcoppell]

Update: We found out it's not Redis issue. It's a bug in the application code.