Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redis health indicators report that Redis is up when the cluster's state is fail #27300

Conversation

hoanvh
Copy link
Contributor

@hoanvh hoanvh commented Jul 13, 2021

Given

a Redis cluster with 3 shards + 3 replicas as:

  • :7001 (master) ~> :7004 (slave)
  • :7002 (master) ~> :7005 (slave)
  • :7003 (master) ~> :7006 (slave)

Kill processes with ports 7003, 7004, 7005, 7006.

check CLUSTER INFO command:

cluster_state:fail
cluster_slots_assigned:16384
cluster_slots_ok:5462
cluster_slots_pfail:10922
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
...

As is:

Although cluster's state is fail but RedisHealthIndicator still mark Health as UP

To be:

If cluster's state is fail, then we will mark Redis Health as DOWN

Solution:

Check cluster_state value, if it equals (ignore case) with fail, then Redis Health is DOWN.

Reference

https://redis.io/commands/cluster-info

@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged label Jul 13, 2021
@wilkinsona wilkinsona changed the title correct RedisHealth status if cluster state is fail RedisHealthIndicator reports that Redis is up when the Redis cluster's state is fail Jul 13, 2021
@wilkinsona wilkinsona added the type: bug A general bug label Jul 13, 2021
@wilkinsona wilkinsona changed the title RedisHealthIndicator reports that Redis is up when the Redis cluster's state is fail Redis health indicators report that Redis is up when the Redis cluster's state is fail Jul 13, 2021
@wilkinsona wilkinsona added this to the 2.4.x milestone Jul 13, 2021
@wilkinsona wilkinsona added for: merge-with-amendments Needs some changes when we merge and removed status: waiting-for-triage An issue we've not yet triaged labels Jul 13, 2021
@wilkinsona wilkinsona self-assigned this Jul 13, 2021
@wilkinsona wilkinsona changed the title Redis health indicators report that Redis is up when the Redis cluster's state is fail Redis health indicators report that Redis is up when the cluster's state is fail Jul 13, 2021
wilkinsona pushed a commit that referenced this pull request Jul 13, 2021
@wilkinsona wilkinsona modified the milestones: 2.4.x, 2.4.9 Jul 13, 2021
@wilkinsona
Copy link
Member

Thanks very much for making your first contribution to Spring Boot, @hoanvh. If you're interested, I polished your changes to bring them into line with the project's conventions and code style in this commit.

@hoanvh hoanvh deleted the issue/correct-redis-health-in-cluster-mode branch July 19, 2021 04:51
@hoanvh
Copy link
Contributor Author

hoanvh commented Jul 21, 2021

@wilkinsona Regarding to this patch, it seems to be available on SB version 2.4.9? We're using SB 2.5.2, so I wonder which version 2.5.x includes it?

@wilkinsona
Copy link
Member

It will be in 2.5.3 as per the forward port issue that’s shown above.

@wilkinsona wilkinsona removed the for: merge-with-amendments Needs some changes when we merge label Jul 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug A general bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants