Zookeeper duration #541

souBel · 2024-01-09T15:04:44Z

Add zokeeper-health and zookeeper-latency parameters

BzSpi · 2024-01-09T15:07:37Z

modules/smart-agent_zookeeper/conf/00-zookeeper-health.yaml

@@ -0,0 +1,16 @@
+module: zookeeper
+name: zookeeper-health


Suggested change

name: zookeeper-health

name: health

BzSpi · 2024-01-09T15:07:56Z

modules/smart-agent_zookeeper/conf/01-zookeeper-latency.yaml

@@ -0,0 +1,22 @@
+module: zookeeper
+name: zookeeper-latency


Suggested change

name: zookeeper-latency

name: latency

BzSpi · 2024-01-09T15:08:41Z

modules/smart-agent_zookeeper/conf/01-zookeeper-latency.yaml

+    lasting_duration: "5m"
+    latency_disabled: "false"
+  major:
+    threshold: 250000


Critical & major thresholds seems ta have too close values.

BzSpi · 2024-01-09T15:09:18Z

modules/smart-agent_zookeeper/conf/00-zookeeper-health.yaml

+name: zookeeper-health
+transformation: false
+aggregation: true
+exclude_not_running_vm: true


Is it necessary ?

BzSpi · 2024-01-09T15:39:11Z

modules/smart-agent_zookeeper/conf/01-zookeeper-latency.yaml

+name: zookeeper-latency
+transformation: false
+aggregation: true
+exclude_not_running_vm: true


Is it necessary ?

BzSpi · 2024-01-09T15:39:35Z

modules/smart-agent_zookeeper/detectors-zookeeper.tf

@@ -26,7 +26,7 @@ EOF
  max_delay = var.heartbeat_max_delay
 }

-resource "signalfx_detector" "zookeeper_health" {
+/*resource "signalfx_detector" "zookeeper_health" {


BzSpi · 2024-01-09T15:40:04Z

modules/smart-agent_zookeeper/variables.tf

@@ -44,7 +44,7 @@ variable "heartbeat_aggregation_function" {
  default     = ""
 }

-# zookeeper_health detector
+/*# zookeeper_health detector


BzSpi · 2024-01-09T15:43:25Z

modules/smart-agent_zookeeper/conf/00-zookeeper-health.yaml

@@ -0,0 +1,16 @@
+module: zookeeper


This detector should aggregate on all servers in the cluster and trigger a major on loss of part of the servers (half ? third ?) and critical on loss of more than that.

For example :

signal = data('gauge.zk_service_health', filter=filter('env', 'preprod') and filter('sfx_monitored', 'true')).mean(by=['plugin_instance']).publish('signal') detect(when(signal < 0.66, lasting='5m', at_least=1)).publish('CRIT') detect(when(signal < 1, lasting='5m', at_least=1)).publish('MAJ')```

BzSpi · 2024-01-09T15:45:11Z

modules/smart-agent_zookeeper/conf/01-zookeeper-latency.yaml

+  signal:
+    metric: "gauge.zk_avg_latency"
+rules:
+  critical:


I'm not sure that a high latency on one server should trigger a critical alter.
Maybe 2 detectors:

one that trigger major and critical if all servers in a cluster have high latency

one that trigger major for a single server high latency

freezeeedos · 2024-01-11T16:45:32Z

Work in progress, still needs some cleanup, but the detector was split as recommended, and behavior is has expected
There's one "major" detector which looks at individual nodes, and a "critical" one which looks at whole clusters

souBel · 2024-01-12T09:48:13Z

Cleanup's details done. Please check our last changes and tell us if all is right now

souBel · 2024-01-15T11:05:58Z

We also split server-health detector : one critical for cluster and one major for single server
Please checkout

souBel · 2024-02-20T08:53:40Z

Hello,
Were you able to checkout if everything is OK with new commits (the last) ?

haedri · 2024-02-20T09:46:45Z

modules/smart-agent_zookeeper/conf/00-zookeeper-server-health.yaml

@@ -0,0 +1,14 @@
+module: zookeeper
+name: server-health
+disabled: false


Not needed, this is the default value

haedri · 2024-02-20T09:46:49Z

modules/smart-agent_zookeeper/conf/01-zookeeper-cluster-health.yaml

@@ -0,0 +1,14 @@
+module: zookeeper
+name: cluster-health
+disabled: false


Not needed, this is the default value

haedri · 2024-02-20T09:46:53Z

modules/smart-agent_zookeeper/conf/02-zookeeper-server-latency.yaml

+module: zookeeper
+name: server-latency
+aggregation: false
+disabled: false


Not needed, this is the default value

haedri · 2024-02-20T09:46:56Z

modules/smart-agent_zookeeper/conf/03-zookeeper-cluster-latency.yaml

+module: zookeeper
+name: cluster-latency
+aggregation: ".mean(by=['kubernetes_cluster'])"
+disabled: false


Not needed, this is the default value

haedri · 2024-02-20T09:47:32Z

modules/smart-agent_zookeeper/conf/03-zookeeper-cluster-latency.yaml

+    comparator: ">"
+    description: "Zookeeper cluster latency is too high"
+    lasting_duration: "5m"
+    latency_disabled: "false"


latency_disabled variable does not exist, I think this line should be deleted

haedri · 2024-02-20T09:47:35Z

modules/smart-agent_zookeeper/conf/02-zookeeper-server-latency.yaml

+    comparator: ">"
+    description: "Zookeeper server latency is too high"
+    lasting_duration: "5m"
+    latency_disabled: "false"


latency_disabled variable does not exist, I think this line should be deleted

haedri · 2024-02-20T09:47:43Z

modules/smart-agent_zookeeper/conf/01-zookeeper-cluster-health.yaml

+    comparator: "=="
+    description: "Zookeeper cluster is not running"
+    lasting_duration: "5m"
+    health_disabled: "false"


health_disabled variable does not exist, I think this line should be deleted

haedri · 2024-02-20T09:47:48Z

modules/smart-agent_zookeeper/conf/00-zookeeper-server-health.yaml

+    comparator: "!="
+    description: "Zookeeper server is not running"
+    lasting_duration: "5m"
+    health_disabled: "false"


health_disabled variable does not exist, I think this line should be deleted

souBel · 2024-03-13T11:18:18Z

Hello,
I updated your recommendations as requested.
Could you check if everything is OK now ?

souBel · 2024-03-28T13:36:24Z

Hello,
Were you able to check it out ?

souBel · 2024-05-06T14:40:54Z

Any update please ?

Soufiane added 7 commits January 3, 2024 15:46

Add zookeeper-health

f538036

Remove old zookeeper health

b9fa05e

Remove old zookeeper health

2f0b068

Remove old health ZK

7bb2c6b

Enabled detector

f90dfae

update duration

a98dd62

Add zookeeper-latency module

2209c16

souBel requested review from BzSpi, cvauvarin, haedri, JMHAUTBOIS-Claranet, pdecat, Shr3ps and tchernomax as code owners January 9, 2024 15:04

BzSpi requested changes Jan 9, 2024

View reviewed changes

Quentin GIBERT added 4 commits January 11, 2024 16:56

zookeeper: divide single and global latency

df37153

zookeeper: divide single and global latency

ae5b8ee

cleanup

75e835a

zookeeper latency: fix aggregation

f1b41ec

cleanup zookeeper config

66b04fd

souBel requested a review from BzSpi January 12, 2024 09:48

Soufiane added 3 commits January 12, 2024 15:59

split health detector

797bfb9

no aggregation for server-health

995a6f8

update cluster-health comparator

8808b17

souBel requested a review from Gati0 January 15, 2024 11:06

haedri requested changes Feb 20, 2024

View reviewed changes

Delete default variables

7d2b1c1

souBel requested a review from haedri March 13, 2024 11:18

souBel assigned haedri, BzSpi and Gati0 May 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Zookeeper duration #541

Zookeeper duration #541

souBel commented Jan 9, 2024

BzSpi Jan 9, 2024

BzSpi Jan 9, 2024

BzSpi Jan 9, 2024

BzSpi Jan 9, 2024

BzSpi Jan 9, 2024

BzSpi Jan 9, 2024

BzSpi Jan 9, 2024

BzSpi Jan 9, 2024

Gati0 Jan 12, 2024

BzSpi Jan 9, 2024

freezeeedos commented Jan 11, 2024

souBel commented Jan 12, 2024

souBel commented Jan 15, 2024

souBel commented Feb 20, 2024

haedri Feb 20, 2024

haedri Feb 20, 2024

haedri Feb 20, 2024

haedri Feb 20, 2024

haedri Feb 20, 2024

haedri Feb 20, 2024

haedri Feb 20, 2024

haedri Feb 20, 2024

souBel commented Mar 13, 2024

souBel commented Mar 28, 2024

souBel commented May 6, 2024

Zookeeper duration #541

Are you sure you want to change the base?

Zookeeper duration #541

Conversation

souBel commented Jan 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freezeeedos commented Jan 11, 2024

souBel commented Jan 12, 2024

souBel commented Jan 15, 2024

souBel commented Feb 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

souBel commented Mar 13, 2024

souBel commented Mar 28, 2024

souBel commented May 6, 2024