✨ Add metrics for total workers and active workers #1125
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: alvaroaleman The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
approved
|
/assign @estroz |
alvaroaleman
force-pushed
the
moar-metrics
branch
from
33ce5b6
to
5cefa42
Compare
alvaroaleman
changed the title
|
/retest |
1 similar comment
|
/retest |
| @@ -169,6 +169,7 @@ func (c *Controller) Start(stop <-chan struct{}) error { | |||
|
|
|||
| // Launch workers to process resources | |||
| c.Log.Info("Starting workers", "worker count", c.MaxConcurrentReconciles) | |||
| ctrlmetrics.WorkerCount.WithLabelValues(c.Name).Set(float64(c.MaxConcurrentReconciles)) | |||
There was a problem hiding this comment.
Can you explain the logic behind collecting this metric?
There was a problem hiding this comment.
- It allows correlating other metrics to the number of workers, for example workqueue_depth
- It allows calculating the
available_workers
|
@estroz PTAL |
|
Without knowing too much about how metrics are collected this looks fine to me. However there are now 7 metrics collected for a controller. Will an increase in the number of metrics impact cluster resources significantly? /lgtm |
k8s-ci-robot
added
the
lgtm
|
Cancel whenever /hold |
k8s-ci-robot
added
the
do-not-merge/hold
No, the way this works is that only the current value of a metric is stored and Prometheus is expected to collect it. The thing to keep in mind is to not expose too many metrics which usually happens when using labels that have a yet-to-be-determined set of values, because every metric+label combinaation actually ends up being an unique metric in Prometheus: https://www.robustperception.io/cardinality-is-key /hold cancel |
k8s-ci-robot
removed
the
do-not-merge/hold
|
/retest |
1 similar comment
|
/retest |
Helps debugging issues around reconciliation duration. Right now its impossible to find out the number of workers from metrics and its also impossible to find out the number of active workers. The only thing we have is
workqueue_unfinished_work_secondswhich if big is probably not good but its not clearly indicating if we have one worker that takes longer, if all workers are blocked etc.