Dynamic label for distribution metrics

[BlueSeaOcean]

I use cloudprober extension and expect to have following metrics with "status" label to indicate whether a response is success or fail.

// counter metric
total{target="google.com", status="success"} 1
total{target="google.com", status="fail"} 1
// distribution metric
total_latency_bucket{target="google.com", status="success", le="0.1"} 1
total_latency_bucket{target="google.com", status="fail", le="15"} 1

For counter metric, I use map to add the dynamic label "status".
res["google.com"] = metrics.NewEventMetrics(time.Now()).
                                 AddMetric("total", statusMap["google.com"])
...
go func (target, em *metrics.EventMetrics, statusCnt *metrics.Map){
 status := p.RunProbeForTarget(...) // return "success" or "fail"
 statusCnt.IncKey(status)
}(target, res["google.com"], statusMap["google.com"])

Is there anyway to add the dynamic label for the distribution metric?

[manugarg]

@BlueSeaOcean one way to do what you're trying to do will be too use two eventmetrics with different labels:

successEM := metrics.NewEventMetrics(time.Now()).AddMetric("total", metrics.NewInt(0))...AddLabel("status", "success")
failEM := metrics.NewEventMetrics(time.Now()).AddMetric("total", metrics.NewInt(0))...AddLabel("status", "fail")

[BlueSeaOcean]

@manugarg My understanding is successEM and failEM in your comment refer to counter metrics.

1. Can this be applied with the same distribution metric? i.e. having one metric with label instead of two metrics.
2. If this is not supported, may I know the reason behind why we should use different metrics instead of having the same metric with label? I prefer to use one metric with label instead of using two metrics because Prometheus supports this and it is simpler to build rules/queries.

[manugarg]

@BlueSeaOcean

My understanding is successEM and failEM in your comment refer to counter metrics.

No, it's a way of segregating metrics by success and failures. Both EMs (EventMetrics -- things that you export) can contain "total", "success", "latency" metrics of their own.

Can this be applied with the same distribution metric? i.e. having one metric with label instead of two metrics.

Metrics (in cloudprober terms) don't have labels EventMetrics do. EventMetrics are collections of metrics with a given set of labels.

If this is not supported, may I know the reason behind why we should use different metrics instead of having the same metric with label? I prefer to use one metric with label instead of using two metrics because Prometheus supports this and it is simpler to build rules/queries.

I think you're getting confused here a bit (or maybe I am not understanding it properly). Having two EventMetrics doesn't mean you'll have two different metric names. See my example in the previous comment. With that you'll see prometheus metrics like this:

total{status="success", probe="probe1"}
success{status="success", probe="probe1"}
total{status="fail", probe="probe1"}
success{status="fail", probe="probe1"}

[BlueSeaOcean]

@manugarg Thank you for clarification. I'll experiment more and let you know.
In this case, can we also have status label based on http response code? Do we need one EM metric for one response code since it's static value when we create the EM metric?
For example,
success{status="204", probe="probe1"}
success{status="200", probe="probe1"}

[manugarg]

@BlueSeaOcean It depends on what you're counting -- what's the measurement here. If want to count status codes, you'll just have one metric say resp_code:

resp_code{code=200} 400
resp_code{code=500} 12
...

See this for example:

cloudprober/probes/http/http.go

Line 394 in 8f0685b

AddMetric("resp-code", result.respCodes.Clone())

If you're trying to count something else by numeric response code, then it becomes more complicated.

[manugarg]

@BlueSeaOcean if you can share what you're trying to do, I can perhaps provide more useful feedback.

Cheers,
Manu

[BlueSeaOcean]

@manugarg Thank you very much. Your suggestion is very close. Does the cloud prober support below where we don't get response code from the server?
For example,

// get_response returns either response code and err is nil,
// or response code is nil and err when we don't get any response code from the server.
resp, err = get_response(...) 

The metrics will look like:
total{status="204", probe="probe1"}
total{status="200", probe="probe1"}
total{status="401", probe="probe1"}
total{status="TIMEOUT", probe="probe1"}

// Corresponding distribution latency metrics
total_latency_bucket{status="204", probe="probe1", le="0.1"} 1
total_latency_bucket{status="200", probe="probe1", le="0.1"} 1
total_latency_bucket{status="401", probe="probe1", le="5"} 1
total_latency_bucket{status="TIMEOUT", probe="probe1", le="15"} 1

[manugarg]

@BlueSeaOcean

Simple counters are easy, e.g. total{..} in the example above is just a status counter. You could use a map for that:

statusCount = metrics.NewMap(status)
If you add it as metric status_count:

status_count{status="204"..}
...

Distributions don't support map values, so you cannot add a dimension to a distribution level. Note that in Cloudprober, latency is added only for successes.

[BlueSeaOcean]

@manugarg Thanks. That's what I explored so far before opened the topic as well. Do we have any plan for distributions to support map values? If not, may I know the justification behind?