Dynamically add labels in a Counter or CounterVec

[nevesgustavo]

Hi Team,
I hope you are having a good weekend, I would like to have an information about CounterVec, is there a way to dynamically add Labels inside an already registered metric?

For example:

Creating a metric using two labels: status and environment

metricName := "test_purposes"
help := "tests"

counter = promauto.NewCounterVec(prometheus.CounterOpts{
		Name: metricName,
		Help: help,
	}, []string{"status", "environment"})

	counter.WithLabelValues("success", "dev").Inc()

But when the status is error I would like to add one more label: reason, to the same metric and it's not possible

counter = promauto.NewCounterVec(prometheus.CounterOpts{
		Name: metricName,
		Help: help,
	}, []string{"status", "environment", "reason"})

	counter.WithLabelValues("success", "dev", "npe").Inc()

Just to explain why I need it, we're migrating an implementation of datadog to prometheus and in the implementations based on datadog we have a lot of microservices sending metrics using different tags (labels) to the same metric, if I could do the same using the prometheus lib it would be great because we could only change the client without any modifications on metric publishers.

[beorn7]

The CounterVec is a primitive for direct instrumentation. If you instrument directly, you should always have the same set of labels, and that's what the library supports (or in fact enforces).

If you just translate metrics from a different system, you would use https://pkg.go.dev/github.com/prometheus/client_golang@v1.12.1/prometheus#NewConstMetric and similar functions as part of a custom Collector implementation. You can the attach all the labels you want. To not trigger registration errors, you can create "unchecked" Collectors, see https://pkg.go.dev/github.com/prometheus/client_golang@v1.12.1/prometheus#Collector .

[nevesgustavo]

Hi @beorn7, thank you so much for your answer, I'll take a look on the documentations that you sent me and try to create an example.

[nevesgustavo]

Hi @beorn7, please let me know if I understood correctly, the NewConstMetric method is used when getting metrics from one system like Datadog, mtail and extracting for Prometheus metrics endpoint right? so in this case we can get all the data from the origin system, like: name, help, value and labels.

Let me try explain my scenario:
1 - Today we are sending logs to Datadog, and not sending the same number of tags in every calls, something like this:

func buildMetricTags(keyValueTags []string, failureReason string) []string {
	if failureReason != "" {
		return append(keyValueTags,
			Status, Error,
			Reason, failureReason)
	}

	return append(keyValueTags, Status, Success)
}

Sometimes sends the Reason tag and sometimes not.

2 - The idea is to create another client to expose metrics for Prometheus, so we will interface with two implementations (datadog and prometheus) and save in both places at this first time, but as you said above the prometheus lib enforces to use always the same number of labels.

So with this scenario I think that we can't use the NewConstMetric because we are not getting data from datadog and we can't depend on this, so I think that refactor everyplace to send the same number of labels is the best choice at this moment, please let me know your thoughts.

Thank you very much for your time.

[beorn7]

You could still use NewConstMetric to create the metrics for Prometheus.

CounterVec etc. is needed for direct instrumentation (mostly) for Prometheus. If you already have instrumented your code so that you can provide metrics for Datadog, then you probably don't want to duplicate every madeUpDataDogAPI.Counter.Inc() line with a counter.Inc() where counter is a prometheus.Counter. Ideally, you can tap into whatever metrics you already have and create them at scrape time with NewConstMetric.

Having said that, it is possible in Prometheus to set an empty label value, with is (almost) completely equivalent to not having set that label at all. So as long as you know the maximum possible label set, you can always provide the whole set, with empty label values for those labels that you don't want to use.

[ekristen]

@beorn7 I have a similar issue. I have a single metric that covers a resource state, sometimes it has 4 static labels, but sometimes I need to add optional labels for further tracking that's instance dependent. The fact that the labels all have to be defined ahead of time is fairly frustrating.

Is there any way around this?

metric{field1="",field2=""}

Sometimes I need to add

metrics{field3="",field4=""}

[ashokrajar]

You can use a function like this to initialise and return GaugeVec or CounterVex just once and then use it to assign dynamic label values to it.

Sample code for GaugeVec

type customMetrics struct {
	customMetricsOne  *prometheus.GaugeVec
	customMetricsTwo  *prometheus.GaugeVec
	Initialized       bool
}

func newCustomMetrics() (m *customMetrics) {
	m = &customMetrics{
		customMetricsOne: prometheus.NewGaugeVec(
		prometheus.GaugeOpts{
			Name:      "metric_name",
			Help:      "Metric Help Doc, can be skipped",
		}, []string{"label1", "label2"}),
		customMetricsTwo: prometheus.NewGaugeVec(
		prometheus.GaugeOpts{
			Name:      "metric_two_name",
			Help:      "Metric Help Doc, can be skipped",
		}, []string{"label3", "label4"}),
	}

	prometheus.MustRegister(
		m. customMetricsOne,
		m. customMetricsTwo,
	)
	m.Initialized = true

	return m
}


func main() {
    if !cm.Initialized {
		cm = *newCustomMetrics()
	}

	cm.customMetricsOne.With(prometheus.Labels{"label1": "label1 value", "label2": "label2 value"}).Set(20.45)
	cm.customMetricsTow.With(prometheus.Labels{"label3": "label3 value", "label4": "label4 value"}).Set(40.45)
}