Prometheus - Allow to split attribute values by a delimiter into new metric series #5136
Comments
|
Note that the string representation of This looks like an invalid transformation of attributers in the prometheus exporter, which simply uses We could possibly add the |
Thanks for the pointers. This is even more confusing because the stored data would be language specific.
Sounds good. |
|
Would you accept a PR? |
|
As you mention above, the current representation is go-specific. So we should be adding clarification in the specification. |
|
I had the same idea. Thanks for opening an issue. |
dashpole
added
the
pkg:exporter:prometheus
|
IIUC, the options we are considering are:
@StarpTech Does prometheus support parsing the proposed format? How would you do that? |
No, the prometheus exporter would need to split |
|
Won't that break totals? E.g. gives 100 as the total. gives 300 as the total. |
|
You're correct, and upon reconsideration, it seems I overlooked an important aspect here. To replicate the same behavior, I simply need to emit the metrics three times, each with its own attribute values. It is funny though, that we all had a different understanding 😄 Thank you! c.Add(ctx, 1, attribute.Key("wg.subgraph.error.extended_code").String("B"))
c.Add(ctx, 1, attribute.Key("wg.subgraph.error.extended_code").String("C"))
c.Add(ctx, 1, attribute.Key("wg.subgraph.error.extended_code").String("D"))I don't know why I didn't test it before. This is the result Search for |
|
I suppose the question remains on how slice attributes are useful for prometheus metrics, or if they should just never be used (and are there purely for historical/specification reasons) |
|
I lean towards "they should never be used". That was the conclusion we came to when discussing potentially adding maps to the common attribute type. This issue did alert me to the fact that our encoding of attributes doesn't follow the spec, though. |
Problem Statement
In OpenTelemetry (OTEL), assigning a list of values to an attribute isn't directly supported. However, there's a helper method called
StringSlicewhich converts the value into a JSON array. It's then up to the consumer to parse it during ingestion.On the other hand, Prometheus doesn't support parsing JSON. Therefore, for optimal compatibility, the data must be prepared and ready at scrape time.
Proposed Solution
I suggest introducing an additional feature called
.WithMultiValues(",")to the Prometheus exporter. This feature would enable the splitting of metric labels by a specified delimiter into different metric series. This approach would allow us to combine the best aspects of both worlds, providing greater flexibility and compatibility. I'm not bound strictly to the delimiter aspect; alternatively, we could also reach a consensus on the output ofStringSlice.This feature is easy to add because it only affects the output of prometheus. We would need to iterate over all metrics attributes, split the value by the delimiter and duplicate the metric by the new label value.
Alternatives
It seems that achieving this functionality solely through the
relabel_configsof Prometheus is not feasible, as it lacks the necessary flexibility. Additionally, adding a preprocessing step to all Prometheus platforms may not always be practical or possible.Therefore, introducing the proposed
.WithMultiValues(",")option directly within the Prometheus exporter would offer a more straightforward and universally applicable solution to splitting metric labels by a delimiter into different metric series. This approach would alleviate the need for additional preprocessing steps and ensure compatibility across various Prometheus platforms.The text was updated successfully, but these errors were encountered: