Epic: Implement UTF-8 character support for label and metric names

[ywwg]

Tracking Issue

Relates to:

Original discussion
Brainstorming doc Quoting Prometheus Names
OTEL Attribute compatibility section

We are creating this issue as part of the OTEL Support milestone so it will be OTEL oriented but the problem is general to prometheus and has been around for a long time.

OTel allows UTF-8 in label names while Prometheus has a much more restrictive set. This is causing friction for users when using Prometheus. In particular, . (dot) is a very common character in OTel and we convert that to _ when converting to Prometheus. For example, service.version becomes service_version.

Proposals have been accepted to do this work:

• https://github.com/prometheus/proposals/blob/main/proposals/2023-08-21-utf8.md
• https://github.com/prometheus/proposals/blob/main/proposals/2023-11-13-utf8-migration.md

Write-side tasks:

• prom/common: update openmetrics_parse, promparse (UTF-8 support in validation, and some parsers and formatters common#537)
• prom/common: content negotiation in scraping (UTF8: Content negotiation (encoding side) common#570)
• prom/client_golang: (Implement changes needed to support UTF8 metric and label names in Prometheus client_golang#1369)
• prom/prometheus: Update some parsers (UTF-8: Add partial support for parsing UTF-8 metric and label names #13271)
• prom/common: update text_parse (go version) (Update expfmt/text_parse.go to support the new UTF-8 syntax common#554)
• prom/prometheus: content negotiation in scraping / remote-write (WIP: UTF8 Content Negotiation for writes #13615)
• prom/pushgateway: (Support UTF-8 in metric and label names pushgateway#623)

Read-side tasks:

• enable querying of old data with new UTF-8 names #13765
• prom/prometheus: tsdb meta.json bump
• prom/prometheus: tsdb multiquerying for otel and other migration cases
• prom/prometheus: content negotiation in querying?
• prom/prometheus: update web/ui parser (UTF-8: Update typescript parser to support new syntax #13564)

Client Libraries:

• Go: no changes required
• Java -- in progress, will be implemented in a feature branch pending OTel spec updates
• UTF-8: Implement support in Python client library client_python#1013
• UTF-8: Implement support in Ruby client library client_ruby#306
• UTF-8: Implement support in Rust client library client_rust#190

Grafana user-visible changes:

Not directly related to Prometheus, but this is the bug I'm using for all the related work, so...

• grafana/agent: needs minimal config and flags for setting globals
• grafana/mimir: flags to enable utf8
• grafana/grafana: Prometheus: Support forthcoming UTF-8 metric and label names in Prometheus grafana/grafana#74875

Other task refs:

• Implement changes needed to support UTF-8 metric and label names in Prometheus common#527
• update docs, like https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md
• Officially update OpenMetrics spec
• Officially update OTEL spec

(This is a clone of #12630 that I own so I can edit the top-level comment and track work here)

[ywwg]

Design work will be needed for the tsdb changes -- determining how to query data that was written with old versions of prometheus will be tricky, for instance if an Otel metric goes from _ to . but doesn't use the U__ quoting syntax in the old metrics.

[spedersen-emailage]

Ran into this issue with Grafana alert labels vs. Prometheus alert labels. Grafana supports -, but Prometheus does not. In order to avoid conflicts or multiple label sets, we'll need to replace - with _ in all of our alerts and notification policies next week.

Consistency between OTel, Prometheus, etc. would be fantastic.

[ywwg]

prometheus/common#537 in progress (name validation, exposition parsers)

[ywwg]

ywwg/common#2 in draft (write-side content negotiation)

[mhausenblas]

Thanks for this! Question concerning "User-visible changes:" section. Should we take into account CNCF projects such as Thanos and Cortex?