Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

> The resource attributes MAY be copied to labels of exported metric families if required by the exporter configuration #6235

Closed
gabrieljones opened this issue Feb 16, 2024 · 7 comments
Labels
Bug Something isn't working

Comments

@gabrieljones
Copy link

Describe the bug
Prometheus metrics exporter is missing config option to copy resource attributes to labels.

The resource attributes MAY be copied to labels of exported metric families if required by the exporter configuration
https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/prometheus_and_openmetrics.md#resource-attributes-1

What did you expect to see?
A config option to copy resource attributes to labels on metric families.

https://github.com/open-telemetry/opentelemetry-java/tree/v1.35.0/sdk-extensions/autoconfigure#prometheus-exporter

What did you see instead?
No such config option

What version and what artifacts are you using?
Artifacts: opentelemetry-sdk-extension-autoconfigure, opentelemetry-exporter-prometheus
Version: v1.35.0

@gabrieljones gabrieljones added the Bug Something isn't working label Feb 16, 2024
@gabrieljones
Copy link
Author

I see #6179 just got merged.

@gabrieljones
Copy link
Author

Looks like you guys already discussed this:
#6179 (comment)

@breedx-splk
Copy link
Contributor

@gabrieljones Are you cool with this outcome, or is there something else you're looking for?

@breedx-splk breedx-splk added the needs author feedback Waiting for additional feedback from the author label Feb 16, 2024
@gabrieljones
Copy link
Author

gabrieljones commented Feb 18, 2024

I'd like to avoid using a customizer here. A customizer would require adding code to 10+ different microservices.

I have a simple function that turns hocon config into System.setProperty calls for all the otel auto config properties.

@github-actions github-actions bot removed the needs author feedback Waiting for additional feedback from the author label Feb 18, 2024
@gabrieljones
Copy link
Author

gabrieljones commented Feb 19, 2024

//application.conf
//...
otel-attributes {
  service-namespace = NAMESPACE-missing //something other than empty string, in case env var below is missing
  service-namespace = ${?NAMESPACE}
  deployment-environment = PAAS_ENV-missing
  deployment-environment = ${?PAAS_ENV}
  service-instance-id = HOSTNAME-missing
  service-instance-id = ${?HOSTNAME}
  service-cluster = CLUSTER_ENV-missing
  service-cluster = ${?CLUSTER_ENV}
  service-version = VERSION_JAR-missing
  service-version = ${?VERSION_JAR}
}

otel {
  exporter.otlp.endpoint = "https://<otlp inbox>"
  service.name = ${APP_NAME}
  resource {
    attributes = "service.namespace="${otel-attributes.service-namespace}
    attributes = ${otel.resource.attributes}",deployment.environment="${otel-attributes.deployment-environment}
    attributes = ${otel.resource.attributes}",service.instance.id="${otel-attributes.service-instance-id}
    attributes = ${otel.resource.attributes}",service.cluster="${otel-attributes.service-cluster}
    attributes = ${otel.resource.attributes}",service.version="${otel-attributes.service-version}
  }
  metrics.exporter = prometheus
  logs.exporter = otlp
}
//Main.scala
//...

    val configRoot: Config = ConfigFactory.load()

    val configOpenTelemetry: Config = configRoot.withOnlyPath("otel")

    configOpenTelemetry.entrySet().asScala.foreach { e =>
      val k = e.getKey
      val v = e.getValue.unwrapped().toString
      // .- is used to denote properties that share a name with a config object name
      // https://github.com/lightbend/config/blob/v1.4.2/HOCON.md#:~:text=there%20is%20an-,intractable%20case,-where%20a%20single
      // eg
      // `otel.traces.sampler.- = parentbased_traceidratio`
      // becomes
      // `System.setProperty("otel.traces.sampler", "parentbased_traceidratio")`
      val km = k.replaceAll("""\.-$""", "")
      System.setProperty(km, v)
    }

    val openTelemetry: OpenTelemetry = AutoConfiguredOpenTelemetrySdk.builder.build().getOpenTelemetrySdk

Result:

System.setProoerty("otel.exporter.otlp.endpoint", "https://<otlp inbox>")
System.setProoerty("otel.logs.exporter", "otlp")
System.setProoerty("otel.resource.attributes", "service.namespace=some-namespace,deployment.environment=dev,service.instance.id=some-deployment-78648ffc65-9p8jj,service.cluster=devc,service.version=3.4.1-55758a5")
System.setProoerty("otel.service.name", "some-service-name")
System.setProoerty("otel.metrics.exporter", "prometheus")

@jack-berg
Copy link
Member

I'd like to avoid using a customizer here. A customizer would require adding code to 10+ different microservices.

The system properties and environment variables we interpret in autoconfigure is governed by the specification. There's currently no spec'd environment variable for configuring the prometheus resource attributes. There's a moratorium in place preventing any non-essential environment variable extensions while file configuration is being worked out. Hence, my comment about adding programmatic configuration but not adding environment variable / system property support.

@gabrieljones
Copy link
Author

Given the moratorium I guess this part of the spec is in limbo until file config is finished.

The resource attributes MAY be copied to labels of exported metric families if required by the exporter configuration
https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/prometheus_and_openmetrics.md#resource-attributes-1

Hardcoding it is then. After #6231 goes in that is.

I don't suppose Prometheus allows for creating "views" over join queries?

This[joins] is not a common pattern in Prometheus, and we recommend copying the most common resource attributes into metric labels.
https://grafana.com/blog/2023/07/20/a-practical-guide-to-data-collection-with-opentelemetry-and-prometheus/#2-copy-common-resource-attributes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants