Allow Prometheus exporter to add resource attributes to metric attributes

[asafm]

Fixes #6108

This PR adds the ability to copy resource attributes to the labels of each data point, per a given regular expression.

Two new configurations have been introduced as part of this PR.

[asafm]

@jack-berg I need your help on 3 open issues before I can convert this to a normal PR:

1. Where do I parse the environment variables? I wanted to add the parsing for the 2 new environment variables.
2. Should I proceed to change the file configuration schema in the relevant repo?
3. Am I correct about the hard-coded attributes that will not be copied? shall this actually be present in the specification?

[codecov]

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (f123d78) 90.97% compared to head (e1e3b7f) 91.04%.
Report is 23 commits behind head on main.

Files	Patch %	Lines
.../exporter/prometheus/Otel2PrometheusConverter.java	93.24%	4 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #6179      +/-   ##
============================================
+ Coverage     90.97%   91.04%   +0.07%     
- Complexity     5670     5686      +16     
============================================
  Files           620      621       +1     
  Lines         16535    16623      +88     
  Branches       1682     1698      +16     
============================================
+ Hits          15042    15134      +92     
+ Misses         1023      998      -25     
- Partials        470      491      +21     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jack-berg]

Where do I parse the environment variables? I wanted to add the parsing for the 2 new environment variables.

https://github.com/open-telemetry/opentelemetry-java/blob/main/exporters/prometheus/src/main/java/io/opentelemetry/exporter/prometheus/internal/PrometheusMetricReaderProvider.java

Should I proceed to change the file configuration schema in the relevant repo?

@codeboten has been working on that. He's merged a number of PRs recently updating the prometheus model, and we're working on the resource attributes part in open-telemetry/opentelemetry-configuration#67.

Am I correct about the hard-coded attributes that will not be copied? shall this actually be present in the specification?

Sorry can you elaborate on this? I think you're referring to specific rules around service.name and service.instance.id since those are conceptually the same as the job name instance in promethues, which are inferred from the scrape config.

[asafm]

@jack-berg Ok, got it - the environment is lowered case, and the _ is replaced with . and then placed in the config map.
Could there be an error then at

| otel.experimental.resource.disabled-keys | OTEL_EXPERIMENTAL_RESOURCE_DISABLED_KEYS | Specify resource attribute keys that are filtered.                                                         |

[asafm]

@jack-berg:

• I removed the hard-coded excludes to match the spec
• PR is ready for review

[jack-berg]

@jack-berg Ok, got it - the environment is lowered case, and the _ is replaced with . and then placed in the config map.
Could there be an error then at

No there shouldn't be. We normalize keys before reading as well. So if you set otel.experimental.resource.disabled-keys, its normalized to otel.experimental.resource.disabled.keys, then when we read otel.experimental.resource.disabled-keys it is also normalized to otel.experimental.resource.disabled.keys

[asafm]

@jack-berg Ok, got it - the environment is lowered case, and the _ is replaced with . and then placed in the config map.
Could there be an error then at

No there shouldn't be. We normalize keys before reading as well. So if you set otel.experimental.resource.disabled-keys, its normalized to otel.experimental.resource.disabled.keys, then when we read otel.experimental.resource.disabled-keys it is also normalized to otel.experimental.resource.disabled.keys

This is the code doing the normalization:

  public static String normalizeEnvironmentVariableKey(String key) {
    return key.toLowerCase(Locale.ROOT).replace("_", ".");
  }

Where is the replacement of "-" to "." ? It's only replacing "_" to ".", no?

[asafm]

@jack-berg All fixed. Take a look

[jack-berg]

Couple of small additional comments. Thanks!

[asafm]

@jack-berg All comments have been fixed (aside from one).

[jack-berg]

Thanks!

[jack-berg]

FYI @open-telemetry/java-approvers I'm planning on merging this for the 1.35.0 release.

[Falmarri]

@jack-berg i'm interested in this feature but I can't figure out how to use it. Are there docs?

[asafm]

@Falmarri Currently the only way to use it is when you build the SDK manually completely. Then you can use the prometheus builder and set the predicate dictating which attributes to copy.
I plan to add a way to modify the exporter to expose all of its properties, so I can recreate the builder from the built exporter. Once that's ready you can modify the default prometheus exporter and add your own predicate.
I know that there is a plan to have that as part of the official opentelemetry configuration, which means you'll be able to configure this with a file.