Help on how to add labels to metrics sent to Prometheus from automatic instrumentation

[angelotessaro]

Hi, I instrumented my Flask app as you can see below (2nd code), using the automatic instrumentation and pushing the metrics to an AWS Managed Prometheus service. The setup works, I was able to visualize the metrics in Grafana, but they don't have the necessary labels to identify them, like the route (ex: "api/post"), request parameters, headers. I was able to add labels by using the Resource class ( provider.resource = Resource.create(resource_attributes)), but the labels set here will be added to every single metric and will not be request-specific. I want labels to identify the requests and be able to filter them in Grafana by the endpoint for example.
I also was able to add a specific label for service_name using an environment variable (OTEL_SERVICE_NAME=my-service).

I saw that all the attributes are already available in the instrumentation module during the requests, both in the "attributes" and "flask_request_environ" below:

def _wrapped_before_request(
    request_hook=None,
    tracer=None,
    excluded_urls=None,
    enable_commenter=True,
    commenter_options=None,
):
    def _before_request():
        if excluded_urls and excluded_urls.url_disabled(flask.request.url):
            return
        flask_request_environ = flask.request.environ
        span_name = get_default_span_name()

        attributes = otel_wsgi.collect_request_attributes(
            flask_request_environ
        )
        if flask.request.url_rule:
            # For 404 that result from no route found, etc, we
            # don't have a url_rule.
            attributes[SpanAttributes.HTTP_ROUTE] = flask.request.url_rule.rule
        span, token = _start_internal_or_server_span(
            tracer=tracer,
            span_name=span_name,
            start_time=flask_request_environ.get(_ENVIRON_STARTTIME_KEY),
            context_carrier=flask_request_environ,
            context_getter=otel_wsgi.wsgi_getter,
            attributes=attributes,
        )

I tried so many different things at this point and I really couldn't crack how to make this work. So my question is, would it be possible to use the already available labels and add them to the metrics being sent to Prometheus? That would help me a lot and I would be unbelievably grateful if someone could help me with that. Thank you very much!

def create_app():
    app = Flask(__name__)
    
    ...

    from opentelemetry.metrics import set_meter_provider
    from opentelemetry.sdk.metrics import MeterProvider
    from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
    from opentelemetry.exporter.prometheus_remote_write import (
    PrometheusRemoteWriteMetricsExporter,
     )
    from requests_auth_aws_sigv4 import AWSSigV4

    aws_auth = AWSSigV4('aps',
        aws_access_key_id=AWS_ACCESS_KEY_ID,
        aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
        region='eu-west-1',
    )
    from opentelemetry.sdk.resources import Resource
    exporter = PrometheusRemoteWriteMetricsExporter(
        endpoint=PROMETHEUS_REMOTE_WRITE_ENDPOINT,
        headers={},
        auth=aws_auth, 
    )

    resource = Resource.create()
    reader = PeriodicExportingMetricReader(exporter, 1000)
    provider = MeterProvider(metric_readers=[reader], resource=resource)
    set_meter_provider(provider)
    FlaskInstrumentor().instrument_app(app, meter_provider=provider)