OpenTelemetry Collector is dropping invalid metric "app_currency_counter"

[Starefossen]

Bug Report

Demo app version: v1.7.0
Collector version: 0.93.0

Symptom

OpenTelemetry Collector is dropping invalid metrics originating from the demo application.

2024-03-05T07:46:50.226Z        error   exporterhelper/common.go:95     Exporting failed. Dropping data.        {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: invalid temporality and type combination for metric \"app_currency_counter\"", "dropped_items": 1}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter.createMetricsExporter.WithQueue.func2.1
        go.opentelemetry.io/collector/exporter@v0.93.0/exporterhelper/common.go:95
go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1
        go.opentelemetry.io/collector/exporter@v0.93.0/exporterhelper/queue_sender.go:117
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue[...]).Consume
        go.opentelemetry.io/collector/exporter@v0.93.0/exporterhelper/internal/bounded_memory_queue.go:57
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*QueueConsumers[...]).Start.func1
        go.opentelemetry.io/collector/exporter@v0.93.0/exporterhelper/internal/consumers.go:43

What is the expected behavior?

Metrics should be accepted by OpenTelemetry Collector and available in Prometheus.

What is the actual behavior?

Some metrics are being dropped by the Collector.

Reproduce

Nothing. Install the demo app and point it to the collector endpoint.

Additional Context

Collector config

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: opentelemetry
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      batch: {}
      memory_limiter:
        check_interval: 5s
        limit_mib: 400
        spike_limit_mib: 100
    exporters:
      prometheusremotewrite:
        endpoint: ***
        tls:
          insecure: true
      otlp/tempo:
        endpoint: ***
        tls:
          insecure: true
      loki:
        endpoint: ***
        tls:
          insecure: true
        default_labels_enabled:
          job: false
    service:
      pipelines:
        metrics:
          receivers:
            - otlp
          processors:
            - memory_limiter
            - batch
          exporters:
            - prometheusremotewrite
        traces:
          receivers:
            - otlp
          processors:
            - memory_limiter
            - batch
          exporters:
            - otlp/tempo
        logs:
          receivers:
            - otlp
          processors:
            - memory_limiter
            - batch
          exporters:
            - loki

[Starefossen]

Might be related:

• [prometheusremotewrite] invalid temporality and type combination when remote write to thanos backend opentelemetry-collector-contrib#15281
• [prometheusremotewrite] exporter complaining about temporality in OTEL pipeline opentelemetry-collector-contrib#30094

[jmichalek132]

I tried to reproduce this locally with running docker compose up --force-recreate --remove-orphans and just small changes in config to use remote write. but I couldn't reproduce it. I also didn't see the app_currency_counter metric in prometheus, do you know by any chance which of the componets should produce it?

[puckpuck]

It looks like a recent commit inadvertently renamed it. Right now, it's called currencyservice_counter_total ... which is not what we want.

I'll write up a PR to get the name back. However this also probably confirms that the original problem for this issue no longer exists as well.

[puckpuck]

I wrote #1470 to fix the metric name problem

[Starefossen]

Awesome. Thanks for looking into this. I'll give the new version a spin and report back (probably a little delay due to KubeCon) 😅

[jmichalek132]

Do you know if the metric is sent as aggregation temporality cumulative or delta . If it's delta than prometheus remote write exporter will refuse it since prometheus never supported detlas, and translation from deltas to cumulative wasn't introduced until recently with https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor.

[puckpuck]

The metric name is now fixed, with #1470 being merged.

I also did some more digging about this issue, and this seems to of been fixed in #1335 where the metric's temporality was changed from Delta to Cummulative (default). That PR was part of the current 1.8.0 release which is also why no one is seeing the issue anymore, as this one is reported on 1.7.0.

Closing this issue as resolved.