Triplicated metrics in Grafana

[jdsanclemente]

Is your question request related to a specific component?

VictoriaMetrics, Grafana

Describe the question in detail

I'm encountering this problem when using VictoriaMetrics as a datasource in graphana: The same metric is being triplicated, showing almost the same values:

This is the JSON code:

{
  "id": 4,
  "gridPos": {
    "h": 6,
    "w": 8,
    "x": 0,
    "y": 26
  },
  "type": "timeseries",
  "title": "RA04 WS sessions ",
  "datasource": {
    "uid": "pz0kQ7OVk",
    "type": "prometheus"
  },
  "fieldConfig": {
    "defaults": {
      "custom": {
        "drawStyle": "line",
        "lineInterpolation": "linear",
        "barAlignment": 0,
        "lineWidth": 1,
        "fillOpacity": 0,
        "gradientMode": "none",
        "spanNulls": false,
        "showPoints": "auto",
        "pointSize": 4,
        "stacking": {
          "mode": "none",
          "group": "A"
        },
        "axisPlacement": "auto",
        "axisLabel": "",
        "axisColorMode": "text",
        "scaleDistribution": {
          "type": "linear"
        },
        "axisCenteredZero": false,
        "hideFrom": {
          "tooltip": false,
          "viz": false,
          "legend": false
        },
        "thresholdsStyle": {
          "mode": "off"
        },
        "lineStyle": {
          "fill": "solid"
        }
      },
      "color": {
        "mode": "palette-classic"
      },
      "mappings": [],
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {
            "color": "green",
            "value": null
          },
          {
            "color": "red",
            "value": 80
          }
        ]
      }
    },
    "overrides": []
  },
  "options": {
    "tooltip": {
      "mode": "single",
      "sort": "none"
    },
    "legend": {
      "showLegend": true,
      "displayMode": "list",
      "placement": "bottom",
      "calcs": []
    }
  },
  "targets": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "pz0kQ7OVk"
      },
      "editorMode": "code",
      "expr": "ra_server_sessions_active{job=~\"raejob04\"}",
      "legendFormat": "{{__name__}}",
      "range": true,
      "refId": "A"
    }
  ]
}

Any idea on how to fix this? Any help would be greatly appreciated, thank you.

Troubleshooting docs

• General - https://docs.victoriametrics.com/troubleshooting/
• vmagent - https://docs.victoriametrics.com/vmagent/#troubleshooting
• vmalert - https://docs.victoriametrics.com/vmalert/#troubleshooting

[xiaozongyang]

hi @jdsanclemente what's your original data in http network response? you may find it by your browser development tools

[zekker6]

Hello @jdsanclemente
Could you elaborate on what is your write path configuration?
It seems like these metrics can either be scraped by mulitiple vmagent instances (for example running in HA mode) or it metrics could be pushed by different application instances without instance label attached.
The fix in the first case is to enable deduplication. In the second case it is needed to ensure that instance label will be present for ingested series to distinguish the source of metrics.

[jdsanclemente]

Hello @jdsanclemente Could you elaborate on what is your write path configuration? It seems like these metrics can either be scraped by mulitiple vmagent instances (for example running in HA mode) or it metrics could be pushed by different application instances without instance label attached. The fix in the first case is to enable deduplication. In the second case it is needed to ensure that instance label will be present for ingested series to distinguish the source of metrics.

The query inspector tells me that these metrics are coming from three diferrent pods:

{
  "request": {
    "url": "api/ds/query",
    "method": "POST",
    "data": {
  "response": {
    "results": {
      "A": {
        "frames": [
          {
                  "name": "Value",
                  "type": "number",
                  "typeInfo": {
                    "frame": "float64"
                  },
                  "labels": {
                    "__name__": "rae_server_sessions_active",
                    "container": "raestatus",
                    "endpoint": "http",
                    "instance": "10.75.22.45:80",
                    "job": "rae02",
                    "namespace": "raestatus",
                    "pod": "raestatus-64d757659f-4l4zw",
                    "prometheus": "monitoring/kube-prometheus-stack-prometheus",
                    "prometheus_replica": "prometheus-kube-prometheus-stack-prometheus-0",
                    "service": "raestatus"
                  },
                  "config": {
                    "displayNameFromDS": "rae_server_sessions_active"
                  }
                }
              ]
 
                  },
                  "labels": {
                    "__name__": "rae_server_sessions_active",
                    "container": "raestatus",
                    "endpoint": "http",
                    "instance": "10.75.22.45:80",
                    "job": "rae02",
                    "namespace": "raestatus",
                    "pod": "raestatus-64d757659f-4l4zw",
                    "prometheus": "monitoring/kube-prometheus-stack-prometheus",
                    "prometheus_replica": "prometheus-kube-prometheus-stack-prometheus-1",
                    "service": "raestatus"
                  },
                  "config": {
                    "displayNameFromDS": "rae_server_sessions_active"
                  }

                  },
                  "labels": {
                    "__name__": "rae_server_sessions_active",
                    "container": "raestatus",
                    "endpoint": "http",
                    "instance": "10.75.22.45:80",
                    "job": "rae02",
                    "namespace": "raestatus",
                    "pod": "raestatus-64d757659f-4l4zw",
                    "prometheus": "monitoring/kube-prometheus-stack-prometheus",
                    "prometheus_replica": "prometheus-kube-prometheus-stack-prometheus-2",
                    "service": "raestatus"
                  },
                  "config": {
                    "displayNameFromDS": "rae_server_sessions_active"
                  }
                }

This is certainly the reason, but how to fix it?

[zekker6]

@jdsanclemente In order to fix that you need to:

1. Remove prometheus_replica label by using relabeling
2. Enable deduplication to enforce storing of 1 copy of each sample scraped by Prometheus.