[kube-prometheus-stack] duplicate dashboard warning when adding JSON of file dashboards

[felipeng]

Describe the bug a clear and concise description of what the bug is.

There is a conflict when adding dashboard via grafana.dashboards and defaultDashboardsEnabled: true, for example adding a dashboard via:

grafana:
  dashboards:
    default:
      evse-grpc:
        json: |
          MY_DASHBOARD_JSON

The grafana/templates/dashboards-json-configmap.yaml will create a ConfigMap and then it will mount it via grafana deployment on grafana container

However, the sidecar.dashboards will also watch the ConfigMaps with grafana_dashboard label and load to Grafana

And the following warning happens because there are two providers adding the same dashboard:

msg="dashboards provisioning provider has no database write permissions because of duplicates" provider=default orgId=1
msg="dashboards provisioning provider has no database write permissions because of duplicates" provider=sidecarProvider orgId=1
msg="the same UID is used more than once" orgId=1 uid=VyVT5qHnz times=2 providers="[default sidecarProvider]"
msg="dashboard title is not unique in folder" orgId=1 title="EVSE GRPC calls" folderID=0 times=2 providers="[default sidecarProvider]"

What's your helm version?

3.14.2

What's your kubectl version?

1.24

Which chart?

kube-prometheus-stack and grafana

What's the chart version?

55.6.0

What happened?

the same dashboard is saved on two different places and two providers are loading to grafana

What you expected to happen?

dashboard added via dashboards should be added to Grafana via sidecard.dashboard or mounted on the grafana container

How to reproduce it?

No response

Enter the changed values of values.yaml?

grafana:
  enabled: true
    dashboards:
      enabled: true
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
      - name: default
        orgId: 1
        options:
          path: /var/lib/grafana/dashboards/default
  dashboards:
    default:
      evse-grpc:
        json: |
          MY_DASHBOARD

Enter the command that you execute and failing/misfunctioning.

helm install prometheus-community/kube-prometheus-stack --version 55.6.0 --values values.yaml

Anything else we need to know?

No response

[drewcm]

I was able to resolve this for my installation by commenting out these 3 lines in charts/grafana/templates/dashboards-json-configmap.yaml:

    {{- if $.Values.sidecar.dashboards.enabled }}
    {{ $.Values.sidecar.dashboards.label }}: {{ $.Values.sidecar.dashboards.labelValue | quote }}
    {{- end }}

Prior to that change, my dashboards were showing up in both /var/lib/grafana/dashboards and /tmp/dashboards. After, they just showed up in /var/lib/grafana/dashboards

This duplicate dashboard behavior did not occur with kps v42.1.0.

The existence of the $.Values.sidecar.dashboards.label label is what tells k8s-sidecar grafana-sc-dashboard to copy the dashboards to /tmp/dashboards, resulting in them being duplicated.