[kube-prometheus-stack] Support for import from jsonnet mixin for etcd (

#2340)

This is needed as the source of truth for etcd alerts and dashboards
is no longer maintained in https://github.com/etcd-io/website. See
https://github.com/etcd-io/etcd/blob/main/contrib/mixin/mixin.libsonnet

Signed-off-by: Sergei Kuzmin <sergeikuzmin@gmail.com>

charts/kube-prometheus-stack/Chart.yaml

@@ -22,7 +22,7 @@ name: kube-prometheus-stack
 sources:
   - https://github.com/prometheus-community/helm-charts
   - https://github.com/prometheus-operator/kube-prometheus
-version: 39.4.0
+version: 39.4.1
 appVersion: 0.58.0
 kubeVersion: ">=1.16.0-0"
 home: https://github.com/prometheus-operator/kube-prometheus

charts/kube-prometheus-stack/hack/README.md

@@ -6,7 +6,7 @@ This script generates prometheus rules set for alertmanager from any properly fo
 
 Currently following imported:
 
-- [prometheus-operator/kube-prometheus rules set](https://github.com/prometheus-operator/kube-prometheus/tree/main/manifests/kubernetes-prometheusRule.yaml)
+- [prometheus-operator/kube-prometheus rules set](https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/kubernetesControlPlane-prometheusRule.yaml)
   - In order to modify these rules:
     - prepare and merge PR into [kubernetes-mixin](https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/rules) master and/or release branch
     - run import inside your fork of [prometheus-operator/kube-prometheus](https://github.com/prometheus-operator/kube-prometheus/tree/main)
@@ -17,13 +17,13 @@ Currently following imported:
      ```
 
     - prepare and merge PR with imported changes into `prometheus-operator/kube-prometheus` master and/or release branch
-    - run sync_prometheus_rules.py inside your fork of this repo
-    - send PR with changes to this repo
-- [etcd-io/website rules set](https://github.com/etcd-io/website/tree/master/content/docs/v3.4.0/etcd-mixin/README.md)
+    - run sync_prometheus_rules.py inside your fork of this repository
+    - send PR with changes to this repository
+- [etcd-io/etcd rules set](https://github.com/etcd-io/etcd/blob/main/contrib/mixin/mixin.libsonnet).
   - In order to modify these rules:
-    - prepare and merge PR into [etcd-io/website](https://github.com/etcd-io/website/blob/master/content/docs/v3.4.0/etcd-mixin/mixin.libsonnet) repo
-    - run sync_prometheus_rules.py inside your fork of this repo
-    - send PR with changes to this repo
+    - prepare and merge PR into [etcd-io/etcd](https://github.com/etcd-io/etcd/blob/main/contrib/mixin/mixin.libsonnet) repository
+    - run sync_prometheus_rules.py inside your fork of this repository
+    - send PR with changes to this repository
 
 ## [sync_grafana_dashboards.py](sync_grafana_dashboards.py)
 
@@ -42,12 +42,17 @@ Currently following imported:
      ```
 
     - prepare and merge PR with imported changes into `prometheus-operator/kube-prometheus` master and/or release branch
-    - run sync_grafana_dashboards.py inside your fork of this repo
-    - send PR with changes to this repo
-- [etcd-io/website dashboard](https://github.com/etcd-io/website/blob/master/content/docs/v3.4.0/op-guide/grafana.json)
+    - run sync_grafana_dashboards.py inside your fork of this repository
+    - send PR with changes to this repository
+
+<!-- textlint-disable -->
+
+- [etcd-io/website dashboard](https://github.com/etcd-io/etcd/blob/main/contrib/mixin/mixin.libsonnet)
   - In order to modify this dashboard:
-    - prepare and merge PR into [etcd-io/website](https://github.com/etcd-io/website/blob/master/content/docs/v3.4.0/op-guide/grafana.json) repo
-    - run sync_grafana_dashboards.py inside your fork of this repo
-    - send PR with changes to this repo
+    - prepare and merge PR into [etcd-io/etcd](https://github.com/etcd-io/etcd/blob/main/contrib/mixin/mixin.libsonnet) repository
+    - run sync_grafana_dashboards.py inside your fork of this repository
+    - send PR with changes to this repository
+
+<!-- textlint-enable -->
 
-[CoreDNS dashboard](https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/k8s-coredns.yaml) is the only dashboard which is maintained in this repo and can be changed without import.
+[CoreDNS dashboard](https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/k8s-coredns.yaml) is the only dashboard which is maintained in this repository and can be changed without import.

charts/kube-prometheus-stack/hack/sync_grafana_dashboards.py

@@ -5,6 +5,7 @@
 import textwrap
 from os import makedirs, path
 
+import _jsonnet
 import requests
 import yaml
 from yaml.representer import SafeRepresenter
@@ -34,9 +35,9 @@ def new_representer(dumper, data):
         'multicluster_key': '.Values.grafana.sidecar.dashboards.multicluster.global.enabled',
     },
     {
-        'source': 'https://raw.githubusercontent.com/etcd-io/website/master/content/en/docs/v3.5/op-guide/grafana.json',
+        'source': 'https://raw.githubusercontent.com/etcd-io/etcd/main/contrib/mixin/mixin.libsonnet',
         'destination': '../templates/grafana/dashboards-1.14',
-        'type': 'json',
+        'type': 'jsonnet_mixin',
         'min_kubernetes': '1.14.0-0',
         'multicluster_key': '(or .Values.grafana.sidecar.dashboards.multicluster.global.enabled .Values.grafana.sidecar.dashboards.multicluster.etcd.enabled)'
     },
@@ -216,8 +217,8 @@ def main():
             for group in groups:
                 for resource, content in group['data'].items():
                     write_group_to_file(resource.replace('.json', ''), content, chart['source'], chart['destination'], chart['min_kubernetes'], chart['max_kubernetes'], chart['multicluster_key'])
-        elif chart['type'] == 'json':
-            json_text = json.loads(raw_text)
+        elif chart['type'] == 'jsonnet_mixin':
+            json_text = json.loads(_jsonnet.evaluate_snippet(chart['source'], raw_text + '.grafanaDashboards'))
             # is it already a dashboard structure or is it nested (etcd case)?
             flat_structure = bool(json_text.get('annotations'))
             if flat_structure:

charts/kube-prometheus-stack/hack/sync_prometheus_rules.py

@@ -1,9 +1,11 @@
 #!/usr/bin/env python3
 """Fetch alerting and aggregation rules from provided urls into this chart."""
+import json
 import re
 import textwrap
 from os import makedirs
 
+import _jsonnet
 import requests
 import yaml
 from yaml.representer import SafeRepresenter
@@ -61,9 +63,10 @@ def new_representer(dumper, data):
         'min_kubernetes': '1.14.0-0'
     },
     {
-        'source': 'https://raw.githubusercontent.com/etcd-io/website/master/content/en/docs/v3.4/op-guide/etcd3_alert.rules.yml',
+        'source': 'https://raw.githubusercontent.com/etcd-io/etcd/main/contrib/mixin/mixin.libsonnet',
         'destination': '../templates/prometheus/rules-1.14',
-        'min_kubernetes': '1.14.0-0'
+        'min_kubernetes': '1.14.0-0',
+        'is_mixin': True
     },
 ]
 
@@ -368,13 +371,16 @@ def main():
             print('Skipping the file, response code %s not equals 200' % response.status_code)
             continue
         raw_text = response.text
-        yaml_text = yaml.full_load(raw_text)
+        if chart.get('is_mixin'):
+            alerts = json.loads(_jsonnet.evaluate_snippet(chart['source'], raw_text + '.prometheusAlerts'))
+        else:
+            alerts = yaml.full_load(raw_text)
 
         if ('max_kubernetes' not in chart):
             chart['max_kubernetes']="9.9.9-9"
 
         # etcd workaround, their file don't have spec level
-        groups = yaml_text['spec']['groups'] if yaml_text.get('spec') else yaml_text['groups']
+        groups = alerts['spec']['groups'] if alerts.get('spec') else alerts['groups']
         for group in groups:
             write_group_to_file(group, chart['source'], chart['destination'], chart['min_kubernetes'], chart['max_kubernetes'])