OpenShift Prometheus Operator: No Metrics, Targets, or Service Discovery Displayed

[konstantinoskolios]

Description

I have installed the Prometheus Operator provided by CoreOS in our OpenShift Cluster, and while the installation and setup seemed successful, I am unable to see any metrics, targets, service discovery, or other expected information in the Prometheus UI.

Environment

• OpenShift Cluster Version: 4.13
• Kubernetes Version: 1.26.7
• Installation Method: Operator

Configuration Files
I have configured the Prometheus Operator, ServiceMonitor, PodMonitor, Deployment for an example application, and a corresponding Service as follows:

• Prometheus Configuration:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  creationTimestamp: '2024-02-05T09:22:56Z'
  generation: 8
  managedFields:
    - apiVersion: monitoring.coreos.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          .: {}
          'f:availableReplicas': {}
          'f:conditions':
            .: {}
            'k:{"type":"Available"}':
              .: {}
              'f:lastTransitionTime': {}
              'f:status': {}
              'f:type': {}
            'k:{"type":"Reconciled"}':
              .: {}
              'f:lastTransitionTime': {}
              'f:status': {}
              'f:type': {}
          'f:paused': {}
          'f:replicas': {}
          'f:shardStatuses':
            .: {}
            'k:{"shardID":"0"}':
              .: {}
              'f:availableReplicas': {}
              'f:replicas': {}
              'f:shardID': {}
              'f:unavailableReplicas': {}
              'f:updatedReplicas': {}
          'f:unavailableReplicas': {}
          'f:updatedReplicas': {}
      manager: PrometheusOperator
      operation: Update
      subresource: status
      time: '2024-02-05T14:31:36Z'
    - apiVersion: monitoring.coreos.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          .: {}
          'f:evaluationInterval': {}
          'f:portName': {}
          'f:scrapeInterval': {}
          'f:serviceMonitorNamespaceSelector': {}
          'f:serviceMonitorSelector': {}
      manager: Mozilla
      operation: Update
      time: '2024-02-05T15:03:29Z'
  name: prometheus
  namespace: namespace-dev
  resourceVersion: '75884116'
  uid: 6865b7a4-c369-4c24-a4f1-68e7d38f85bd
spec:
  evaluationInterval: 30s
  portName: web
  scrapeInterval: 30s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
status:
  availableReplicas: 1
  conditions:
    - lastTransitionTime: '2024-02-05T14:31:36Z'
      status: 'True'
      type: Available
    - lastTransitionTime: '2024-02-05T14:31:04Z'
      status: 'True'
      type: Reconciled
  paused: false
  replicas: 1
  shardStatuses:
    - availableReplicas: 1
      replicas: 1
      shardID: '0'
      unavailableReplicas: 0
      updatedReplicas: 1
  unavailableReplicas: 0
  updatedReplicas: 1

• Service Monitor Configuration:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"monitoring.coreos.com/v1","kind":"ServiceMonitor","metadata":{"annotations":{},"labels":{"team":"frontend"},"name":"example-app","namespace":"namespace-dev"},"spec":{"endpoints":[{"port":"web"}],"selector":{"matchLabels":{"app":"example-app"}}}}
  resourceVersion: '75824311'
  name: example-app
  uid: 6fa8f7c4-7555-44ee-a500-246013501cfe
  creationTimestamp: '2024-02-05T13:55:15Z'
  generation: 1
  managedFields:
    - apiVersion: monitoring.coreos.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
          'f:labels':
            .: {}
            'f:team': {}
        'f:spec':
          .: {}
          'f:endpoints': {}
          'f:selector': {}
      manager: kubectl-client-side-apply
      operation: Update
      time: '2024-02-05T13:55:15Z'
  namespace: namespace-dev
  labels:
    team: frontend
spec:
  endpoints:
    - port: web
  selector:
    matchLabels:
      app: example-app

• Pod Monitor Configuration:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"monitoring.coreos.com/v1","kind":"PodMonitor","metadata":{"annotations":{},"labels":{"team":"frontend"},"name":"example-app","namespace":"namespace-dev"},"spec":{"podMetricsEndpoints":[{"port":"web"}],"selector":{"matchLabels":{"app":"example-app"}}}}
  resourceVersion: '75824417'
  name: example-app
  uid: fe97f130-d990-4ddc-8d5c-84fb5398156a
  creationTimestamp: '2024-02-05T13:55:21Z'
  generation: 1
  managedFields:
    - apiVersion: monitoring.coreos.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
          'f:labels':
            .: {}
            'f:team': {}
        'f:spec':
          .: {}
          'f:podMetricsEndpoints': {}
          'f:selector': {}
      manager: kubectl-client-side-apply
      operation: Update
      time: '2024-02-05T13:55:21Z'
  namespace: namespace-dev
  labels:
    team: frontend
spec:
  podMetricsEndpoints:
    - port: web
  selector:
    matchLabels:
      app: example-app

• Example App Deployment:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: example-app
  namespace:namespace-dev
  uid: 1393addd-e918-4bd7-a7fe-a6b31732c25a
  resourceVersion: '75824344'
  generation: 1
  creationTimestamp: '2024-02-05T13:54:57Z'
  annotations:
    deployment.kubernetes.io/revision: '1'
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"example-app","namespace":"namespace-dev"},"spec":{"replicas":3,"selector":{"matchLabels":{"app":"example-app"}},"template":{"metadata":{"labels":{"app":"example-app"}},"spec":{"containers":[{"image":"fabxc/instrumented_app","name":"example-app","ports":[{"containerPort":8080,"name":"web"}]}]}}}}
  managedFields:
    - manager: kubectl-client-side-apply
      operation: Update
      apiVersion: apps/v1
      time: '2024-02-05T13:54:57Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
        'f:spec':
          'f:progressDeadlineSeconds': {}
          'f:replicas': {}
          'f:revisionHistoryLimit': {}
          'f:selector': {}
          'f:strategy':
            'f:rollingUpdate':
              .: {}
              'f:maxSurge': {}
              'f:maxUnavailable': {}
            'f:type': {}
          'f:template':
            'f:metadata':
              'f:labels':
                .: {}
                'f:app': {}
            'f:spec':
              'f:containers':
                'k:{"name":"example-app"}':
                  .: {}
                  'f:image': {}
                  'f:imagePullPolicy': {}
                  'f:name': {}
                  'f:ports':
                    .: {}
                    'k:{"containerPort":8080,"protocol":"TCP"}':
                      .: {}
                      'f:containerPort': {}
                      'f:name': {}
                      'f:protocol': {}
                  'f:resources': {}
                  'f:terminationMessagePath': {}
                  'f:terminationMessagePolicy': {}
              'f:dnsPolicy': {}
              'f:restartPolicy': {}
              'f:schedulerName': {}
              'f:securityContext': {}
              'f:terminationGracePeriodSeconds': {}
    - manager: kube-controller-manager
      operation: Update
      apiVersion: apps/v1
      time: '2024-02-05T13:55:16Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:deployment.kubernetes.io/revision': {}
        'f:status':
          'f:availableReplicas': {}
          'f:conditions':
            .: {}
            'k:{"type":"Available"}':
              .: {}
              'f:lastTransitionTime': {}
              'f:lastUpdateTime': {}
              'f:message': {}
              'f:reason': {}
              'f:status': {}
              'f:type': {}
            'k:{"type":"Progressing"}':
              .: {}
              'f:lastTransitionTime': {}
              'f:lastUpdateTime': {}
              'f:message': {}
              'f:reason': {}
              'f:status': {}
              'f:type': {}
          'f:observedGeneration': {}
          'f:readyReplicas': {}
          'f:replicas': {}
          'f:updatedReplicas': {}
      subresource: status
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: example-app
    spec:
      containers:
        - name: example-app
          image: fabxc/instrumented_app
          ports:
            - name: web
              containerPort: 8080
              protocol: TCP
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: Always
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
status:
  observedGeneration: 1
  replicas: 3
  updatedReplicas: 3
  readyReplicas: 3
  availableReplicas: 3
  conditions:
    - type: Available
      status: 'True'
      lastUpdateTime: '2024-02-05T13:55:16Z'
      lastTransitionTime: '2024-02-05T13:55:16Z'
      reason: MinimumReplicasAvailable
      message: Deployment has minimum availability.
    - type: Progressing
      status: 'True'
      lastUpdateTime: '2024-02-05T13:55:16Z'
      lastTransitionTime: '2024-02-05T13:54:57Z'
      reason: NewReplicaSetAvailable
      message: ReplicaSet "example-app-796f6d7c65" has successfully progressed.

• Example App Service:

kind: Service
apiVersion: v1
metadata:
  name: example-app
  namespace: namespace-dev
  uid: a8ced819-804c-40a4-9ea9-8df331c034e3
  resourceVersion: '75824214'
  creationTimestamp: '2024-02-05T13:55:09Z'
  labels:
    app: example-app
    namespace: namespace-dev
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"example-app","namespace":"namespace-dev"},"name":"example-app","namespace":"namespace-dev"},"spec":{"ports":[{"name":"web","port":8080}],"selector":{"app":"example-app"}}}
  managedFields:
    - manager: kubectl-client-side-apply
      operation: Update
      apiVersion: v1
      time: '2024-02-05T13:55:09Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
          'f:labels':
            .: {}
            'f:app': {}
            'f:namespace': {}
        'f:spec':
          'f:internalTrafficPolicy': {}
          'f:ports':
            .: {}
            'k:{"port":8080,"protocol":"TCP"}':
              .: {}
              'f:name': {}
              'f:port': {}
              'f:protocol': {}
              'f:targetPort': {}
          'f:selector': {}
          'f:sessionAffinity': {}
          'f:type': {}
spec:
  clusterIP: 172.30.94.104
  ipFamilies:
    - IPv4
  ports:
    - name: web
      protocol: TCP
      port: 8080
      targetPort: 8080
  internalTrafficPolicy: Cluster
  clusterIPs:
    - 172.30.94.104
  type: ClusterIP
  ipFamilyPolicy: SingleStack
  sessionAffinity: None
  selector:
    app: example-app
status:
  loadBalancer: {}

• Network Policy:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-all
  namespace: namespace-dev
  uid: 41e45d6f-9226-4659-9999-2443984bdf96
  resourceVersion: '75676249'
  generation: 1
  creationTimestamp: '2024-02-05T11:05:43Z'
  managedFields:
    - manager: Mozilla
      operation: Update
      apiVersion: networking.k8s.io/v1
      time: '2024-02-05T11:05:43Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          'f:egress': {}
          'f:ingress': {}
          'f:policyTypes': {}
spec:
  podSelector: {}
  ingress:
    - {}
  egress:
    - {}
  policyTypes:
    - Ingress
    - Egress
status: {}

The Prometheus configuration can be found under Status > Configurations (via the Prometheus UI):

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4
  evaluation_interval: 30s
  external_labels:
    prometheus: namespace-dev/prometheus
    prometheus_replica: prometheus-prometheus-0
scrape_configs:
- job_name: serviceMonitor/namespace-dev/example-app/0
  honor_timestamps: true
  track_timestamps_staleness: false
  scrape_interval: 30s
  scrape_timeout: 10s
  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4
  metrics_path: /metrics
  scheme: http
  enable_compression: true
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (example-app);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: web
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: web
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - namespace-dev
- job_name: serviceMonitor/namespace-dev/prometheus-exporter-operator-controller-manager-metrics-monitor/0
  honor_timestamps: true
  track_timestamps_staleness: false
  scrape_interval: 30s
  scrape_timeout: 10s
  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4
  metrics_path: /metrics
  scheme: http
  enable_compression: true
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_control_plane, __meta_kubernetes_service_labelpresent_control_plane]
    separator: ;
    regex: (controller-manager);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - namespace-dev
- job_name: serviceMonitor/namespace-dev/prometheus-operator/0
  honor_timestamps: true
  track_timestamps_staleness: false
  scrape_interval: 30s
  scrape_timeout: 10s
  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4
  metrics_path: /metrics
  scheme: http
  enable_compression: true
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_k8s_app, __meta_kubernetes_service_labelpresent_k8s_app]
    separator: ;
    regex: (prometheus-operator);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - namespace-dev

[konstantinoskolios]

The problem was i didn't have the necessary permissions to view the metrics :)!