Cannot add the external nodes as targets to prometheus-operator in kubernetes

[bharathpantala]

Cannot add the external nodes as targets to prometheus-operator in kubernetes

My prom release on k8s is : prometheus-operator

Followed the same https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/additional-scrape-config.md

Can someone give some doc to add external node-exporter as targets to prometheus-operator in kubernetes.

[slashpai]

May be there are better ways but I tested this way in local cluster and it works. We will create the the service without selector and then define the endpoints separately. And then create servicemonitor. Treat this as example and tweak settings as per your need :)

1. Create service object for external node-exporter with selector

kind: Service
apiVersion: v1
metadata:
 name: node-exporter-external
 labels:
    app: node-exporter-external
spec:
 type: ClusterIP
 ports:
 - name: metrics
   port: 9100
   targetPort: 9100

2. Create endpoint object separately

kind: Endpoints
apiVersion: v1
metadata:
 name: node-exporter-external
subsets:
- addresses:
  - ip: <node external ip>
  ports:
  - name: metrics
    port: 9100
    protocol: TCP

3. Create servicemonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: node-exporter-external
  name: node-exporter-external
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    port: metrics
  namespaceSelector:
    matchNames:
    - monitoring
  selector:
    matchLabels:
      app: node-exporter-external

[ikozar]

The solution doesn't work in my case.
I want to add to prometheus additional node-exported from external host.
I found value "endpoints" in "charts/prometheus-node-exporter/values.yaml", but it also doesn't work

I was able to add an external node export after applying the following endpoint

apiVersion: v1
kind: Endpoints
metadata:
  name: my-prometeus-prometheus-node-exporter
  namespace: monitoring
  labels:
    app: prometheus-node-exporter
    heritage: Helm
    release: my-prometeus
    chart: prometheus-node-exporter-3.3.0
subsets:
  - addresses:
      - ip: <minikube node IP>
      - ip: <external IP>
    ports:
      - name: http-metrics
        port: 9100
        protocol: TCP

prometheus discovered external exporter and queried metrics
but, unfortunately, after some time the external IP was lost and getting metrics from external node was stopped

[xiu]

Since prometheus-operator 0.65, we have the ScrapeConfig CRD. With it, you currently can define targets via the static_config, file_sd_config and http_sd_config parameters. Here's an example with static_config:

apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
  name: my-scrape-config
spec:
  staticConfigs:
  - targets:
    - "10.x.y.z:1234"

Documentation: https://prometheus-operator.dev/docs/operator/api/#monitoring.coreos.com/v1alpha1.ScrapeConfig

[hpruszyn]

@xiu do I need any additional config for the ScrapeConfig CRD resource to work? I'm trying to configure it for some external targets but it is not showing in promethus.

I'm using the kube prometheus helm chart. And I've also used scrapeConfigSelectorNilUsesHelmValues: false like for other CRDs. Chart version is 50.2.0.

I can see that in Promethus CRD no scrapeConfigSelector attribute is created so probably there is a bug.

[slashpai]

Try setting scrapeConfigSelector: {}, note empty label selector will select all objects

[hpruszyn]

I’ve tried it already but without success.

[slashpai]

are you using latest CRD version? If not update to latest CRD version, ScrapeConfig is supported from v0.65 version of operator so you need to use corresponding helm chart which use this version or above.

Check prometheus-operator logs incase if you find some issues there, if configs are wrong scrapeconfig wont be selected

[hpruszyn]

Yes. kube-prometheus-stack v50.2.0 (latest) with operator in version 0.67.1. Logs are empty.

I think that the operator does not discover the ScrapeConfig and it is related to some problems with auto created Promethus configuration. Promethus (CRD) doesn't have any configuration starting with scrapeConfig*, however other related to pod, service, rule monitoring are available:

➜  ~ kubectl get Prometheus -n monitoring -o yaml
apiVersion: v1
items:
- apiVersion: monitoring.coreos.com/v1
  kind: Prometheus
  metadata:
    annotations:
      meta.helm.sh/release-name: kube-prometheus-stack
      meta.helm.sh/release-namespace: monitoring
    creationTimestamp: "2023-09-04T16:00:50Z"
    generation: 1
    labels:
      app: kube-prometheus-stack-prometheus
      app.kubernetes.io/instance: kube-prometheus-stack
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/part-of: kube-prometheus-stack
      app.kubernetes.io/version: 50.2.0
      chart: kube-prometheus-stack-50.2.0
      heritage: Helm
      release: kube-prometheus-stack
    name: kube-prometheus-stack-prometheus
    namespace: monitoring
    resourceVersion: "425106521"
    uid: 80233654-642d-462f-b475-9c4e46e45be4
  spec:
    affinity: [...]
    alerting: [...]
    enableAdminAPI: false
    evaluationInterval: 30s
    externalUrl: http://kube-prometheus-stack-prometheus.monitoring:9090
    image: quay.io/prometheus/prometheus:v2.46.0
    listenLocal: false
    logFormat: logfmt
    logLevel: info
    paused: false
    podMonitorNamespaceSelector: {}
    podMonitorSelector: {}
    portName: http-web
    probeNamespaceSelector: {}
    probeSelector: {}
    replicas: 1
    resources: [...]
    retention: 7d
    routePrefix: /
    ruleNamespaceSelector: {}
    ruleSelector: {}
    scrapeInterval: 30s
    securityContext: [...]
    serviceAccountName: [...]
    serviceMonitorNamespaceSelector: {}
    serviceMonitorSelector: {}
    shards: 1
    storage: [...]
    version: v2.46.0
    walCompression: true
kind: List
metadata:
  resourceVersion: ""

[slashpai]

Looks like you need to install latest CRD's, I am not sure how helm chart installs them.
https://github.com/prometheus-operator/prometheus-operator/tree/release-0.67/example/prometheus-operator-crd-full

[hpruszyn]

CRD refresh did the trick. Thank you very much @slashpai !

[slashpai]

Closing this discussion since ScrapeConfig is available now for this
https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/scrapeconfig.md