How to debug ScrapeConfig creation

[LeoQuote]

I've created ScrapeConfig with

apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
  labels:
    app.kubernetes.io/managed-by: Helm
  name: vault
spec:
  httpSDConfigs:
  - url: https://my-sd/api/prometheus-service-discovery/?tags=vault&job=vault
  metricsPath: /v1/sys/metrics

but no targets nor config about it was found in prometheus config.

I assure that there's no prometheus reload failed found, and I saw log in prometheus operator

level=info ts=2023-12-18T10:50:26.091561053Z caller=operator.go:975 component=prometheusoperator key=monitoring/kube-prom-kube-prometheus-prometheus msg="sync prometheus"

and logs in prometheus pod

ts=2023-12-18T10:50:56.109Z caller=main.go:1267 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml totalDuration=178.264749ms db_storage=1.512µs remote_storage=122.98µs web_handler=471ns query_engine=812ns scrape=10.717142ms scrape_sd=5.545892ms notify=12.724µs notify_sd=350.486µs rules=136.849586ms tracing=7.895µs

but no config about it found in configuration

basic info:

prometheus-operator version: v0.70.0

prometheus version: v2.48.1

scrapeConfigSelector: {}

[slashpai]

I am not familiar with helm chart how its been configured, a quick question has CRDs been updated too along with installing new prometheus-operator?

[LeoQuote]

yes, I installed the CRDs with kubectl in example folder in this very project

[slashpai]

Are prometheus pod and scrape config created in same namespace?

[LeoQuote]

No, but I don’t think this is relevant, I can try to apply ScrapeConfig in the same namespace with Prometheus operator and Prometheus instance.

I have set scrapeConfigNamespaceSelector and scrapeConfigSelector to {} , meaning any ScrapeConfig in any namespace would be discovered, podMonitor and serviceMonitor can be discovered correctly.

[LeoQuote]

maybe Things could be simpler after #6030

[slashpai]

In past we had similar question on servicemonitor something to do how helm values are set
#5917 (comment)

Is there something similar for scrape config as well?

[LeoQuote]

Thanks for the issue you provided!

I also noticed about this helm value and I did set it to false

    scrapeConfigSelectorNilUsesHelmValues: false
    scrapeConfigSelector: {}
    scrapeConfigNamespaceSelector: {}

I just checked the Prometheus resource in monitoring namespace and found the scrapeConfigNamespaceSelector spec is unset , which would make ScrapeConfig in other namespace discoverable, I set it manually by kubectl edit, after that, ScrapeConfig can be discovered correctly.

I thought it would be working since it's identical with podmonitor , etc...

turns out there could be some bugs in helm chart, I'll try to submit a fix pr on helm chart if no one has done so.

[slashpai]

Glad that helped 🙂

[LeoQuote]

Thanks @slashpai for the help, the debug process would be:

1. try not to use helm, check the Prometheus resource directly, check the scrapeConfigNamespaceSelector and scrapeConfigSelector to see if they are all set properly.
2. check the prometheus and prometheus operator log to see if there's any errors.
3. wait for feat: emit events for invalid configurations #6030 to merge , after that, there would be more info in event section (hopefully.)

reference:

• prometheus operator api: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#scrapeconfigspec
• helm-chart readme: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

[slashpai]

I think it would be worth adding a documentation on helm chart repo since it's pretty confusing and difficult to debug

[LeoQuote]

You're right, I've added a few references