Reconciliation Loop in the 0.73.0 release on KIND environments

[diranged]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

Description

(moved from #6284 (comment))

In the 0.73.0 release, if I run the operator in a Kind environment (1.26.6, and 1.29.2 both tested) I get a constant loop where the operator keeps killing and resyncing the AlertManagers and ThanosRulers:

level=info ts=2024-04-06T00:01:17.533270048Z caller=operator.go:471 component=thanos-controller key=observability/default msg="sync thanos-ruler"
level=info ts=2024-04-06T00:01:17.614336673Z caller=operator.go:766 component=prometheus-controller key=observability/scraper msg="sync prometheus"
level=warn ts=2024-04-06T00:01:17.629025757Z caller=klog.go:106 component=k8s_client_runtime func=Warning msg="spec.template.spec.containers[1].ports[0]: duplicate port definition with spec.template.spec.initContainers[0].ports[0]"
level=info ts=2024-04-06T00:01:17.629379048Z caller=operator.go:572 component=alertmanager-controller key=observability/alertmanager msg="sync alertmanager"
level=info ts=2024-04-06T00:01:17.71029034Z caller=operator.go:471 component=thanos-controller key=observability/default msg="sync thanos-ruler"
level=warn ts=2024-04-06T00:01:17.738319007Z caller=klog.go:106 component=k8s_client_runtime func=Warning msg="spec.template.spec.containers[1].ports[0]: duplicate port definition with spec.template.spec.initContainers[0].ports[0]"
level=info ts=2024-04-06T00:01:17.741981132Z caller=operator.go:572 component=alertmanager-controller key=observability/alertmanager msg="sync alertmanager"
level=warn ts=2024-04-06T00:01:17.768118048Z caller=promcfg.go:1374 component=prometheus-controller msg="'bearerTokenFile' is deprecated, use 'authorization' instead." version=v2.51.1 service_monitor=observability/alertmanager-kube-promethe-coredns
level=warn ts=2024-04-06T00:01:17.768276923Z caller=promcfg.go:1374 component=prometheus-controller msg="'bearerTokenFile' is deprecated, use 'authorization' instead." version=v2.51.1 service_monitor=observability/prometheus-kube-prometheus-apiserver
level=warn ts=2024-04-06T00:01:17.768290298Z caller=promcfg.go:1374 component=prometheus-controller msg="'bearerTokenFile' is deprecated, use 'authorization' instead." version=v2.51.1 service_monitor=observability/prometheus-kube-prometheus-coredns
level=warn ts=2024-04-06T00:01:17.768301673Z caller=promcfg.go:1374 component=prometheus-controller msg="'bearerTokenFile' is deprecated, use 'authorization' instead." version=v2.51.1 service_monitor=observability/prometheus-kube-prometheus-kube-controller-manager
level=warn ts=2024-04-06T00:01:17.76831184Z caller=promcfg.go:1374 component=prometheus-controller msg="'bearerTokenFile' is deprecated, use 'authorization' instead." version=v2.51.1 service_monitor=observability/prometheus-kube-prometheus-kube-etcd
level=warn ts=2024-04-06T00:01:17.768322382Z caller=promcfg.go:1374 component=prometheus-controller msg="'bearerTokenFile' is deprecated, use 'authorization' instead." version=v2.51.1 service_monitor=observability/prometheus-kube-prometheus-kube-proxy
level=warn ts=2024-04-06T00:01:17.768333298Z caller=promcfg.go:1374 component=prometheus-controller msg="'bearerTokenFile' is deprecated, use 'authorization' instead." version=v2.51.1 service_monitor=observability/prometheus-kube-prometheus-kube-scheduler
level=warn ts=2024-04-06T00:01:17.826784257Z caller=klog.go:106 component=k8s_client_runtime func=Warning msg="spec.template.spec.containers[1].ports[0]: duplicate port definition with spec.template.spec.initContainers[0].ports[0]"
level=info ts=2024-04-06T00:01:17.831138965Z caller=operator.go:766 component=prometheus-controller key=observability/scraper msg="sync prometheus"
level=warn ts=2024-04-06T00:01:17.863252798Z caller=klog.go:106 component=k8s_client_runtime func=Warning msg="spec.template.spec.containers[1].ports[0]: duplicate port definition with spec.template.spec.initContainers[0].ports[0]"
level=info ts=2024-04-06T00:01:17.863601923Z caller=operator.go:572 component=alertmanager-controller key=observability/alertmanager msg="sync alertmanager"
level=info ts=2024-04-06T00:01:17.910792007Z caller=operator.go:471 component=thanos-controller key=observability/default msg="sync thanos-ruler"
level=warn ts=2024-04-06T00:01:18.008474923Z caller=klog.go:106 component=k8s_client_runtime func=Warning msg="spec.template.spec.containers[1].ports[0]: duplicate port definition with spec.template.spec.initContainers[0].ports[0]"
level=info ts=2024-04-06T00:01:18.009143465Z caller=operator.go:572 component=alertmanager-controller key=observability/alertmanager msg="sync alertmanager"
level=warn ts=2024-04-06T00:01:18.043482424Z caller=promcfg.go:1374 component=prometheus-controller msg="'bearerTokenFile' is deprecated, use 'authorization' instead." version=v2.51.1 service_monitor=observability/alertmanager-kube-promethe-coredns
level=warn ts=2024-04-06T00:01:18.043598215Z caller=promcfg.go:1374 component=prometheus-controller msg="'bearerTokenFile' is deprecated, use 'authorization' instead." version=v2.51.1 service_monitor=observability/prometheus-kube-prometheus-apiserver
level=warn ts=2024-04-06T00:01:18.04361659Z caller=promcfg.go:1374 component=prometheus-controller msg="'bearerTokenFile' is deprecated, use 'authorization' instead." version=v2.51.1 service_monitor=observability/prometheus-kube-prometheus-coredns
level=warn ts=2024-04-06T00:01:18.043627507Z caller=promcfg.go:1374 component=prometheus-controller msg="'bearerTokenFile' is deprecated, use 'authorization' instead." version=v2.51.1 service_monitor=observability/prometheus-kube-prometheus-kube-controller-manager
level=warn ts=2024-04-06T00:01:18.043645965Z caller=promcfg.go:1374 component=prometheus-controller msg="'bearerTokenFile' is deprecated, use 'authorization' instead." version=v2.51.1 service_monitor=observability/prometheus-kube-prometheus-kube-etcd
level=warn ts=2024-04-06T00:01:18.043658674Z caller=promcfg.go:1374 component=prometheus-controller msg="'bearerTokenFile' is deprecated, use 'authorization' instead." version=v2.51.1 service_monitor=observability/prometheus-kube-prometheus-kube-proxy
level=warn ts=2024-04-06T00:01:18.043670382Z caller=promcfg.go:1374 component=prometheus-controller msg="'bearerTokenFile' is deprecated, use 'authorization' instead." version=v2.51.1 service_monitor=observability/prometheus-kube-prometheus-kube-scheduler
level=warn ts=2024-04-06T00:01:18.158803007Z caller=klog.go:106 component=k8s_client_runtime func=Warning msg="spec.template.spec.containers[1].ports[0]: duplicate port definition with spec.template.spec.initContainers[0].ports[0]"
level=info ts=2024-04-06T00:01:18.15981009Z caller=operator.go:572 component=alertmanager-controller key=observability/alertmanager msg="sync alertmanager"
level=warn ts=2024-04-06T00:01:18.187363549Z caller=klog.go:106 component=k8s_client_runtime func=Warning msg="spec.template.spec.containers[1].ports[0]: duplicate port definition with spec.template.spec.initContainers[0].ports[0]"
level=info ts=2024-04-06T00:01:18.187736132Z caller=operator.go:766 component=prometheus-controller key=observability/scraper msg="sync prometheus"

If I back out to the 0.72.0 release, it works fine. I've also tried building the images locally on my laptop and I get the same behavior ... building 0.72.0 works, but anything newer fails.

Steps to Reproduce

Install the prometheus-operator on a local KIND environment, create an AlertManager and ThanosRuler object, and watch.

Expected Result

It should work as it did in the 0.72.0 release - reconciling only when appropriate.

Actual Result

Reconciliation loop.

Prometheus Operator Version

0.73.0

Kubernetes Version

1.26.6, 1.29.2

Kubernetes Cluster Type

kind

How did you deploy Prometheus-Operator?

helm chart:prometheus-community/kube-prometheus-stack

Manifests

No response

prometheus-operator log output

logs above

Anything else?

No response

[simonpasquier]

Can you share the promehteus-operator deployment manifest as well as the AlertManager and ThanosRuler manifests?