Accessing prometheus alertmanager

[martin0598]

Hello there,
I set up a Flux cluster on premise with bitbucket. Everything works like a charm. Now I want to add monitoring and alerting using prometheus/grafana. I used flux2-monitoring-example to integrate prometheus/grafana in my cluster. Monitoring flux with the three dashboards works fine.

Now I want to add alerting. In kube-prometheus-stack/release.yaml I enabled alertmanager. I get an additional pod:

• alertmanager-kube-prometheus-stack-alertmanager-0

after a port forward I can access alertmanager UI

 kubectl -n monitoring port-forward svc/kube-prometheus-stack-alertmanager 9093:9093

I add Provider, Secret and Alert

apiVersion: notification.toolkit.fluxcd.io/v1beta2
kind: Provider
metadata:
  name: alertmanager
  # namespace: monitoring - wird durch kustomization gesetzt
spec:
  type: alertmanager
  secretRef:
    name: alertmanager-address

---
apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-address
  # namespace: monitoring - wird durch kustomization gesetzt
stringData:
  address: "http://admin:flux@http://kube-prometheus-stack-alertmanager.monitoring:9093/api/v2/alerts/"
---
apiVersion: notification.toolkit.fluxcd.io/v1beta2
kind: Alert
metadata:
  name: prometheus
spec:
  summary: "Cluster addons impacted in us-east-2"
  providerRef:
    name: alertmanager
  eventSeverity: error
  eventSources:
    - kind: GitRepository
      name: '*'
      namespace: 'flux-system'
    - kind: Kustomization
      name: '*'
      namespace: 'flux-system'
    - kind: HelmRelease
      name: '*'
      namespace: 'flux-system'

to force an alert I misconfigure the URL in a GitRepository.
As expected, my flux log states an error for this. However, no alerts arrive in alertmanager.

notification-controllers log state:

{"level":"error","ts":"2023-10-31T12:50:55.652Z","logger":"event-server","msg":"failed to send notification","reconciler kind":"GitRepository","name":"my-chart","namespace":"flux-system","error":"postMessage failed: failed to execute request: context deadline exceeded"}

in alertmanager I see no log entries.

alert manager is accesible from notification-controller

kubectl -n flux-system exec -it notification-controller-d7d76744f-g2lgg -- wget -qO- http://kube-prometheus-stack-alertmanager.monitoring:9093

gives a reasonable output.

I did not find a working example for my use case. Can you please give advice how to get this working?

Best regards,
Martin

[antonblr]

@martin0598 - you have http two times in the address: http://admin:flux@http://kube-prometheus-stack....

Also, did you setup admin:flux user/password in Alertmanager? I also testing the same setup right now - haven't deployed it yet. My provider yaml looks like this (I'm trying to use alertmanager address inline, instead of via Secret):

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: alertmanager
  namespace: flux-system
spec:
  type: alertmanager
  address: http://kube-prometheus-stack-alertmanager.monitoring:9093/api/v2/alerts/

[martin0598]

Thanks for your answer, got this working now

[jaythamke]

@martin0598 how did you setup admin:flux basic auth in secret. Where did you get that? Could you point me to the docs? Thanks