Notification controller not publishing Reconciliation Succeeded events

[vcalmic]

Hi everyone,

We've set up the following GCP Pub/Sub alert and provider. However, events like Reconciliation Succeeded are not being published to Pub/Sub. Only events whose message mentions that K8s resources have been changed are published.

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: pubsub-alert
  namespace: flux-system
spec:
  address: GCP_PROJECT_ID
  channel: GCP_PUBSUB_TOPIC_ID
  secretRef:
    name: pubsub-alert
  type: googlepubsub
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: pubsub-alert
  namespace: flux-system
spec:
  eventMetadata:
    Cluster: gke-cluster
    Environment: sbx
  eventSources:
    - apiVersion: kustomize.toolkit.fluxcd.io/v1
      kind: Kustomization
      name: "*"
  providerRef:
    name: pubsub-alert
  summary: Flux Kustomization Event

Example of which event gets published:

{'involvedObject': {'kind': 'Kustomization', 'namespace': 'flux-system', 'name': 'kyverno', 'uid': 'a6c5de09-7a9f-47b5-9f07-30a3d0d44147', 'apiVersion': 'kustomize.toolkit.fluxcd.io/v1', 'resourceVersion': '63727811'}, 'severity': 'info', 'timestamp': '2024-01-24T11:29:51Z', 'message': 'ClusterRole/kyverno:admission-controller configured\nClusterRole/kyverno:background-controller configured\nClusterRole/kyverno:cleanup-controller configured\nClusterRole/kyverno:reports-controller configured', 'reason': 'Progressing', 'metadata': {'Cluster': 'gke-cluster', 'Environment': 'sbx', 'revision': 'sbx@sha256:6c7d48f73ecb184291ec23be776129ebf1da613913ac7e32ba387a7ca378c86d', 'summary': 'Flux Kustomization Event'}, 'reportingController': 'kustomize-controller', 'reportingInstance': 'kustomize-controller-57cf68fb6d-nz8x2'}.

Is there a way to publish Reconciliation Succeeded events as well?

---

Some logs from the notification controller

{"level":"info","ts":"2024-01-24T11:28:43.670Z","logger":"event-server","msg":"dispatching event","eventInvolvedObject":{"kind":"Kustomization","namespace":"flux-system","name":"certmanager","uid":"cd891cc0-ac6d-4401-ba83-ef2693939c6e","apiVersion":"kustomize.toolkit.fluxcd.io/v1","resourceVersion":"63727516"},"message":"Reconciliation finished in 4.20548393s, next run in 10m0s"}
{"level":"info","ts":"2024-01-24T11:28:50.951Z","logger":"event-server","msg":"dispatching event","eventInvolvedObject":{"kind":"Kustomization","namespace":"flux-system","name":"externalsecrets","uid":"a97402e8-5a72-46c2-a1cf-371fa78c2873","apiVersion":"kustomize.toolkit.fluxcd.io/v1","resourceVersion":"63727368"},"message":"Reconciliation finished in 2.125098351s, next run in 10m0s"}

[stefanprodan]

This is the intended behaviour.

[vcalmic]

@stefanprodan Thank you for your quick response. Is it possible to configure it to publish this event as well?

[vcalmic]

In our CI/CD pipeline, we are triggering the reconciliation of Kustomizations and we would like to use Pub/Sub to check whether the reconciliation was a success or not.

[stefanprodan]

The Alert API is meant to notify when the cluster state drifts from the desired state, the fact that the controller went over a reconcile loop that resulted in no operations is not subject to altering. If you need to capture these events, you could write your own HTTP receiver and use an Alert Provider that forwards all events in raw JSON format to that endpoint. Docs here: https://fluxcd.io/flux/components/notification/providers/#generic-webhook