Report HelmRelease reconciliation status to Gitlab CI pipeline

[heretic098]

Hi, I wanted to ask if other people have come up with a good solution to the following problem. I have a Gitlab CI pipeline which results in a helm chart being packaged and deployed to the Gitlab project's helm "package registry".

For reference, here is the shell script that the Gitlab CI pipeline step calls in order to post the created helm chart:

#!/bin/sh
apk add openssl curl jq
cd $CI_PROJECT_DIR
echo "curl -L -w '{\"code\": \"%{http_code}\"}' --request POST --form \"chart=@$CI_PROJECT_NAME-$HELM_TAG.tgz\" --user $CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/packages/helm/api/$RELEASE_CHANNEL/charts"
echo $(curl -L -w '{"code": "%{http_code}"}' --request POST --form "chart=@$CI_PROJECT_NAME-$HELM_TAG.tgz" --user $CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/packages/helm/api/$RELEASE_CHANNEL/charts | jq -sr)

I have a Flux HelmRelease custom resource set up to pull the latest Helm release from the registry and apply it. It is set up like this:

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: myapp-helmrelease
  namespace: default
spec:
  interval: 5m
  chart:
    spec:
      chart: myapp
      version: "*"
      sourceRef:
        kind: HelmRepository
        name: myapp
        namespace: default
      interval: 1m
  upgrade:
    timeout: "900s"
    remediation:
      remediateLastFailure: true
      retries: 3
  valuesFrom:
    - kind: ConfigMap
      name: myapp-helm-overrides
      optional: false

I want to trigger some automated testing once the Flux helmrelease reconcilation has completed but I am struggling to work out how to programatically report back to the pipeline that the reconciliation is completed. I've seen the Flux alerting docs which suggest it is possible to report back on the status of a Kustomization. I don't think this does what I want; to report back on a HelmRelease. In addition, my helm chart and source code live in separate git repositories to my Flux manifests. So I think that if I wanted to report back to my Flux manifests repo / merge request pipelines on whether or not my reconcilliations had worked as part of my pipeline on that repo then this is exactly what I would want but that isn't today's problem.

I am using Google Kubernetes Engine (GKE). I'm considering a couple of options:

1. Add a pipeline step to Gitlab CI to use the Flux CLI to check the reconciliation status and wait until it finishes before proceeding - I don't like this because I don't really want to have my pipelines using kubectl and having access to the cluster like this
2. Have flux emit an alert to a custom webhook when HelmRelease events happen, write my own webhook receiver to receive them plus webhooks from Gitlab to be triggered when the pipeline starts and then work out some way to pause the pipeline so that the webhook reciever can resume it using Gitlab's API - seems very heavy and I'm not totally sure it will work
3. Implement GKE's managed prometheus and then check for pod restarts by talking to the prometheus API - probably my best option
4. Give up and wait five minutes before starting the testing - seems like a cop out

Here's the yaml for the Flux alert config for option 2, just sending to Slack so I could get a look at the kind of information it puts out:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: flux-helm-watcher
  namespace: default
spec:
  summary: "dev-designer Helm state"
  providerRef:
    name: slack
  eventSeverity: info
  eventSources:
    - kind: HelmRelease
      name: '*'

Does anyone have a better suggestion they would care to offer please?

[heretic098]

Having throught about this a little more, I think it would be possible to use Flux's built in alerting to do what I need by sending the alerts to a generic webhook hosted on Google Cloudrun e.g. like this: https://cloud.google.com/run/docs/triggering/webhooks. I'd get by cloudrun job to spit out a message to a Google pubsub topic and have the pipeline wait until the message arrives giving the status of the reconciliation.

What would be really nice is if flux could directly talk to pubsub instead. As flux is installed inside GKE and running that cluster, it can usually directly talk to pubsub through Google's APIs or potentially be given permission to do so if needed using GKE workload identity.

[heretic098]

I was able to get Google's managed prometheus to pick up the metrics Flux provides using a PodMonitor resource kindly provided by the Google managed prometheus team. I was also able to get the metrics in a Google monitoring dashboard by adapting the PromQL from manifests/monitoring/monitoring-config/podmonitor.yaml. This lets me view the overall status of the Flux installations in all the GKE clusters installed in a particular project through one screen.

However I still cannot get direct notifications of reconciliation statuses to work as this information isn't directly exported in the prometheus metrics (I think). I've been trying to get the Flux notification controller to talk to the Google mananged prometheus alertmanager pod in order to push the alerts through there as the docs suggest that this is possible. However unathenticated connections do not let the alerts process correctly, I get log messages from the notification controller like this:

{"level":"error","ts":"2023-02-10T15:22:29.714Z","logger":"event-server","msg":"failed to send notification","reconciler kind":"HelmRelease","name":"myapp","namespace":"default","error":"postMessage failed: failed to execute request: context deadline exceeded"}
{"level":"error","ts":"2023-02-10T15:22:31.546Z","logger":"event-server","msg":"failed to send notification","reconciler kind":"HelmRelease","name":"myapp","namespace":"default","error":"postMessage failed: failed to execute request: context deadline exceeded"}

Having checked through the code, it looks like I need to use bearer authentication but there is no documented way to do that in the Flux docs.

Can someone please suggest how I can go about setting up Flux to use bearer authentication? For reference, here are the manifests that I am applying to set up Flux alerts to go to the GMP alert manager:

---
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Provider
metadata:
  name: alertmanager
  namespace: default
spec:
  type: alertmanager
  secretRef:
    name: alertmanager-address
---
apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-address
  namespace: default
stringData:
    address: https://alertmanager.gmp-system/api/v2/alerts/"
---
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: flux-helm-prometheus
  namespace: default
spec:
  eventSeverity: info
  eventSources:
  - kind: HelmRelease
    name: '*'
  providerRef:
    name: alertmanager
  summary: myapp Helm state

[stefanprodan]

You can set HTTP headers, please see https://fluxcd.io/flux/components/notification/provider/#http-headers-example

[tolikkostin]

@heretic098 have you solved this task finally? I need to implement something similar. Could you share how your solution works?
Thanks in advance.

[heretic098]

I have not solved this. I don't think that setting headers will solve it as the token expires. In the end I used the flux notification controller to send the alerts directly to Slack in order to get notifications when reconciliation was happening.

My GKE clusters are running with private endpoints so I have written a script to create the necessary tunnels to the control plane via my bastion host. The plan is to run that script in my pipeline and then run flux reconcile inside the pipeline step.

I suspect this doesn't solve your problem for which I apologise.

[heretic098]

Although I now see https://fluxcd.io/flux/components/notification/providers/#google-pubsub which is more along the lines of what I want so perhaps the best thing to do is to try that.