[QUESTION] Use reloader with Custom Resource Definitions that create deployments?

[romancin]

I am trying to use reloader with a Deployment that is created by a Custom Resource Definition managed by an operator. I annotated the deployment using the CRD. Reloader creates a new replica set correctly, but when the operator reconciles, it rolls back this change. Is there I way to use reloader for this cases? Thank you very much.

[MuneebAijaz]

Sorry i need some more information around this to understand.

Can you share the exact change which is reverted?

[romancin]

Of course, I will try to explain it better. I am trying to manage custom configuration changes for Victoria Metrics Agent using reloader. We are deploying the Agent using Victoria Metrics Operator. To deploy a VMAgent using Victoria metrics operator, we create a CRD like this:

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "95"
    reloader.stakater.com/auto: "true"
  labels:
    app.kubernetes.io/instance: vmagent
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: victoria-metrics-k8s-stack
    app.kubernetes.io/version: v1.94.0
    helm.sh/chart: victoria-metrics-k8s-stack-0.18.5
  name: vmagent
  namespace: victoriametrics
spec:
  additionalScrapeConfigs:
    key: additional-scrape-configs.yaml
    name: additional-scrape-configs
  extraArgs:
    promscrape.streamParse: "true"
    remoteWrite.streamAggr.config: /etc/vm/stream-aggr-configs/stream-aggr-configs.config
    remoteWrite.streamAggr.keepInput: "true"
  image:
    tag: v1.94.0
  replicaCount: 2
  resources:
    limits:
      memory: 1024Mi
    requests:
      cpu: "0.5"
      memory: 512Mi
  scrapeInterval: 60s
  selectAllByDefault: true
  volumeMounts:
  - mountPath: /etc/vm/stream-aggr-configs
    name: stream-aggr-configs
    readOnly: true
  volumes:
  - configMap:
      defaultMode: 420
      name: stream-aggr-configs
    name: stream-aggr-configs

As you can see on it, we are putting the annotation reloader.stakater.com/auto: "true". We want that when changes are made to stream-aggr-configs configMap, reloader takes care of reloading VMAgent. When this CRD is applied, it creates this deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "95"
    deployment.kubernetes.io/revision: "7"
    reloader.stakater.com/auto: "true"
  labels:
    app.kubernetes.io/component: monitoring
    app.kubernetes.io/instance: vmagent
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: vmagent
    app.kubernetes.io/version: v1.94.0
    helm.sh/chart: victoria-metrics-k8s-stack-0.18.5
    managed-by: vm-operator
  name: vmagent-vmagent
  namespace: victoriametrics
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: monitoring
      app.kubernetes.io/instance: vmagent
      app.kubernetes.io/name: vmagent
      managed-by: vm-operator
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: monitoring
        app.kubernetes.io/instance: vmagent
        app.kubernetes.io/name: vmagent
        managed-by: vm-operator
    spec:
      containers:
      - args:
        - --reload-url=http://localhost:8429/-/reload
        - --config-envsubst-file=/etc/vmagent/config_out/vmagent.env.yaml
        - --watched-dir=/etc/vm/relabeling
        - --watched-dir=/etc/vm/stream-aggr
        - --config-file=/etc/vmagent/config/vmagent.yaml.gz
        command:
        - /bin/prometheus-config-reloader
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        image: quay.io/prometheus-operator/prometheus-config-reloader:v0.68.0
        imagePullPolicy: IfNotPresent
        name: config-reloader
        resources:
          limits:
            cpu: 100m
            memory: 25Mi
          requests:
            cpu: 100m
            memory: 25Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - mountPath: /etc/vmagent/config_out
          name: config-out
        - mountPath: /etc/vm/relabeling
          name: relabeling-assets
          readOnly: true
        - mountPath: /etc/vm/stream-aggr
          name: stream-aggr-conf
          readOnly: true
        - mountPath: /etc/vmagent/config
          name: config
      - args:
        - -httpListenAddr=:8429
        - -promscrape.config=/etc/vmagent/config_out/vmagent.env.yaml
        - -promscrape.streamParse=true
        - -remoteWrite.maxDiskUsagePerURL=1073741824
        - -remoteWrite.streamAggr.config=/etc/vm/stream-aggr-configs/stream-aggr-configs.config
        - -remoteWrite.streamAggr.keepInput=true
        - -remoteWrite.tmpDataPath=/tmp/vmagent-remotewrite-data
        image: victoriametrics/vmagent:v1.94.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 10
          httpGet:
            path: /health
            port: 8429
            scheme: HTTP
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 5
        name: vmagent
        ports:
        - containerPort: 8429
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 10
          httpGet:
            path: /health
            port: 8429
            scheme: HTTP
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          limits:
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - mountPath: /tmp/vmagent-remotewrite-data
          name: persistent-queue-data
        - mountPath: /etc/vm/stream-aggr-configs
          name: stream-aggr-configs
          readOnly: true
        - mountPath: /etc/vmagent/config_out
          name: config-out
          readOnly: true
        - mountPath: /etc/vmagent-tls/certs
          name: tls-assets
          readOnly: true
        - mountPath: /etc/vm/relabeling
          name: relabeling-assets
          readOnly: true
        - mountPath: /etc/vm/stream-aggr
          name: stream-aggr-conf
          readOnly: true
        - mountPath: /etc/vmagent/config
          name: config
          readOnly: true
      dnsPolicy: ClusterFirst
      initContainers:
      - args:
        - -c
        - gunzip -c /etc/vmagent/config/vmagent.yaml.gz > /etc/vmagent/config_out/vmagent.env.yaml
        command:
        - /bin/sh
        image: quay.io/prometheus-operator/prometheus-config-reloader:v0.68.0
        imagePullPolicy: IfNotPresent
        name: config-init
        resources:
          limits:
            cpu: 100m
            memory: 25Mi
          requests:
            cpu: 100m
            memory: 25Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/vmagent/config
          name: config
        - mountPath: /etc/vmagent/config_out
          name: config-out
      restartPolicy: Always
      schedulerName: default-scheduler
      serviceAccount: vmagent-vmagent
      serviceAccountName: vmagent-vmagent
      terminationGracePeriodSeconds: 30
      volumes:
      - name: persistent-queue-data
      - configMap:
          defaultMode: 420
          name: stream-aggr-configs
        name: stream-aggr-configs
      - name: tls-assets
        secret:
          defaultMode: 420
          secretName: tls-assets-vmagent-vmagent
      - name: config-out
      - configMap:
          defaultMode: 420
          name: relabelings-assets-vmagent-vmagent
        name: relabeling-assets
      - configMap:
          defaultMode: 420
          name: stream-aggr-vmagent-vmagent
        name: stream-aggr-conf
      - name: config
        secret:
          defaultMode: 420
          secretName: vmagent-vmagent

When a change to the configMap is made, I can see in reloader that it triggers a reload, but against the Deployment:

time="2023-10-20T13:21:17Z" level=info msg="Changes detected in 'stream-aggr-configs' of type 'CONFIGMAP' in namespace 'victoriametrics', Updated 'vmagent-vmagent' of type 'Deployment' in namespace 'victoriametrics'"

But just before this, here comes the problem. If I execute kubectl describe deploy ... I can see this:

...
events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  Reloaded           5m12s  reloader-configMaps    Changes detected in 'stream-aggr-configs' of type 'CONFIGMAP' in namespace 'victoriametrics', Updated 'vmagent-vmagent' of type 'Deployment' in namespace 'victoriametrics'
  Normal  ScalingReplicaSet  5m12s  deployment-controller  Scaled up replica set vmagent-vmagent-5d777c9478 to 1
  Normal  ScalingReplicaSet  5m11s  deployment-controller  Scaled down replica set vmagent-vmagent-5d777c9478 to 0

After one second, the new replicaSet is scaled down. I suppose the operator is reverting the change, as something behind it (reloader) is making changes to the object, so the operator reconciles it again.

Can this be solved?

Thank you!

[faizanahmad055]

Reloader cannot control how the operator reconciles the custom resources. In your situation, if the old pod is deleted and either the operator or reloader creates the new pod, you should still be fine because the new pod will have the new changes from configmap.