Not able to take etcd backup using cronjob

[kavita1205]

Hi Team,

I am trying to take backup of our existing kubernetes cluster v 1.24 using cronjob. In our company we have 5 master servers, so I wanted to take backup of each master server. So, I am running a script (which will take backup) in configmap and passing this configmap in cronjob.
The issue is , this cronjob is not able to take backup. So, can someone help me here.

Below are the steps which I followed to create cronjob.

1. Building backup tool using Docker file
2. Create Configmap
3. Deploy Cronjob

Dockerfile:

FROM alpine:latest

ARG ETCD_VERSION=v3.5.5

ENV ETCDCTL_ENDPOINTS "https://10.150.***.***:2379"
ENV ETCDCTL_CACERT "/etc/kubernetes/pki/etcd/ca.crt"
ENV ETCDCTL_KEY "/etc/kubernetes/pki/etcd/client.key"
ENV ETCDCTL_CERT "/etc/kubernetes/pki/etcd/client.crt"

RUN apk add --update --no-cache bash ca-certificates tzdata openssl

RUN wget [https://github.com/etcd-io/etcd/releases/download/${ETCD_VERSION}/etcd-${ETCD_VERSION}-linux-amd64.tar.gz](https://github.com/etcd-io/etcd/releases/download/$%7BETCD_VERSION%7D/etcd-$%7BETCD_VERSION%7D-linux-amd64.tar.gz)
&& tar xzf etcd-${ETCD_VERSION}-linux-amd64.tar.gz
&& mv etcd-${ETCD_VERSION}-linux-amd64/etcdctl /usr/local/bin/etcdctl
&& rm -rf etcd-${ETCD_VERSION}-linux-amd64*

ENTRYPOINT ["/bin/bash"]

Configuration of Configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: etcd-backup-script-configmap
  namespace: etcd-backup
data:
  etcd-backup.sh: |
    #!/bin/sh
    export ETCDCTL_API=3
    DIRBACKUP="/data/etcd-backup"
    if [ ! -d "$DIRBACKUP" ];then
      mkdir -p "$DIRBACKUP"
    fi
    etcdctl member list -wjson | jq -r '.members[].clientURLs | .[]' | while read member ; do
      etcdctl --endpoints=${member} snapshot save /data/etcd-backup/etcd-snapshot-${member}-$(date +%Y-%m-%d_%H:%M:%S_%Z).db; 
    done
```
3. Configuration of cronjob
```
apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: etcd-backup
spec:
  schedule: "*/5 * * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  concurrencyPolicy: Allow
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: etcd-backup
            image: artifacts.corp.****.com/etcd-backup:v3.5.5
            imagePullPolicy: IfNotPresent
            env:
            - name: ETCDCTL_API
              value: "3"
            - name: ETCDCTL_ENDPOINTS
              value: "https://10.150.***.***:2379,https://10.150.***.**:2379,https://10.150.***.***:2379,https://10.150.***.***:2379,https://10.150.***.***:2379"
            - name: ETCDCTL_CACERT
              value: "/etc/kubernetes/pki/etcd/ca.crt"
            - name: ETCDCTL_CERT
              value: "/etc/kubernetes/pki/etcd/client.crt"
            - name: ETCDCTL_KEY
              value: "/etc/kubernetes/pki/etcd/client.key"
            - name: ARTIFACTORY_API_KEY
              valueFrom:
                secretKeyRef:
                  name: etcd-backup-artifactory-secret
                  key: ARTIFACTORY_API_KEY
            volumeMounts:
            - mountPath: /etc/kubernetes/pki/etcd
              name: etcd-certs
              readOnly: true
            - mountPath: /data/etcd-backup
              name: etcd-backup
            - mountPath: /etc/localtime
              name: local-timezone
            - mountPath: /tmp/etcd-backup.sh
              subPath: etcd-backup.sh
              name: etcd-backup-script-configmap
          restartPolicy: OnFailure
          hostNetwork: true
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                    - key: node-role.kubernetes.io/master
                      operator: Exists
          tolerations:
            - key: node-role.kubernetes.io/master
              effect: NoSchedule
              operator: Exists
          volumes:
          - name: etcd-certs
            hostPath:
              path: /etc/kubernetes/pki/etcd
              type: Directory
          - name: etcd-backup
            hostPath:
              path: /data/etcd-backup
              type: DirectoryOrCreate
          - name: local-timezone
            hostPath:
              path: /usr/share/zoneinfo/America/Los_Angeles
          - name: etcd-backup-script-configmap
            configMap:
              name: etcd-backup-script-configmap
              defaultMode: 0777 
          imagePullSecrets:
            - name: etcd-backup-regcred
 ```

[kavita1205]

@tjungblu can you please look into it and help me here.

[kavita1205]

So, Is there any issue in my code?

[tjungblu]

Can't tell, I can not look into your cluster. I would exec into a debug pod and run the respective commands one by one.
I would sort "hanging without logs" into a firewall rule or an infinite loop.

[kavita1205]

One query, you told me we can take backup of multiple servers using below command. but when I am passing this command in my script then it seems it is not working. however, I am passing all IP address in env: ETCDCTL_ENDPOINTS. Do you have any idea how we can run this command in script or do I need to pass all endpoints here within this script.
```
etcdctl member list -wjson | jq -r '.members[].clientURLs | .[]' | while read member ; do
etcdctl --endpoints=${member} snapshot save /data/etcd-backup/etcd-snapshot-${member}-$(date +%Y-%m-%d_%H:%M:%S_%Z).db;
done

[kavita1205]

actually pod seems to be in completed status due to which it is not allowing me to do exec and check why this command is not running.

[tjungblu]

again, I suggest you to debug the script by running it one by one and see where it hangs. If it exits fine without any output it means the loop member list does not return anything.