Backup commands run forever causing excessive disk usage

[smlx]

Redis persistence works like this:

• Redis forks. We now have a child and a parent process.
• The child starts to write the dataset to a temporary RDB file.
• When the child is done writing the new RDB file, it replaces the old one.

Meanwhile, the persistent redis backup command in this repository is:

/bin/sh -c "/bin/busybox tar -cf - -C /data .u

---

k8up can sometimes lose track of executed commands such that inside a redis pod you'll see multiple backup commands like this:

  426 user      0:00 /bin/busybox tar -cf - -C /data .
  854 user      0:00 /bin/busybox tar -cf - -C /data .

When tar is executed like this to send data to stdout it will open the files in /data, bundle them into a file, and then start writing to the stdout pipe. Once the stdout pipe fills (as can happen if k8up pods restart or otherwise stop reading off the exec pipe), tar will be blocked but will retain an open file handle to /data/restic.rdb.

Redis will rotate restic.rdb files as outlined above, but the disk space will never actually be reclaimed while tar holds the open file handle.

This will cause the disk space in /data to be filled with old phantom copies of redis.rdb and we get disk usage alerts.

I think the best solution will be to set a timeout on the tar command so that it cannot get into the "infinite runtime" state if the stdout pipe fills. Maybe setting the backup command to something like this to enforce a 4 hour maximum runtime?

/bin/sh -c "timeout 14400 tar -cf - -C /data .