Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement cmd_mkbackupdir option #123

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

nioncode
Copy link

@nioncode nioncode commented Jan 1, 2016

This implements the proposed cmd_mkbackupdir option from #20.

Previously, you'd have to create btrfs subvolumes for possible snapshot directories (.sync and lowest_interval.0) for all hosts manually before running rsnapshot for the first time.
Now, you can point cmd_mkbackupdir to an executable that creates the directory for you whenever its needed.

This allows us to take advantage of btrfs features:

  • use cmd_cp to use btrfs subvolume snapshot instead of a slow cp operation -> instant cp
  • use cmd_rm to use btrfs subvolume delete instead of slow rm -> instant removal of old backups
  • use cmd_mkbackupdir to create subvolumes which are necessary for the two above

Overall, this reduces the backup time to the differential rsync operation, all additional IO operations are now instant.
Previously, my hourly backup rotation (+ deletion of old backups) took 17s for my server and 4m7s for my desktop backup. Both of them are now done in one second.
The more files you have in your backup, the more time you will save.

Note: At the beginning, I used read-only snapshots, but this breaks when the .sync directory is missing and gets recreated from the latest snapshot. Then, the .sync directory would have been read-only and rsync fails.

I am using this with the following config options:

cmd_cp              /usr/local/sbin/rsnapshot-cp-btrfs
cmd_rm              /usr/local/sbin/rsnapshot-rm-btrfs
cmd_mkbackupdir     /usr/local/sbin/rsnapshot-mkbackupdir-btrfs

rsnapshot-cp-btrfs:

#!/bin/sh
# rsnapshot will pass the last backup directory
# as $2 and the destination as $3. $1 contains
# additional cp arguments ('-al') that we ignore.
btrfs subvolume snapshot $2 $3

rsnapshot-rm-btrfs:

#!/bin/sh
# rsnapshot will pass the directory to be deleted as $2.
# $1 contains rm arguments ('-rf') that we ignore.
btrfs subvolume delete $2

rsnapshot-mkbackupdir-btrfs

#!/bin/sh
# rsnapshot will pass the directory to be created as $4.
# The other arguments contain additional options:
# $1: "-m" string
# $2: permissions for directory (e.g. 0755)
# $3: "-p" string
btrfs subvolume create $4
chmod $2 

 * Similar to cmd_cp it allows the user to customize the command that gets
   called whenever a backup directory gets created, e.g. '.sync' or
   'interval.0'.
 * This only applies to created backup directories, not to all created
   directories, i.e. the root directory will still be created with mkpath.
@MTecknology
Copy link

@nioncode Thanks for keeping your clone open for all these years!
From my perspective, these look like good changes. I suspect the only thing that /should/ be holding it back is the lack of additional tests, to prevent regression (and the lack of repo maintenance).

Is there any chance you're still interested in having this merged and could write some tests for these changes?

@sgpinkus
Copy link
Member

sgpinkus commented Dec 6, 2019

I don't have any experience with BTRFS to see at a glance why this is useful. But since it is a fairly large change, necessary to have tests.

@nioncode
Copy link
Author

nioncode commented Dec 6, 2019

It's useful, because as stated in the original post, it can significantly reduce backup times when using btrfs as rming and cping the backup directories is now instant instead of possibly taking many minutes, because hardlinks have to be created / removed for each file.

Unfortunately, I don't have enough time to add test cases for this (and personally I don't use rsnapshot anymore). The only thing I can say is that I used my fork for over 3 years without any problems. Further, the code only changes the behavior of rsnapshot if the cmd_mkbackupdir configuration option is set and is thus perfectly backwards compatible and safe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants