heavy-sync

heavy-sync is created to address the pain with current tools (s3cmd sync, gsutil rsync, aws s3 sync) when dealing with large buckets (containing millions of objects). Goals:

• Everything that can be retried, should be retried
• Everything that can be resumed, should be resumed
• Ultimately, every bucket synchronization that can be finished, should be finished

Supported service providers: Amazon Simple Storage Service (AWS S3), Google Cloud Storage (GCS).

How it works

1. Generate a list of objects (files) at the source and the destination. Both lists are stored in a SQLite database.
2. Copy objects that are new or have changed (based on the md5 checksum) over. Each sucessfull transfer is recorded in the SQLite database.
3. Delete, from the destination bucket, objects that have disappeared from the source bucket.
4. If a non-zero threshold is provided, delete old object versions that are older than the threshold.

Usually, step 2 takes the bulk of the time and is where other tools will fail (timeout, broken pipe or just hang). When they fail, the whole operation needs to be started from scratch. For big buckets this can be a show stopper, as the synchronization is unlikely to finish without any interruption.

heavy-sync can resume a previously interrupted sync. As long as step 1 finishes sucessfully, progress is never lost. Transient errors are retried forever. 404 errors are ignored. Other errors should cause the program to exit with an non-zero code. Of course, the object listings may get out of date during the sync (it can take hours for the listing and days for the rest), but that's generally not a problem.

heavy-check

heavy-check chooses a random file in the source bucket, then tries downloading that file from the destination bucket, and verifies the checksum. Attention is paid so that randomness is preserved without first loading the whole object list into memory.

If heavy-sync is used to backup a bucket from one service to another, heavy-check can be used to check the backup. Normally, your certainty about your backups reduces over time. With a daily heavy-check, however, it will be increasing instead. TODO: boost this certainty by checking more than one file at a time.