rclone with SeaweedFS

Installation

See https://rclone.org/install/

On mac: brew install rclone

Configuration

See https://rclone.org/s3/

Set config ~/.config/rclone/rclone.conf:

[swfs]
type = s3
provider = Other
access_key_id = any-key-id
secret_access_key = any-access-key
endpoint = http://localhost:8333
upload_cutoff = 50Mi
chunk_size = 50Mi
force_path_style = true

Execute commands

copy files

rclone  --log-level INFO copy  --checksum --fast-list /Users/kmlebedev/files swfs:/bucket-name/files

Client-side encryption

Installation local KMS API

git clone https://github.com/kmlebedev/local-kms.git
cd local-kms
go install

run local-kms

local-kms 
INFO[2023-07-24 15:34:23.876] Local KMS Version Unknown (Commit Hash Unknown) 
INFO[2023-07-24 15:34:23.992] No file found at path /init/seed.yaml; skipping seeding. 
INFO[2023-07-24 15:34:23.992] Data will be stored in /tmp/local-kms        
INFO[2023-07-24 15:34:23.992] Local KMS started on 0.0.0.0:8080 

create master key

aws kms create-key --endpoint=http://localhost:8080
{
    "KeyMetadata": {
        "AWSAccountId": "111122223333",
        "KeyId": "5beb0309-d1ec-45ea-895a-52bbecbc8bde",
        "Arn": "arn:aws:kms:eu-west-2:111122223333:key/5beb0309-d1ec-45ea-895a-52bbecbc8bde",
        "CreationDate": "2023-07-03T14:24:36+05:00",
        "Enabled": true,
        "KeyUsage": "ENCRYPT_DECRYPT",
        "KeyState": "Enabled",
        "Origin": "AWS_KMS",
        "KeyManager": "CUSTOMER",
        "CustomerMasterKeySpec": "SYMMETRIC_DEFAULT",
        "KeySpec": "SYMMETRIC_DEFAULT",
        "EncryptionAlgorithms": [
            "SYMMETRIC_DEFAULT"
        ]
    }
}

Copy files with encryption

rclone  --log-level INFO copy --cse-kms-master-key-id 5beb0309-d1ec-45ea-895a-52bbecbc8bde --kms-endpoint http://localhost:8080 --ignore-size --checksum --fast-list /Users/kmlebedev/files swfs:/bucket-name/files