Skip to content

Async Filer Metadata Backup

Chris Lu edited this page Nov 27, 2021 · 6 revisions

It is desirable to have a copy of the filer metadata. However, it is not always easy to setup replications for filer metadata store:

  • The store does support stream replication.
  • The store is embedded leveldb, RocksDB, etc.
  • It is just complicated.

Async filer metadata backup to a different store

We can continuously backup filer metadata to a different store, without running a filer instance.

         Metadata Changes
 Filer --------------------> `weed filer.meta.backup` ---> A backup store(LevelDB/RocksDB/
                                                                        Mysql/Postgres/
                                                                        Redis/Cassanra/...)

Just need to configure the backup store the same way as filer.toml.

weed filer.meta.backup can be stopped and resumed. The metadata backup progress is tracked in the backup store itself. So you can pause/resume any time.

It is worth noting that the backup store can be different from the source filer store. E.g., you can use a cheaper on disk LevelDB as a remote store to backup Redis.

         Metadata Changes
 Filer --------------------> `weed filer.meta.backup` ---> A backup store(LevelDB)
on Redis

You can even do this:

         Metadata Changes                                                           sqlite streaming  
 Filer --------------------> `weed filer.meta.backup` ---> A backup store(Sqlite)  ------------------> S3
on Redis

Introduction

API

Configuration

Filer

Advanced Filer Configurations

Cloud Drive

AWS S3 API

AWS IAM

Machine Learning

HDFS

Replication and Backup

Messaging

Use Cases

Operations

Advanced

Security

Misc Use Case Examples

Clone this wiki locally