Replies: 13 comments 23 replies
-
Without entering the "pgbackrest vs Barman" debate (BTW, I am one of the creators of Barman), as of today you can actually rely on the Kubernetes way to take backups, using the standard Kubernetes API for Volume Snapshots: https://cloudnative-pg.io/documentation/current/backup_volumesnapshot/ CloudNativePG is the first Postgres operator that natively supports this amazing feature that Kubernetes has standardized, abstracting and automatically bringing features like incremental and differential backup and restore. FYI, I will be writing a blog article about this feature in the next few days, showing how I have been able to restore a 4.5 TB database in 2 minutes. |
Beta Was this translation helpful? Give feedback.
-
Thank you for creating barman, we used the traditional barman version 1 and 2 without issue a few years back for some pg clusters before moving all our backups to object storage! -- Cool, we actually did the combined approach of physical backups in combination with WAL for a while before pgbackrest had its bundle feature. Combining an external system (block-level snapshots) with an integrated postgres-aware backup solution was not very convenient however. We dropped this approach due to issues related to retention, backup validation, encryption and multi-repo support that's all managed & controlled by a single tool (pgbackrest) now. I think pgbackrest could add some really nice capabilities to CloudNativePG, including differential and incremental backup support without requiring the underlying storage to support snapshots. |
Beta Was this translation helpful? Give feedback.
-
Hi, is there any plan to include pgbackrest? |
Beta Was this translation helpful? Give feedback.
-
Not at the moment. We are still committed to continue our progress with Kubernetes native solutions. I am however curious to understand which storage solutions you are using. This is a list of all storage drivers that natively support snapshots: https://kubernetes-csi.github.io/docs/drivers.html (look for the 'Snapshot' column). |
Beta Was this translation helpful? Give feedback.
-
I also wanted to comment in favour of pgbackrest. The incremental backups using the WAL, multi repo support, selective restore etc. all exceed volume snapshots for me, which are as others mentioned really icky to handle. In terms of retention, capabilities, etc.. |
Beta Was this translation helpful? Give feedback.
-
I have an existing cluster that uses pgbackrest for backups and wal archiving. Without support for pgbackrest, I don't see an easy solution for us moving over to cloudnative-pg. |
Beta Was this translation helpful? Give feedback.
-
Can you please elaborate why you say it is not an easy solution? Is this existing cluster already in Kubernetes or outside Kubernetes? |
Beta Was this translation helpful? Give feedback.
-
Can you please provide more information about "incremental backups using WAL"? Also, interested in the reasons behind 'selective restore' capability. |
Beta Was this translation helpful? Give feedback.
-
Hi,
Ok, that's clearer as 'incremental backups using WAL' didn't make any sense, as we are talking about base backups here. For this purpose, we have decided to delegate, for the time being, incremental and differential to Kubernetes itself. I cannot talk for the entire group of maintainers (we will have a meeting and decide), but to be honest I'd rather prioritize different features at the moment, than add another way (non Kubernetes way) to perform incremental/differential backup of instances. However, I won't stop anyone willing to contribute to this feature. My view on this has been clear for many years (and you can easily find public evidence on the topic): incremental/differential backup (and restore) should be part of Postgres' pg_basebackup (there's work going on by Robert Haas in this direction to add this to the core in fact).
Cool, thanks for confirming this (I am well aware of this feature as we wanted to implement it in Barman too many years ago). The direction we've taken is to actually eliminate the issue at the source, by advocating for the microservice database. Ideally, you should have a single database in CloudNativePG (backup and restore is one of the reasons for separating databases in different instances, not the only one). This is explained in the FAQ section. |
Beta Was this translation helpful? Give feedback.
-
I'm surprised by this request. This is more or less like asking for Patroni support in CNPG to manage failover. Of course, nobody will ask for Patroni because the operator already manages failover very well. My 2 cents. |
Beta Was this translation helpful? Give feedback.
-
Since support or implementation of pgbackrest is not on the roadmap currently, do you think you could open the API a bit for people to experiment? E.g. allow extensions to the |
Beta Was this translation helpful? Give feedback.
-
Just a quick update. We have started working on a prototype of generic interface for CloudNativePG, called CNPG-I. See: https://github.com/cloudnative-pg/cnpg-i Instead of adding new backup tools to the operator, our general idea is to actually even strip Barman Cloud from it, and use plugins that adhere to a common interface. For compatibility, obviously we will maintain the Barman Cloud backup support - as either embedded or as a plugin. |
Beta Was this translation helpful? Give feedback.
-
@MannerMan How did the evaluation go? |
Beta Was this translation helpful? Give feedback.
-
Hello!
First of all, thanks for developing cloudnative-pg operator, it's a fantastic well-designed operator for Postgres on Kubernetes.
We have a large deployment of postgres servers (900+) which we're evaluating moving to Kubernetes. We have looked into the various operators available and have found cloudnative-pg to be the best one available. We do have one thing blocking us from using it however, and that is the barman-cloud backup. Unfortunately barman-cloud is not on-par with pgbackrest features and performance. We have tried using barman for our setup previously and it simply cannot handle our amount of data-files. In pgbackrest they were able to solve this however, and improve performance with their bundle feature. Barman is also missing features like multi-repo support, backup encryption which we use currently.
It would be fantastic if cloudnative-pg could support pgbackrest, which is arguably the most robust and feature-complete backup tool for Postgres currently available!
Beta Was this translation helpful? Give feedback.
All reactions