feat: prevent failovers when disk space is exhausted #4404
+792
−28
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
backport-requested ◀️
|
❗ By default, the pull request is configured to backport to all release branches.
|
leonardoce
force-pushed
the
dev/space
branch
3 times, most recently
from
4012706
to
126566d
Compare
|
I tested this using Longhorn in a Fedora VM, but any storage enforcing the PV capacity will do the trick. To test the patch, you need to finish your WAL storage. To keep things easy, I used: apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: cluster-example
spec:
instances: 1
storage:
size: 256MiAnd then: CREATE TABLE storage_area (t text);
-- repeat the following query 20-30 times (you need to be fast!)
INSERT INTO storage_area (t) (select repeat('Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do', 5*1024*1024));With the predefined WAL settings, you'll finish your WAL disk space before you finish the space for PGDATA. |
gbartolini
reviewed
May 13, 2024
gbartolini
reviewed
May 13, 2024
armru
reviewed
May 15, 2024
armru
force-pushed
the
dev/space
branch
2 times, most recently
from
9b2cea6
to
e625d8f
Compare
|
/test limit=local |
|
@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/9110497781 |
armru
approved these changes
May 20, 2024
jsilvela
reviewed
May 20, 2024
jsilvela
reviewed
May 20, 2024
leonardoce
force-pushed
the
dev/space
branch
2 times, most recently
from
172cfe7
to
3cdc43a
Compare
jsilvela
approved these changes
May 21, 2024
jsilvela
reviewed
May 21, 2024
jsilvela
reviewed
May 21, 2024
leonardoce
changed the title
|
E2e tests are green!!!! |
leonardoce
force-pushed
the
dev/space
branch
2 times, most recently
from
b10e33f
to
8b3ab1f
Compare
|
I started an entire E2e run, both local and on cloud: https://github.com/EnterpriseDB/cloudnative-pg/actions/runs/9286574473 |
PostgreSQL will shutdown cleanly when there is no enough disk space to store WAL files. The operator was not recognizing this condition and, since the primary failed, was performing a failover to the most advanced replica. This action will not fix the underlying issue. Only a manual disk resize, initiated by the user, can ultimately lead to a fully working PostgreSQL cluster. This patch makes the instance manager recognize this condition, and report it back to the operator. Upon detecting it, the operator will fence the primary instance and set a phase describing the situation. Since the primary instance is fenced, no failovers will be done. Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enteprisedb.com>
Add an e2e to test the recovery in case a primary runs out of disk space. Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enteprisedb.com>
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Signed-off-by: Jaime Silvela <jaime.silvela@enterprisedb.com>
Signed-off-by: Jaime Silvela <jaime.silvela@enterprisedb.com>
Co-authored-by: Jaime Silvela <jaime.silvela@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@gmail.com>
Co-authored-by: Jaime Silvela <jaime.silvela@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@gmail.com>
Co-authored-by: Jaime Silvela <jaime.silvela@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@gmail.com>
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Signed-off-by: Jaime Silvela <jaime.silvela@enterprisedb.com>
Signed-off-by: Jaime Silvela <jaime.silvela@enterprisedb.com>
Signed-off-by: Jaime Silvela <jaime.silvela@enterprisedb.com>
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Signed-off-by: Gabriele Bartolini <gabriele.bartolini@enterprisedb.com>
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PostgreSQL will shut down cleanly when there is not enough disk space to store WAL files.
The operator did not recognize this condition and, since the primary failed, was performing a failover to the most advanced replica. This action will not fix the underlying issue.
Only a manual disk resize, initiated by the user, can ultimately lead to a fully working PostgreSQL cluster.
This patch makes the instance manager recognize this condition and report it to the operator. Upon detecting it, the operator will not trigger a switchover and set a phase describing the situation.
After the PVCs are resized, the cluster will restart working correctly.
Closes: #4521