[Bug]: stopBackup should handle disconnects

[csdaraujo]

Is there an existing issue already for this bug?

• I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

• I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

• I have read the troubleshooting guide and I think this is a new bug.

Contact Details

caetano@heroiclabs.com

Version

older in 1.22.x

What version of Kubernetes are you using?

1.29

What is your Kubernetes environment?

Cloud: Amazon EKS

How did you install the operator?

YAML manifest

What happened?

Some backups are kept on started state blocking newer backups. One of the causes seems to be that connection is closed within the backupConnection struct and there is no re-connecting strategy. Sample log:

{
  "level": "error",
  "ts": "2024-05-05T10:44:04Z",
  "msg": "while stopping PostgreSQL physical backup",
  "step": "stop",
  "logging_pod": "mycluster-1",
  "error": "sql: connection is already closed",
  "stacktrace": "github.com/cloudnative-pg/cloudnative-pg/pkg/management/log.(*logger).Error\n\tpkg/management/log/log.go:128\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres/webserver.(*backupConnection).stopBackup.func2\n\tpkg/management/postgres/webserver/backup_connection.go:214\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres/webserver.(*backupConnection).executeWithLock\n\tpkg/management/postgres/webserver/backup_connection.go:94\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres/webserver.(*backupConnection).stopBackup\n\tpkg/management/postgres/webserver/backup_connection.go:212"
}

cloudnative-pg/pkg/management/postgres/webserver/backup_connection.go

Line 213 in ae692bf

if err := row.Scan(&bc.data.EndLSN, &bc.data.LabelFile, &bc.data.SpcmapFile); err != nil {

Not sure if my reasoning is right but looks like a reconnect could solve the issue we are facing.

Cluster resource

No response

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct