Give possibility for restoring just a single DC

[karol-kokoszka]

scylladb/scylla-cluster-tests#7359

Right now, there is no option in the Scylla Manager restore task to restore just a single data center (DC) from the backup location. This could lead to problematic situations, particularly when:

Encryption at Rest (EaR) is enabled,
Two DCs use different encryption keys,
Encryption keys are stored in different cloud regions, and
There is only one backup location available.
To address this, we would need to make the encryption keys multi-regional to facilitate the restoration process in such scenarios.

The location flag may not be very intuitive, as the [dc] part defines the destination DC, not the source DC data. We need to discuss during the manager planning to determine if a new flag specifying the source is necessary.
If we can restore just a single DC, then we can restore DC by DC, avoiding the need to create multi-regional keys.

(cc: @tzach)

[tzach]

To address this, we would need to make the encryption keys multi-regional to facilitate the restoration process in such scenarios.

Agree, but how this is a Scylla Manager issue to fix?

[karol-kokoszka]

We could potentially address the problem by allowing to restore just a single DC from the location bucket.
It's something what we doesn't support at the moment (possibly by a mistake).

[rayakurl]

@tzach - we need a resolution. for now almost all sct tests are failing since they are multi DC. We will add a couple of pipelines for a single DC + encryption but are are disabling the multi DC jobs as they constantly failing. @mikliapko as discussed, please create a task for the new pipelines. and disable the multi DC ones for now. Thanks

[karol-kokoszka]

grooming notes

The initial idea is to add new flag to the restore CLI, so that it's possible to define the origin DC from the backup location.
Then, data from this DC is going to be restored to specified destination.

@mikliapko SCT will have to be updated to test the scenario with restoring single DC.