Increase the default batch size for restore

[Michal-Leszczynski]

As mentioned in scylladb/scylladb#18234 (comment). Using batch size lesser than shard_cnt results in not utilizing all available shards on the node.

Although current implementation allows for a better control over restore batch size (so also shard utilization), the default value (2) is definitely too low for a full cluster restore. We should introduce a special value (0) for batch size which translates to 2 * shard_cnt for each node and make it the default.

cc: @karol-kokoszka @tzach

[karol-kokoszka]

Grooming notes:

We need to have SCT test covering the resotre process.
We want to use it as a benchmark for the further changes/improvements to just compare different values for batch size (that is valid for this issue) and to compare the behavior when the number of parallel transfers is changed.

We want to have the job available in jenkins.
We need to have a possibility of choosing the manager branch that will be executed against the test.
We need to have a possibility of selecting Scylla version.

The goal is to check and observe the metrics during the restore and measure the impact of different values for batch size and transfers to the whole restore process.

It's enough to use just 1TB of data on the cluster.

---

There is not much (if any) scylla manager development work required.
The effort will be mostly on testing and building the CI job.

The most important part is to collect the metrics from SCT run. Make sure that this part of SCT is working.

@mikliapko To keep this issue on his plate and start working on it soon.

@Michal-Leszczynski we need to find the old restore SCT test and put a link.

[Michal-Leszczynski]

@mikliapko I think that this is the job: https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/michal-leszczynski/job/sct-feature-test-4TB-single-node-backup-and-restore-3-nodes-ilya/

[mikliapko]

@karol-kokoszka @Michal-Leszczynski
Could you please elaborate a bit on metrics we should collect from such kind of runs?

From what I see in current test for 4Tb restore, it returns the data only about:

• backup run time;
• restore run time.

Do we need anything else?

[Michal-Leszczynski]

@mikliapko it would be good to get both SM and Scylla metrics.
Mentioned run didn't collect them automatically (not sure why), so I had to collect them manually by logging into monitor node and copying the files. Manually collected metrics can be found here.

[mikliapko]

The requested job is ready:

• Jenkins https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/mikita/job/sct/job/sct-feature-test-1TB-single-node-backup-and-restore-3-nodes
• Argus https://argus.scylladb.com/workspace?state=WyI5Zjc4Nzg0YS00MGMwLTRmOGUtOWVmNC00YWNiYjQyMWJlZWMiXQ

I will add the pipeline in our Jenkins folder after PR merged.

@Michal-Leszczynski @karol-kokoszka Please, check it out in Jenkins/Argus. Let me know if you want to have something else in this job.

[Michal-Leszczynski]

@mikliapko I validated that the metrics are there. The only strange thing is that making schema changes visible in grafana doesn't seem to work, but that's not a big deal. Since this job does not provide a way to change restore task params in jenkins job params, we will still need to: fork the repo, change restore task params and provide this fork in job config, but that's also manageable.

[mikliapko]

@mikliapko I validated that the metrics are there. The only strange thing is that making schema changes visible in grafana doesn't seem to work, but that's not a big deal. Since this job does not provide a way to change restore task params in jenkins job params, we will still need to: fork the repo, change restore task params and provide this fork in job config, but that's also manageable.

@Michal-Leszczynski
Please, provide me the list of restore parameters you want to have editable in pipeline. I'll check how to integrate them and how much effort is needed.

[Michal-Leszczynski]

I think that it would be useful to have:

• keyspace
• batch-size
• parallel

But even then, if we want to change backup size or cluster topology, we need to do it by forking repo and changing yaml config files, so adding them probably won't solve the whole problem, but will be a step in a convenient direction.

cc: @karol-kokoszka

[karol-kokoszka]

Maybe it's good if jenkins job would allow to specify all the arguments for sctool restore ?
Modifying backup size and cluster topology would be a great option too, the question is @mikliapko how difficult you think it is to have it ?

[mikliapko]

@karol-kokoszka @Michal-Leszczynski

Yes, I fully agree about the convenience such parameterized solution would provide us. I'll consider how to address all these points in SCT. Will keep you both informed here.

[mikliapko]

So, from my brief look at how it's done in SCT, we can make keyspaces_num, batch_size and parallel parameters configurable from pipeline.

About cluster topology: if to talk only about number of DB nodes in singleDC cluster, it seems like making this param configurable is not a big deal as well.

Backup size - a bit harder as the current approach comes with hardcoded in configuration file stress commands where the amount of db entries is defined. As a way around, we can introduce a couple of jobs for different amount of data where all the rest requested parameters would be configurable.

@karol-kokoszka @Michal-Leszczynski

[mikliapko]

@karol-kokoszka @Michal-Leszczynski

About cluster topology - what type of parameters you would like to have adjustable - number of nodes, multiDC/singleDC, tablets? and how critical it is?

Backup size - a bit harder as the current approach comes with hardcoded in configuration file stress commands where the amount of db entries is defined. As a way around, we can introduce a couple of jobs for different amount of data where all the rest requested parameters would be configurable.

It can be an option to define several configuration files for different backup sizes, for example, 500Gb, 1Tb, 2Tb or anything else you'd like to have.

[karol-kokoszka]

@mikliapko The more what we can configure is better.
But it's not a blocker.

MultiDC option would be good to have.
Number of nodes - not critical, but nice to have.
Tablets - yes, some enum like "VNODES, TABLETS, mixed"

[mikliapko]

Backup size - a bit harder as the current approach comes with hardcoded in configuration file stress commands where the amount of db entries is defined. As a way around, we can introduce a couple of jobs for different amount of data where all the rest requested parameters would be configurable.

It can be an option to define several configuration files for different backup sizes, for example, 500Gb, 1Tb, 2Tb or anything else you'd like to have.

@karol-kokoszka What do you think about backup sizes?

[Michal-Leszczynski]

It can be an option to define several configuration files for different backup sizes, for example, 500Gb, 1Tb, 2Tb or anything else you'd like to have.

Those backup sizes look good, but maybe just add a 5TB as well.