Unclear timezone semantics when used with cron

[rzetelskik]

sctool and service API accept timezone flag, which, according to the docs [1][2][3], specifies the timezone of cron and window flags.

--timezone
Timezone of –cron and –window flag values. The default value is taken from this system, namely ‘TZ’ envvar or ‘/etc/localtime’ file.

Default value: UTC

However, cron also accepts the extended syntax allowing for specifying TZ=<timezone> and CRON_TZ=<timezone>. It's undocumented in scylla-manager task docs, and it's not clear what the outcome of using it is, especially when used in tandem with timezone.

For instance, it's not clear what the difference between the following would be, or what behaviour should be expected:

$ sctool repair -c=scylla/scylla --cron="0 23 * * SAT" --timezone="Europe/Warsaw"

$ sctool repair -c=scylla/scylla --cron="TZ=Europe/Warsaw 0 23 * * SAT"

$ sctool repair -c=scylla/scylla --cron="TZ=UTC 0 23 * * SAT" --timezone="Europe/Warsaw"

$ sctool repair -c=scylla/scylla --cron="TZ=Europe/Warsaw 0 23 * * SAT" --timezone="Europe/Warsaw"

$ sctool repair -c=scylla/scylla --cron="TZ=Europe/Warsaw CRON_TZ=UTC 0 23 * * SAT" --timezone="Europe/Warsaw"

I'd expect this to either be documented, or the additional syntax to be explicitly forbidden.

For reference, Kubernetes ran into a similar issue with CronJob [4], and the docs now specify that the extended syntax is not officially supported, and using the syntax results in a validation error in newer releases.

[1] - https://manager.docs.scylladb.com/stable/sctool/repair.html#timezone
[2] - https://manager.docs.scylladb.com/stable/sctool/backup.html#timezone
[3] - https://manager.docs.scylladb.com/stable/sctool/restore.html#timezone
[4] - https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#unsupported-timezone-specification

[rzetelskik]

cc @karol-kokoszka

[karol-kokoszka]

Candidate for 3.2.9 (or 3.3.1)

[karol-kokoszka]

grooming note

We must extend the documentation and explain what are the timezones that can be defined in task responsible for.
We must explain that cron specification allows to include the timezone.

We must understand how definining/not defining the timzeone in CRON impacts the scheduled task execution.
There is a timezone available for the manager overall (must be checked what is it about).
And the third timezone defines task.

So 3 timezones, per CRON, per task, per manager.

The goal of this task is to understand their meaning and impact + document them.