Unreachable hosts not reset on platform group failure.

[wxtim]

Description

When all hosts in a platform are uncontactable we remove all the hosts on that platform from the list of bad hosts to allow submission retries.

It doesn't look like this happens when all the platforms in a group are exhausted leading to submit retries being ineffective at handling short term network blips.

Reproducible Example

# ~/.ssh/config
# Set up an alias to allow you to "break" comms with the host
Host mymachine
    HostName=<some-valid-host-name>

# global.cylc
[platforms]
    [[myplatform]]
        hosts = mymachine
        install target = localhost

[platform groups]
    [[mygroup]]
        platforms = myplatform

# flow.cylc
[scheduler]
    allow implicit tasks = True
    cycle point format = %Y

[scheduling]
    initial cycle point = 1311
    final cycle point = 1344
    [[graph]]
        R1 = print-config
        P1Y = foo[-P1Y] => foo

[runtime]
    [[print-config]]
        script = cylc config
    [[foo]]
        script = sleep 10
        platform = mygroup
        submission retry delays = PT5S, PT5S, PT5S, PT5S, PT5S, PT5S

CYLC_CONF_PATH=/the/global/config/above cylc vip 

1. Watch the workflow logs until the first attempt at foo has succeeded, then remove the lines in the ssh config.
2. Watch the workflow logs for submit-failure, then restore the ssh config.
3. Watch as the submit retries fail to actually retry, because the host is still marked as bad.

Expected Behaviour

If all the hosts of all the platforms in a group are bad, all the hosts of all the platforms should be removed from the bad-hosts set to allow resubmission.

Note on logging

Logging, especially error logging is not alway clear where selection of hosts from platform groups has failed:

[ColemanTom]

Thanks for finding a nice simple reproducer. I would not have thought to use the .ssh/config file. Hopefully this isn't a hard fix.

[wxtim]

Thanks for finding a nice simple reproducer. I would not have thought to use the .ssh/config file.

There aren't very many things deliberately breaking your SSH config is good for, but it's a very easy way to simulate broken connections.