Using FallbackForecaster for fallback by hierarchy level

[fkiraly]

Transplanting a question by @marrov from discord to here, since I don't know @ninedigits' alias on discord.

---

Original question by @marrov:

Quick dev idea/help: I was planning on using the new FallbackForecaster to default to a simple moving average if an AutoETS forecast of my multi index df failed. The failing is expected to happen only on some levels, but most of them will be fine. The thing is that Fallback seems to take any failure and switches the forecast for all hierarchy levels to the simple MA. I have zero time to look into thus but just wanted to post for future reference or in case someone is interested. I think having something that would behave as I wanted to over a multiindex df would be a good idea

[fkiraly]

wrap it in ForecastByLevel, that should do the trick with current components (hopefully)

I think it might also be good to ping @ninedigits for ideas (he does not seem to be on discord)

[marrov]

Versions:

sktime==0.27.0
skbase==0.7.5

Minimal example:

from sktime.forecasting.ets import AutoETS
from sktime.forecasting.naive import NaiveForecaster
from sktime.forecasting.base import ForecastingHorizon
from sktime.utils._testing.hierarchical import _make_hierarchical
from sktime.forecasting.compose import ForecastByLevel, FallbackForecaster

# Similar df to my example: 5 levels and 100k rows with period index with ~100 timepoints
df = _make_hierarchical(
    hierarchy_levels = (2, 2, 5, 5, 10),
    max_timepoints = 100,
    min_timepoints = 100,
    same_cutoff = True,
    n_columns = 1,
    all_positive = True,
    index_type = "period",
    random_state = 0,
    add_nan = False,
)

fh =  ForecastingHorizon((range(1, 13)))

model = ForecastByLevel(
    FallbackForecaster(
        forecasters=[
            ("autoets", AutoETS(auto=True, sp=12, n_jobs=-1)),
            ("naive", NaiveForecaster()),
        ],
    )
)

model_config = {
    "backend:parallel": "joblib",
    "backend:parallel:params": {"backend": "loky", "n_jobs": -1},
}

model.set_config(**model_config)

model.fit(y=df, fh=fh)

In my local laptop this takes ~6 minutes without the config (commenting out the second to last line) and ~5 minutes with the parallel config). Am I doing something wrong here?

[fkiraly]

odd. Opened an issue here:
#6225

[ninedigits]

@fkiraly @marrov

I'm not sure if this is the sole issue here, but I do see at least one problem with your setup. You're running a multiprocess job on top of another multiprocess job.

("autoets", AutoETS(auto=True, sp=12, n_jobs=-1)),

...

model_config = {
    "backend:parallel": "joblib",
    "backend:parallel:params": {"backend": "loky", "n_jobs": -1},
}

model.set_config(**model_config)

This might be locking up some of your cores or degrading performance. This is the change I made:

("autoets", AutoETS(auto=True, sp=12, n_jobs=1))

When I run the code with this line it takes about 2 min to complete with multiprocessing enabled and 13 minutes without. I can dig a bit deeper to see if there's other issues. When I do my multiprocessing, I usually use the multiprocessing library. I can try this setup to see if there are any performance differences.

[fkiraly]

oh, so it was already parallelized? Astutely observed!
I moved your answer to the issue #6225.

[marrov]

Good catch, I'm getting more reasonable run times of ~1 min now.