Statsforecast AutoArima behaves differently when using sktime's evaluate vs. temporal_train_test_split

[markross1]

Hello,

I've been experimenting with SKTime's evaluate function to cross-validate forecasting models. I've encountered differing results when using the Statsforecast AutoArima output from evaluate compared to results obtained using the temporal_train_test_split function.

The discrepancy becomes more pronounced when I include an exogenous regressor. In my tests, the SKTime AutoArima model behaves as expected, but the Statsforecast AutoArima model produces unexpected results. However, when using Statsforecast AutoArima with temporal_train_test_split, I get results that match closely with SKTime's AutoArima. This issue seems to specifically arise with the Statsforecast AutoArima with the evaluate function and an exogenous regressor.

SKTime with evaluate:

StatsForecast with evaluate:

StatsForecast with train-test-split

Does anyone have insights on why the Statsforecast AutoArima model behaves differently with the evaluate function with exogenous regressors in this example?

Below is an example that should demonstrate this (unless this is computer-specific somehow):

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# sktime imports
from sktime.split import SingleWindowSplitter, temporal_train_test_split
from sktime.forecasting.base import ForecastingHorizon
from sktime.forecasting.arima import AutoARIMA
from sktime.forecasting.statsforecast import StatsForecastAutoARIMA
from sktime.utils.plotting import plot_series
from sktime.forecasting.model_evaluation import evaluate
from sktime.datasets import load_airline


# generate exogenous variable
def generate_seats(y):
    np.random.seed(42)
    base_seats = 1.1 * y
    load_factor = np.random.uniform(low=0.82, high=0.85, size=y.shape)
    seats = y / load_factor
    seats = np.maximum(seats, base_seats)
    seats = np.ceil(seats)
    df = pd.DataFrame({'PAX': y, 'SEATS': seats, 'LF': y / seats})
    return df

# variables
y = load_airline()
X = generate_seats(y)[['SEATS']]

sw = SingleWindowSplitter(fh=list(range(1, 37)))

# Evaluate SKTime AutoARIMA
results = evaluate(forecaster=AutoARIMA(sp=12), y=y, X=X, cv=sw, return_data=True);

y_train = results['y_train'][0]
y_test = results['y_test'][0]
y_pred = results['y_pred'][0]
plot_series(y_train, y_test, y_pred, labels=['train', 'test', 'pred']);
plt.show()


# Statsforecast AutoArima with evaluate (this should show different results)
results = evaluate(forecaster=StatsForecastAutoARIMA(sp=12), y=y, X=X, cv=sw, return_data=True);
y_train = results['y_train'][0]
y_test = results['y_test'][0]
y_pred = results['y_pred'][0]
plot_series(y_train, y_test, y_pred, labels=['train', 'test', 'pred']);
plt.show()

## Statsforecast AutoArima manually (this should match closely with AutoArima and evaluate)
y_train, y_test, X_train, X_test = temporal_train_test_split(y=y, X=X, fh=list(range(1, 37)))
y_pred = StatsForecastAutoARIMA(sp=12).fit_predict(y=y_train, X=X_train, X_pred=X_test, fh=ForecastingHorizon(y_test.index, is_relative=False))
plot_series(y_train, y_test, y_pred, labels=['train', 'test', 'pred']);
plt.show()

Thank you in advance for your help.
-Marko

[fkiraly]

well, that is very odd.

To me, it looks like the statsforecast y_pred is identical with the first three years of y_train.

This is clearly a bug, opened bug report here: #5894

[ComputerVision804]

Well!..
The discrepancy you're experiencing between the results of the AutoArima model from the Statsforecast package and the AutoArima model from the sktime package could be due to the way the two packages handle exogenous variables.
In the Statsforecast package, the AutoArima function does not directly support exogenous variables. When you pass an exogenous variable to the AutoArima function, it will be ignored. This is why you're getting unexpected results when using the AutoArima model from the Statsforecast package with an exogenous variable.
On the other hand, the AutoArima model from the sktime package does support exogenous variables. When you pass an exogenous variable to the AutoArima function in sktime, it will be used to fit the model. This is why you're getting results that match closely with the AutoArima model from Statsforecast when using the temporal_train_test_split function.
To get consistent results across both packages, you can either:
Use the AutoArima model from the sktime package, which supports exogenous variables.
Use the AutoArima model from the Statsforecast package, but manually add the exogenous variable to the model using the xreg argument in the arima function from the statsmodels package.

[fkiraly]

That sounds inaccurate to me, or at least discrepant to some things, could you explain, @ComputerVision804?

• the sktime interface to statsforecast has the tag ignores-exogeneous-X: False, so it claims it uses exogeneous data, it is not ignored.
• how would the claimed difference above - which is between the pmdarima interfacing and the statsforecast interfacing variants - explain a difference between using evaluate, or not using evaluate?

[ComputerVision804]

You're right, the ignores-exogeneous-X tag in sktime for the statsforecast interface indicates it considers exogenous features (denoted by X) during forecasting. This differs from the pmdarima interface for ARIMA models, which typically ignores them (ignores-exogeneous-X: True).
Here's how this distinction can explain a difference between using evaluate and not using it:
Without evaluate:

When you fit the model without using evaluate, the statsforecast interface in sktime takes into account both the endogenous data (y) and any provided exogenous data (X) for training.
The model uses this combined information to learn the underlying relationships between variables and generate forecasts based solely on the endogenous data (y).
With evaluate:
If you use evaluate with the statsforecast interface, the training process might differ. Here's why:
evaluate is typically used for hyperparameter tuning or model selection.
During evaluation, sktime might split the data into training and validation sets.
The statsforecast interface, respecting the ignores-exogeneous-X tag, might only use the endogenous data (y) in the training set for hyperparameter tuning or model selection within evaluate. This is because the final model, trained later, will consider both endogenous and exogenous data.
Impact on Differences:
Without evaluate: The model directly uses all available information (y and X) for training, potentially leading to better forecasts when exogenous features are relevant.
With evaluate: The model used for hyperparameter tuning might not capture the full impact of exogenous features, potentially leading to suboptimal hyperparameter choices for the final model.