Issue with updating exogenous features in Recursive Predictions using ForecastingPipeline and YtoX

[jpss95]

Hello sktime community,

I am working on a project involving time series forecasting using sktime's ForecastingPipeline and YtoX, inspired by @fkiraly response in this #discussion #5064 . I am attempting to implement a recursive prediction using transformers to generate exogenous features that update based on previous predictions.

However, I am encountering an issue where it appears that the features are not recalculated correctly for each prediction step. Below, I share a relevant snippet of my code and the output I observe:

from sktime.datasets import load_airline
from sktime.transformations.series.summarize import WindowSummarizer
from sktime.transformations.compose import YtoX
from sktime.forecasting.compose import make_reduction, ForecastingPipeline, TransformedTargetForecaster
from sktime.forecasting.model_selection import temporal_train_test_split
from sklearn.ensemble import RandomForestRegressor

# Load data with no exogenous features
y = load_airline()
y_train, y_test = temporal_train_test_split(y, test_size=0.2)

# WindowSummarizer configuration
kwargs = {
    "lag_feature": {"lag": [1], "mean": [[1, 3], [3, 6]], "std": [[1, 4]]},
    "truncate": 'bfill',
}

# Custom forecaster with logging
class LoggedForecaster(TransformedTargetForecaster):
    def _fit(self, y, X=None, fh=None):
        print("Fitting model...")
        print("y: ", y.head())
        print("X: ", X.head() if X is not None else "No exogenous X")
        return super()._fit(y, X=X, fh=fh)

    def _predict(self, fh=None, X=None):
        print("Predicting...")
        print("X: ", X.head() if X is not None else "No exogenous X")
        return super()._predict(fh, X=X)

# Create forecaster with make_reduction
forecaster = make_reduction(
    RandomForestRegressor(),
    strategy="recursive",
    pooling="global",
    window_length=12,
)

# Forecaster with logging
logged_forecaster = LoggedForecaster(steps=[("model", forecaster)])

# Create pipeline
pipe = ForecastingPipeline(
    steps=[
        ("ytox", YtoX()),
        ("summarizer", WindowSummarizer(**kwargs)),
        ("forecaster", logged_forecaster),
    ]
)

# Fit and forecast
pipe.fit(y=y_train)
pipe.predict(fh=range(1, 7))

Output when predicting:

Predicting...
X:          Number of airline passengers_lag_1  \
1958-08                               491.0   
1958-09                                 NaN   
1958-10                                 NaN   
1958-11                                 NaN   
1958-12                                 NaN   

Number of airline passengers_mean_1_3  \
1958-08                             429.666667   
1958-09                                    NaN   
1958-10                                    NaN   
1958-11                                    NaN   
1958-12                                    NaN   

Number of airline passengers_mean_3_6  \
1958-08                             344.500000   
1958-09                             361.000000   
1958-10                             386.166667   
1958-11                                    NaN   
1958-12                                    NaN   

Number of airline passengers_std_1_4  
1958-08                             66.424769  
1958-09                                   NaN   
1958-10                                   NaN   
1958-11                                   NaN   
1958-12                                   NaN   

Questions for the community:

• The logs show the presence of null values in the generated exogenous data, but perhaps it's because the logs are printing the data only once and in the first iteration. Is there a way to confirm if the features are updating beyond what the initial logs display during predictions? It's crucial to ensure that the model isn't inadvertently processing null values.
• Is there any additional configuration necessary in ForecastingPipeline or WindowSummarizer to ensure that features update correctly with each new prediction?
• Has anyone faced a similar issue and could share their approach to solving it?
• Are there recommended best practices when using derived features in a recursive prediction context?

Any help or suggestions would be greatly appreciated to resolve this issue.

Thank you in advance.

[fkiraly]

Thanks for reporting!

Looking at the code, the X as in the LoggedForecaster are as I would expect them. You asked the WindowSummarizer to produce a few features, among them lag of one. So, when you try to forecast 7 steps ahead, the "lag of one" will be available only for one step ahead, and the remaining entries for that feature will be nan.

I think the confusion comes from you thinking this will produce a recursive prediction - but it will not. YtoX just produces one batch of features, possibly lagged forward if you request a lag in the next step.

If you want recursive forecasts, you need to use make_reduction or RecursiveReductionForecaster.