Use transformations for future-unknown exogenous features for use with ForecastX

[yarnabrina]

Hi, is there a way in sktime to allow transformations of exogenous features, but not all of them are available during prediction? As of now, it's failing for me with ValueError from sklearn validation check during transform call.

To give some context, I've a set of exogenous features, and only a proper subset will be known in some shape or form for future, others are unknown. So I pass them accordingly to ForecastX. If that's it, and there's no transformations, it's all fine.

But if I add transformations, for example scaling for X features, predictions start failing complaining that no features are passed during predictions. All my transformations have a dedicated ColumnSelect step before the transformer itself.

Is there any conditional transformer to check if columns present or not and transform only if present?

Sample Code

from sklearn.preprocessing import StandardScaler
from sktime.datasets import load_longley
from sktime.forecasting.arima import ARIMA
from sktime.forecasting.base import ForecastingHorizon
from sktime.forecasting.compose import ForecastingPipeline, ForecastX
from sktime.split import temporal_train_test_split
from sktime.transformations.compose import FeatureUnion, TransformerPipeline
from sktime.transformations.series.adapt import TabularToSeriesAdaptor
from sktime.transformations.series.func_transform import FunctionTransformer
from sktime.transformations.series.subset import ColumnSelect

y, X = load_longley()

future_known_exogenous = ["GNPDEFL", "GNP"]
future_unknown_exogenous = ["UNEMP", "ARMED", "POP"]

fh = ForecastingHorizon([1, 2, 3, 4])
y_train, y_test, X_train, X_test = temporal_train_test_split(y, X=X, test_size=fh.max())

sklearn_scaler = StandardScaler()
sktime_scaler = TabularToSeriesAdaptor(sklearn_scaler)

future_known_selector = ColumnSelect(columns=future_known_exogenous)
future_unknown_selector = ColumnSelect(columns=future_unknown_exogenous)

future_known_transformer = TransformerPipeline([future_known_selector, sktime_scaler])
future_unknown_transformer = TransformerPipeline([future_unknown_selector, sktime_scaler])

exogenous_transformer = FeatureUnion(
    [("known", future_known_transformer), ("unknown", future_unknown_transformer)]
)


def create_prefix_remover(prefix: str) -> FunctionTransformer:
    return FunctionTransformer(
        func=lambda data_frame: data_frame.rename(
            columns=lambda column_name: column_name.removeprefix(f"{prefix}__")
        )
    )


future_known_renamer = create_prefix_remover("known")
future_unknown_renamer = create_prefix_remover("unknown")

exogenous_pipeline = TransformerPipeline(
    [exogenous_transformer, future_known_renamer, future_unknown_renamer]
)

model = ARIMA()
forecaster = ForecastX(model.clone(), model.clone(), columns=future_unknown_exogenous)

unfitted_pipeline = ForecastingPipeline([exogenous_pipeline, forecaster])
fitted_pipeline = unfitted_pipeline.fit(y_train, X=X_train, fh=fh)

fitted_pipeline.predict(fh=fh, X=X_test.drop(columns=future_unknown_exogenous))

In the last line, if I pass full X_test it does not fail as transformation step is happy and forecasting step ignores it completely. The issue is I will not have those columns in any of actual scenario.

Note: I'm not using dunder methods mainly because I like being explicit, but also because I don't know how to convert this to that for all steps. A lot of stuff here needs to be configurable for me, e.g. if there are X or not, if yes, are all future-known or are few future-unknown, whether transform X or not, whether transform y or not, etc. These are configurable based on inputs from a GUI, not just in tuning sense. If there's a easy way to convert this to dunder or graphical, I'll be happy to learn.

[fkiraly]

FYI @benHeid, implicitly a pipeline/graphical issue

[yarnabrina]

@fkiraly @benHeid any suggestion regarding this? I could not find any workaround so far for this issue.

[imSanko]

How about he transformation pipeline expects all columns to be present during the transform call, even if some of them are not available for future predictions. One way to solve this issue is to conditionally apply transformations based on whether the columns are present or not.

from sklearn.base import TransformerMixin

class ConditionalTransformer(TransformerMixin):
    def __init__(self, transformer, columns):
        self.transformer = transformer
        self.columns = columns

    def fit(self, X, y=None):
        return self

    def transform(self, X, y=None):
        if set(self.columns).issubset(X.columns):
            return self.transformer.transform(X)
        else:
            return X

# Usage example
sklearn_scaler = StandardScaler()
sktime_scaler = TabularToSeriesAdaptor(sklearn_scaler)

future_known_selector = ColumnSelect(columns=future_known_exogenous)
future_unknown_selector = ColumnSelect(columns=future_unknown_exogenous)

future_known_transformer = ConditionalTransformer(sktime_scaler, future_known_exogenous)
future_unknown_transformer = ConditionalTransformer(sktime_scaler, future_unknown_exogenous)

exogenous_transformer = FeatureUnion(
    [("known", future_known_transformer), ("unknown", future_unknown_transformer)]
)

[yarnabrina]

Thanks, it's a nice idea. It doesn't work for me as it is though.

>>> exogenous_transformer = FeatureUnion(
...     [("known", future_known_transformer), ("unknown", future_unknown_transformer)]
... )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/anirban/sktime-fork/sktime/transformations/compose/_featureunion.py", line 83, in __init__
    self.transformer_list_ = self._check_estimators(
  File "/home/anirban/sktime-fork/sktime/base/_meta.py", line 296, in _check_estimators
    raise TypeError(msg)
TypeError: Invalid 'steps' attribute, 'steps' should be a list of estimators, or a list of (string, estimator) tuples. All estimators in 'steps' must be of type BaseTransformer.

[imSanko]

So I guess we need to ensure that all transformers in the FeatureUnion are of type BaseTransformer. In the ConditionalTransformer we created, it inherits from BaseTransformer but doesn't correctly implement the required methods (fit and transform).

from sktime.transformations.base import BaseTransformer
from sklearn.base import TransformerMixin

class ConditionalTransformer(BaseTransformer, TransformerMixin):
    def __init__(self, transformer, columns):
        self.transformer = transformer
        self.columns = columns

    def fit(self, X, y=None):
        return self.transformer.fit(X[self.columns], y)

    def transform(self, X, y=None):
        if set(self.columns).issubset(X.columns):
            return self.transformer.transform(X[self.columns])
        else:
            return X


sklearn_scaler = StandardScaler()
sktime_scaler = TabularToSeriesAdaptor(sklearn_scaler)

future_known_selector = ColumnSelect(columns=future_known_exogenous)
future_unknown_selector = ColumnSelect(columns=future_unknown_exogenous)

future_known_transformer = ConditionalTransformer(sktime_scaler, future_known_exogenous)
future_unknown_transformer = ConditionalTransformer(sktime_scaler, future_unknown_exogenous)

exogenous_transformer = FeatureUnion(
    [("known", future_known_transformer), ("unknown", future_unknown_transformer)]
)

[yarnabrina]

@imSanko thanks once again, but unfortunately it still fails. I am getting this error, which I believe is coming from fit of BaseTransformer.

NotImplementedError: abstract method

Is it working for you, can you please share the end-to-end reproducible code please? I really like your idea, and I think you can also contribute your working solution to sktime. Probably we don't need to inherit from the mixin class separately, as BaseTransformer already has public fit method calling private _fit method. It'll probably be something like @ninedigits did for FallbackForecaster in #5897.

Also, this seems not very far from this issue #5936 in my view. FYI @fkiraly @benHeid.

[fkiraly]

hm, yes - this is the same as, I guess, OptionalPassthrough with Subset in a chain with Condition?