[BUG] ForecastX is failing with forecaster_X_exogeneous="complement"

[yarnabrina]

Reproducible Code

from sktime.datasets import load_longley

y, X = load_longley()

cols_to_forecast = ["GNPDEFL", "GNP"]
fh = [1, 2, 3, 4]

from sktime.split import temporal_train_test_split

y_train, _, X_train, X_test = temporal_train_test_split(y, X, test_size=max(fh))

from sktime.forecasting.arima import ARIMA

forecaster = ARIMA()

from sktime.forecasting.compose import ForecastX

pipeline = ForecastX(
    forecaster.clone(),
    forecaster_X=forecaster.clone(),
    columns=cols_to_forecast,
    forecaster_X_exogeneous="complement",
)

pipeline.fit(y_train, X=X_train, fh=fh)

pipeline.predict(X=X_test.drop(columns=cols_to_forecast))

Error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/to/env/lib/python3.10/site-packages/sktime/forecasting/base/_base.py", line 448, in predict
    y_pred = self._predict(fh=fh, X=X_inner)
  File "/path/to/env/lib/python3.10/site-packages/sktime/forecasting/compose/_pipeline.py", line 1609, in _predict
    X = self._get_forecaster_X_prediction(fh=fh, X=X)
  File "/path/to/env/lib/python3.10/site-packages/sktime/forecasting/compose/_pipeline.py", line 1559, in _get_forecaster_X_prediction
    X_for_fcX = self._get_X_for_fcX(X)
  File "/path/to/env/lib/python3.10/site-packages/sktime/forecasting/compose/_pipeline.py", line 1586, in _get_X_for_fcX
    X_for_fcX = X.drop(columns=self.columns)
  File "/path/to/env/lib/python3.10/site-packages/pandas/core/frame.py", line 5581, in drop
    return super().drop(
  File "/path/to/env/lib/python3.10/site-packages/pandas/core/generic.py", line 4788, in drop
    obj = obj._drop_axis(labels, axis, level=level, errors=errors)
  File "/path/to/env/lib/python3.10/site-packages/pandas/core/generic.py", line 4830, in _drop_axis
    new_axis = axis.drop(labels, errors=errors)
  File "/path/to/env/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 7070, in drop
    raise KeyError(f"{labels[mask].tolist()} not found in axis")
KeyError: "['GNPDEFL', 'GNP'] not found in axis"

Version

System:
    python: 3.10.13 (main, Sep 11 2023, 08:39:02) [Clang 14.0.6 ]
executable: /path/to/env/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
          pip: 24.0
       sktime: 0.29.0
      sklearn: 1.4.2
       skbase: 0.7.7
        numpy: 1.26.4
        scipy: 1.13.0
       pandas: 2.2.2
   matplotlib: 3.8.4
       joblib: 1.4.0
        numba: 0.59.1
  statsmodels: 0.14.2
     pmdarima: 2.0.4
statsforecast: 1.7.4
      tsfresh: None
      tslearn: None
        torch: None
   tensorflow: None
tensorflow_probability: None

Note

If forecaster_X_exogeneous="complement" is removed from the reproducible code above, it starts working.

[fnhirwa]

I can confirm the bug, is reproducible on macOS, python: 3.9.18 on the main branch
I am going to start looking into this item.