Can fairlearn mitigate Tensorflow/Keras models?

[eliasbenussi]

Hi,

Thank you very much for making this package, so far I have had a great experience using it!

I have recently tried to use fairlearn to mitigate a simple TensorFlow model created using the functional API from Keras.

Mitigating using ExponentiatedGradient didn't seem to work out of the box (due to predict yielding probabilities instead of a prediction in tf), however wrapping the tf model in a wrapper with custom fit and predict functions did the trick.

However, I am now trying to mitigate my model using ThresholdOptimizer and I am not figuring out how to make this work. I get the following error:

  File "training.py", line 151, in <module>
    mitigator.fit(train_ds.X_df, train_ds.y_df, sensitive_features=sex_train)
  File "/Users/me/.pyenv/versions/tf-3.8.2/lib/python3.8/site-packages/fairlearn/postprocessing/_threshold_optimizer.py", line 208, in fit
    self.estimator_ = clone(self.estimator)
  File "/Users/me/.pyenv/versions/tf-3.8.2/lib/python3.8/site-packages/sklearn/base.py", line 64, in clone
    raise TypeError("Cannot clone object '%s' (type %s): "
TypeError: Cannot clone object '<keras_wrapper.KerasWrapper object at 0x15a230d00>' (type <class 'keras_wrapper.KerasWrapper'>): it does not seem to be a scikit-learn estimator as it does not implement a 'get_params' methods.

And it seems to happen when sklearn.clone is called on the model in the file fairlearn/fairlearn/postprocessing/_threshold_optimizer.py.

Is there a workaround?
Apologies if this question has already been asked somewhere, I tried to look for one but couldn't find anything.

On a more general note, what are the plans for wider support of TensorFlow models?

[adrinjalali]

Have you tried using https://github.com/adriangb/scikeras to wrap your TF model and then pass it to ExponentiatedGradient?

[eliasbenussi]

Thank you @adrinjalali, this solved my problem and it was exactly the answer I was hoping to get!

[VishakBharadwaj94]

Hello I'm a noob here, but I'm trying to use scikeras to wrap an already trained model for mitigation, but I always keep getting shape errors. What would be the right way to do this?

[hildeweerts]

Hi @VishakBharadwaj94, it's a bit hard to see what's wrong without some more info - do you have a minimum working example? I personally don't have a lot of experience with Keras, but do you get shape errors with .predict_proba() as well? In that case folks at SciKeras might be able to help you out.