[BUG] NaNs in model outputs for intfloat/multilingual-e5-large-instruct

[saattrupdan]

🐛 Describe the bug

When benchmarking intfloat/multilingual-e5-large-instruct we encounter the following error:

scandeval.exceptions.InvalidBenchmark: NaN value detected in model outputs, even with mixed precision disabled.

If we force the dtype to be fp32 then we don't encounter the error, but if we examine the raw scores we see that NaNs still appear, affecting the scores negatively.

Operating System

Linux

Device

CUDA GPU

Python version

3.11.x

ScandEval version

12.6.1