[FEATURE REQUEST] Add finetuning of generative models

[saattrupdan]

🚀 The feature, motivation and pitch

Adding finetuning evaluation of generative models would give a better idea of how well these models would work when tuned to a particular problem, especially when comparing them with closed source solutions, that don’t have this option.

Alternatives

No response

Additional context

Using the SFTTrainer with completions only seems to do the trick, probably with PEFT:
https://huggingface.co/docs/trl/v0.8.1/en/sft_trainer#train-on-completions-only