[SegGPT] Fix loss calculation

[EduardoPach]

What does this PR do?

This PR fixes #30419 and ensures that the loss is being correctly calculated.

While working on this PR I not only noticed that SegGptLoss was broken, but it was being incorrectly calculated (shame on SegGpt contributor 😞). Proposed solution include:

• Passing labels to SegGptModel to correctly perform the forward pass when training in In-Context Painting style
• Changed SegGptLoss forward method and its docstrings accordingly so that the output is the same as the one obtained in the original implementation.

Note

While running test_modeling_seggpt with is_training = True I found that gradient_checkpointing is also not working due to type_token_semantic parameter that is not used in the forward pass and is controlled by the embedding_type the model's forward and by default we use the type_token_instance just like the original implementation. Hence, we could probably move the embedding_type to config to allow gradient_checkpointing or remove it entirely as in the original implementation is not clear what is the use case for type_token_semantic

c.c. @amyeroberts

[amyeroberts]

Thanks for working on this!

Have we verified the loss value with that of the original model?

[EduardoPach]

Thanks for working on this!

Have we verified the loss value with that of the original model?

Yeap, also added some tests to make sure this won't be an issue again 🙂.

Regarding gradinet_checkpoint while training, should this be resolved in this PR or a new one? It will probably involve re-uploading the checkpoint to the hub as we would need to take the type_token_semantic parameter from the weights

[amyeroberts]

Regarding gradinet_checkpoint while training, should this be resolved in this PR or a new one? It will probably involve re-uploading the checkpoint to the hub as we would need to take the type_token_semantic parameter from the weights

This should be done in a separate PR.

[amyeroberts]

Thanks for fixing!