[GroundingDino] Adding ms_deform_attn kernels

[EduardoPach]

What does this PR do?

This PR fixes #30765 (comment) by adding the missing ms_deformable_attn kernels to GroundingDino

Who can review it?

@NielsRogge

[NielsRogge]

Do we need new kernels for Grounding DINO? Can't we reuse the ones from Deformable DETR? https://github.com/huggingface/transformers/tree/main/src/transformers/kernels/deformable_detr

[EduardoPach]

We could, but then the same would be true to Deta https://github.com/huggingface/transformers/tree/main/src/transformers/kernels/deta as it's just a copy and paste

[NielsRogge]

Ok, tbh I have no idea why they were duplicated for Deta if they're exactly the same as Deformable DETR.

So I'd advocate to just share the kernel among these 3 models in case they're the same (pinging @amyeroberts here).

[amyeroberts]

They were probably duplicated because the kernels used to live at the model level i.e. src/transformers/models/model_name/ - and there shouldn't be imports from one modeling file to another.

Happy to have them shared i.e. grounding DINO just points to deformable DETR kernels in the modeling file.

There's a possible issue where changes might be made to the kernels to e.g. update deformable DETR which isn't compatible with other dependant models. This would effectively be hidden as it's not obvious from the kernel paths here. The likelihood of this happening is so low, I think we can just ignore it for the moment.

[EduardoPach]

GroundingDino pointing to DeformableDetr kernels here bc1f8cb

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[amyeroberts]

Thanks for fixing!