Fix image post-processing for OWLv2

[jla524]

What does this PR do?

Fixes #30131 (issue)

Who can review?

@NielsRogge @nisyad-ms

[amyeroberts]

Thanks for updating our example, @jla524!

This is something which should really be handled within the image processor of the model itself, specifically the rescaling of the box dimensions

[amyeroberts]

Looks great - thanks for taking the time to update the example as well!

Technically this is a breaking change. However, as many users have highlighted this confusing behaviour and it should always have been done in the post-processing method, I think it's OK to add directly.

Final request is to add a test here so that we check the images and boxes are rescaled as expected in the post processing method. Once that's done I think we're good to merge!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[jla524]

Looks great - thanks for taking the time to update the example as well!

Technically this is a breaking change. However, as many users have highlighted this confusing behaviour and it should always have been done in the post-processing method, I think it's OK to add directly.

Final request is to add a test here so that we check the images and boxes are rescaled as expected in the post processing method. Once that's done I think we're good to merge!

Added a test for the resize. I'm not sure if this is the right approach. Let me know what you think!

[amyeroberts]

Looks great - thanks for iterating on this!

For the test, have you visualized the bounding boxes to confirm they are sensible? It would be good to check this + bboxes on the examples to make sure everything is a-ok before merge. Otherwise looks good to go!

[jla524]

Looks great - thanks for iterating on this!

For the test, have you visualized the bounding boxes to confirm they are sensible? It would be good to check this + bboxes on the examples to make sure everything is a-ok before merge. Otherwise looks good to go!

Yes, the boxes look reasonable. One thing to note is that the box on the right is slightly out of bound. The box has a x_max of 642.32, whereas the image itself is 640 pixels.

[amyeroberts]

Thanks for fixing, adding tests and verifying the outputs!

[ydshieh]

Hi @jla524 Thank you for the fix.

Our doctesting for this model now has one failure. See below.

python3 -m pytest -v --make-reports doc_tests_gpu_docs_source_en_model_doc_owlv2.md --doctest-modules docs/source/en/model/doc/owlv2.md failures:
owlv2.md -sv --doctest-continue-on-failure --doctest-glob="*.md"

(on a T4 GPU machine)

I think it's expected? Would you like to take a look to confirm and open a pull request to update docs/source/en/model/doc/owlv2.md if that is the way to go?

Expected:
    Detected a photo of a cat with confidence 0.614 at location [341.67, 17.54, 642.32, 278.51]
    Detected a photo of a cat with confidence 0.665 at location [6.75, 38.97, 326.62, 354.85]
Got:
    Detected a photo of a cat with confidence 0.614 at location [341.67, 23.39, 642.32, 371.35]
    Detected a photo of a cat with confidence 0.665 at location [6.75, 51.96, 326.62, 473.13]

[jla524]

@ydshieh Yes, I'll send in a PR to fix this.