handle contour with zero area exceptions #11909

zovelsanj · 2024-04-10T04:03:35Z

In case the contour is very small (like a single pixel contour), it will make a line instead of polygonal detection region. Thus, when creating polygon from that contour, the area of the polygon (and thus distance) will be zero, resulting in an empty box. This results the following error during minimum area calculation:

bounding_box = cv2.minAreaRect(contour)
cv2.error: OpenCV(4.6.0) D:\a\opencv-python\opencv-python\opencv\modules\imgproc\src\convhull.cpp:143: error: (-215:Assertion failed) total >= 0 && (depth == CV_32F || depth == CV_32S) in function 'cv::convexHull'

This error was found specifically with det_box_type='poly', but it may also occur in the case of det_box_type='quad'. Here is the figure to replicate the error.

For privacy reasons, I have to erase few areas in this image. The erroneous detection region coordinates are:
[[909 353], [908 353], [907 352], [906 352], [905 353], [901 353], [901 354], [902 353], [907 353], [908 354]]

The contour made by these coordinates is shown below (by red line).

In case any two coordinates of a contour are same, it will make a triangular region instead of rectangular one. Thus, when creating polygon from that contour, the area of the box (and thus distance) will be zero, resulting in an empty box. This results the following error during minimum area calculation: ``` bounding_box = cv2.minAreaRect(contour) cv2.error: OpenCV(4.6.0) D:\a\opencv-python\opencv-python\opencv\modules\imgproc\src\convhull.cpp:143: error: (-215:Assertion failed) total >= 0 && (depth == CV_32F || depth == CV_32S) in function 'cv::convexHull' ```

paddle-bot · 2024-04-10T04:03:40Z

Thanks for your contribution!

GreatV · 2024-04-16T01:50:47Z

Would it be possible to provide some images that reproduce this problem?

zovelsanj · 2024-04-16T04:53:46Z

Would it be possible to provide some images that reproduce this problem?

@GreatV, I have edited the description with the images and the contour output for this specific case.

jzhang533 · 2024-04-16T06:42:37Z

@GreatV CI passed after rerun.
CI is suffering from network access issues.
We'll need to migrate CI in PaddleOCR to github action. it would be more transparent, and no need to worry about unstable network.

GreatV · 2024-04-16T15:06:42Z

Hi @zovelsanj, this fix does not seem to work well.

paddleocr --image_dir=322704193-d4be89fc-13e4-448d-a614-20cb8a037ad5.png --lang=japan --det_box_type=poly

  File "/home/greatx/repos/PaddleOCR/venv/bin/paddleocr", line 33, in <module>
    sys.exit(load_entry_point('paddleocr==2.8.0', 'console_scripts', 'paddleocr')())
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/paddleocr.py", line 807, in main
    result = engine.ocr(
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/paddleocr.py", line 681, in ocr
    dt_boxes, rec_res, _ = self.__call__(img, cls)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/tools/infer/predict_system.py", line 76, in __call__
    dt_boxes, elapse = self.text_detector(img)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/tools/infer/predict_det.py", line 352, in __call__
    dt_boxes, elapse = self.predict(img)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/tools/infer/predict_det.py", line 273, in predict
    post_result = self.postprocess_op(preds, shape_list)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/ppocr/postprocess/db_postprocess.py", line 241, in __call__
    boxes, scores = self.polygons_from_bitmap(pred[batch_index],
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/ppocr/postprocess/db_postprocess.py", line 84, in polygons_from_bitmap
    box = self.unclip(points, self.unclip_ratio)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/ppocr/postprocess/db_postprocess.py", line 160, in unclip
    expanded = np.array(offset.Execute(distance))
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.

zovelsanj · 2024-04-16T15:30:40Z

Hi @zovelsanj, this fix does not seem to work well.

paddleocr --image_dir=322704193-d4be89fc-13e4-448d-a614-20cb8a037ad5.png --lang=japan --det_box_type=poly

  File "/home/greatx/repos/PaddleOCR/venv/bin/paddleocr", line 33, in <module>
    sys.exit(load_entry_point('paddleocr==2.8.0', 'console_scripts', 'paddleocr')())
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/paddleocr.py", line 807, in main
    result = engine.ocr(
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/paddleocr.py", line 681, in ocr
    dt_boxes, rec_res, _ = self.__call__(img, cls)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/tools/infer/predict_system.py", line 76, in __call__
    dt_boxes, elapse = self.text_detector(img)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/tools/infer/predict_det.py", line 352, in __call__
    dt_boxes, elapse = self.predict(img)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/tools/infer/predict_det.py", line 273, in predict
    post_result = self.postprocess_op(preds, shape_list)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/ppocr/postprocess/db_postprocess.py", line 241, in __call__
    boxes, scores = self.polygons_from_bitmap(pred[batch_index],
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/ppocr/postprocess/db_postprocess.py", line 84, in polygons_from_bitmap
    box = self.unclip(points, self.unclip_ratio)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/ppocr/postprocess/db_postprocess.py", line 160, in unclip
    expanded = np.array(offset.Execute(distance))
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.

Hi @GreatV, thanks for pointing out. I tested it and I think the error is due to db_unclip_ratio. When db_unclip_ratio = 2.5 it works and with the lower value like db_unclip_ratio = 1.8 the exact same error occurs. I think it is the issue with det_box_type=poly and not this specific fix type. Can you please verify this from your side?

GreatV · 2024-04-17T01:12:27Z

The det_box_type='poly' mode may have a lot of bugs that need to be double-checked and fixed.

zovelsanj · 2024-04-17T06:59:48Z

The det_box_type='poly' mode may have a lot of bugs that need to be double-checked and fixed.

ok, it seems so. This fix is still mergeable, isn't it?

GreatV · 2024-04-17T07:46:57Z

Hi @zovelsanj, would you be willing to try to fix these issues to make det_box_type='poly' mode work correctly as well?

GreatV · 2024-04-17T07:56:54Z

db_unclip_ratio = 2.5 still not work.

paddleocr --image_dir=322704193-d4be89fc-13e4-448d-a614-20cb8a037ad5.png --lang=japan --det_box_type=poly --det_db_unclip_ratio=2.5

  File "/home/greatx/repos/PaddleOCR/venv/bin/paddleocr", line 33, in <module>
    sys.exit(load_entry_point('paddleocr==2.8.0', 'console_scripts', 'paddleocr')())
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/paddleocr.py", line 807, in main
    result = engine.ocr(
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/paddleocr.py", line 681, in ocr
    dt_boxes, rec_res, _ = self.__call__(img, cls)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/tools/infer/predict_system.py", line 76, in __call__
    dt_boxes, elapse = self.text_detector(img)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/tools/infer/predict_det.py", line 352, in __call__
    dt_boxes, elapse = self.predict(img)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/tools/infer/predict_det.py", line 277, in predict
    dt_boxes = self.filter_tag_det_res_only_clip(dt_boxes, ori_im.shape)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/tools/infer/predict_det.py", line 217, in filter_tag_det_res_only_clip
    dt_boxes = np.array(dt_boxes_new)
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (56,) + inhomogeneous part.

zovelsanj · 2024-04-17T11:16:21Z

db_unclip_ratio = 2.5 still not work.

paddleocr --image_dir=322704193-d4be89fc-13e4-448d-a614-20cb8a037ad5.png --lang=japan --det_box_type=poly --det_db_unclip_ratio=2.5

  File "/home/greatx/repos/PaddleOCR/venv/bin/paddleocr", line 33, in <module>
    sys.exit(load_entry_point('paddleocr==2.8.0', 'console_scripts', 'paddleocr')())
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/paddleocr.py", line 807, in main
    result = engine.ocr(
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/paddleocr.py", line 681, in ocr
    dt_boxes, rec_res, _ = self.__call__(img, cls)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/tools/infer/predict_system.py", line 76, in __call__
    dt_boxes, elapse = self.text_detector(img)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/tools/infer/predict_det.py", line 352, in __call__
    dt_boxes, elapse = self.predict(img)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/tools/infer/predict_det.py", line 277, in predict
    dt_boxes = self.filter_tag_det_res_only_clip(dt_boxes, ori_im.shape)
  File "/home/greatx/repos/PaddleOCR/venv/lib/python3.10/site-packages/paddleocr-2.8.0-py3.10.egg/paddleocr/tools/infer/predict_det.py", line 217, in filter_tag_det_res_only_clip
    dt_boxes = np.array(dt_boxes_new)
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (56,) + inhomogeneous part.

Thanks @GreatV. Last time, I checked det_db_unclip_ratio=2.5 with chinese model which works. However, det_db_unclip_ratio=2.5 for japanese language gave this same error. When I increased it to >=2.8 it worked for me. Here is the detection output image ( det_db_unclip_ratio=2.5 with chinese (Green boxes) and det_db_unclip_ratio=2.8 with japanese (Red boxes) languages):

Can you verify this from your side?

zovelsanj · 2024-04-17T11:18:52Z

Hi @zovelsanj, would you be willing to try to fix these issues to make det_box_type='poly' mode work correctly as well?

Hi @GreatV, minimum area polygons may fix this poly issue. I can try that. Since that is a different issue, I will create another pull request.

GreatV

I still can't reproduce it successfully, so I may need to wait until you submit your next PR that addresses the other issues before I can check this again.

zovelsanj · 2024-04-18T04:27:14Z

I still can't reproduce it successfully, so I may need to wait until you submit your next PR that addresses the other issues before I can check this again.

Did you check with higher values of det_db_unclip_ratio=2.5 like 3, 3.5? I think specifically for poly decreasing the det_db_unclip_ratio is squashing the polygons to lines resulting in zero area exceptions. I will try to address that in the next pull request. Can you try the higher values of det_db_unclip_ratio and merge close this pull request?

GreatV · 2024-04-18T13:54:30Z

Hi @zovelsanj. This is weird, no matter how I set it up it doesn't work in this mode. Would you be able to provide your script for running it?

zovelsanj · 2024-04-19T07:37:16Z

Hi @zovelsanj. This is weird, no matter how I set it up it doesn't work in this mode. Would you be able to provide your script for running it?

Hi @GreatV, its indeed weird that it doesn't work on your part. Here is how i am using:

def ocr(image_dir):
        ''' ocr for images in a given image dir. language codes: `ch`, `en`, `fr`, `german`, `korean`, `japan` '''
        japan_det_model_dir = <path_to/japan_det_model_dir>
        ch_det_model_dir = <path_to/det_model_4>

        japan_rec_model_dir = <path_to/japan_rec_model_dir>
        ch_rec_model_dir = <path_to/ch_rec_model_dir>

        cls_model_dir = <path_to/cls_model>

        engine_japan = PaddleOCR(use_angle_cls=True, lang='japan', det_model_dir=japan_det_model_dir, rec_model_dir=japan_rec_model_dir, cls_model_dir=cls_model_dir, ocr_version='PP-OCRv4', det_box_type='poly', det_db_unclip_ratio=2.8)
        engine_ch = PaddleOCR(use_angle_cls=True, lang='ch', det_model_dir=ch_det_model_dir, rec_model_dir=ch_rec_model_dir, cls_model_dir=cls_model_dir, det_box_type='poly', det_db_unclip_ratio=2.5)
        images = sorted(glob.glob(image_dir+'/**/*.*', recursive=True))

        for image in images:
            results_japan = engine_japan.ocr(image, det=True, rec=True, cls=True)
            results_ch = engine_ch.ocr(image, det=True, rec=True, cls=True)

Nothing complicated, just straight forward. Let me know if you are able to solve the issue on your side.

GreatV · 2024-04-19T08:01:51Z

still not work

from paddleocr import PaddleOCR


def ocr(image):
    ''' ocr for images in a given image dir. language codes: `ch`, `en`, `fr`, `german`, `korean`, `japan` '''
    engine_japan = PaddleOCR(use_angle_cls=True, lang='japan', ocr_version='PP-OCRv4', det_box_type='poly', det_db_unclip_ratio=2.8)
    results_japan = engine_japan.ocr(image, det=True, rec=True, cls=True)

    print(results_japan)

if __name__ == "__main__":
    ocr("./default_3_75.png")

zovelsanj · 2024-04-19T08:39:21Z

For the image you provided, i am getting error on chinese model with det_db_unclip_ratio=1.8 and works when it is set to 3.5. Its weird that the same code works on mine and not on yours. Here is the successful detection with det_db_unclip_ratio=3.5:

zovelsanj · 2024-05-18T05:01:54Z

Hi @GreatV, I have fixed this issue as well. It is now merged in main with #12108. I am closing this pull request.

GreatV · 2024-05-18T05:07:39Z

@zovelsanj, thank you for your contribution.

paddle-bot bot added the contributor label Apr 10, 2024

paddle-bot bot assigned Sunting78 Apr 10, 2024

zovelsanj changed the title ~~handle triangular contour exceptions~~ handle contour with zero area exceptions Apr 16, 2024

GreatV self-assigned this Apr 16, 2024

GreatV self-requested a review April 16, 2024 15:07

GreatV reviewed Apr 17, 2024

View reviewed changes

zovelsanj closed this May 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

handle contour with zero area exceptions #11909

handle contour with zero area exceptions #11909

zovelsanj commented Apr 10, 2024 •

edited

paddle-bot bot commented Apr 10, 2024

GreatV commented Apr 16, 2024

zovelsanj commented Apr 16, 2024

jzhang533 commented Apr 16, 2024

GreatV commented Apr 16, 2024

zovelsanj commented Apr 16, 2024

GreatV commented Apr 17, 2024

zovelsanj commented Apr 17, 2024

GreatV commented Apr 17, 2024

GreatV commented Apr 17, 2024

zovelsanj commented Apr 17, 2024

zovelsanj commented Apr 17, 2024

GreatV left a comment

zovelsanj commented Apr 18, 2024

GreatV commented Apr 18, 2024

zovelsanj commented Apr 19, 2024

GreatV commented Apr 19, 2024

zovelsanj commented Apr 19, 2024

zovelsanj commented May 18, 2024

GreatV commented May 18, 2024

handle contour with zero area exceptions #11909

handle contour with zero area exceptions #11909

Conversation

zovelsanj commented Apr 10, 2024 • edited

paddle-bot bot commented Apr 10, 2024

GreatV commented Apr 16, 2024

zovelsanj commented Apr 16, 2024

jzhang533 commented Apr 16, 2024

GreatV commented Apr 16, 2024

zovelsanj commented Apr 16, 2024

GreatV commented Apr 17, 2024

zovelsanj commented Apr 17, 2024

GreatV commented Apr 17, 2024

GreatV commented Apr 17, 2024

zovelsanj commented Apr 17, 2024

zovelsanj commented Apr 17, 2024

GreatV left a comment

Choose a reason for hiding this comment

zovelsanj commented Apr 18, 2024

GreatV commented Apr 18, 2024

zovelsanj commented Apr 19, 2024

GreatV commented Apr 19, 2024

zovelsanj commented Apr 19, 2024

zovelsanj commented May 18, 2024

GreatV commented May 18, 2024

zovelsanj commented Apr 10, 2024 •

edited