-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support slice function, padding the mask to the full images and t… #881
base: develop
Are you sure you want to change the base?
Add support slice function, padding the mask to the full images and t… #881
Conversation
…hen concatenate all the predictions together
Hi @SkalskiP 👋 I was discussing with Tal Yagev, who is a Roboflow user, about the issue of large images consuming too much RAM. I tried several approaches to reduce RAM consumption, such as integrating a function to merge the mask in a batch, incremental stacking with lists, generator expressions, and others. However, none of these approaches solved the peak of the RAM that broke the code. After further investigation, I found that the issue was with the NMS (non-maximum suppression) in the segmentation function that consumed a lot of RAM when you have slices of the image. Therefore, I suggested two changes and would like to hear your perspective on them. The first change is to modify the code from this:
to this:
The reason for this change is that we don't need to implement NMS in the full image if we're processing slices of the images. The second change is optional and adds a conditional in the with_nms function, where the user can decide whether they want to implement NMS with a mask or with a bounding box.
With these changes, we have improved the results significantly. I integrated a memory_profiler function in the demo to check the improvements.
What do you think Piotr ? On the other hand, I would like to create a cookbook of this functionality because I believe it will be of great help to people who work with large images and need to detect small objects. |
Description
This pull request introduces support for InferenceSlicer that focuses on segmentation. This is the second part of the discussion in the #678 issue.
First, I added a conditional statement to pad images that were smaller than the slice size, such as corners. Next, I created the
_apply_padding_to_slice
function to apply padding to a slice using the letterbox resizing method.We are adjusting the masks to align them with the image coordinate system, which represents the entire image. Most of the masks are empty except for the specific area we want to focus on. This simplifies the process of merging all the masks, but it also requires more memory since the size of each mask will be the same as the original image. Besides, we are currently using a
concatenate
function to join all the masks in themerge
functionThe main difficulty lies in the high computational cost and time required by this method. The algorithm needs to fit every mask detection in the entire image, resulting in a significantly larger size of the data.
In my opinion, this PR is not ready for production yet, as we need to handle the computational cost for the mask. However, I would like to hear your perspective @SkalskiP
Currently, the method is consuming a lot of RAM.
In addition, this PR encompasses comprehensive unit tests for the new move_masks functionality and a demo that demonstrates the algorithm's application and effectiveness in real-world scenarios.
Type of change
New feature (non-breaking change which adds functionality)
How has this change been tested
I created a demo to showcase this functionality here