[Full images datamanager] fix image indexing with cv.undistort and implement eval indices logic #2683
Conversation
maturk
changed the title
| @@ -150,19 +164,19 @@ def cache_images(self, cache_images_option): | |||
| image = cv2.undistort(image, K, distortion_params, None, newK) # type: ignore | |||
| # crop the image and update the intrinsics accordingly | |||
| x, y, w, h = roi | |||
| image = image[y : y + h, x : x + w] | |||
| image = image[y : y + h + 1, x : x + w + 1] | |||
There was a problem hiding this comment.
this should preserve 1) input.shape == output.shape
|
also added option for mipnerf360 style evaluation in colmap dataparser to hold out every n'th image in train set for eval dataset. Useful for comparing with inria eval scripts |
|
Reverted sampling strategies in colmap dataparser in favour of the new PR #2709 which also implements them. This PR fixes 1) the output image size bug after cv.undistort and 2) implements the logic for "eval_image_indices" and "eval_num_images_to_sample_from" in the datamanager config. |
|
How does this PR interact with #2726 ? |
|
isn't it OK that after the undistortion the shape of the image changes? |
|
@maturk For the first issue: do you have sources indicating the roi is in (left, upper, right, lower) convention? I think it is in x, y, w, h convention. If you search code on github for sample codes of using |
|
The fix should be done upstream: w and h returned by the latest version of icvGetRectangles are off by 1 pixel, see opencv/opencv#24831 |
|
@f-dy okay wow this is actually pretty crazy XD Never would I have thought that I would find a bug in OpenCV! Thank you for making the issue and even tracking down the problem in the source code. |
|
I would recommend adding a unit test to test_datamanager.py, so that we're sure to get the thing right in the end. |
|
And btw we should probably not use cv.undistort, which uses bilinear interpolation, and write our own function that calls cv2.remap with INTER_CUBIC instead. bilinear interpolation creates uneven blur in the image, especially for small distortion amounts, i.e with a small k1 you would see rings of blur in the undistorted images. I'll add some code that demonstrates that |
|
So with opencv 4.8+ this is no longer an issue? |
|
Still an issue, even with 4.9. |
|
Any suggestions what we should do with this issue with the cropped pixel? |
|
@maturk Let's wait opencv's team to fix this bug, I believe they will fix it very soon. Once they fix the issue, we can bump up opencv version dependency again. |
gate the fix you did with if 4.8<=opencv_version<4.10, since the issue is marked for OpenCV 4.10, and add a unit test, so that we can adjust the gate if they don't fix it in 4.10 |
Two things:
1) fix new undistorted image shape not being the same as input
from my understanding in
newK, roi = cv2.getOptimalNewCameraMatrix(K, distortion_params, (image.shape[1], image.shape[0]), 0)the roi that is returned is in pixel notations(px_start, py_start, px_end, py_end)When slicing an image we should instead do
image[y : y + h + 1, x : x + w + 1]to preserve the same shape as the input image. Otherwise we crop the last pixel and output_image.shape != input_image.shape :D2) add eval indices logic from config
There are a few nice eval options in the config that haven't been implemented. I implemented an easy version for
eval_num_images_to_sample_fromandeval_image_indiceswhich I think are the most important ones for evaluating.