✨ add 360 and ortho cameras

[ThomasParistech]

Update 'pixel_to_ray' method to handle orthographic and environment (360°) cameras

[Tom94]

Thank you for the contribution. It's great to have support for additional camera modes.

Two small things I think would be prudent to have before I hit the merge button:

• the inverse of the new camera modes should be implemented in pos_to_pixel. This will allow correct motion vectors to be computed, and thus DLSS to work.
• a GUI drop-down within the Advanced camera settings section would be great.
• code style: the new code uses different spacing and newline rules from the rest of the codebase -- that needs to be aligned.

You don't have to do these if you don't want to; I'd be happy to implement these changes/additions myself in the coming weeks as cycles come up. Cheers!

[ThomasParistech]

Thanks for your quick reply!

• Since Vulkan isn't installed on my laptop I can't check if DLSS is still working correctly
• I've skipped the parallax in the Environment camera mode because the function may not be invertible if the parallax shift goes outside of the sphere of radius parallax.z. What do you think?
• I've replaced the tabulations at the beginning of new lines

[ThomasParistech]

Let me know if you still have some fixes in mind !

[gateway]

Hate to jump on this but has this been added, I have a insta 360 1 Inch camera I wanted to test out, also if @ThomasParistech could provide some documentation on best usage of 360 data that would be awesome.

[ThomasParistech]

Hate to jump on this but has this been added, I have a insta 360 1 Inch camera I wanted to test out, also if @ThomasParistech could provide some documentation on best usage of 360 data that would be awesome.

The goal of this PR is to support the rendering of 360 views once the model has been trained with normal images

Camera models are implemented, but input data is currently considered as coming from a standard perspective camera

But we could as well add the possibility to define the camera type to use during training, or even have multimodal training.
For instance by handling an optional camera_type flag in the transform.json

[colinpeng-datascience]

Hi! Thanks for the great work. How does it work headlessly, can I generate a 360 video from the perspective camera path file?

[gateway]

any chance to merg this into the branch or how can I test this.. documentation would be appreciated :)

[ThomasParistech]

The 360 camera model is actually quite simple.
if it helps, I have just written a Medium article which explains well how to map between 3D and 2D in the case of a spherical/360 image :)
https://medium.com/check-visit-computer-vision/understanding-360-images-8e0fcf0ee861

[aaronhdgolf]

Adding 360 support would be great, Agisoft currently supports 360 images.

As well would be great to get agisoft support direct.

[ThomasParistech]

@Tom94 Any suggestion / improvement to do to help this PR being merged ? :)
(other than resolving update conflicts)

Many thanks !

[Tom94]

Hi, sorry for the radio silence -- I actually wanted to merge it a long time ago, but ended up deciding against it in favor of a different approach.

Instant NGP nowadays has a Lens parameter to control how pixels are mapped to directions, and, importantly, this parameter works for both training and rendering. Among others, there already is a latlong and a equirectangular lens for 360 rendering, so only the orthogonal lens from this PR is missing. If you'd like, you could implement it on top of the latest version of the code, but otherwise I'm afraid the codebase has diverged too much from this PR to merge it.

[ThomasParistech]

Hi, sorry for the radio silence -- I actually wanted to merge it a long time ago, but ended up deciding against it in favor of a different approach.

Instant NGP nowadays has a Lens parameter to control how pixels are mapped to directions, and, importantly, this parameter works for both training and rendering. Among others, there already is a latlong and a equirectangular lens for 360 rendering, so only the orthogonal lens from this PR is missing. If you'd like, you could implement it on top of the latest version of the code, but otherwise I'm afraid the codebase has diverged too much from this PR to merge it.

Thanks for the reply 👍
Ok, then I might open a PR for the orthographic lens when I have time :)