Image Lens Reprojection Tool

What is lens-reproject?

lens-reproject is a tool to reproject images taken with a known lens to a new lens. The process unprojects every pixel coordinate back to the light ray in spherical coordinates -- with the origin set to the center of projection -- after which it projects the light ray back onto the sensor using the new lens parameters. A variety of lenses is supported:

• Rectilinear (The default perspective projection lenses).
• Equisolid (Fisheye)
• Equidistant (Fisheye)

Prerequisites

On Linux systems, install libjpeg:

sudo apt install libjpeg-turbo8-dev

Build

You will need a C++ compiler and CMake. All other dependencies are included as submodules.

git clone https://github.com/IDLabMedia/image-lens-reproject.git
cd image-lens-reproject
git submodule update --init --recursive
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
cmake --build . --config Release

You will find the exectable files with the required library files next to in in a bin/ subdirectory.

Updating your local copy from Git

git pull origin main
git submodule update

CLI interface

This program has two ways of being used:

• through Blender-style JSON configuration files.
• by manually specifying the lens characteristics on the command line.

Originally, this tool was made to convert images generated by Blender, from one lens type to another. However, as this tool had potential to be used in other contexts, where no configuration files from Blender were available, simpliy specifying which lens-type the input image has directly on the command-line was very desirable. To specify the input image lens-type directly on the command-line, use the --no-configs argument along with --i-... flags under the Input optics section. When using equirectangular panoraminc input images, the --rotation flag might prove useful.

Usage:
  ./bin/reproject [OPTION...]

 Color processing options:
      --exposure EV   Exposure compensation in stops (EV) to brigthen or 
                      darken the pictures. (default: 0.0)
      --reinhard max  Use reinhard tonemapping with given maximum value 
                      (after exposure processing) on the output images. 
                      (default: 1.0)

 Filter files options:
      --filter-prefix prefix  Only include files starting with (default: 
                              "")
      --filter-suffix suffix  Only include files ending with (default: "")

 Input optics.
   These are usually inferred by the config JSONs. When specifying
   --no-configs, lens information needs to be passed through these
   command line options options:
      --i-rectilinear focal_length,sensor_width
                                Input rectilinear images with given 
                                focal_length,sensor_width tuple.
      --i-equisolid focal_length,sensor_width,fov
                                Input equisolid images with given 
                                focal_length,sensor_width,fov tuple.
      --i-equidistant fov       Input equidistant images with given fov 
                                value.
      --i-equirectangular long_min,long_max,lat_min,lat_max (radians)
                                Input equirectangular images with given 
                                longitude min,max and latitude min,max 
                                value or 'full'.

 Input/output options:
      --input-cfg json-file     Input JSON file containing lens and camera 
                                settings of the input images.
      --output-cfg json-file    Output JSON file containing lens and camera 
                                settings of the output images.
      --no-configs width,height
                                Work without reading and writing config 
                                files. Requires you to specify the input 
                                lens through the input-optics flags 
                                (staring with -i-...) and the expected 
                                resolution of the input images here.
  -i, --input-dir file          Input directory containing images to 
                                reproject.
      --single file             A single input file to convert.
  -o, --output-dir file         Output directory to put the reprojected 
                                images.
      --exr                     Output EXR files. Color and depth.
      --png                     Output PNG files. Color only.

 Output optics options:
      --no-reproject            Do not reproject at all.
      --rectilinear focal_length,sensor_width
                                Output rectilinear images with given 
                                focal_length,sensor_width tuple.
      --equisolid focal_length,sensor_width,fov
                                Output equisolid images with given 
                                focal_length,sensor_width,fov tuple.
      --equidistant fov         Output equidistant images with given fov 
                                value.
      --equirectangular longitude_min,longitude_max,latitude_min,latitude_max
                                Output equirectangular images with given 
                                longitude min,max and latitude min,max 
                                value or 'full'.
      --rotation pan, pitch, roll (degrees)
                                Specify a rotation (default: 0.0)

 Runtime options:
      --skip-if-exists    Skip if the output file already exists.
  -j, --parallel threads  Number of parallel images to process. (default: 
                          1)
      --dry-run           Do not actually reproject images. Only produce 
                          config.
  -h, --help              Show help

 Sampling options:
  -s, --samples number          Number of samples per dimension for 
                                interpolating (default: 1)
      --nn                      Nearest neighbor interpolation
      --bl                      Bilinear interpolation
      --bc                      Bicubic interpolation (default)
      --scale percentage        Output scale, as a fraction of the input 
                                size. It is recommended to increase 
                                --samples to prevent aliassing in case you 
                                are downscaling. Eg: --scale 0.5 --samples 
                                2 or --scale 0.33334 --samples 3 or --scale 
                                0.25 --samples 4. Final dimensions are 
                                rounded towards zero. (default: 1.0)
      --output-resolution width,height
                                A fixed output resolution. Overwrites the 
                                behavior of the 'scale' parameter.

Configuration JSON

The configuration JSON files required in the input of the CLI interface are flexible. The application will read in the JSON, extract the lens information from the "camera", "resolution", and "sensor_size" keys in the JSON root object, rewrite the values of those keys to match the new settings, and finally write out the updated JSON to a new file. This way, all custom information is left untouched and it is more easy to integrate it in other pipelines and systems.

The "camera" object in the root object follows the Blender camera settings structure. This is not always the clearest, but we chose it as we work a lot with Blender and it was straightforward to export the camera and lens information straight from Blender. Next we show the different lens configuration templates for the JSON file:

rectilinear

{
  "camera": {
    "focal_length": 36.0,
    "lens_unit": "MILLIMETERS",
    "projection_matrix": [
      [ 2.0, 0.0, 0.0, 0.0 ],
      [ 0.0, 2.0, 0.0, 0.0 ],
      [ 0.0, 0.0, 0.0, 0.0 ],
      [ 0.0, 0.0, 0.0, 1.0 ]
    ],
    "type": "PERSP"
  },
  "resolution": [ 2048, 2048 ],
  "sensor_size": [ 36.0, 36.0 ]
}

(Note that more elements may be present in the root object, as the tool just ignores them, but will copy them over to the output JSON.)

equidistant

{
  "camera": {
    "type": "PANO",
    "panorama_type": "FISHEYE_EQUIDISTANT",
    "fisheye_fov": 3.1415927410125732
  },
  "resolution": [ 2048, 2048 ],
  "sensor_size": [ 36.0, 36.0 ]
}

Note: This lens is not expressed by a focal length. Instead the lens perfectly fits a circle of projection on the sensor. The equidistant lens is only specified by a field of view (fov).

equisolid

{
  "camera": {
    "type": "PANO",
    "panorama_type": "FISHEYE_EQUISOLID",
    "lens": 12.5,
    "fisheye_fov": 3.1415927410125732
  },
  "resolution": [ 2048, 2048 ],
  "sensor_size": [ 36.0, 36.0 ]
}

Note: lens is the focal length of the lens, expressed in the same unit as the sensor_size element (typically millimeters). The naming is taken from Blender.

License

See the LICENSE file.

Credits

To cite this paper:

@inproceedings{courteaux2022silvr,
 title = {{SILVR: A Synthetic Immersive Large-Volume Plenoptic Dataset}},
 author = {Courteaux, Martijn and Artois, Julie and De Pauw, Stijn and Lambert, Peter and Van Wallendael, Glenn},
 year = {2022},
 doi = {10.1145/3524273.3532890},
 publisher = {Association for Computing Machinery},
 url = {https://doi.org/10.1145/3524273.3532890},
 address = {New York, NY, USA},
 month = {jun},
 numpages = {6},
 isbn = {978-1-4503-9283-9/22/06},
 booktitle = {13th ACM Multimedia Systems Conference (MMSys '22)},
 location = {Athlone, Ireland}
}