News

• 04/04/2023 dataset preview release: 2 synthetic scenes available
• 15/04/2023 code release: 3D reconstruction and novel view synthesis part

TODO

• Full dataset release
• Code release for 3D reconstruction and novel view synthesis
• Code release for intrinsic decomposition and scene editing

I²-SDF: Intrinsic Indoor Scene Reconstruction and Editing via Raytracing in Neural SDFs (CVPR 2023)

Project Page | Paper | Dataset

Setup

Installation

conda env create -f environment.yml
conda activate i2sdf

Data preparation

Download our synthetic dataset and extract them into data/synthetic. If you want to run on your customized dataset, we provide a brief introduction to our data convention here.

Dataset

We provide a high-quality synthetic indoor scene multi-view dataset, with ground truth camera pose and geometry annotations. See HERE for data conventions. Click HERE to download.

3D Reconstruction and Novel View Synthesis

Training

python main_recon.py --conf config/<config_file>.yml --scan_id <scan_id> -d <gpu_id> -v <version>

Note: config/synthetic.yml doesn't contain light mask network, while config/synthetic_light.yml contains.

Evaluation

Novel view synthesis

python main_recon.py --conf config/<config_file>.yml --scan_id <scan_id> -d <gpu_id> -v <version> --test [--is_val] [--full]

The optional flag --is_val evaluates on the validation set instead of training set, --full produces full-resolution rendered images without downsampling.

View Interpolation

python main_recon.py --conf config/<config_file>.yml --scan_id <scan_id> -d <gpu_id> -v <version> --test --test_mode interpolate --inter_id <view_id_0> <view_id_1> [--full]

Generates a view interpolation video between 2 views. Requires ffmpeg being installed.

The number of frames and frame rate of the video can be specified by options.

Mesh Extraction

python main_recon.py --conf config/<config_file>.yml --scan_id <scan_id> -d <gpu_id> -v <version> --test --test_mode mesh

Intrinsic Decomposition and Scene Editing

Brewing🍺, code coming soon.

Citation

If you find our work is useful, please consider cite:

@inproceedings{zhu2023i2sdf,
    title = {I$^2$-SDF: Intrinsic Indoor Scene Reconstruction and Editing via Raytracing in Neural SDFs},
    author = {Jingsen Zhu and Yuchi Huo and Qi Ye and Fujun Luan and Jifan Li and Dianbing Xi and Lisha Wang and Rui Tang and Wei Hua and Hujun Bao and Rui Wang},
    booktitle = {CVPR},
    year = {2023}
}

Acknowledgement

• This repository is built upon Pytorch lightning.
• Thanks to Lior Yariv for her excellent work VolSDF.
• Thanks to Scalable-NISR team for providing their real-world dataset.