UPST-NeRF: Universal Photorealistic Style Transfer of Neural Radiance Fields for 3D Scene

UPST-NeRF(see our paper and project page )is capable of rendering photorealistic stylized novel views with a consistent appearance at various view angles in 3D space.

Qualitative comparisons

Installation

git clone https://github.com/semchan/UPST-NeRF.git
cd UPST-NeRF
pip install -r requirements.txt

Pytorch and torch_scatter installation is machine dependent, please install the correct version for your machine.

Dependencies (click to expand)

• PyTorch, numpy, torch_scatter: main computation.
• scipy, lpips: SSIM and LPIPS evaluation.
• tqdm: progress bar.
• mmcv: config system.
• opencv-python: image processing.
• imageio, imageio-ffmpeg: images and videos I/O.

Download: datasets, trained models, and rendered test views

Directory structure for the datasets (click to expand; only list used files)

data
├── coco     # Link: http://cocodataset.org/#download
│   └── [mscoco2017]
│       ├── [train]
│           └── r_*.png
   
├── nerf_synthetic     # Link: https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1
│   └── [chair|drums|ficus|hotdog|lego|materials|mic|ship]
│       ├── [train|val|test]
│       │   └── r_*.png
│       └── transforms_[train|val|test].json
│
│
└── nerf_llff_data     # Link: https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1
    └── [fern|flower|fortress|horns|leaves|orchids|room|trex]

Synthetic-NeRF datasets

We use the datasets organized by NeRF. Download links:

• Synthetic-NeRF dataset (manually extract the nerf_synthetic.zip to data/)

LLFF dataset

We use the LLFF dataset organized by NeRF. Download link: nerf_llff_data.

Train

To train fern scene and evaluate testset PSNR at the end of training, run:

$ python run_upst.py  --config configs/llff/fern.py  --style_img ./style_images/your_image_name.jpg

Evaluation

To only evaluate the trained fern, run:

$ python run_upst.py --config configs/llff/fern.py --style_img ./style_images/your_image_name.jpg --render_style --render_only --render_test --render_video

We also share some checkpoints for the 3D senes on llff dataset in baidu disk. You can download and put it into "./logs" for evaluation.

link：https://pan.baidu.com/s/18z70qCdRXjm7j1EyCh63Gw

code：1234

Acknowledgement

Thanks very much for the excellent work of DirectVoxGO, our code base is origined from an awesome DirectVoxGO implementation.