refine-pixsfm work with colmap #2715

Guangyun-Xu · 2024-01-02T12:06:57Z

Refer to this example, use pixsfm to optimize colmap results

jb-ye · 2024-01-03T15:22:20Z

@Guangyun-Xu Do you have side-by-side visualizations w/o pixsfm? I am curious if it produces better GS models.

Guangyun-Xu · 2024-01-04T13:22:19Z

@jb-ye
I tested it on the poster dataset(download by : ns-download-data nerfstudio --capture-name=poster),and PSNR,SSIM and LPIPS are all improved.

gs	PSNR ↑	SSIM ↑	LPIPS ↓
colmap	33.3519	0.9627	0.1157
colmap+pixsfm	34.3037	0.9662	0.1026

thomakah · 2024-01-10T19:58:56Z

Not sure if I'm missing something but don't you need to pass in refine_pixsfm=self.refine_pixsfm when colmap_utils.run_colmap is called in colmap_converter_to_nerfstudio_dataset.py?

I tried running this without that change (as-is) and it looks like pixsfm was not run

Guangyun-Xu · 2024-01-11T04:01:06Z

Not sure if I'm missing something but don't you need to pass in refine_pixsfm=self.refine_pixsfm when colmap_utils.run_colmap is called in colmap_converter_to_nerfstudio_dataset.py?

I tried running this without that change (as-is) and it looks like pixsfm was not run

@thomakah Thanks for your advice. I forgot to add the file colmap_converter_to_nerfstudio_dataset.py before committing. Now you can try again.

jb-ye · 2024-01-11T18:54:30Z

@Guangyun-Xu Could you give a full step-by-step instruction that reproduces your results on poster dataset including the installation instructions for pixsfm (versions, links)?

thomakah · 2024-01-11T19:16:07Z

@Guangyun-Xu I made that change locally before I wrote my comment and then received an error in PixSfM (installed from a recent version via mamba (conda-associated installer, working off conda-forge).

+1 to @jb-ye 's request. It could be that we are working off of different versions

Guangyun-Xu · 2024-01-12T03:13:55Z

@Guangyun-Xu I made that change locally before I wrote my comment and then received an error in PixSfM (installed from a recent version via mamba (conda-associated installer, working off conda-forge).

+1 to @jb-ye 's request. It could be that we are working off of different versions

OK.

Guangyun-Xu · 2024-01-12T12:40:52Z

@jb-ye @thomakah

In order to use pixsfm, we need to install these packages. I refer to the Dockerfile for these installation instructions：

into conda environment

colmap (used in pycolmap)

git clone --branch 3.8 https://github.com/colmap/colmap.git --single-branch

cd colmap && mkdir build && cd build

cmake .. -DCUDA_ENABLED=ON -DCMAKE_CUDA_ARCHITECTURES=${YOU CUDA ARCHITECTURES, RTX40xx is 89}

 make -j `nproc`
 
 make install

pycolmap (used in hloc)

git clone --branch v0.4.0 --recursive https://github.com/colmap/pycolmap.git

cd pycolmap

pip install .

hloc (used in pixsfm)

git clone --branch master --recursive https://github.com/cvg/Hierarchical-Localization.git

cd Hierarchical-Localization

git checkout v1.4

pip install -e .

pyceres (used in pixsfm)

git clone --branch v1.0 --recursive https://github.com/cvg/pyceres.git

cd pyceres

pip install -e .

omegaconf(used in pixsfm)
- ```
pip install omegaconf
```

pixsfm

git clone --recursive https://github.com/cvg/pixel-perfect-sfm.git

cd pixel-perfect-sfm

git reset --hard 40f7c1339328b2a0c7cf71f76623fb848e0c0357

git clean -df

pip install -e .

Guangyun-Xu · 2024-01-12T12:57:44Z

To process data by colmap and then refine by pixsfm, we should run commend like this:

# download Poster dataset
ns-download-data nerfstudio --capture-name=poster

# processing data
ns-process-data images --data data/nerfstudio/poster/images/ --output-dir data/colmap_pixsfm/poster/ --sfm-tool colmap --refine-pixsfm

The log like this:

[20:07:24] 🎉 Done copying images with prefix 'frame_'.                                        process_data_utils.py:3
33                                                                                                                    
[20:07:31] 🎉 Done extracting COLMAP features.                                                       colmap_utils.py:1
81                                                                                                                    
[20:10:03] 🎉 Done matching COLMAP features.                                                         colmap_utils.py:1
95                                                                                                                    
[20:13:00] 🎉 Done COLMAP bundle adjustment.                                                         colmap_utils.py:2
17                                                                                                                    
[20:13:10] 🎉 Done refining intrinsics.                                                              colmap_utils.py:2
28                                                                                                                    
[2024/01/12 20:13:11 pixsfm.features.models.s2dnet INFO] Loading S2DNet checkpoint at /home/guangyun/Project/open_sour
ce/pixel-perfect-sfm/pixsfm/features/models/checkpoints/s2dnet_weights.pth.                                           
[2024/01/12 20:13:11 pixsfm INFO] Loaded dense extractor with configuration:                                          
{'cache_format': 'chunked',                                                                                           
 'device': 'auto',                                                                                                    
 'dtype': 'half',                                                                                                     
 'fast_image_load': False,                                                                                            
 'l2_normalize': True,                                                                                                
 'load_cache_on_init': False,                                                                                         
 'max_edge': 1600,                                                                                                    
 'model': {'name': 's2dnet'},                                                                                         
 'overwrite_cache': False,                                                                                            
 'patch_size': 16,                                                                                                    
 'pyr_scales': [1.0],                                                                                                 
 'resize': 'LANCZOS',                                                                                                 
 'sparse': True,                                                                                                      
 'use_cache': True}                                                                                                   
[2024/01/12 20:13:13 pixsfm INFO] Building matching graph...                                                          
[2024/01/12 20:13:14 pixsfm INFO] Extracting dense features...                                                        
100%|███████████████████████████████████████████████████████████████████████████████| 226/226 [01:21<00:00,  2.78it/s]
[2024/01/12 20:14:36 pixsfm INFO] Loading featuremaps from H5 File.                                                   
100%[████████████████████] 226/226 [00:00, 750.830it/s]                                                               
[2024/01/12 20:14:36 pixsfm INFO] Computing tracks...                                                                 
[2024/01/12 20:14:36 pixsfm INFO] # graph nodes: 498551                                                               
[2024/01/12 20:14:36 pixsfm INFO] # graph edges: 1705513                                                              
[2024/01/12 20:14:37 pixsfm INFO] # tracks: 67819                                                                     
[2024/01/12 20:14:37 pixsfm INFO] Start feature-metric keypoint adjustment.                                           
100%[████████████████████] 498551/498551 [06:39, 1248.80it/s]                                                         
[2024/01/12 20:21:17 pixsfm INFO] KA Time: 399.224s, cost change: 0.0220461 --> 0.0207581                             
[2024/01/12 20:21:32 pixsfm INFO] Extracting references and costmaps.                                                 
100%[████████████████████] 31352/31352 [01:00, 519.287it/s]                                                           
[2024/01/12 20:22:32 pixsfm INFO] Start costmap bundle adjustment.                                                    
100%[████████████████████] 101/101 [00:44, 2.27344it/s]                                                               
[2024/01/12 20:23:17 pixsfm INFO] BA Time: 44.4456s, cost change: 0.0741727 --> 0.0687257                             
[20:23:19] 🎉 Done refining pixsfm                                                                   colmap_utils.py:2
37                                                                                                                    
           🎉 🎉 🎉 All DONE 🎉 🎉 🎉                                                images_to_nerfstudio_dataset.py:1
33                                                                                                                    
           Starting with 226 images                                                  images_to_nerfstudio_dataset.py:1
36                                                                                                                    
           Colmap matched 226 images                                                 images_to_nerfstudio_dataset.py:1
36                                                                                                                    
           COLMAP found poses for all images, CONGRATS!                              images_to_nerfstudio_dataset.py:1
36

Guangyun-Xu · 2024-01-12T13:03:36Z

Tain and eval:

ns-train gaussian-splatting --data data/colmap_pixsfm/poster/

ns-eval --load-config {path to config} --output-path poster_colmap_pixsfm.json

Eval result:

{
  "experiment_name": "poster",
  "method_name": "gaussian-splatting",
  "checkpoint": "outputs/poster/gaussian-splatting/2024-01-12_202737/nerfstudio_models/step-000029999.ckpt",
  "results": {
    "psnr": 34.955841064453125,
    "psnr_std": 1.5855902433395386,
    "ssim": 0.9688587784767151,
    "ssim_std": 0.009276634082198143,
    "lpips": 0.10827375203371048,
    "lpips_std": 0.039100054651498795,
    "num_rays_per_sec": 38027596.0,
    "num_rays_per_sec_std": 8173328.0,
    "fps": 73.56842803955078,
    "fps_std": 15.812171936035156
  }
}

jb-ye · 2024-01-12T15:53:06Z

That's really nice! @Guangyun-Xu Thanks

ichsan2895 · 2024-02-06T15:11:23Z

Whats up with this PR?
Maybe update this PR with the current commit?

ichsan2895 · 2024-02-07T11:52:43Z

Is Colmap 3.9.1 compatible with this PR?

ichsan2895 · 2024-02-07T13:15:01Z

Is Colmap 3.9.1 compatible with this PR?

So far, I can not build pixsfm from source if I use colmap 3.9.1
So I revert back to colmap 3.8

Edit:
confirmed in pixsfm repo that we need colmap==3.8
cvg/pixel-perfect-sfm#136

ichsan2895 · 2024-02-08T04:52:59Z

I got unknown error..

This is playroom dataset which is a sample from Inria GS github repo.

!ns-process-data images --data /path/to/playroom/images \
    --output-dir /path/to/playroom_pixsfm \
    --sfm-tool colmap --refine-pixsfm

The error:

[2024/02/08 04:06:11 pixsfm INFO] Computing tracks...
[2024/02/08 04:06:11 pixsfm INFO] # graph nodes: 457588
[2024/02/08 04:06:11 pixsfm INFO] # graph edges: 874930
[2024/02/08 04:06:12 pixsfm INFO] # tracks: 77316
*** Aborted at 1707365172 (unix time) try "date -d @1707365172" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x0) received by PID 15762 (TID 0x7f6badfbd000) from PID 0; stack trace: ***
    @     0x7f6babc19046 (unknown)
    @     0x7f6bae000520 (unknown)
    @     0x7f6bae165e94 (unknown)
    @     0x7f6b058d5214 (unknown)
    @     0x7f6b0589b530 (unknown)
    @     0x555d3f221e0e (unknown)
    @     0x555d3f2185eb _PyObject_MakeTpCall
    @     0x555d3f22de1d (unknown)
    @     0x555d3f220501 _PyObject_GenericGetAttrWithDict
    @     0x555d3f21ea1d PyObject_GetAttr
    @     0x7f6b059ca0b9 (unknown)
    @     0x7f6b058e12bd (unknown)
    @     0x7f6b0589b530 (unknown)
    @     0x555d3f221e0e (unknown)
    @     0x555d3f2185eb _PyObject_MakeTpCall
    @     0x555d3f230910 (unknown)
    @     0x555d3f2c9e5b PyObject_CallObject
    @     0x7f6ab25bef25 (unknown)
    @     0x7f6ab29f05a0 (unknown)
    @     0x7f6ab2665790 (unknown)
    @     0x7f6ab25659ea (unknown)
    @     0x555d3f221e0e (unknown)
    @     0x555d3f2185eb _PyObject_MakeTpCall
    @     0x555d3f230910 (unknown)
    @     0x555d3f2c9e5b PyObject_CallObject
    @     0x7f6ab25bef25 (unknown)
    @     0x7f6ab262b501 (unknown)
    @     0x7f6ab26bc374 (unknown)
    @     0x7f6ab25659ea (unknown)
    @     0x555d3f221e0e (unknown)
    @     0x555d3f2185eb _PyObject_MakeTpCall
    @     0x555d3f230910 (unknown)

using :

Ubuntu 22.04 LTS
python 3.10
torch-2.0.1+cu118
colmap 3.8 (built with CUDA)
hloc 1.5
pycolmap 0.6.1 (built from source)
pyceres v1.0 (built from source)
pixsfm 1.0 (built from source)

Guangyun-Xu · 2024-02-08T05:13:01Z

I got unknown error..

This is playroom dataset which is a sample from Inria GS github repo.

!ns-process-data images --data /path/to/playroom/images \
    --output-dir /path/to/playroom_pixsfm \
    --sfm-tool colmap --refine-pixsfm

The error:

[2024/02/08 04:06:11 pixsfm INFO] Computing tracks...
[2024/02/08 04:06:11 pixsfm INFO] # graph nodes: 457588
[2024/02/08 04:06:11 pixsfm INFO] # graph edges: 874930
[2024/02/08 04:06:12 pixsfm INFO] # tracks: 77316
*** Aborted at 1707365172 (unix time) try "date -d @1707365172" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x0) received by PID 15762 (TID 0x7f6badfbd000) from PID 0; stack trace: ***
    @     0x7f6babc19046 (unknown)
    @     0x7f6bae000520 (unknown)
    @     0x7f6bae165e94 (unknown)
    @     0x7f6b058d5214 (unknown)
    @     0x7f6b0589b530 (unknown)
    @     0x555d3f221e0e (unknown)
    @     0x555d3f2185eb _PyObject_MakeTpCall
    @     0x555d3f22de1d (unknown)
    @     0x555d3f220501 _PyObject_GenericGetAttrWithDict
    @     0x555d3f21ea1d PyObject_GetAttr
    @     0x7f6b059ca0b9 (unknown)
    @     0x7f6b058e12bd (unknown)
    @     0x7f6b0589b530 (unknown)
    @     0x555d3f221e0e (unknown)
    @     0x555d3f2185eb _PyObject_MakeTpCall
    @     0x555d3f230910 (unknown)
    @     0x555d3f2c9e5b PyObject_CallObject
    @     0x7f6ab25bef25 (unknown)
    @     0x7f6ab29f05a0 (unknown)
    @     0x7f6ab2665790 (unknown)
    @     0x7f6ab25659ea (unknown)
    @     0x555d3f221e0e (unknown)
    @     0x555d3f2185eb _PyObject_MakeTpCall
    @     0x555d3f230910 (unknown)
    @     0x555d3f2c9e5b PyObject_CallObject
    @     0x7f6ab25bef25 (unknown)
    @     0x7f6ab262b501 (unknown)
    @     0x7f6ab26bc374 (unknown)
    @     0x7f6ab25659ea (unknown)
    @     0x555d3f221e0e (unknown)
    @     0x555d3f2185eb _PyObject_MakeTpCall
    @     0x555d3f230910 (unknown)

using :

Ubuntu 22.04 LTS
python 3.10
torch-2.0.1+cu118
colmap 3.8 (built with CUDA)
hloc 1.5
pycolmap 0.6.1 (built from source)
pyceres v1.0 (built from source)
pixsfm 1.0 (built from source)

@ichsan2895 Maybe CUDA out of memory？Or you can try CUDA_VISIBLE_DEVICES=0, multiple GPUs may cause some problems.

Due to the Spring Festival, I am very sorry for not being able to deal with your problems in time.

ichsan2895 · 2024-02-10T06:12:12Z

@Guangyun-Xu hello, I know the problem, we must build from source colmap 3.8 and pixsfm with ceres-solver 2.1.0 instead of ceres-solver 2.2.0. See this bug report: cvg/pixel-perfect-sfm#112

If you have newer Ubuntu (for example >= 23.04) and use apt-get install for installing ceres, the version will be 2.2.0 instead of 2.1.0.

So, we must built from source ceres-2.1.0, the rebuild colmap 3.8 and pixsfm

Guangyun-Xu added 7 commits January 2, 2024 20:00

refine-pixsfm work with colmap

534d633

import pycolmap by "try"

f9be142

fix typo

19d035d

update

13e6de7

Use Dict instead of Dictconfig

19ad7a9

reformat

84de9bd

ignore some line for pyright

5344279

Guangyun-Xu added 2 commits January 8, 2024 15:40

Merge branch 'main' into main

0016271

Merge branch 'nerfstudio-project:main' into main

7d73160

Guangyun-Xu added 2 commits January 11, 2024 11:49

Merge branch 'main' into main

91d311c

pass refine_pixsfm into run_colmap

b29fa76

Merge branch 'main' into main

fe0a63b

Merge branch 'main' into main

1eedf7d

Merge branch 'main' into main

c05b86f

Merge branch 'main' into main

8878834

Merge branch 'nerfstudio-project:main' into main

1dde17b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refine-pixsfm work with colmap #2715

refine-pixsfm work with colmap #2715

Guangyun-Xu commented Jan 2, 2024

jb-ye commented Jan 3, 2024

Guangyun-Xu commented Jan 4, 2024

thomakah commented Jan 10, 2024

Guangyun-Xu commented Jan 11, 2024

jb-ye commented Jan 11, 2024

thomakah commented Jan 11, 2024

Guangyun-Xu commented Jan 12, 2024

Guangyun-Xu commented Jan 12, 2024

Guangyun-Xu commented Jan 12, 2024

Guangyun-Xu commented Jan 12, 2024

jb-ye commented Jan 12, 2024

ichsan2895 commented Feb 6, 2024

ichsan2895 commented Feb 7, 2024

ichsan2895 commented Feb 7, 2024 •

edited

ichsan2895 commented Feb 8, 2024 •

edited

Guangyun-Xu commented Feb 8, 2024

ichsan2895 commented Feb 10, 2024

refine-pixsfm work with colmap #2715

Are you sure you want to change the base?

refine-pixsfm work with colmap #2715

Conversation

Guangyun-Xu commented Jan 2, 2024

jb-ye commented Jan 3, 2024

Guangyun-Xu commented Jan 4, 2024

thomakah commented Jan 10, 2024

Guangyun-Xu commented Jan 11, 2024

jb-ye commented Jan 11, 2024

thomakah commented Jan 11, 2024

Guangyun-Xu commented Jan 12, 2024

Guangyun-Xu commented Jan 12, 2024

Guangyun-Xu commented Jan 12, 2024

Guangyun-Xu commented Jan 12, 2024

jb-ye commented Jan 12, 2024

ichsan2895 commented Feb 6, 2024

ichsan2895 commented Feb 7, 2024

ichsan2895 commented Feb 7, 2024 • edited

ichsan2895 commented Feb 8, 2024 • edited

Guangyun-Xu commented Feb 8, 2024

ichsan2895 commented Feb 10, 2024

ichsan2895 commented Feb 7, 2024 •

edited

ichsan2895 commented Feb 8, 2024 •

edited