Skip to content
/ sfd2 Public

[CVPR 2023] SFD2: Semantic-guided Feature Detection and Description. Embedding semantics into local features implicitly for long-term visual localization

Notifications You must be signed in to change notification settings

feixue94/sfd2

Repository files navigation

SFD2: Semantic-guided Feature Detection and Description

In this work, we propose to leverage global instances, which are robust to illumination and season changes for both coarse and fine localization. For coarse localization, instead of performing global reference search directly, we search for reference images from recognized global instances progressively. The recognized instances are further utilized for instance-wise feature detection and matching to enhance the localization accuracy.

Dependencies

  • Python 3 >= 3.6
  • PyTorch >= 1.8
  • OpenCV >= 3.4
  • NumPy >= 1.18
  • segmentation-models-pytorch = 0.1.3
  • colmap
  • pycolmap = 0.0.1

Data preparation

  • training data. We use the same training dataset as R2D2. Please download the training dataset following the instructions provided by R2D2.
  • segmentation model. ConvXt is used to provide semantic labels and semantic-aware features for stability learning in the training process.
  • local feature model. SuperPoint is used to provide local reliability in the training process.

Pretrained weights

Pretrained weight of SFD2 is in the weights directory. If you want to retrain the model, please also download the weights of ConvXt and SuperPoint from here and put them nto the weights directory.

Localization results

Please download datasets e.g. Aachen_v1.1, RobotCar-Seasons v2, and Extended-CMU-Seasons from the visualization benchmark for evaluation.

  • localization on Aachen_v1.1
./test_aachenv_1_1

you will get results like this:

Day Night
88.2 / 96.0 / 98.7 78.0 / 92.1 / 99.5
  • localization on RobotCar-Seasons
./test_robotcar

you will get results like this:

day night night-rain
56.9 / 81.6 / 97.4 27.6 / 66.2 / 90.2 43.0 / 71.1 / 90.0
  • localization on Extended CMU-Seasons
./test_ecmu

you will get results like this:

urban suburban park
95.0 / 97.5 / 98.6 90.5 / 92.7 / 95.3 86.4 / 89.1 / 91.2

Training

./train.sh

BibTeX Citation

If you use any ideas from the paper or code from this repo, please consider citing:

@inproceedings{xue2023sfd2,
  author    = {Fei Xue and Ignas Budvytis and Roberto Cipolla},
  title     = {SFD2: Semantic-guided Feature Detection and Description},
  booktitle = {CVPR},
  year      = {2023}
}

Acknowledgements

Part of the code is from previous excellent works including SuperPoint, R2D2 , HLoc, ConvXt, LBR. You can find more details from their released repositories if you are interested in their works.

About

[CVPR 2023] SFD2: Semantic-guided Feature Detection and Description. Embedding semantics into local features implicitly for long-term visual localization

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published