Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localization

Siyan Dong* · Shuzhe Wang* · Shaohui Liu · Lulu Cai · Qingnan Fan · Juho Kannala · Yanchao Yang

Paper | Online Demo (Coming Soon)

Reloc3r is a simple yet effective camera pose estimation framework that combines a pre-trained two-view relative camera pose regression network with a multi-view motion averaging module.

Trained on approximately 8 million posed image pairs, Reloc3r achieves surprisingly good performance and generalization ability, producing high-quality camera pose estimates in real-time.

TODO List

• Release pre-trained weights and inference code.
• Release evaluation code for ScanNet1500 and Cambridge datasets.
• Release demo code for wild images and videos.
• Release evaluation code for other datasets.
• Release the accelerated version for visual localization.
• Release Gradio Demo.
• Release training code and data.

Installation

1. Clone Reloc3r

git clone --recursive https://github.com/ffrivera0/reloc3r.git
cd reloc3r
# if you have already cloned reloc3r:
# git submodule update --init --recursive

2. Create the environment using conda

conda create -n reloc3r python=3.11 cmake=3.14.0
conda activate reloc3r 
conda install pytorch torchvision pytorch-cuda=12.1 -c pytorch -c nvidia  # use the correct version of cuda for your system
pip install -r requirements.txt
# optional: you can also install additional packages to:
# - add support for HEIC images
pip install -r requirements_optional.txt

3. Optional: Compile the cuda kernels for RoPE

# Reloc3r relies on RoPE positional embeddings for which you can compile some cuda kernels for faster runtime.
cd croco/models/curope/
python setup.py build_ext --inplace
cd ../../../

4. Download the Reloc3r-224/512 checkpoints here, and place them under ./checkpoints/.

Relative Pose Estimation on ScanNet1500

Download the dataset here and unzip it to ./data/scannet1500/. Then run the following script. You will obtain results similar to those presented in our paper.

bash scripts/eval_relpose.sh

Visual Localization on Cambridge

Download the dataset here and unzip it to ./data/cambridge/. Then run the following script. You will obtain results similar to those presented in our paper.

bash scripts/eval_visloc.sh

Demo for Wild Images

In the demos below, you can run Reloc3r on your own data.

For relative pose estimation, try the demo code in wild_relpose.py. We provide some image pairs used in our paper.

# replace the args with your paths
python wild_relpose.py --v1_path ./data/wild_images/zurich0.jpg --v2_path ./data/wild_images/zurich1.jpg --output_folder ./data/wild_images/

Visualize the relative pose

# replace the args with your paths
python visualization.py --mode relpose --pose_path ./data/wild_images/pose2to1.txt

For visual localization, the demo code in wild_visloc.py estimates absolute camera poses from sampled frames in self-captured videos.

Important: The demo uses the first and last frames as the database, which requires overlapping regions among all images. This demo does not support linear motion. We provide some videos as examples.

# replace the args with your paths
python wild_visloc.py --video_path ./data/wild_video/ids.MOV --output_folder ./data/wild_video

Visualize the absolute poses

# replace the args with your paths
python visualization.py --mode visloc --pose_folder ./data/wild_video/ids_poses/

Citation

If you find our work helpful in your research, please consider citing:

@article{reloc3r,
  title={Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localization},
  author={Dong, Siyan and Wang, Shuzhe and Liu, Shaohui and Cai, Lulu and Fan, Qingnan and Kannala, Juho and Yang, Yanchao},
  journal={arXiv preprint arXiv:2412.08376},
  year={2024}
}

Acknowledgments

Our implementation is based on several awesome repositories:

• Croco
• DUSt3R

We thank the respective authors for open-sourcing their code.