MaRINeR: Enhancing Novel Views by Matching Rendered Images with Nearby References

Project Page | Paper | Video

This repository contains code for the paper "MaRINeR: Enhancing Novel Views by Matching Rendered Images with Nearby References", ECCV 2024.

Table of Content

• Demo
• Installation
• Data and model weights
• Usage
• Training
• Evaluation
• Acknowledgement

Demo

We provide a demo using Google Colab!

Installation

conda (Recommended) - Clone the repository and then create and activate a mariner conda environment using the provided environment definition:

conda env create -f environment.yml
conda activate mariner

Dependencies

• torchvision==0.16.0
• tensorboard===2.17.0
• lightning==2.2.1
• opencv-python==4.10.0.84
• jsonargparse==4.31.0
• jsonargparse[signatures]>=4.26.1
• erqa==1.1.2
• lpips==0.1.4
• tabulate==0.9.0
• rich==13.7.1
• numpy==1.26.4

Data and model weights

We train and test our model on data from LaMAR. Additionally we test the model on data from MeshLoc (renderings), 12Scenes, NeRF, IBRnet (renderings), Nerfacto.

• Demo dataset
• Train dataset
• Test datasets
• Download the weights and place them in pretrained_weights/

Usage

Prediction

python mariner/main.py predict \
    -c configs/MaRINeR.yml \
    --ckpt_path /path/to/the/project/pretrained_weights/mariner.ckpt \
    --data_dir /path/to/the/demo_data

The data_dir folder needs to contain input and ref subfolders with the corresponding images. Creates the outputs in a subfolder out in data_dir.

/data_dir/
├── input
│   ├── 00000.png
│   ├── 00001.png
│   ├── 00002.png
└── ref
    ├── 00000.png
    ├── 00001.png
    ├── 00002.png

Optional: Larger resolution

For images with resolution much larger than 160, Real-ERSGAN can be used as pipeline extension.

Pull the submodule and download the pretrained weights:

git submodule update --init --recursive
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P thirdparty/Real-ESRGAN/weights
cd thirdparty/Real-ESRGAN && python setup.py develop

Predict with:

python scripts/predict_scaled.py \
     -c configs/MaRINeR.yml \
     --ckpt_path /path/to/the/project/pretrained_weights/mariner.ckpt \
     --data_dir /path/to/the/demo_data

Training

To train the model with the default config, 32 GB of VRAM are needed. Parameters, such as the batch_size, can be changed in configs/MaRINeR.yml or added as options e.g --batch_size 2.

python mariner/main.py fit \
     -c configs/MaRINeR.yml \
     --train_data_dir /path/to/the/train_data/CAB_merged_LIN/train \
     --test_data_dir /path/to/the/train_data/CAB_merged_LIN/test

Evaluation

Download the test datasets and select one of the available datasets for evaluation:

dataset = { CAB_ref_gt, CAB_ref_lvl_1, LIN_ref_lvl_1, HGE_ref_lvl_1, IBRnet, 12SCENES_apt_1_living_ref_lvl_10, NeRF}

python scripts/eval_metrics.py \
    --images /path/to/the/test_data/dataset/out \
            /path/to/the/test_data/dataset/gt

Citation

If you find our work useful, please consider citing:

@inproceedings{bösiger2024marinerenhancingnovelviews,
      title={MaRINeR: Enhancing Novel Views by Matching Rendered Images with Nearby References}, 
      author={Lukas Bösiger and Mihai Dusmanu and Marc Pollefeys and Zuria Bauer},
      year={2024},
      booktitle={European Conference on Computer Vision (ECCV)},
      url={https://arxiv.org/abs/2407.13745}, 
}

Acknowledgement

We borrow some code from MASA-SR.