Skip to content

We simulate traveling back in time with a modern camera to rephotograph famous historical subjects.

License

MIT and 2 other licenses found

Licenses found

MIT
LICENSE
Unknown
LICENSE-NVIDIA
MIT
LICENSE-STYLEGAN2
Notifications You must be signed in to change notification settings

gessyoo/Time-Travel-Rephotography.github.io

 
 

Repository files navigation

[SIGGRAPH Asia 2021] Time-Travel Rephotography

Open in Colab

Many historical people were only ever captured by old, faded, black and white photos, that are distorted due to the limitations of early cameras and the passage of time. This paper simulates traveling back in time with a modern camera to rephotograph famous subjects. Unlike conventional image restoration filters which apply independent operations like denoising, colorization, and superresolution, we leverage the StyleGAN2 framework to project old photos into the space of modern high-resolution photos, achieving all of these effects in a unified framework. A unique challenge with this approach is retaining the identity and pose of the subject in the original photo, while discarding the many artifacts frequently seen in low-quality antique photos. Our comparisons to current state-of-the-art restoration filters show significant improvements and compelling results for a variety of important historical people.

Time-Travel Rephotography
Xuan Luo, Xuaner Zhang, Paul Yoo, Ricardo Martin-Brualla, Jason Lawrence, and Steven M. Seitz
In SIGGRAPH Asia 2021.

Demo

We provide an easy-to-get-started demo using Google Colab! The Colab will allow you to try our method on the sample Abraham Lincoln photo or your own photos using Cloud GPUs on Google Colab.

Open in Colab

Or you can run our method on your own machine following the instructions below.

Prerequisite

  • Pull third-party packages.

sudo apt-get install build-essential cmake pkg-config # necessary for dlib to compile sudo apt-get install libx11-dev libatlas-base-dev sudo apt-get install libgtk-3-dev libboost-python-dev

git submodule update --init --recursive
  • Install python packages.
    conda create --name rephotography python=3.8.5
    conda activate rephotography
    conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1 -c pytorch
    pip install -r requirements.txt
    

Quick Start

Run our method on the example photo of Abraham Lincoln.

  • Download models:
    ./scripts/download_checkpoints.sh
    
  • Run:
    ./scripts/run.sh b "dataset/Abraham Lincoln_01.png" 0.75 
    
  • You can inspect the optimization process by
    tensorboard --logdir "log/Abraham Lincoln_01"
    
  • You can find your results as below.
    results/
      Abraham Lincoln_01/       # intermediate outputs for histogram matching and face parsing
      Abraham Lincoln_01_b.png  # the input after matching the histogram of the sibling image
      Abraham Lincoln_01-b-G0.75-init(10,18)-s256-vgg1-vggface0.3-eye0.1-color1.0e+10-cx0.1(relu3_4,relu2_2,relu1_2)-NR5.0e+04-lr0.1_0.01-c32-wp(250,750)-init.png        # the sibling image
      Abraham Lincoln_01-b-G0.75-init(10,18)-s256-vgg1-vggface0.3-eye0.1-color1.0e+10-cx0.1(relu3_4,relu2_2,relu1_2)-NR5.0e+04-lr0.1_0.01-c32-wp(250,750)-init.pt         # the sibing latent codes and initialized noise maps
      Abraham Lincoln_01-b-G0.75-init(10,18)-s256-vgg1-vggface0.3-eye0.1-color1.0e+10-cx0.1(relu3_4,relu2_2,relu1_2)-NR5.0e+04-lr0.1_0.01-c32-wp(250,750).png             # the output result
      Abraham Lincoln_01-b-G0.75-init(10,18)-s256-vgg1-vggface0.3-eye0.1-color1.0e+10-cx0.1(relu3_4,relu2_2,relu1_2)-NR5.0e+04-lr0.1_0.01-c32-wp(250,750).pt              # the final optimized latent codes and noise maps
      Abraham Lincoln_01-b-G0.75-init(10,18)-s256-vgg1-vggface0.3-eye0.1-color1.0e+10-cx0.1(relu3_4,relu2_2,relu1_2)-NR5.0e+04-lr0.1_0.01-c32-wp(250,750)-rand.png        # the result with the final latent codes but random noise maps
    
    

Run on Your Own Image

  • Crop and align the head regions of your images:

    python -m tools.data.align_images <input_raw_image_dir> <aligned_image_dir>
    
  • Run:

    ./scripts/run.sh <spectral_sensitivity> <input_image_path> <blur_radius>
    

    The spectral_sensitivity can be b (blue-sensitive), gb (orthochromatic), or g (panchromatic). You can roughly estimate the spectral_sensitivity of your photo as follows. Use the blue-sensitive model for photos before 1873, manually select between blue-sensitive and orthochromatic for images from 1873 to 1906 and among all models for photos taken afterwards.

    The blur_radius is the estimated gaussian blur radius in pixels if the input photot is resized to 1024x1024.

Historical Wiki Face Dataset

Path Size Description
Historical Wiki Face Dataset.zip 148 MB Images
spectral_sensitivity.json 6 KB Spectral sensitivity (b, gb, or g).
blur_radius.json 6 KB Blur radius in pixels

The jsons are dictionares that map input names to the corresponding spectral sensitivity or blur radius. Due to copyright constraints, Historical Wiki Face Dataset.zip contains all images in the Historical Wiki Face Dataset that were used in our user study except the photo of Mao Zedong. You can download it separately and crop it as above.

Citation

If you find our code useful, please consider citing our paper:

@article{Luo-Rephotography-2021,
  author    = {Luo, Xuan and Zhang, Xuaner and Yoo, Paul and Martin-Brualla, Ricardo and Lawrence, Jason and Seitz, Steven M.},
  title     = {Time-Travel Rephotography},
  journal = {ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH Asia 2021)},
  publisher = {ACM New York, NY, USA},
  volume = {40},
  number = {6},
  articleno = {213},
  doi = {https://doi.org/10.1145/3478513.3480485},
  year = {2021},
  month = {12}
}

License

This work is licensed under MIT License. See LICENSE for details.

Codes for the StyleGAN2 model come from https://github.com/rosinality/stylegan2-pytorch.

Acknowledgments

We thank Nick Brandreth for capturing the dry plate photos. We thank Bo Zhang, Qingnan Fan, Roy Or-El, Aleksander Holynski and Keunhong Park for insightful advice. We thank Xiaojie Feng for his contributions on the colab demo.

About

We simulate traveling back in time with a modern camera to rephotograph famous historical subjects.

Resources

License

MIT and 2 other licenses found

Licenses found

MIT
LICENSE
Unknown
LICENSE-NVIDIA
MIT
LICENSE-STYLEGAN2

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 88.1%
  • Cuda 9.1%
  • Shell 1.4%
  • C++ 1.4%