FcF-Inpainting

Jitesh Jain^†, Yuqian Zhou^†, Ning Yu, Humphrey Shi, WACV 2023

^† Equal Contribution

[Project Page] [arXiv] [pdf] [BibTeX]

This repo contains the code for our paper Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand.

News

• [October 6, 2022]: You can host your own FcF-Inpainting demo using streamlit by following the instructions here.
• [September 5, 2022]: FcF-Inpainting is now available in the image inpainting tool Lama Cleaner. Thanks to @Sanster for integrating FcF-Inpainting into Lama Cleaner!
• [August 16, 2022]: FcF-Inpainting is accepted to WACV 2023!
• [August 5, 2022]: Project Page, ArXiv Preprint and GitHub Repo are public!

Contents

1. Setup Instructions
2. Dataset Preparation
3. Training and Evaluation
4. Citing FcF-Inpainting

1. Setup Instructions

• Clone the repo:

  git clone https://github.com/SHI-Labs/FcF-Inpainting.git
  cd FcF-Inpainting

• Create a conda environment:

  conda create --name fcfgan python=3.7
  conda activate fcfgan

• Install Pytorch 1.7.1 and other dependencies:

  pip3 install -r requirements.txt
  export TORCH_HOME=$(pwd) && export PYTHONPATH=.

• Download the models for the high receptive perceptual loss:

  mkdir -p ade20k/ade20k-resnet50dilated-ppm_deepsup/
  wget -P ade20k/ade20k-resnet50dilated-ppm_deepsup/ http://sceneparsing.csail.mit.edu/model/pytorch/ade20k-resnet50dilated-ppm_deepsup/encoder_epoch_20.pth

2. Dataset Preparation

CelebA-HQ Dataset

Training Data

• Download data256x256.zip from gdrive.

  mkdir -p datasets/
  # unzip & split into train/test/visualization
  bash tools/prepare_celebahq.sh
  
  datasets
  ├── celeba-hq-dataset
  │   ├── train_256
  │   ├── val_source_256
  │   ├── visual_test_source_256

Evaluation Data

• Generate 2k (image, mask) pairs to be used for evaluation.

  bash tools/prepare_celebahq_evaluation.sh

Places2 Dataset

Training Data

• Download the Places2 dataset:

  mkdir -p datasets/
  mkdir datasets/places2_dataset/
  wget http://data.csail.mit.edu/places/places365/train_large_places365challenge.tar
  tar -xvf train_large_places365challenge.tar -C datasets/places2_dataset/
  mv datasets/places2_datasets/data_large datasets/places2_dataset/train
  
  wget http://data.csail.mit.edu/places/places365/val_large.tar
  tar -xvf val_large.tar -C datasets/places2_dataset/
  mv datasets/places2_dataset/val_large datasets/places2_dataset/val
  
  datasets
  ├── places2_dataset
  │   ├── train
  │   ├── val

• Generate 10k (image, mask) pairs to be used for validation during training.

  bash tools/prepare_places_val.sh

Evaluation Data

Irregular Mask Strategy

• Generate 30k (image, mask) pairs to be used for evaluation.

  bash tools/prepare_places_evaluation.sh

Segmentation Mask strategy

• Install Detectron2-v0.5.

  python -m pip install detectron2==0.5 -f \
  https://dl.fbaipublicfiles.com/detectron2/wheels/cu110/torch1.7/index.html

• Download networks for segmentation masks:

  mkdir -p ade20k/ade20k-resnet50dilated-ppm_deepsup/
  wget -P ade20k/ade20k-resnet50dilated-ppm_deepsup/ http://sceneparsing.csail.mit.edu/model/pytorch/ade20k-resnet50dilated-ppm_deepsup/encoder_epoch_20.pth
  wget -P ade20k/ade20k-resnet50dilated-ppm_deepsup/ http://sceneparsing.csail.mit.edu/model/pytorch/ade20k-resnet50dilated-ppm_deepsup/decoder_epoch_20.pth

• Generate (image, mask) pairs to be used for segmentation mask based evaluation.

  bash tools/prepare_places_segm_evaluation.sh

Note: The pairs are only generated for images with detected instances.

3. Training and Evaluation

Training on 256x256

• Execute the following command to start training for 25M images on 8 gpus with 16 images per gpu:

  python train.py \
      --outdir=training-runs-inp \
      --img_data=datasets/places2_dataset/train \
      --gpus 8 \
      --kimg 25000 \
      --gamma 10 \
      --aug 'noaug' \
      --metrics True \
      --eval_img_data datasets/places2_dataset/evaluation/random_segm_256
      --batch 128

Note: If the process hangs on Setting up PyTorch plugin ..., refer to this issue.

Evaluation

Pretrained Models

checkpoint	Description
places_512.pkl	Model trained on 512x512 for 25M Places2 images
places.pkl	Model trained on 256x256 for 25M Places2 images
celeba-hq.pkl	Model trained on 256x256 for 25M CelebA-HQ images

• Run the following command to calculate the metric scores (fid, ssim and lpips) using 8 gpus:

  python evaluate.py \
      --img_data=datasets/places2_dataset/evaluation/random_segm_256 \
      --network=[path-to-checkpoint] \
      --num_gpus=8

Demo

• Run the following command and find the results in the visualizations/ folder:

  python demo.py \
  --img_data=datasets/demo/places2 \
  --network=[path-to-checkpoint] \
  --resolution 256

4. Citing FcF-Inpainting

@inproceedings{jain2022keys,
  title={Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand},
  author={Jitesh Jain and Yuqian Zhou and Ning Yu and Humphrey Shi},
  booktitle={WACV},
  year={2023}
} 

Acknowledgement

Code is heavily based on the following repositories: stylegan2-ada-pytorch and lama.