Instance Eraser

Our project aims to develop an algorithm inspired by Google's MagicEraser feature to remove specified objects from images and seamlessly replace them with background elements. We provide a user-friendly interface where users can upload their image and select the object class to remove. The algorithm involves instance segmentation, removing instances of the specified object category, and reconstructing the image by filling the removed areas with appropriate background information. This algorithm addresses the need to remove unwanted objects while preserving the background, benefiting applications such as landscape photography, medical imaging, and satellite imaging.

Project Demo

A small video demo of the Instance eraser streamlit application.

instance_eraser_demo-2023-06-15_18.49.30.mp4

Method

Our method for developing the "Instance Eraser" algorithm involves the following steps:

1. Dataset Generation: We utilize existing benchmark datasets for training the image segmentation model. Additionally, we modify the dataset by randomly whitening out background regions to create inputs for training the image inpainting models.

2. Instance Segmentation: We employ techniques like Mask R-CNN with a U-Net decoder or pretrained Mask R-CNN models to segment the input image into different regions. For UI we are using pretrained MaskRCNN as it performed better compared to out trained MaskRCNN with U-Net model.

3. Instance Removal: Users specify the object class they want to remove, and based on the image segmentation results, we identify and remove all instances of the specified object category from the image. The removed object pixels are replaced with white background pixels.

4. Image Reconstruction: The instance-removed image serves as input to generative models like Pix2Pix (supervised learning) with a ResNet generator, Pix2Pix with a U-Net generator, or CycleGAN (unsupervised learning). These models are trained to reconstruct the image by filling the removed areas with meaningful background information while preserving visual coherence.

We experimented with all the mentioned models and the best model which was able to recover background was Pix2pix with ResNet generator. Although we were not able to achieve state of the art results, we were able to recover the background which is contextually relevant. With more experimentation and training we can achieve good results.

Prerequisites

• Linux, Windows or macOS
• Python 3
• CPU or NVIDIA GPU + CUDA CuDNN

Getting Started

Installation

• Clone this repo:

git clone https://github.com/rkg266/instance-eraser-Pix2Pix
cd Instance_Eraser
python3 -m venv myenv
source myenv/bin/activate

• Install PyTorch and 0.4+ and other dependencies (e.g., torchvision, visdom and dominate).
• For pip users, please type the command pip3 install -r requirements.txt.

Generating Datasets

• Download the Train2017 and Val2017 of COCO dataset from COCO Website
• Change the paths accordinly in generation/generate_imgs.py python file and run:

python3 generation/generate_imgs.py

Training and Testing

• Change the hyperparameters and dataset path accordingly and run the train.py file

python3 generation/train.py

Executing Streamlit

streamlit run ui_main.py

Now, in your browser, open http://localhost:8501/

The test images are present in the test_images folder.