to_annotate.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os \n",
    "import json \n",
    "from skimage import io\n",
    "import skimage\n",
    "import tqdm\n",
    "from collections import defaultdict\n",
    "\n",
    "path_instagram = '.../...' #@@@ OVERRIDE: Path of folder with images downloaded from IG for test set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TEST SET"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Data Collection and Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The aim of this project is to perform fashion instance segmentation into the wild. Fashion instance segmentation means detetcting each garment in a picture with a ploygon or mask. \"In to the wild\" refers to the quality of the images: the aim is not to detect garments in \"street\" photos, but in \"wild\" images. Street photos are out-of-the-studio good quality pictures, that usually focus on one professional model, have sophisticated backgrounds, different lighting conditions, and minor occlusion due to various yet standard poses. On the opposite side there are \"wild\" photos. They have no constraints at all: they are user-created, hence they may have heavy occlusion, bad lighting, cropping, and an overall poor quality.\n",
    "\n",
    "**Data Collection**  \n",
    "The train and validation set are built starting from publicly available fashion datasets.  \n",
    "The test set, used to evaluate the segmentation model's performance, is built from scratch. First, a number of images from Instagram has been downloaded. Different keyworkds have been used as the selection creterion, resulting into a very wild dataset with 38,913 images with various size, occlusion, lighting, resolution, zoom-in, and content. Regarding this last aspect, many images don't have any garment at all. \n",
    "\n",
    "**Data Cleaning**  \n",
    "To clean the 38,913 wild dataset, first no-color images have been removed to be consistent with the segmentation model that takes as input RGB images. By no-color images, I consider both black-and-white and greyscale images. Even if these two terms are used interchangeably, they mean two different things: a black-and-white image simply consists of two colors — black and white (1 channel), while a greyscale one is composed exclusively of shades of gray, varying from black to white (3 channels). The script to detect color and no-color images is: ```color_detector.py```\n",
    "\n",
    "Secondly, a face detector, MTCNN - Multi-task Cascaded CNN [(Zhang, 2016)](https://arxiv.org/abs/1604.02878), has been applied to get the ids of images with at least one person. The cascade structure makes this network pretty fast. The motivation behind this step is that many images in the initial IG dataset show objects or landscapes. Therefore they should be removed, since they don't contain any garment. Moreover, uploading in the annotation tool images with at least one face would increase the probability to find garments and save a lot of time. However, since it would be interesting to test the performance of the model even on photos of clothes not on the human body, some pictures satisfying this condition are retrieved from the list of no-face ids and add to the face-images' pool. The script to detect face and no-face images is: ```face_detector.py```\n",
    "\n",
    "After these two steps, the number of potential test images to annotate is 24,033 (61% of IG downloaded images).\n",
    "  \n",
    "Number of potential images to annotate for test set:\n",
    "| Initial number | 38.913 \n",
    "|---|---|\n",
    "| Potential number | 24.033|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_ids = [i for i in sorted(os.listdir(path_instagram)) if not i.startswith('.')]\n",
    "assert len(all_ids) == len(set(all_ids)) == 38913"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. NO-COLOR\n",
    "To be consistent in training the model, select only color images.  \n",
    "Problem: How to detect an image as black and white or greyscale? \n",
    "Implement an algorithm that checks the following 3 conditions:\n",
    "1. ```Number of channels < 3```: rare case, many times images have 3 channels but they look like b/w because they have many shadows of grey pixels.\n",
    "2. ```(R == G == B).all```: not always True\n",
    "3. ```Channel variance < threshold```: define a threshold T, if variance between channels is lower, then it is a greyscale immage \n",
    "\n",
    "SCRIPT: ```color_detector.py```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the color detector, output: txt files with the ids of no-color images\n",
    "ids_color_lists = ['ids_maybe_color', 'ids_bw_grey', 'ids_no_idea_color', 'ids_monocolor']\n",
    "ids_problem_color = {}\n",
    "for name in ids_color_lists:\n",
    "    with open(name+'.txt', 'r') as f:\n",
    "        ids_problem_color[name] = [i.strip() for i in f.readlines()]\n",
    "        \n",
    "for k,v in ids_problem_color.items():\n",
    "    print(k, len(v))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove from potential test images: ids_bw_grey\n",
    "num_potential_imgs = len(all_ids)\n",
    "print('# Potential Images TEST: {}'.format(num_potential_imgs))\n",
    "ids_no_bw = [i for i in all_ids if i not in ids_problem_color['ids_bw_grey']]\n",
    "assert (len(all_ids)-len(ids_problem_color['ids_bw_grey'])) == len(ids_no_bw)\n",
    "num_potential_imgs = len(ids_no_bw)\n",
    "print('# Potential Images TEST: {}'.format(num_potential_imgs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. NO-FACE\n",
    "\n",
    "To avoid uploading in the annotation tool images of objects or landscapes without any garment.   \n",
    "Method: Multi-task Cascade CNN - a network with three convolutional networks that learns simultaneously face detection and keypoints alignment. Accuracy: ~ 95%\n",
    "\n",
    "SCRIPT: ```face_detector.py```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the face detector, output; txt file with the ids of no-face images\n",
    "with open('ids_no_face.txt', 'r') as f:\n",
    "    ids_no_face = [i.strip() for i in f.readlines()]\n",
    "\n",
    "print('ids_no_face', len(ids_no_face))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove from potential test images: ids_no_face\n",
    "print('# Potential Images TEST: {}'.format(num_potential_imgs))\n",
    "ids_no_bw_face = [i for i in ids_no_bw if i not in ids_no_face]\n",
    "assert (len(all_ids)-len(set(sorted(ids_no_face+ids_problem_color['ids_bw_grey'])))) == len(ids_no_bw_face)\n",
    "num_potential_imgs = len(ids_no_bw_face)\n",
    "print('# Potential Images TEST: {}'.format(num_potential_imgs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Data Annotation\n",
    "\n",
    "Annotator: [VGG Image Annotator (VIA)](#https://www.robots.ox.ac.uk/~vgg/software/via/)  \n",
    "Download: [Link](#https://www.robots.ox.ac.uk/~vgg/software/via/downloads/via-2.0.11.zip)\n",
    "\n",
    "Given the 24,033 images to use as test set, only 500 images have been annotated. The annotation tool used in this project is the Visual Geometry Group Image Annotator (VIA), a software that runs as an offline application in most modern web browsers.  \n",
    "\n",
    "During the annotation phase, images have been discarded following these criteria: \n",
    "1. Skip images/instances if clothes are not human recognizable; \n",
    "2. Skip images/instances if clothes are too much occluded;\n",
    "3. Skip images/instances if clothes are too small (precise segmentation is impossible);\n",
    "4. As per DeepFashion2, include in the segmentation polygon even the area of the item that is occluded by another item/human body part.\n",
    "\n",
    "The VIA annotation tool returns annotations in VIA format. However, the segmentation model needs as input annotations in COCO format. To convert the annotation file from VIA to COCO use the function in ```converters.ipynb```.\n",
    "\n",
    "#### <font color='red'>Following cells contain code to manipulate the VIA json file with annotations. Customize or delete the code according to your needs.</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# MERGE JSONS: I saved annotations in 5 different jsosn, use this code to merge the 5 jsons in 1 json\n",
    "paths_DLCV = [\n",
    "    '/Users/francescabianchessi/THESIS/ANNOTATING_DF2/DLCV_501.json',\n",
    "    '/Users/francescabianchessi/THESIS/ANNOTATING_DF2/DLCV_502_1000.json',\n",
    "    '/Users/francescabianchessi/THESIS/ANNOTATING_DF2/DLCV_1001_2009.json',\n",
    "    '/Users/francescabianchessi/THESIS/ANNOTATING_DF2/DLCV_38000_38912.json'\n",
    "]\n",
    "\n",
    "list_jsons = [json.load(open(p)) for p in paths_DLCV]\n",
    "z = list_jsons[0]['_via_img_metadata'] | list_jsons[1]['_via_img_metadata'] | list_jsons[2]['_via_img_metadata'] | list_jsons[3]['_via_img_metadata'] \n",
    "\n",
    "new_js = {}\n",
    "new_js['_via_settings'] = list_jsons[0]['_via_settings']\n",
    "new_js['_via_settings']['project']['name'] = 'DLCV_merge'\n",
    "new_js['_via_img_metadata'] = z\n",
    "new_js['_via_attributes'] = list_jsons[0]['_via_attributes']\n",
    "new_js['_via_data_format_version'] = list_jsons[0]['_via_data_format_version']\n",
    "new_js['_via_image_id_list'] = sorted([l for j in list_jsons for l in j['_via_image_id_list']])\n",
    "\n",
    "with open('DLCV_VIA_merge_pre_delete.json', 'w') as f:\n",
    "    json.dump(new_js, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# REMOVE DATA: Remove no-annotated images still included in the merged json file\n",
    "js = json.load(open('/Users/francescabianchessi/THESIS/ANNOTATING_DF2/DLCV_merge_pre_delate.json'))\n",
    "meta_data = {k:v for k,v in js['_via_img_metadata'].items() if len(js['_via_img_metadata'][k]['regions'])>0}\n",
    "image_list = [name for name in js['_via_image_id_list'] if name in meta_data.keys()]\n",
    "new_js = {}\n",
    "new_js['_via_settings'] = js['_via_settings']\n",
    "new_js['_via_settings']['project']['name'] = 'DLCV_merge_post_delete'\n",
    "new_js['_via_img_metadata'] = meta_data\n",
    "new_js['_via_attributes'] = js['_via_attributes']\n",
    "new_js['_via_data_format_version'] = js['_via_data_format_version']\n",
    "new_js['_via_image_id_list'] = sorted(image_list)\n",
    "\n",
    "with open('DLCV_merge_post_delete.json', 'w') as f:\n",
    "    json.dump(new_js, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1893\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "500"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# TRACK annotation progress: save the annotations from VIA and check how many annotations you have done so far\n",
    "path = '/Users/francescabianchessi/THESIS/ANNOTATING_DF2/via_test_reviewed.json'\n",
    "j = json.load(open(path)) #'_via_img_metadata', '_via_attributes', '_via_data_format_version', '_via_image_id_list'\n",
    "\n",
    "masks_count = []\n",
    "for k, v in j['_via_img_metadata'].items():\n",
    "    masks_count.append((int(k.split('.jpg')[0]), len(j['_via_img_metadata'][k]['regions'])))\n",
    "\n",
    "print(sum([t[1] for t in masks_count]))\n",
    "\n",
    "ids_segmented = []\n",
    "for t in masks_count:\n",
    "    if t[1] > 0:\n",
    "        ids_segmented.append(t[0])\n",
    "len(ids_segmented)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check all images have annotations!\n",
    "for d in j['_via_img_metadata'].values():\n",
    "    for r in d['regions']:\n",
    "        try:\n",
    "            assert r['region_attributes']['clothing'] or r['region_attributes']['accessories'] or r['region_attributes']['shoes']\n",
    "        except: \n",
    "            print(d['filename'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TRAIN - VALIDATION SET"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Data Collection and Cleaning: DeepFashion2 \n",
    "\n",
    "DeepFashion2 (DF2) is the only publicly-available fashion dataset in the instance segmentation literature with both wild and street photos. The street photos are retrieved from online shopping/fahsion websites and are referred to as \"shop\" images. The wild photos are pictures taken by consumers and are referred to as \"user\" images. The actual reason why DF2 contains also wild images is that the final goal of the researchers behind this dataset was the task of consumer-to-shop clothes retrieval. \n",
    "\n",
    "Idea: create a training/validation set with both street and wild images from DF2 to perform instance segmentation on IG wild test set.\n",
    "\n",
    "DF2 offers 191,961 images, a part of the original training set, and 32,153 images, half of the original validation set. All of them are annotated with the 13 categories defined by DF2 authors. For a deeper analysis of this dataset look at ```summary_datasets.ipynb```.  \n",
    "In this project 30 categories are considered, hence all the DF2 images have been re-annotated in accordance with the new categorization. \n",
    "Images to further annotate have been retrieved rom the 32k validation set because its smaller size makes it easier to unzip and manipulate. \n",
    "\n",
    "\n",
    "### 2. Data Annotation \n",
    "\n",
    "Annotator: [VGG Image Annotator (VIA)](#https://www.robots.ox.ac.uk/~vgg/software/via/)  \n",
    "Download: [Link](#https://www.robots.ox.ac.uk/~vgg/software/via/downloads/via-2.0.11.zip)\n",
    "\n",
    "Given the 32k images to use as train/validation set, only 8,500 images have been annotated. The annotation tool used in this project is the Visual Geometry Group Image Annotator (VIA), a software that runs as an offline application in most modern web browsers.\n",
    "\n",
    "During the annotation phase, images have been discarded following these criteria: \n",
    "1. Skip images/instances if clothes are not human recognizable; \n",
    "2. Skip images/instances if clothes are too much occluded;\n",
    "3. Skip images/instances if clothes are too small (precise segmentation is impossible);\n",
    "4. As per DeepFashion2, include in the segmentation polygon even the area of the item that is occluded by another item/human body part.\n",
    "\n",
    "The VIA annotation tool returns annotations in VIA format. However, the segmentation model needs as input annotations in COCO format. To convert the annotation file from VIA to COCO use the function in ```converters.ipynb```.\n",
    "\n",
    "#### <font color='red'>Following cells contain code to track the progress made during annotation.</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### USERS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5511 5511\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5511/5511 [00:00<00:00, 181449.17it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "| Tot    | 6510   |\n",
      "| No Use | 6   |\n",
      "| Done   | 6504   | 99.91% | 118.25%\n",
      "\n",
      "| # Masks      | 8822\n",
      "| AVG(# Masks) | 1.3563960639606396\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# AIM: TRACK - Track annotation's progress\n",
    "c = '/Users/francescabianchessi/THESIS/ANNOTATING_DF2/df2_user_6510.json'\n",
    "jcon = json.load(open(c)) #'_via_img_metadata', '_via_attributes', '_via_data_format_version', '_via_image_id_list'\n",
    "print(len(jcon['_via_image_id_list']), len(jcon['_via_img_metadata']))\n",
    "\n",
    "masks_count = []\n",
    "limit = 6510 # ID of last image you have annotated\n",
    "removed = 0\n",
    "for k,v in tqdm.tqdm(jcon['_via_img_metadata'].items()):\n",
    "    if int(k) <= limit:\n",
    "        try: # if mask_0 has no clothing, neither the others\n",
    "            v['regions'][0]['region_attributes']['clothing']\n",
    "            masks_count.append((int(k.split('.jpg')[0]), len(jcon['_via_img_metadata'][k]['regions'])))\n",
    "        except:\n",
    "            removed += 1\n",
    "\n",
    "num_masks = sum([t[1] for t in masks_count])\n",
    "done = limit - removed\n",
    "print(\"\"\"\n",
    "| Tot    | {}   |\n",
    "| No Use | {}   |\n",
    "| Done   | {}   | {:.2%} | {:.2%}\n",
    "\n",
    "| # Masks      | {}\n",
    "| AVG(# Masks) | {}\n",
    "\"\"\".format(limit, removed, done, done/limit, done/5500, num_masks, num_masks/done ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### SHOP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3001 3001\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3001/3001 [00:00<00:00, 347940.80it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "| Tot    | 3001   |\n",
      "| No Use | 0   |\n",
      "| Done   | 3001   | 100.00% | 54.56%\n",
      "\n",
      "| # Masks      | 6902\n",
      "| AVG(# Masks) | 2.299900033322226\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# AIM: TRACK - Track annotation's progress\n",
    "s = '/Users/francescabianchessi/THESIS/ANNOTATING_DF2/post_delete.json'\n",
    "jshop = json.load(open(s)) #'_via_img_metadata', '_via_attributes', '_via_data_format_version', '_via_image_id_list'\n",
    "print(len(jshop['_via_image_id_list']), len(jshop['_via_img_metadata']))\n",
    "\n",
    "masks_count = []\n",
    "limit = 3001 # ID of last image you have annotated\n",
    "removed = 0\n",
    "for k,v in tqdm.tqdm(jshop['_via_img_metadata'].items()):\n",
    "    if int(k) <= limit:\n",
    "        try: # if mask_0 has no clothing, neither the others\n",
    "            v['regions'][0]['region_attributes']['clothing']\n",
    "            masks_count.append((int(k.split('.jpg')[0]), len(jshop['_via_img_metadata'][k]['regions'])))\n",
    "        except:\n",
    "            removed += 1\n",
    "\n",
    "num_masks = sum([t[1] for t in masks_count])\n",
    "done = limit - removed\n",
    "print(\"\"\"\n",
    "| Tot    | {}   |\n",
    "| No Use | {}   |\n",
    "| Done   | {}   | {:.2%} | {:.2%}\n",
    "\n",
    "| # Masks      | {}\n",
    "| AVG(# Masks) | {}\n",
    "\"\"\".format(limit, removed, done, done/limit, done/5500, num_masks, num_masks/done ))"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "40e58e24db7711c8a4184049570588e5df3102e3a293043e4b369eb002d4f5a0"
  },
  "kernelspec": {
   "display_name": "Python 3.7.7 64-bit ('base': conda)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}