Facial_Recognition_101.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How does Facial Recognition work?\n",
    "<h2></h2>\n",
    "<center><img src=\"./notebook_images/kyle-glenn-MbPDSi0ILMo-unsplash.jpg\" alt=\"FacePicture\" style=\"width: 60%;\"/></center>\n",
    "\n",
    "Photo by Kyle Glenn on Unsplash\n",
    "\n",
    "<a style=\"background-color:black;color:white;text-decoration:none;padding:4px 6px;font-family:-apple-system, BlinkMacSystemFont, &quot;San Francisco&quot;, &quot;Helvetica Neue&quot;, Helvetica, Ubuntu, Roboto, Noto, &quot;Segoe UI&quot;, Arial, sans-serif;font-size:12px;font-weight:bold;line-height:1.2;display:inline-block;border-radius:3px\" href=\"https://unsplash.com/@kylejglenn?utm_medium=referral&amp;utm_campaign=photographer-credit&amp;utm_content=creditBadge\" target=\"_blank\" rel=\"noopener noreferrer\" title=\"Download free do whatever you want high-resolution photos from Kyle Glenn\"><span style=\"display:inline-block;padding:2px 3px\"><svg xmlns=\"http://www.w3.org/2000/svg\" style=\"height:12px;width:auto;position:relative;vertical-align:middle;top:-2px;fill:white\" viewBox=\"0 0 32 32\"><title>unsplash-logo</title><path d=\"M10 9V0h12v9H10zm12 5h10v18H0V14h10v9h12v-9z\"></path></svg></span><span style=\"display:inline-block;padding:2px 3px\">Kyle Glenn</span></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everyone's face is different, and yet they are all roughly the same.\n",
    "\n",
    "It is easy for humans, and even animals, to recognize faces but how would you instruct a computer to do the same?\n",
    "\n",
    "<font size=\"4\">That is the question we are going to answer in this notebook.</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./notebook_images/BigNineCover-193x300.jpg\" alt=\"FacePicture\" style=\"width: 25%;\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amy Webb - author of *The Big Nine* talks about the 9 global companies that are responsible for the majority of the AI research being done today.  The US has 6, which Amy calls the *g-mafia*, and they are:\n",
    "\n",
    "* Google\n",
    "* Microsoft\n",
    "* Amazon\n",
    "* IBM\n",
    "* Apple\n",
    "* Facebook\n",
    "\n",
    "and 3 in China called the BAT:\n",
    "\n",
    "* Baidu\n",
    "* Alibaba\n",
    "* Tencent\n",
    "\n",
    "Amy Webb points out that these companies, and the countries and governments these companies operate in are shaping the future of the world we live in.  \n",
    "\n",
    "One aspect of that shaping is facial recognition.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Controversy of Facial Recognition\n",
    "\n",
    "<div style=\"height: 20px\"></div>\n",
    "    \n",
    "<center><font size=\"5\">Who owns your face? </font></center>\n",
    "\n",
    "This exact question was asked in an [ACM article](https://cacm.acm.org/news/237592-who-owns-your-face/fulltext) and there is no good answer right now.  We give away our most recognizable attribute every day on social media platforms. Digital representations of our faces are being collected, bought, sold and stored because there are no real restrictions on it.   \n",
    "\n",
    "[My Medium article](https://towardsdatascience.com/using-linkedin-profile-pictures-for-facial-recognition-8be709e8fac) shows on how to perform facial recognition on a single LinkedIn Profile picture.\n",
    "\n",
    "<div style=\"height: 20px\"></div>\n",
    "    \n",
    "<center><font size=\"5\">Social Credit Scoring System </font></center>\n",
    "\n",
    "Today China is working toward a 'Social Credit Scoring System' as reference by Amy Webb and others:\n",
    "\n",
    "* See *The Big Nine*\n",
    "\n",
    "* https://www.cbc.ca/radio/thecurrent/the-current-for-march-7-2019-1.5046443/how-china-s-social-credit-system-blocked-millions-of-people-from-travelling-1.5046445\n",
    "\n",
    "* https://www.wired.com/story/china-social-credit-score-system/\n",
    "\n",
    "According to the official Chinese blueprint is states:\n",
    "\n",
    "\"... it will ‘allow the trustworthy to roam everywhere under heaven while making it hard for the discredited to take a single step. ...\"\n",
    "\n",
    "To do this China is using Facial Recognition, and other technologies, on a massive scale.  China is using this technology to social engineer behavior.\n",
    "\n",
    "However, so too are the g-mafia.  Maybe not to create a social credit scoring system, but for what purpose are they using our most highly personal and highly recognizable attribute - our faces.\n",
    "\n",
    "\n",
    "*The Big Nine* is a non-technical book that everyone can read and understand.  I would highly recommend this book to anyone wanting to understand the state of AI today.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The purpose of the overview is not to unnecessarily scare people - but being a little afraid concerning how for profit companies with an allegiance to shareholders and not to the country in which they do business, should be concerning.\n",
    "\n",
    "This notebook will show how facial recognition works and demonstrate just how easy it is to train a model to recognize anyone with as a little as a single picture.\n",
    "\n",
    "If Facial Recognition technology is so pervasive it would be in everyone's best interest to understand at least the basics of how it work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "This notebook will show code snippets of the relevant parts of the facial recognition process.  For full details see the scripts that are in the github repo backing this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Facial Recognition Background"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In 2001 Paul Viola and Michael Jones created a object detection algorithm.  Although generic its focus was on faces.  Since then, more reliable solutions have been discovered.  \n",
    "\n",
    "In 2005 an algorithm called:  **histogram of oriented gradients** or just **HOG** for short was developed and it was much more reliable and faster.  \n",
    "Since then CNN models have been trained to locate faces - with even greater accuracy but at the expense of compute time.  Using the **HOG** algorithm is a good balance between accuracy and compute effiency.  \n",
    "\n",
    "Using a **HOG** algorithm can even be run on small computing devices such as a Raspberry PI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Facial recognition can be broken down into the following steps:\n",
    "\n",
    "* <font size=\"4\">Find all faces in a picture</font>\n",
    "\n",
    "<br>\n",
    "\n",
    "* <font size=\"4\">Determine 68 Facial Landmarks</font>\n",
    "\n",
    "<br>\n",
    "\n",
    "* <font size=\"4\">Encode 68 Facial Landmarks into vector of 128 values</font>\n",
    "\n",
    "<br>\n",
    "\n",
    "* <font size=\"4\">Compare the unique features of the new, previously unseen face - with the faces that the model has been trained on to see which of the known faces the new face is most similiar to.</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Finding all of the faces"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./notebook_images/just_a_face.png\" alt=\"FacePicture\" style=\"width: 40%;\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find faces in a picture we start by converting any color picture to black and white.  The RGB values provide no additional information.\n",
    "\n",
    "Then, for every pixel examine the pixels immediately adjacent and find the pixel that is the darkest and 'draw' an arrow in that direction - from lightest to darkest.\n",
    "\n",
    "Doing this for every pixel shows the flow of light intensity from light to darker.  These arrows are called gradients.  The gradients, or changes in pixel density, are a better way to analyze images because lighter or darker images are not as impacted by changes in color or brightness.  \n",
    "\n",
    "If we looked just at pixel values a really bright or light picture would be considered different from a dark picture even if they are of the same person.  However the gradients would remain relatively the same.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keeping track of the gradients for every single pixel is a lot of data - therefore images are generally broken up into 16x16 segments.  In each segment, the direction of the gradients are tallied and the 16x16 image segment is replaced by the most prevalent gradient."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./notebook_images/hog_example.png\" alt=\"FacePicture\" style=\"width: 100%;\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets look at what the resulting HOG would look like from the video camera"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-16T19:32:57.105035Z",
     "start_time": "2020-01-16T19:32:36.097153Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-01-16 13:32:38.982 Python[1163:163734] ApplePersistenceIgnoreState: Existing state will not be touched. New state will be written to (null)\n"
     ]
    }
   ],
   "source": [
    "%%bash \n",
    "python video_hog.py -t 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "from skimage.feature import hog\n",
    "\n",
    "fd, hog_image = hog(image, orientations=8, pixels_per_cell=(16, 16),\n",
    "                        cells_per_block=(1, 1), visualize=True, multichannel=True)\n",
    "cv2.imshow(\"HOG\", hog_image)\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now - lets apply facial detection to show where the faces are detected.\n",
    "\n",
    "The libraries *face_recognition* and *dlib* provide a programatic interface to the pre-trained models that can take a HOG and determine the area of the face."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./notebook_images/hog_face_example.png\" alt=\"FacePicture\" style=\"width: 100%;\"/></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-16T19:33:47.611110Z",
     "start_time": "2020-01-16T19:33:20.413609Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-01-16 13:33:24.321 Python[1165:164606] ApplePersistenceIgnoreState: Existing state will not be touched. New state will be written to (null)\n"
     ]
    }
   ],
   "source": [
    "%%bash \n",
    "python video_hog_face_detect.py -t 25"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "def detect_mark_faces(frame, hog_image):\n",
    "    boxes = face_recognition.face_locations(frame, model='hog')\n",
    "    for (top, right, bottom, left) in boxes:\n",
    "        # draw the predicted face name on the image\n",
    "        cv2.rectangle(image, (left, top), (right, bottom), (0, 255, 0), 2)\n",
    "\n",
    "        cv2.rectangle(hog_image, (left, top), (right, bottom), (255, 255, 255), 2)\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Facial Landmarks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To account for different poses of faces, we will use an algorithm called, *face landmark estimation*.  The library dlib comes with a pretrained facial landmark detector.\n",
    "\n",
    "The idea is to come up with 68 points, called landmarks, that exist on every face.  For example the outline of the eyes, bridge of the nose, top of the chin.  We use a machine learning model to find these specific points. \n",
    "\n",
    "Once the algorithm knows where the landmarks are, the algorithm can determine how to transform the picture to normalize the picture so we are making similar comparisons with other face encodings.\n",
    "\n",
    "68 Point Facial Landmarks\n",
    "\n",
    "<center><img src=\"./notebook_images/facial_landmarks_68markup.jpg\" alt=\"68Landmarks\" style=\"width: 50%;\"/></center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./notebook_images/facial_landmarks.png\" alt=\"FacePicture\" style=\"width: 50%;\"/></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-16T19:34:39.251525Z",
     "start_time": "2020-01-16T19:34:08.697976Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-01-16 13:34:10.687 Python[1167:165586] ApplePersistenceIgnoreState: Existing state will not be touched. New state will be written to (null)\n"
     ]
    }
   ],
   "source": [
    "%%bash \n",
    "python face_landmarks.py -t 30"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "import dlib\n",
    "\n",
    "predictor = dlib.shape_predictor(\"shape_predictor_68_face_landmarks.dat\")\n",
    "shape = predictor(gray, rect)\n",
    "\n",
    "for (x, y) in shape:\n",
    "    cv2.circle(image, (x, y), 2, (0, 255, 0), -1)\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding Facial Landmarks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of the 68 facial landmark features - what are the best features and measurements to determine facial recognition?\n",
    "\n",
    "68 landmark features are used as input to a CNN (Convolutional Neural Network), that outputs a  128 element vector.  \n",
    "\n",
    "What do the 128 values represent?\n",
    "\n",
    "Like many neural networks - we do not really know.  It turns out that Deep Learning Neural Networks are actually much better at determining how the interaction of the 68 landmark features should be combined than a person would be.\n",
    "\n",
    "The process of training a CNN to output accurate face encodings from any face image requires a great deal of training data.  However, once the neural network has been trained, and the weigths are known - generating the 128 element vector for a new image is relatively fast."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-07T16:35:25.560179Z",
     "start_time": "2019-08-07T16:35:25.556801Z"
    }
   },
   "source": [
    "<center><img src=\"./notebook_images/landmarks_cnn_128.png\" alt=\"cnn\" style=\"width: 100%;\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets take a look at some of the encoding values in realtime.  In this case of the 128 we will look at indexes:\n",
    "\n",
    "* Index 0\n",
    "\n",
    "* Index 32\n",
    "\n",
    "* Index 64\n",
    "\n",
    "* Index 96\n",
    "\n",
    "* Index 127\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./notebook_images/landmark_encoding_values.png\" alt=\"cnn\" style=\"width: 50%;\"/></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-16T19:35:44.605767Z",
     "start_time": "2020-01-16T19:35:23.131691Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-01-16 13:35:25.836 Python[1169:166742] ApplePersistenceIgnoreState: Existing state will not be touched. New state will be written to (null)\n"
     ]
    }
   ],
   "source": [
    "%%bash \n",
    "python encodings_display.py -t 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "import face_recognition\n",
    "\n",
    "boxes = face_recognition.face_locations(image, model='hog')\n",
    "encodings = face_recognition.face_encodings(image, boxes)\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Encode Facial Images\n",
    "\n",
    "Lets take some pictures and add the encodings to the collection of existing encodings.\n",
    "\n",
    "This dataset is from the PyImageSearch article and includes characters from Jurassic Park.\n",
    "\n",
    "[PyImageSearch Face Recognition Article](https://www.pyimagesearch.com/2018/06/18/face-recognition-with-opencv-python-and-deep-learning/)\n",
    "\n",
    "We will use pictures from 6 characters and a set of my pictures.  We will create encodings for every picture and label the encoding the name of the person.  After we have all of the encodings we will be ready to perform facial recognition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-07T17:56:56.887726Z",
     "start_time": "2019-08-07T17:56:56.881798Z"
    }
   },
   "source": [
    "<table>\n",
    "<td> <img src=\"./notebook_images/alan_grant_collage.png\" alt=\"Drawing\" style=\"width: 250px;\"/> </td>\n",
    "<td> <img src=\"./notebook_images/claire_dearing_collage.png\" alt=\"Drawing\" style=\"width: 250px;\"/> </td>\n",
    "<td> <img src=\"./notebook_images/ellie_sattler_collage.png\" alt=\"Drawing\" style=\"width: 250px;\"/> </td>\n",
    "</tr>\n",
    "    <tr>\n",
    "<td> <img src=\"./notebook_images/ian_malcolm_collage.png\" alt=\"Drawing\" style=\"width: 250px;\"/> </td>\n",
    "<td> <img src=\"./notebook_images/john_hammond_collage.png\" alt=\"Drawing\" style=\"width: 250px;\"/> </td>\n",
    "<td> <img src=\"./notebook_images/owen_grady_collage.png\" alt=\"Drawing\" style=\"width: 250px;\"/> </td>\n",
    "</tr>\n",
    "\n",
    "\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have included my own set of images\n",
    "\n",
    "\n",
    "<center><img src=\"./notebook_images/pat_ryan_collage.png\" alt=\"cnn\" style=\"width: 50%;\"/></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-16T03:16:42.067898Z",
     "start_time": "2020-01-16T03:16:42.052942Z"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "tree -d images/dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-23T17:47:15.155749Z",
     "start_time": "2019-09-23T17:46:06.547159Z"
    }
   },
   "outputs": [],
   "source": [
    "# %%bash\n",
    "# only run this if we need recreate the base encodings\n",
    "# python create_facial_encodings.py -d images/dataset -e encodings/facial_encodings.pkl -r true\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Determining the person's identity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last step is to determine the person in the datastore of known people who has the 'closest' measurements to the image we are testing.\n",
    "\n",
    "For this, a simple Euclidean distance calculation can be performed.\n",
    "\n",
    "distance(f1, f2) = $\\sqrt{(f1_1-f2_1)^2 + (f1_2-f2_2)^2 + ... + (f1_{128}-f2_{128})^2}$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-06T21:16:06.205800Z",
     "start_time": "2019-08-06T21:16:06.202183Z"
    }
   },
   "source": [
    "\n",
    "We take the unknown facial image encodings and compare that against the known facial image encodings by calculating a distance.  The *threshold* is a parameter that can be changed to fine tune when the distance is considered a match or not.  The default value is 0.6.  If you to reduce the threshold this will reduce the false positives. This will make it more restrictive on when it determines there is a facial match. Lower the threshold value to around 0.5.  You might need to work with this threshold to get the performance you would like.\n",
    "\n",
    "If the distance between the two facial encodings is less than the threshold, 0.6 by default, then this is considered a match.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The facial_recognition library has the matching built in, but we could also use the *k-nearest neighbors* classifier from scikit learn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using a picture that has never been encoded, lets calculate the distance from all of the faces that have been encoded.\n",
    "\n",
    "<center><img src=\"./images/pat_ryan_linkedin/pat.ryan.smaller.png\" alt=\"PRLinkedIn\" style=\"width: 25%;\"/></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-16T19:36:43.616883Z",
     "start_time": "2020-01-16T19:36:42.249031Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0.32661354282556054, 'pat_ryan')\n",
      "(0.3396320380221242, 'pat_ryan')\n",
      "(0.37282213715155005, 'pat_ryan')\n",
      "(0.3873396941553151, 'pat_ryan')\n",
      "(0.3931766472928196, 'pat_ryan')\n",
      "(0.39550867208939877, 'pat_ryan')\n",
      "(0.4046523844511506, 'pat_ryan')\n",
      "(0.40950235763100534, 'pat_ryan')\n",
      "(0.41354160490816366, 'pat_ryan')\n",
      "(0.41416512717769977, 'pat_ryan')\n",
      "(0.4285352991745805, 'pat_ryan')\n",
      "(0.4396553971704904, 'pat_ryan')\n",
      "(0.44511289292837664, 'pat_ryan')\n",
      "(0.4516888372607261, 'pat_ryan')\n",
      "(0.4582808748855473, 'pat_ryan')\n",
      "(0.47817264865584863, 'pat_ryan')\n",
      "(0.5072786967667339, 'pat_ryan')\n",
      "(0.5260529562656461, 'pat_ryan')\n",
      "(0.5267452089307003, 'pat_ryan')\n",
      "(0.5373723124691484, 'pat_ryan')\n",
      "(0.5558160607623518, 'pat_ryan')\n",
      "(0.5688919798438286, 'pat_ryan')\n",
      "(0.5762997434396107, 'pat_ryan')\n",
      "(0.5899434975941679, 'john_hammond')\n",
      "(0.5915515133378403, 'pat_ryan')\n",
      "(0.5931512383511677, 'pat_ryan')\n",
      "(0.5942929020745742, 'pat_ryan')\n",
      "(0.6014384591381164, 'pat_ryan')\n",
      "(0.6066298357006913, 'pat_ryan')\n",
      "(0.6181329489347294, 'john_hammond')\n",
      "(0.6220354071101851, 'pat_ryan')\n",
      "(0.6288404317148943, 'claire_dearing')\n",
      "(0.6289871731418288, 'pat_ryan')\n",
      "(0.6362522359834057, 'ellie_sattler')\n",
      "(0.6385852659752145, 'ian_malcolm')\n",
      "(0.6396490817349119, 'pat_ryan')\n",
      "(0.6400514475302866, 'pat_ryan')\n",
      "(0.6402352852332767, 'pat_ryan')\n",
      "(0.64030341755603, 'ian_malcolm')\n",
      "(0.6481591277972383, 'john_hammond')\n",
      "(0.6492390421627161, 'pat_ryan')\n",
      "(0.6536934378322224, 'pat_ryan')\n",
      "(0.6558663581518828, 'john_hammond')\n",
      "(0.6581323325011266, 'ian_malcolm')\n",
      "(0.6581323325011266, 'ian_malcolm')\n",
      "(0.6611961217552441, 'john_hammond')\n",
      "(0.6652856736609556, 'ian_malcolm')\n",
      "(0.6726445797686037, 'john_hammond')\n",
      "(0.6738603174865447, 'ian_malcolm')\n",
      "(0.6754032846722595, 'ian_malcolm')\n",
      "(0.6754032846722595, 'ian_malcolm')\n",
      "(0.6755407451981639, 'ian_malcolm')\n",
      "(0.6789201915554416, 'ian_malcolm')\n",
      "(0.681139355836158, 'ian_malcolm')\n",
      "(0.6816251804707946, 'ian_malcolm')\n",
      "(0.6841373820722909, 'ellie_sattler')\n",
      "(0.6843887549303747, 'ian_malcolm')\n",
      "(0.6849450403046492, 'john_hammond')\n",
      "(0.6873216752444695, 'ian_malcolm')\n",
      "(0.6887553474545683, 'john_hammond')\n",
      "(0.6903494012773076, 'john_hammond')\n",
      "(0.6971487067982899, 'john_hammond')\n",
      "(0.6978596452421465, 'ian_malcolm')\n",
      "(0.6982708782484495, 'john_hammond')\n",
      "(0.6983248753777227, 'ellie_sattler')\n",
      "(0.7001808805877384, 'ian_malcolm')\n",
      "(0.7001808805877384, 'ian_malcolm')\n",
      "(0.701353800886472, 'ian_malcolm')\n",
      "(0.7017763348150203, 'john_hammond')\n",
      "(0.7019545247238835, 'john_hammond')\n",
      "(0.7022215877966421, 'ian_malcolm')\n",
      "(0.7040783667754974, 'john_hammond')\n",
      "(0.7047482299784956, 'ian_malcolm')\n",
      "(0.7061260851523208, 'ian_malcolm')\n",
      "(0.7061470053051588, 'john_hammond')\n",
      "(0.7072557144388278, 'ian_malcolm')\n",
      "(0.7087620421533146, 'ian_malcolm')\n",
      "(0.7115716155491275, 'ian_malcolm')\n",
      "(0.7122029496466573, 'ian_malcolm')\n",
      "(0.7123813562614477, 'ian_malcolm')\n",
      "(0.7135636699559729, 'owen_grady')\n",
      "(0.7137778158421559, 'john_hammond')\n",
      "(0.7143550502721355, 'ian_malcolm')\n",
      "(0.7167953332923148, 'john_hammond')\n",
      "(0.7202781886971112, 'ian_malcolm')\n",
      "(0.72130388217646, 'john_hammond')\n",
      "(0.7219957492191751, 'ian_malcolm')\n",
      "(0.7225244872314113, 'ian_malcolm')\n",
      "(0.7225244872314113, 'ian_malcolm')\n",
      "(0.7256969159897252, 'ellie_sattler')\n",
      "(0.7274605592415497, 'ellie_sattler')\n",
      "(0.7324149530990302, 'ian_malcolm')\n",
      "(0.7324961002398358, 'ian_malcolm')\n",
      "(0.732556235529592, 'owen_grady')\n",
      "(0.733399393509923, 'ian_malcolm')\n",
      "(0.7366774453898368, 'ian_malcolm')\n",
      "(0.7381962198080141, 'ian_malcolm')\n",
      "(0.7443288459034428, 'alan_grant')\n",
      "(0.7451278640082822, 'pat_ryan')\n",
      "(0.7472491180698301, 'ellie_sattler')\n",
      "(0.7511026943137312, 'ian_malcolm')\n",
      "(0.7537443358406553, 'alan_grant')\n",
      "(0.7551361712768093, 'ian_malcolm')\n",
      "(0.7559334139469732, 'ellie_sattler')\n",
      "(0.7563007017975717, 'ellie_sattler')\n",
      "(0.757133594810266, 'alan_grant')\n",
      "(0.759590405134342, 'owen_grady')\n",
      "(0.7629392808798456, 'ellie_sattler')\n",
      "(0.7639089398353653, 'owen_grady')\n",
      "(0.7702533099167744, 'ellie_sattler')\n",
      "(0.7703596102042019, 'claire_dearing')\n",
      "(0.7703596102042019, 'claire_dearing')\n",
      "(0.7722257290773955, 'ellie_sattler')\n",
      "(0.7732675781961792, 'ellie_sattler')\n",
      "(0.7737063726411468, 'owen_grady')\n",
      "(0.779903178657211, 'ellie_sattler')\n",
      "(0.7809397744884499, 'ellie_sattler')\n",
      "(0.7831769431441765, 'owen_grady')\n",
      "(0.7862161745682563, 'ellie_sattler')\n",
      "(0.7863713057263593, 'owen_grady')\n",
      "(0.7863713057263593, 'owen_grady')\n",
      "(0.7870171249521919, 'ellie_sattler')\n",
      "(0.7916459685209076, 'alan_grant')\n",
      "(0.7918876598713653, 'ellie_sattler')\n",
      "(0.7930599274872371, 'ellie_sattler')\n",
      "(0.7937062637997291, 'owen_grady')\n",
      "(0.7937062637997291, 'owen_grady')\n",
      "(0.7943186129855621, 'owen_grady')\n",
      "(0.7947613286783151, 'ellie_sattler')\n",
      "(0.795255654170381, 'ian_malcolm')\n",
      "(0.7965248381553185, 'claire_dearing')\n",
      "(0.7977771148402735, 'ellie_sattler')\n",
      "(0.7992363658471222, 'ellie_sattler')\n",
      "(0.800827048799034, 'ellie_sattler')\n",
      "(0.8022163340322579, 'alan_grant')\n",
      "(0.8029255317646946, 'alan_grant')\n",
      "(0.8030492527829846, 'ellie_sattler')\n",
      "(0.8078828699240788, 'owen_grady')\n",
      "(0.8086514463039463, 'owen_grady')\n",
      "(0.80879694784646, 'ellie_sattler')\n",
      "(0.8101565932282602, 'claire_dearing')\n",
      "(0.8101642557390123, 'owen_grady')\n",
      "(0.8102646492699561, 'owen_grady')\n",
      "(0.8126179226328122, 'owen_grady')\n",
      "(0.8148532573329225, 'owen_grady')\n",
      "(0.8176063902115231, 'claire_dearing')\n",
      "(0.8187407163530488, 'ellie_sattler')\n",
      "(0.8192669135654769, 'owen_grady')\n",
      "(0.8234456542713161, 'owen_grady')\n",
      "(0.8242871990223037, 'pat_ryan')\n",
      "(0.825827133544525, 'owen_grady')\n",
      "(0.826198887588872, 'ellie_sattler')\n",
      "(0.8294462461761851, 'ellie_sattler')\n",
      "(0.8307478739427074, 'alan_grant')\n",
      "(0.8307825511724806, 'owen_grady')\n",
      "(0.830934080921409, 'alan_grant')\n",
      "(0.8327168631645407, 'ellie_sattler')\n",
      "(0.8343748230667891, 'owen_grady')\n",
      "(0.8365165329224562, 'owen_grady')\n",
      "(0.8366126576305927, 'alan_grant')\n",
      "(0.8379516899957707, 'owen_grady')\n",
      "(0.8393708537649552, 'claire_dearing')\n",
      "(0.8402371204504558, 'owen_grady')\n",
      "(0.8407201136417455, 'owen_grady')\n",
      "(0.8433564601528631, 'alan_grant')\n",
      "(0.8443177773854363, 'owen_grady')\n",
      "(0.8466517612666864, 'owen_grady')\n",
      "(0.8472068231193731, 'claire_dearing')\n",
      "(0.8473602411447202, 'owen_grady')\n",
      "(0.8478961302145941, 'owen_grady')\n",
      "(0.8480047799103918, 'ellie_sattler')\n",
      "(0.8490815609015173, 'owen_grady')\n",
      "(0.8495946245961127, 'owen_grady')\n",
      "(0.851037056647449, 'claire_dearing')\n",
      "(0.85109996402033, 'owen_grady')\n",
      "(0.8511573030523145, 'claire_dearing')\n",
      "(0.8553775123632641, 'claire_dearing')\n",
      "(0.8587910344058308, 'alan_grant')\n",
      "(0.8621468505013381, 'claire_dearing')\n",
      "(0.8665529098278729, 'claire_dearing')\n",
      "(0.8676009386437253, 'claire_dearing')\n",
      "(0.8685000092266124, 'claire_dearing')\n",
      "(0.8705604206845643, 'claire_dearing')\n",
      "(0.8711266136186187, 'claire_dearing')\n",
      "(0.8716554035047909, 'claire_dearing')\n",
      "(0.8719671283157854, 'claire_dearing')\n",
      "(0.8719671283157854, 'claire_dearing')\n",
      "(0.8727409181228036, 'owen_grady')\n",
      "(0.876461279200852, 'claire_dearing')\n",
      "(0.8786673162190405, 'claire_dearing')\n",
      "(0.8797817889181145, 'claire_dearing')\n",
      "(0.8803113172786617, 'claire_dearing')\n",
      "(0.8814350993311177, 'owen_grady')\n",
      "(0.8831168132327869, 'claire_dearing')\n",
      "(0.8836551879518364, 'claire_dearing')\n",
      "(0.8857108339181285, 'alan_grant')\n",
      "(0.8897424145060763, 'claire_dearing')\n",
      "(0.8902727732829087, 'claire_dearing')\n",
      "(0.8918453330289444, 'claire_dearing')\n",
      "(0.892855469263077, 'claire_dearing')\n",
      "(0.894628257632909, 'claire_dearing')\n",
      "(0.8963296538209279, 'claire_dearing')\n",
      "(0.896433204528148, 'claire_dearing')\n",
      "(0.9016910637788597, 'claire_dearing')\n",
      "(0.9027038113184612, 'claire_dearing')\n",
      "(0.9056350447138798, 'claire_dearing')\n",
      "(0.9065656105099739, 'claire_dearing')\n",
      "(0.9081654496767444, 'claire_dearing')\n",
      "(0.9099733773754091, 'claire_dearing')\n",
      "(0.9107371709950579, 'claire_dearing')\n",
      "(0.9230943263329686, 'claire_dearing')\n",
      "(0.9328742287811348, 'claire_dearing')\n",
      "(0.937011912960577, 'claire_dearing')\n",
      "(0.937011912960577, 'claire_dearing')\n",
      "(0.9488232373450203, 'claire_dearing')\n",
      "(0.9935973296143709, 'claire_dearing')\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python calculate_encoding_distance.py -i images/pat_ryan_linkedin/pat.ryan.smaller.png -e encodings/facial_encodings.pkl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "\n",
    "from face_recognition.api import face_distance\n",
    "\n",
    "distance_results = face_distance(encodings, new_encoded_image)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets try with real time video feed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-16T19:38:16.277354Z",
     "start_time": "2020-01-16T19:37:29.642599Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] loading encodings...\n",
      "[INFO] starting video stream...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-01-16 13:37:32.207 Python[1173:168384] ApplePersistenceIgnoreState: Existing state will not be touched. New state will be written to (null)\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python video_facial_recognition.py -e encodings/facial_encodings.pkl --distance-tolerance 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Photo Booth\n",
    "\n",
    "<center><img src=\"./notebook_images/photo_booth.jpg\" alt=\"PRLinkedIn\" style=\"width: 75%;\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets have someone come up and we can capture new pictures, calcuate the encodings and perform facial recognition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Capture New Facial Images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-16T19:39:13.690502Z",
     "start_time": "2020-01-16T19:38:42.078387Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Grab image: 0\n",
      "Grab image: 1\n",
      "Grab image: 2\n",
      "Grab image: 3\n",
      "Grab image: 4\n",
      "Grab image: 5\n",
      "Grab image: 6\n",
      "Grab image: 7\n",
      "Grab image: 8\n",
      "Grab image: 9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-01-16 13:38:43.397 Python[1175:169497] ApplePersistenceIgnoreState: Existing state will not be touched. New state will be written to (null)\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python capture_webcam_face_images.py -d images/dataset -n ernest_t_bass -c 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the images are in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-16T19:39:34.344846Z",
     "start_time": "2020-01-16T19:39:34.326780Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "images/dataset\n",
      "├── alan_grant\n",
      "├── claire_dearing\n",
      "├── ellie_sattler\n",
      "├── ernest_t_bass\n",
      "├── ian_malcolm\n",
      "├── john_hammond\n",
      "├── owen_grady\n",
      "└── pat_ryan\n",
      "\n",
      "8 directories\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "tree -d images/dataset "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "<td> <img src=\"./images/dataset/ernest_t_bass/ernest_t_bass_0.png\" alt=\"Drawing\" style=\"width: 250px;\"/> </td>\n",
    "<td> <img src=\"./images/dataset/ernest_t_bass/ernest_t_bass_1.png\" alt=\"Drawing\" style=\"width: 250px;\"/> </td>\n",
    "<td> <img src=\"./images/dataset/ernest_t_bass/ernest_t_bass_2.png\" alt=\"Drawing\" style=\"width: 250px;\"/> </td>\n",
    "</tr>\n",
    "    <tr>\n",
    "<td> <img src=\"./images/dataset/ernest_t_bass/ernest_t_bass_3.png\" alt=\"Drawing\" style=\"width: 250px;\"/> </td>\n",
    "<td> <img src=\"./images/dataset/ernest_t_bass/ernest_t_bass_4.png\" alt=\"Drawing\" style=\"width: 250px;\"/> </td>\n",
    "<td> <img src=\"./images/dataset/ernest_t_bass/ernest_t_bass_5.png\" alt=\"Drawing\" style=\"width: 250px;\"/> </td>\n",
    "</tr>\n",
    "    <tr>\n",
    "<td> <img src=\"./images/dataset/ernest_t_bass/ernest_t_bass_6.png\" alt=\"Drawing\" style=\"width: 250px;\"/> </td>\n",
    "<td> <img src=\"./images/dataset/ernest_t_bass/ernest_t_bass_7.png\" alt=\"Drawing\" style=\"width: 250px;\"/> </td>\n",
    "<td> <img src=\"./images/dataset/ernest_t_bass/ernest_t_bass_8.png\" alt=\"Drawing\" style=\"width: 250px;\"/> </td>\n",
    "</tr>\n",
    "    <tr>\n",
    "<td> <img src=\"./images/dataset/ernest_t_bass/ernest_t_bass_9.png\" alt=\"Drawing\" style=\"width: 250px;\"/> </td>\n",
    "<td> </td>\n",
    "<td> </td>\n",
    "</tr>\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Facial Encodings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-16T19:39:48.024371Z",
     "start_time": "2020-01-16T19:39:42.950089Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] quantifying faces...\n",
      "[INFO] processing image [ernest_t_bass] 1/10\n",
      "[INFO] processing image [ernest_t_bass] 2/10\n",
      "[INFO] processing image [ernest_t_bass] 3/10\n",
      "[INFO] processing image [ernest_t_bass] 4/10\n",
      "[INFO] processing image [ernest_t_bass] 5/10\n",
      "[INFO] processing image [ernest_t_bass] 6/10\n",
      "[INFO] processing image [ernest_t_bass] 7/10\n",
      "[INFO] processing image [ernest_t_bass] 8/10\n",
      "[INFO] processing image [ernest_t_bass] 9/10\n",
      "[INFO] processing image [ernest_t_bass] 10/10\n",
      "Encoding dataset took: 0.06332361300786336 minutes\n",
      "[INFO] serializing encodings...\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# only run this if we need recreate the base encodings\n",
    "python create_facial_encodings.py -d images/dataset/ernest_t_bass -e encodings/facial_encodings.pkl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run Facial Recognition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-16T19:42:19.226917Z",
     "start_time": "2020-01-16T19:39:54.639689Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] loading encodings...\n",
      "[INFO] starting video stream...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-01-16 13:39:57.230 Python[1182:170945] ApplePersistenceIgnoreState: Existing state will not be touched. New state will be written to (null)\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python video_facial_recognition.py -e encodings/facial_encodings.pkl --distance-tolerance 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove named encodings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-23T17:56:53.240720Z",
     "start_time": "2019-09-23T17:56:53.113459Z"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "python remove_name_from_encodings.py -n ernest_t_bass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Closing Thoughts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Facial recognition is the technology Genie that we cannot put back into the bottle.  \n",
    "\n",
    "While facial recognition at the scale of China will likely use other technologies, todays libraries make it incredibly easy to perform facial recognition.\n",
    "\n",
    "You may never have to perform facial recognition in your work, but it still is important to understand how this technology can be incorporated into different applications and how easy it is.\n",
    "\n",
    "### What else can we do with this\n",
    "\n",
    "Have you ever heard people say, 'oh you look just like your father', or 'I cannot tell you and your sister apart sometimes'\n",
    "\n",
    "With the information in this notebook, you can find out just how close to, or similar to, another person you really are.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Sources:\n",
    "\n",
    "### PyImageSearch\n",
    "PyImageSearch.com is an amazing resource for Deep Learning and computer vision. Adrian has a tremendous amount of free articles.  I would also encourage you to check out the books he has written.  The books that I have read have been really well done and worth the money.\n",
    "\n",
    "[PyImageSearch Face Recognition Article](https://www.pyimagesearch.com/2018/06/18/face-recognition-with-opencv-python-and-deep-learning/)\n",
    "\n",
    "### Machine Learning Is Fun\n",
    "Adam is the person that created the *face_recognition* library that makes all of this so easy.  His book is also a very good read and covers much more than just computer vision.  A lot of inspiration for this notebook came from his book and articles.\n",
    "\n",
    "[Machine Learning Is Fun](https://www.machinelearningisfun.com)\n",
    "\n",
    "### Scikit Image\n",
    "As usual, the scikit documentation does not disappoint.  This example shows how to display the HOG image\n",
    "[Scikit Image](https://scikit-image.org/docs/dev/auto_examples/features_detection/plot_hog.html)\n",
    "\n",
    "### Italojs\n",
    "Thank you for sharing your github repo with the example code to display the facial landmarks.\n",
    "[Italojs Github](https://github.com/youngsoul/facial-landmarks-recognition-)\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}