healthcare-fraud-identification-using-PCA-anomaly-detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Healthcare insurance fraud identification using PCA anomaly detection\n",
    "\n",
    "1. [Background](#background)\n",
    "1. [Setup](#setup)\n",
    "1. [Data](#data)\n",
    "    1. [Obtain data](#datasetfiles)\n",
    "1. [Feature Engineering](#feateng)\n",
    "    1. [Missing values](#missing)\n",
    "    1. [Categorical features](#catfeat)\n",
    "        1. [Gender](#gender)\n",
    "        1. [Age Group](#age)\n",
    "    1. [NLP for Textual features](#nlp)\n",
    "        1. [Diagnosis Descriptions](#diagnosis)\n",
    "        1. [Procedure Descriptions](#procedure)\n",
    "1. [Split train & test data](#split)\n",
    "    1. [Standardize](#standardize)\n",
    "    1. [PCA](#pca)\n",
    "1. [Calculate the Mahalanobis distance](#md)\n",
    "1. [Unsupervised Anomaly Detection](#ad)\n",
    "1. [Understanding Anomaly](#understandinganomaly)\n",
    "1. [(Optional) Deploy PCA](#deployendpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Background <a name=\"background\"></a>\n",
    "Medicare is a federal healthcare program created in 1965 with the passage of the Social Security Amendments to ensure that citizens 65 and older as well as younger persons with certain disabilities have access to quality healthcare. Medicare is administered by the Centers for Medicare and Medicaid Services (CMS). CMS manages Medicare programs by selecting official Medicare administrative contractors (MACs) to process the Medicare claims associated with various parts of Medicare. We propose a solution to apply unsupervised outlier techniques at post-payment stage to detect fraudulent patterns of received insurance claims. \n",
    "\n",
    "Health care insurance fraud is a pressing problem, causing substantial and increasing costs in medical insurance programs. Due to large amounts of claims submitted, review of individual claims becomes a difficult task and encourages the employment of automated pre-payment controls and better post-payment decision support tools to enable subject matter expert analysis. We will demonstrate the unsupervised anomalous outlier techniques on a minimal set of metrics made available in the CMS Medicare inpatient claims from 2008. Once more data is available as extracts from different systems  -Medicaid Information Management systems(MMIS), Medicaid Statistical Information Systems(MSIS), Medicaid Reference data such as Provider Files, Death Master Files, etc. - there is an opportunity to build a database of metrics to make the fraud detection technique more robust. The method can be used to flag claims as a targeting method for further investigation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Setup <a name=\"setup\"></a>\n",
    "\n",
    "To begin, we'll install the Python libraries we'll need for the remainder of the exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upgrade numpy to latest version. Should be numpy==1.15.0 or higher to use quantile attribute\n",
    "import sys\n",
    "!{sys.executable} -m pip install --upgrade numpy\n",
    "\n",
    "#If thenumpy version prints less than 1.15.0\n",
    "#Go to Jupyter notebook menu on the top, click on kernal and click \"Restart and Clear Output\". Start from the beginning again.\n",
    "import numpy as np\n",
    "print(np.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install a pip columnize package in the current Jupyter kernel\n",
    "import sys\n",
    "!{sys.executable} -m pip install columnize gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# unInstall seaborn==0.8 in the Jupyter kernel. seaborn==0.9 or higher is required for seaborn pair plot that we plan to use for visulization in this notebook.\n",
    "import sys\n",
    "!{sys.executable} -m pip uninstall seaborn -y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install seaborn==0.9 in the Jupyter kernel\n",
    "import sys\n",
    "!{sys.executable} -m pip install seaborn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check if sns resolves to 0.9.0. If it still shows version 0.8.0, \n",
    "#go to Jupyter notebook menu on the top, click on kernal and click \"Restart and Clear Output\" \n",
    "import seaborn as sns\n",
    "print(sns.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll import the Python libraries we'll need for the remainder of the exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np                                # For matrix operations and numerical processing\n",
    "import pandas as pd                               # For munging tabular data\n",
    "import boto3                                      #enables Python developers to create, configure, and manage AWS services\n",
    "from IPython.display import display               # For displaying outputs in the notebook\n",
    "import matplotlib.pyplot as plt                   #for interactive plots and simple cases of programmatic plot generation\n",
    "%matplotlib inline                                \n",
    "from time import gmtime, strftime                 # For labeling SageMaker models, endpoints, etc.\n",
    "import sys                                        #provides access to some variables used or maintained by the interpreter\n",
    "import os                                         # For manipulating filepath names\n",
    "import sagemaker                                  #open source library for training and deploying machine-learned models on Amazon SageMaker\n",
    "import time                                       #provides various time-related functions\n",
    "import warnings                                   #allows you to handle all warnings with the standard logging\n",
    "import io                                         #interface to access files and streams\n",
    "import sagemaker.amazon.common as smac            #provides common function used for training and deploying machine-learned models on Amazon SageMaker\n",
    "warnings.filterwarnings(action = 'ignore')        #warnings filter controls whether warnings are ignored, displayed\n",
    "from sklearn.model_selection import train_test_split #Quick utility to split data into train and test set\n",
    "import gensim                                     #topic modelling library for Python that provides access to Word2Vec\n",
    "import columnize                                  #format a simple (i.e. not nested) list into aligned columns.\n",
    "from gensim.models import Word2Vec                #topic modelling library for Python that provides access to Word2Vec\n",
    "from sklearn.manifold import TSNE                 #containing T-SNE algorithms used to project high dimensional space into lower dimesional space\n",
    "from numpy.linalg import inv                      #Compute the dot product of two or more arrays in a single function call\n",
    "import scipy.stats                                #contains a large number of probability distributions for statistical analysis\n",
    "import scipy as sp                                #collection of mathematical algorithms\n",
    "import seaborn as sns                             #data visualization library based on matplotlib\n",
    "import mxnet as mx                                #open-source deep learning software framework, used to train, and deploy deep neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook was created and tested on an ml.m4.xlarge and ml.t3.xlarge notebook instance.\n",
    "\n",
    "We first need to specify some AWS credentials; specifically data locations and access roles. This is the only cell of this notebook that you will need to edit. In particular, we need the following data:\n",
    "\n",
    "* `bucket` - An S3 bucket accessible by this account.\n",
    "  * Used to store input training data and model data output.\n",
    "  * Should be withing the same region as this notebook instance, training, and hosting.\n",
    "* `prefix` - The location in the bucket where this notebook's input and and output data will be stored. (The default value is sufficient.)\n",
    "* `role` - The IAM Role ARN used to give training and hosting access to your data.\n",
    "  * See documentation on how to create these.\n",
    "  * The script below will try to determine an appropriate Role ARN.\n",
    "  \n",
    "#### Note: Pick a unique s3 bucket name before you execute the cell below, update 'bucket' variable with new name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import get_execution_role\n",
    "import boto3, os\n",
    "s3 = boto3.resource('s3')\n",
    "\n",
    "role = get_execution_role()\n",
    "# Assign a unique name to the bucket. S3 buckets should have unique global name.\n",
    "bucket = '<unique-alias>-aim302-30-may-2019'\n",
    "prefix = 'sagemaker/healthcare-fraud-detection'\n",
    "s3.create_bucket(Bucket=bucket)\n",
    "\n",
    "print('Training input/output will be stored in {}/{}'.format(bucket, prefix))\n",
    "print('\\nIAM Role: {}'.format(role))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Data<a name=\"data\"></a>\n",
    "\n",
    "The dataset we'll be using in this example was downloaded from following link.\n",
    "https://www.cms.gov/Research-Statistics-Data-and-Systems/Downloadable-Public-Use-Files/BSAPUFS/Inpatient_Claims.html\n",
    "\n",
    "The data set is the public available Basic Stand Alone (BSA) Inpatient Public Use Files (PUF) named “CMS 2008 BSA Inpatient Claims PUF”. The file contains Medicare inpatient claims from 2008. Each record is an inpatient claim incurred by a 5% sample of Medicare beneficiaries. The file contains seven (7) variables: A primary claim key indexing the records and six (6) analytic variables. One of the analytic variables, claim cost, is provided in two forms, (a) as an integer category and (b) as a dollar average. There are some demographic and claim-related variables provided in this PUF. \n",
    "\n",
    "However, as beneficiary identities are not provided, it is not possible to link claims that belong to the same beneficiary in the CMS 2008 BSA Inpatient Claims PUF. Without linking beneficiary Id to the claims, it is not possible to create features such as 'amount reimbursed over time', 'average reimbursement per visit' etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3A. Obtain data<a name=\"datasetfiles\"></a>\n",
    "\n",
    "We will use the following link to download claims dataset.\n",
    "\n",
    "https://www.cms.gov/Research-Statistics-Data-and-Systems/Downloadable-Public-Use-Files/BSAPUFS/Downloads/2008_BSA_Inpatient_Claims_PUF.zip\n",
    "\n",
    "The data dictionary required to interpret codes in dataset have been constructed from following pdf document.\n",
    "\n",
    "https://www.cms.gov/Research-Statistics-Data-and-Systems/Downloadable-Public-Use-Files/BSAPUFS/Downloads/2008_BSA_Inpatient_Claims_PUF_DataDic_CB.pdf\n",
    "\n",
    "Following dictionary files are already avaliable in data folder in the notebook.\n",
    "- `ColumnNames.csv` - column description\n",
    "- `DiagnosisRelatedGroupNames.csv` - dictionary for procedure codes\n",
    "- `InternationalClassificationOfDiseasesNames.csv` - dictionary of diagnosis codes\n",
    "- `LengthOfStayDays.csv` - dictionary of length of stay\n",
    "- `AgeGroup.csv` - dictionary of age group\n",
    "- `Gender.csv` - dictionary of gender"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download claims data file from CMS site."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!wget https://www.cms.gov/Research-Statistics-Data-and-Systems/Downloadable-Public-Use-Files/BSAPUFS/Downloads/2008_BSA_Inpatient_Claims_PUF.zip\n",
    "!unzip -o ./2008_BSA_Inpatient_Claims_PUF-backup.zip -d data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The data file have been extrcated under data folder locally on Sagemaker notebook volume in the data folder.\n",
    "\n",
    "- `2008_BSA_Inpatient_Claims_PUF.csv` - claims data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's begin exploring data:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Feature Engineering <a name=\"feateng\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read the ColumnNames csv file to identify meaningful names for column labels in the claim data\n",
    "colnames = pd.read_csv(\"./data/ColumnNames.csv\")\n",
    "colnames[colnames.columns[-1]] = colnames[colnames.columns[-1]].map(lambda x: x.replace('\"','').strip())\n",
    "display(colnames)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read claims data file\n",
    "df_cms_claims_data = pd.read_csv('./data/2008_BSA_Inpatient_Claims_PUF.csv')\n",
    "df_cms_claims_data.columns = colnames[colnames.columns[-1]].ravel()\n",
    "pd.set_option('display.max_columns', 500)\n",
    "\n",
    "# print the shape of the data file\n",
    "print('Shape:', df_cms_claims_data.shape)\n",
    "\n",
    "# show the top few rows\n",
    "display(df_cms_claims_data.head())\n",
    "\n",
    "# describe the data object\n",
    "display(df_cms_claims_data.describe())\n",
    "\n",
    "# check the datatype for each column\n",
    "display(df_cms_claims_data.dtypes)\n",
    "\n",
    "# check null value for each column\n",
    "display(df_cms_claims_data.isnull().mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### You might have observed some 'NaN' and mean value(0.469985) for ICD9 primary procedure code in print results above.\n",
    "We need fix to 'NaN' in ICD9 primary procedure code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4A. Missing values<a name=\"missing\"></a>\n",
    "Do I have missing values? How are they expressed in the data? Should I withhold samples with missing values? Or should I replace them? If so, which values should they be replaced with?\n",
    "\n",
    "Based on results of isnull.mean(), it is clear that 'ICD9 primary procedure code' has a non zero mean and it is so because it has NaN values. The NaN values corresponds to \"No Procedure Performed\" in the  in the 'ICD9 primary procedure code' dictionary. Let's replace NaN values with a numeric code for \"No Procedure Performed\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Fill NaN with -1 for \"No Procedure Performed\"\n",
    "procedue_na = -1\n",
    "df_cms_claims_data['ICD9 primary procedure code'].fillna(procedue_na, inplace = True)\n",
    "\n",
    "#convert procedure code from float to int64\n",
    "df_cms_claims_data['ICD9 primary procedure code'] = df_cms_claims_data['ICD9 primary procedure code'].astype(np.int64)\n",
    "\n",
    "\n",
    "#check count of null values to ensure dataframe is updated\n",
    "display(df_cms_claims_data.isnull().mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4B. Categorical features <a name=\"catfeat\"></a>\n",
    "\n",
    "Munging categorical data is another essential process during data preprocessing. It is necessary to convert categorical features to a numerical representation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### a. Gender <a name=\"gender\"></a>\n",
    "\n",
    "Since gender is already binary and coded as 1 for Male and 2 for Female, no pre-processing is required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read gender dictionary csv file\n",
    "dict_gender = pd.read_csv(\"./data/Gender.csv\")\n",
    "dict_gender.columns = dict_gender.columns.to_series().apply(lambda x: x.strip())\n",
    "display(dict_gender)\n",
    "\n",
    "# join the beneficiary category code with age group definition and describe the distribution amongst different age group in claims dataset\n",
    "tmp_gender_counts = dict_gender.set_index('bene_sex_ident_cd').join( df_cms_claims_data['Beneficiary gender code'].value_counts() )\n",
    "tmp_gender_counts['percentage'] = tmp_gender_counts['Beneficiary gender code']/tmp_gender_counts['Beneficiary gender code'].sum()*100\n",
    "\n",
    "# project gender distribution in the dataset on the bar graph\n",
    "plt.bar(tmp_gender_counts.index, tmp_gender_counts['percentage'].tolist());\n",
    "plt.xticks(tmp_gender_counts.index, tmp_gender_counts['Beneficiary gender'].tolist(), rotation=45)\n",
    "plt.ylabel('Percentage claims')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### You may have observed a slight imbalance in claims distribution for male and female records in above bar graph.\n",
    "Nothing concerning hear. But, we may use this information later in result analysis to justify our anomaly hypothesis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### b. Age Group <a name=\"age\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#read AgeGroup csv file into a pandas dataframe\n",
    "dict_age_group = pd.read_csv(\"./data/AgeGroup.csv\", sep=',')\n",
    "dict_age_group.columns = dict_age_group.columns.to_series().apply(lambda x: x.strip())\n",
    "display(dict_age_group)\n",
    "\n",
    "# join the beneficiary category code with age group definition and describe the distribution amongst different age group in claims dataset\n",
    "tmp_age_sorted_counts = dict_age_group.set_index('BENE_AGE_CAT_CD').join( df_cms_claims_data['Beneficiary Age category code'].value_counts() )\n",
    "tmp_age_sorted_counts['percentage'] = tmp_age_sorted_counts['Beneficiary Age category code']/tmp_age_sorted_counts['Beneficiary Age category code'].sum()*100\n",
    "\n",
    "# project age group distribution in the dataset on the bar graph\n",
    "plt.bar(tmp_age_sorted_counts.index, tmp_age_sorted_counts['percentage'].tolist());\n",
    "plt.xticks(tmp_age_sorted_counts.index, tmp_age_sorted_counts['Age Group'].tolist(), rotation=45)\n",
    "plt.ylabel('Percentage claims')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### You might have observed a slight imbalance in age group group distribution. \n",
    "Nothing concerning in above distribution. Small imbalance is OK."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4B. NLP for Textual features <a name=\"nlp\"></a>\n",
    "\n",
    "All physician and hospital claims include one or more diagnosis codes. The ICD-9-CM diagnosis coding system is used since October, 2012.\n",
    "\n",
    "Hospital inpatient claims also include one or more procedure codes that represent the services performed. The ICD-9-CM diagnosis coding system is used since October, 2012.\n",
    "\n",
    "The codes are numeric number representing the phrases describing the diagnosis and the procedures itself. The code iteself is numberic but doesn't capture context of a word in a document, semantic and syntactic similarity, relation with other words, etc.\n",
    "\n",
    "For diagnosis and procedure codes there is a option to consider it as categorical code and apply one hot encoding to it. That categorical data is defined as variables with a finite set of label values. We apply a technique called one hot encoding to do binarization of such values. In one hot encode we create one column for each label value and mark it as 0 or 1 as applicable to sample record. In case of dignosis code and procedure code it will give us a sparse matrix. Again, the code iteself will be numberic but doesn't capture context of a word in a document, semantic and syntactic similarity, relation with other words, etc.\n",
    "\n",
    "Inorder to capture, capture context of a word in a document, semantic and syntactic similarity, relation with other words, etc. we use a technique called word embedding to convert every word in a phrase into a vector of floating point numbers. We then average the vector for each word in a phrase to derive vector for a phrase. We will use this approach for both diagnosis and procedure descriptions to extract features.\n",
    "\n",
    "Word2Vec is a specific method to derieve word embeddings. It can be done using two methods (both involving Neural Networks): Skip Gram and Common Bag Of Words (CBOW)\n",
    "\n",
    "CBOW Model: This method takes the context of each word as the input and tries to predict the word corresponding to the context.\n",
    "\n",
    "Skip-Gram model: This method uses the target word (whose representation we want to generate) to predict the context and in the process, we produce the representations.\n",
    "\n",
    "Both model have their own advantages and disadvantages. Skip Gram works well with small amount of data and is found to represent rare words well.\n",
    "\n",
    "On the other hand, CBOW is faster and has better representations for more frequent words.\n",
    "\n",
    "In our use case, we will use CBOW model to derieve wordtovec for pharases used to describe procedure and diagnosis code description."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### a. Diagnosis Descriptions <a name=\"diagnosis\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read the diagnosis names into dataframe, clean up values and display records\n",
    "dict_diagnosis_names = pd.read_csv('./data/DiagnosisRelatedGroupNames.csv', sep=', \"', skiprows=1, names=['Base DRG code','Diagnosis related group']);\n",
    "dict_diagnosis_names['Diagnosis related group'] = dict_diagnosis_names['Diagnosis related group'].map(lambda x: x.replace('\"',''));\n",
    "display(dict_diagnosis_names.head())\n",
    "display(dict_diagnosis_names.dtypes)\n",
    "\n",
    "\n",
    "# join the Base DRG code in dictionay with Base DRG code in claims dataframe and describe the distribution amongst different age group in claims dataset\n",
    "tmp_diagnosis_counts = dict_diagnosis_names.set_index('Base DRG code').join( df_cms_claims_data['Base DRG code'].value_counts() )\n",
    "tmp_diagnosis_counts['percentage'] = tmp_diagnosis_counts['Base DRG code']/tmp_diagnosis_counts['Base DRG code'].sum()*100\n",
    "\n",
    "# project age group distribution in the dataset on the bar graph\n",
    "plt.figure(figsize=(100,20))\n",
    "plt.rc('xtick', labelsize=16)\n",
    "plt.bar(tmp_diagnosis_counts.index, tmp_diagnosis_counts['percentage'].tolist());\n",
    "#plt.xticks(tmp_diagnosis_counts.index, tmp_diagnosis_counts['Diagnosis related group'].tolist(), rotation=45)\n",
    "plt.ylabel('Percentage claims')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Observe the distribution of different diagnosis code in above bar graph printed from claims dataset.\n",
    "Next, let's do text processing on diagnosis descriptions to make some of the acronyms more meaningful for word embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to run pre processing on diagnosis descriptions\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize \n",
    "\n",
    "def text_preprocessing(phrase):\n",
    "    phrase = phrase.lower()\n",
    "    phrase = phrase.replace('&', 'and')\n",
    "    #phrase = phrase.replace('non-', 'non') #This is to ensure non-critical, doesn't get handled as {'non', 'critical'}\n",
    "    phrase = phrase.replace(',','')\n",
    "    phrase = phrase.replace('w/o','without').replace(' w ',' with ').replace('/',' ')\n",
    "    phrase = phrase.replace(' maj ',' major ')\n",
    "    phrase = phrase.replace(' proc ', ' procedure ')\n",
    "    phrase = phrase.replace('o.r.', 'operating room')\n",
    "    sentence = phrase.split(' ')\n",
    "    return sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run pre processing on diagnosis related group\n",
    "tmp_diagnosis_tokenized = dict_diagnosis_names['Diagnosis related group'].map(lambda x: text_preprocessing(x))\n",
    "display(tmp_diagnosis_tokenized.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build a map of number of tokens in a phrase to number of phrases with speific number of phrases\n",
    "diagnosis_phrase_lengths = tmp_diagnosis_tokenized.map(lambda x: len(x)).value_counts().sort_index()\n",
    "\n",
    "# plot count of tokens in phrase on x-axis and number of phrases with a specific count of tokens in y-axis\n",
    "plt.bar(np.arange(1,1+len(diagnosis_phrase_lengths)), diagnosis_phrase_lengths)\n",
    "plt.xlabel('Number of Tokens');\n",
    "plt.ylabel('Phrases');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Word to vec hyperparameters explained\n",
    "\n",
    "**size:** The size of the dense vector that is to represent each token or word. If you have very limited data, then size should be a much smaller value. If you have lots of data, its good to experiment with various sizes. A value of 100–150 has worked well for me for similarity lookups.\n",
    "\n",
    "**window:** The maximum distance between the target word and its neighboring word. If your neighbor’s position is greater than the maximum window width to the left or the right, then some neighbors are not considered as being related to the target word. In theory, a smaller window should give you terms that are more related. If you have lots of data, then the window size should not matter too much, as long as its not overly narrow or overly broad. If you are not too sure about this, just use the default value.\n",
    "\n",
    "**min_count:** Minimium frequency count of words. The model would ignore words that do not satisfy the min_count.Extremely infrequent words are usually unimportant, so its best to get rid of those. Unless your dataset is really tiny, this does not really affect the model.\n",
    "\n",
    "**workers:** How many threads to use behind the scenes?\n",
    "\n",
    "**iter:** How many epochs to train for? I typically use 10 or more for a small to medium dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# traing wordtovec model on diagnosis description tokens\n",
    "model_drg = Word2Vec(tmp_diagnosis_tokenized, min_count = 1, size = 72, window = 5, iter = 30)\n",
    "print(model_drg)\n",
    "diagnosis_words = list(model_drg.wv.vocab)\n",
    "print(columnize.columnize(diagnosis_words, displaywidth=80, ljust=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### t-Distributed Stochastic Neighbor Embedding (t-SNE)\n",
    "\n",
    "t-Distributed Stochastic Neighbor Embedding (t-SNE) is a non-linear technique for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot TSNE visualization\n",
    "from sklearn.manifold import TSNE\n",
    "def tsne_plot(model):\n",
    "    \"Creates and TSNE model and plots it\"\n",
    "    labels = []\n",
    "    tokens = []\n",
    "\n",
    "    for word in model.wv.vocab:\n",
    "        tokens.append(model[word])\n",
    "        labels.append(word)\n",
    "    \n",
    "    tsne_model = TSNE(perplexity=10, n_components=2, init='pca', n_iter=2500, random_state=10)\n",
    "    new_values = tsne_model.fit_transform(tokens)\n",
    "\n",
    "    x = []\n",
    "    y = []\n",
    "    for value in new_values:\n",
    "        x.append(value[0])\n",
    "        y.append(value[1])\n",
    "        \n",
    "    plt.figure(figsize=(16, 16)) \n",
    "    for i in range(len(x)):\n",
    "        plt.scatter(x[i],y[i])\n",
    "        plt.annotate(labels[i],\n",
    "                     xy=(x[i], y[i]),\n",
    "                     xytext=(5, 2),\n",
    "                     textcoords='offset points',\n",
    "                     ha='right',\n",
    "                     va='bottom')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot t_SNE chart for diagnosis word to vector. \n",
    "#2D visual plot of word embeddings derieved from diagnosis description.\n",
    "tsne_plot(model_drg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# test most similiar for some word from model_drg.wv.keywords\n",
    "model_drg.most_similar('diagnosis')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extract diagnosis words that starts with non\n",
    "#display(tmp_diagnosis_tokenized.head())\n",
    "series_diagnosis = pd.Series(diagnosis_words)\n",
    "diagnosis_words_with_non = series_diagnosis[series_diagnosis.map(lambda x: 'non' in x)]\n",
    "display(diagnosis_words_with_non)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check similarity between diagnosis words with opposite severity\n",
    "for i in diagnosis_words_with_non:\n",
    "    a, not_a = i.replace('non-','').replace('non',''), i\n",
    "    if a in diagnosis_words:\n",
    "        print('Cosine similarity between', a, not_a, ':', model_drg.wv.similarity(a, not_a))\n",
    "        print('')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate mean of word vector in a phrase to derieve phrase vector for diagnosis description\n",
    "# we plan to use mean of word vectors to derive word embeddings for a phrase\n",
    "values = []\n",
    "index =[]\n",
    "#iterate through list of strings in each diagnosis phrase\n",
    "for i, v in pd.Series(tmp_diagnosis_tokenized).items():\n",
    "    #calculate mean of all word embeddings in each diagnosis phrase\n",
    "    values.append(model_drg[v].mean(axis =0))\n",
    "    index.append(i)\n",
    "tmp_diagnosis_phrase_vector = pd.DataFrame({'Base DRG code':index, 'DRG_VECTOR':values})\n",
    "display(tmp_diagnosis_phrase_vector.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# expand tmp_diagnosis_phrase_vector into dataframe\n",
    "# every scalar value in phrase vector will be considered a feature\n",
    "diagnosis_features = tmp_diagnosis_phrase_vector['DRG_VECTOR'].apply(pd.Series)\n",
    "\n",
    "# rename each variable in diagnosis_features use DRG_F as prefix\n",
    "diagnosis_features = diagnosis_features.rename(columns = lambda x : 'DRG_F' + str(x + 1))\n",
    "\n",
    "# view the diagnosis_features dataframe\n",
    "display(diagnosis_features.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### b. Procedure Descriptions <a name=\"procedure\"></a>\n",
    "\n",
    "Apply the same process that we used for diagnosis description to procedure description to build a feature vector for procedure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#read procedure code and procedure description mappings\n",
    "dict_procedure_names = pd.read_csv('./data/InternationalClassificationOfDiseasesNames.csv', \n",
    "                              sep=', \"', \n",
    "                              skiprows=1, \n",
    "                              names=['ICD9 primary procedure code','International Classification of Diseases'])\n",
    "\n",
    "#replace double quotes with blank\n",
    "dict_procedure_names = dict_procedure_names.applymap(lambda x: x.replace('\"',''))\n",
    "\n",
    "# replace -1 as code for 'No procedure performed'. In the dictionary the code is set as blank.\n",
    "dict_procedure_names.iloc[0]['ICD9 primary procedure code'] = procedue_na\n",
    "\n",
    "# convert procedure code from float to int64\n",
    "dict_procedure_names['ICD9 primary procedure code'] = dict_procedure_names['ICD9 primary procedure code'].astype(np.int64)\n",
    "\n",
    "\n",
    "# check the datatype for each column\n",
    "display(dict_procedure_names.dtypes)\n",
    "\n",
    "# display first five records in dictionary\n",
    "display(dict_procedure_names.head())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(dict_procedure_names.count())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# join the ICD9 primary procedure code in dictionay with ICD9 primary procedure code in claims dataframe and plot the distribution of procedures in the claims dataset\n",
    "tmp_procedure_counts = dict_procedure_names.set_index('ICD9 primary procedure code').join(df_cms_claims_data['ICD9 primary procedure code'].value_counts() )\n",
    "tmp_procedure_counts['percentage'] = tmp_procedure_counts['ICD9 primary procedure code']/tmp_procedure_counts['ICD9 primary procedure code'].sum()*100\n",
    "\n",
    "# project procedure distribution in the claims dataset on the bar graph\n",
    "plt.figure(figsize=(100,25))\n",
    "plt.rc('xtick', labelsize=16)\n",
    "plt.bar(tmp_procedure_counts.index, tmp_procedure_counts['percentage'].tolist());\n",
    "#plt.xticks(tmp_diagnosis_counts.index, tmp_diagnosis_counts['Diagnosis related group'].tolist(), rotation=45)\n",
    "plt.ylabel('Percentage claims')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run pre processing on procedure descriptions\n",
    "tmp_procedure_tokenized = dict_procedure_names['International Classification of Diseases'].map(lambda x: text_preprocessing(x))\n",
    "display(tmp_procedure_tokenized.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build a map of number of tokens in a phrase to number of phrases with speific number of phrases\n",
    "procedure_phrase_lengths = tmp_procedure_tokenized.map(lambda x: len(x)).value_counts().sort_index()\n",
    "\n",
    "# plot count of tokens in phrase on x-axis and number of phrases with a specific count of tokens in y-axis\n",
    "plt.bar(np.arange(1,1+len(procedure_phrase_lengths)), procedure_phrase_lengths)\n",
    "plt.xlabel('Number of Tokens');\n",
    "plt.ylabel('Phrases');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# traing wordtovec model on procedure description tokens\n",
    "model_prc = Word2Vec(tmp_procedure_tokenized, min_count = 1, size = 72, window = 5, iter = 100)\n",
    "print(model_prc)\n",
    "procedure_words = list(model_prc.wv.vocab)\n",
    "print(columnize.columnize(procedure_words, displaywidth=80, ljust=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot t_SNE chart for procedure word to vector\n",
    "# #2D visual plot of word embeddings derieved from procedure description.\n",
    "tsne_plot(model_prc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# test most similiar for some word from model_prc.wv.keywords\n",
    "model_prc.most_similar('nonoperative')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extract procedure words that starts with non\n",
    "#display(tmp_procedure_tokenized.head())\n",
    "series_procedure = pd.Series(procedure_words)\n",
    "procedure_words_with_non = series_procedure[series_procedure.map(lambda x: 'non' in x)]\n",
    "display(procedure_words_with_non)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check similarity between procedure words with opposite severity\n",
    "for i in procedure_words_with_non:\n",
    "    a, not_a = i.replace('non-','').replace('non',''), i\n",
    "    if a in procedure_words:\n",
    "        print('Cosine similarity between', a, not_a, ':', model_prc.wv.similarity(a, not_a))\n",
    "        print('')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate mean of word vector in a phrase to derieve phrase vector for procedure description \n",
    "values = []\n",
    "index =[]\n",
    "#iterate through list of strings in each procedure phrase\n",
    "for i, v in pd.Series(tmp_procedure_tokenized).items():\n",
    "    #calculate mean of all word embeddings in each procedure phrase\n",
    "    values.append(model_prc[v].mean(axis =0))\n",
    "    index.append(i)\n",
    "tmp_procedure_phrase_vector = pd.DataFrame({'ICD9 primary procedure code':index, 'PRC_VECTOR':values})\n",
    "display(tmp_procedure_phrase_vector.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# expand tmp_procedure_phrase_vector into dataframe\n",
    "# every scalar value in phrase vector will be considered a feature\n",
    "procedure_features = tmp_procedure_phrase_vector['PRC_VECTOR'].apply(pd.Series)\n",
    "\n",
    "# rename each variable in procedure_features use PRC_F as prefix\n",
    "procedure_features = procedure_features.rename(columns = lambda x : 'PRC_F' + str(x + 1))\n",
    "\n",
    "# view the procedure_features dataframe\n",
    "display(procedure_features.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#merge diagnosis word embeddings derived using word2vec in the base claims data as new features.\n",
    "tmp_join_claim_diagnosis = pd.merge(df_cms_claims_data, diagnosis_features, how='inner', left_on = 'Base DRG code', right_index = True)\n",
    "display(tmp_join_claim_diagnosis.head())\n",
    "\n",
    "#merge procedure word embeddings derived using word2vec in the base claims data as new features.\n",
    "tmp_join_claim_procedure = pd.merge(tmp_join_claim_diagnosis, procedure_features, how='inner', left_on = 'ICD9 primary procedure code', right_index = True)\n",
    "display(tmp_join_claim_procedure.head())\n",
    "\n",
    "#assign new feature set with procedure and diagnosis work embeddings to a new claims feature dataframe\n",
    "#aggregate all the features extrcated so far to build a final claims feature set for training\n",
    "claims_features = tmp_join_claim_procedure\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Split train and test: train only on normal data <a name=\"split\"></a>\n",
    "\n",
    "We want to split our data into training and test sets. We want to ensure that in this random split we have samples that cover the distribution of payments. We perform a stratified shuffle split on the DRG quintile payment amount code, taking 30% of the data for testing and 70% for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedShuffleSplit\n",
    "\n",
    "X = claims_features.drop(['Encrypted PUF ID','ICD9 primary procedure code','Base DRG code'], axis=1)\n",
    "strata = claims_features['DRG quintile payment amount code']\n",
    "sss = StratifiedShuffleSplit(n_splits=1, test_size=0.3, random_state=0)\n",
    "splits = sss.split(X, strata)\n",
    "for train_index, test_index in splits:\n",
    "    X_train, X_test = X.iloc[train_index], X.iloc[test_index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(X.head())\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5A. Standardize data based on training sample <a name=\"standardize\"></a>\n",
    "\n",
    "Because the PCA algorithm that we will use later for training maximizes the orthogonal variances of one's data, it is important to standardize the training data to have zero-mean and unit-variance prior to performing PCA. This way your PCA algorithm is idempotent to such rescalings, and prevent variables of large scale from dominating the PCA projection.\n",
    "\n",
    "$$ \\tilde{X} = \\frac{X-\\mu_x}{\\sigma_z} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_obs, n_features = X_train.shape\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(X_train)\n",
    "X_stndrd_train = scaler.transform(X_train)\n",
    "X_stndrd_train = pd.DataFrame(X_stndrd_train, index=X_train.index, columns=X_train.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5B. PCA <a name=\"pca\"></a>\n",
    "\n",
    "Principal Component Analysis (PCA) is an unsupervised method for taking a data set where features have multi-collinearity and creating a decorrelated data set, by finding the linear combination of vectors which maximize the data's variances in orthogonal dimensions. \n",
    "\n",
    "#### PCA  on Amazon SageMaker \n",
    "\n",
    "The built-in PCA algorithm of SageMaker solves for the singular values, $s$, and for the Principal Components, $V$, of our data set. Here we'll perform SageMaker PCA on our standardized training dataset $\\tilde{X}$, and then we'll use its outputs to project our correlated dataset into a decorrelated one.\n",
    "\n",
    "$$ s, V = \\rm{PCA}(\\tilde{X})$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert data to binary stream.\n",
    "matrx_train = X_stndrd_train.as_matrix().astype('float32')\n",
    "import io\n",
    "import sagemaker.amazon.common as smac\n",
    "buf_train = io.BytesIO()\n",
    "smac.write_numpy_to_dense_tensor(buf_train, matrx_train)\n",
    "buf_train.seek(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to upload the file object to our Amazon S3 bucket. We specify two paths: one to where our uploaded matrix will reside, and one to where Amazon SageMaker will write the output. Amazon SageMaker will create folders within the paths that do not already exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "key = 'healthcare_fraud_identification_feature_store'\n",
    "prefix = 'sagemaker/pca-anol'\n",
    "\n",
    "boto3.resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'train', key)).upload_fileobj(buf_train)\n",
    "s3_train_data = 's3://{}/{}/train/{}'.format(bucket, prefix, key)\n",
    "print('uploaded training data location: {}'.format(s3_train_data))\n",
    "\n",
    "output_location = 's3://{}/{}/output/model'.format(bucket, prefix)\n",
    "print('training artifacts will be uploaded to: {}'.format(output_location))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "# select the algorithm container based on this notebook's current location\n",
    "\n",
    "region_name = boto3.Session().region_name\n",
    "container = get_image_uri(region_name, 'pca')\n",
    "\n",
    "print('Using SageMaker PCA container: {} ({})'.format(container, region_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Start the Amazon Sagemaker Session and set training parameters for Estimator API\n",
    "\n",
    "Instance type should be one of the following and number of instances can be greater than 1. Option to train on P instance type family to use GPUs for training\n",
    "\n",
    "#### [ml.p2.xlarge, ml.m5.4xlarge, ml.m4.16xlarge, ml.p3.16xlarge, ml.m5.large, ml.p2.16xlarge, ml.c4.2xlarge, ml.c5.2xlarge, ml.c4.4xlarge, ml.c5.4xlarge, ml.c4.8xlarge, ml.c5.9xlarge, ml.c5.xlarge, ml.c4.xlarge, ml.c5.18xlarge, ml.p3.2xlarge, ml.m5.xlarge, ml.m4.10xlarge, ml.m5.12xlarge, ml.m4.xlarge, ml.m5.24xlarge, ml.m4.2xlarge, ml.p2.8xlarge, ml.m5.2xlarge, ml.p3.8xlarge, ml.m4.4xlarge]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_obs, feature_dim = np.shape(matrx_train)\n",
    "num_components = feature_dim-1\n",
    "num_instances=2\n",
    "instance_type = 'ml.c5.2xlarge'\n",
    "algorithm_mode='regular'\n",
    "platform='sagemaker'\n",
    "\n",
    "\n",
    "start = time.time()\n",
    "\n",
    "sess = sagemaker.Session()\n",
    "\n",
    "pca = sagemaker.estimator.Estimator(container,\n",
    "                                    role, \n",
    "                                    train_instance_count=num_instances, \n",
    "                                    train_instance_type=instance_type,\n",
    "                                    output_path=output_location,\n",
    "                                    sagemaker_session=sess)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Specify the hyperparameters for your training job and start the training using Amazon SageMaker fit API call\n",
    "\n",
    "Training will take approximately 4-5 minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca.set_hyperparameters(feature_dim=feature_dim,\n",
    "                        num_components=num_components,\n",
    "                        subtract_mean=False,\n",
    "                        algorithm_mode='regular',\n",
    "                        mini_batch_size=200)\n",
    "\n",
    "print('Start timestamp of launch: '+ str(start))\n",
    "pca.fit({'train': s3_train_data})\n",
    "\n",
    "stop = time.time()\n",
    "total_time = stop-start\n",
    "print('%2.2f minutes' %(total_time/60))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the training job is complete, SageMaker writes the model artifact to the specified S3 output location. Let's download and unpack returned PCA model artifact. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_name = pca.latest_training_job.name\n",
    "os.system('aws s3 cp {}/{}/output/model.tar.gz ./'.format(output_location, job_name))\n",
    "!tar xvzf model.tar.gz\n",
    "\n",
    "pca_model = mx.ndarray.load('model_algo-1')\n",
    "print('PCA model artifact:', pca_model.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SageMaker PCA artifact contains $V$, the eigenvector principal components in *increasing* order of $s$, their singular values. A component's singular value is equal to the standard deviation that the component explains, i.e., the squared value of a singular component is equal to the variance that component explains. Therefore to calculate the proportion of variance of the data that each component explains, take the square of the singular value and divide it by the sum of all the singular values squared:\n",
    "$$ \\rm{component \\,}i \\% \\rm{\\,variance\\,  explained} =  100\\cdot\\frac{s_i^s}{\\sum_{p=1}^P s_p^2} $$\n",
    "\n",
    "\n",
    "First, we'll reverse this returned ordering, so that instead we have the components which explain the most variance come first, i.e., reorder the components in decreasing order of their singular values.\n",
    "\n",
    "\n",
    "PCA can be further used to reduce the dimensionality of the problem. We have $P$ features and $P-1$ components, but we'll see in the plot below that many of the components don't contribute much to the explained variance of the data. We will keep only the $K$ leading components of $V$ which  explain 95% of the variance in our data. \n",
    "\n",
    "We will denote this reduced matrix as $V_K$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "singular_values = pca_model['s'].asnumpy()[::-1]\n",
    "pc_reversedorder = pd.DataFrame(pca_model['v'].asnumpy())\n",
    "pc = pc_reversedorder[list(pc_reversedorder.columns[::-1])]\n",
    "\n",
    "eigenvalues = np.power(singular_values,2)\n",
    "explained_var_pct = eigenvalues/np.sum(eigenvalues) *100\n",
    "explained_var_cum = np.cumsum(explained_var_pct)\n",
    "\n",
    "var_threshold = 95\n",
    "n_components = np.min([np.where(explained_var_cum>=var_threshold)[0][0], n_features-1])\n",
    "print('%i components explain %2.2f%% of the data\\'s variance.' %(n_components+1, explained_var_cum[n_components]))\n",
    "\n",
    "fig= plt.figure(figsize=[14,8])\n",
    "width = 0.5\n",
    "ax1 = fig.add_subplot(111)\n",
    "ax1.bar(np.arange(0,len(singular_values)), singular_values, align='edge',  color='darkgreen', label='Singular Values', alpha=0.5, width=width);\n",
    "ax1.set_ylabel('Singular Values', fontsize=17);\n",
    "ax1.set_xlabel('Principal Component', fontsize=17);\n",
    "ax1.legend(loc='upper right', fontsize=14)\n",
    "ax2 = ax1.twinx()\n",
    "ax2.plot(np.arange(0,len(explained_var_cum)), explained_var_cum, color='black',  label='Cumulative');\n",
    "ax2.plot([0, n_components], [var_threshold, var_threshold], 'r:')\n",
    "ax2.plot([n_components, n_components], [0, var_threshold], 'r:')\n",
    "ax2.set_ylabel('% Variance Explained', fontsize=17);\n",
    "ax2.legend(loc='right', fontsize=14)\n",
    "ax2.set_ylim([0, 100])\n",
    "ax2.set_xlim([0,len(eigenvalues)])\n",
    "plt.title('Dimensionality Reduction', fontsize=20);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will now work with the reduced matrix that includes components that explains 95% of variance in the data\n",
    "Vk = pc[pc.columns[:n_components+1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Calculate the Mahalanobis distance <a name=\"md\"></a>\n",
    "\n",
    "Above, we used the singular values returned by PCA to keep the $K$ principal component vectors  that explain 95% of the data's variance, and stored them in dataframe $V_K$.\n",
    "\n",
    "We use $V_K$ to tranform the data into an decorrelated dataset, by taking their matrix dot product:\n",
    "$$ Z = \\tilde{X} V_K $$\n",
    "\n",
    "To detect anomaly data points, we want to measure how far a data point is from the distribution of the projected data. The farther a point lays from the distribution, the more anomalous it is.\n",
    "\n",
    "Even though we have $K$ dimensions instead of $P$, this is still a multi-variate distribution. We will use the Mahalanobis distance [Mahalanobis, 1936](https://insa.nic.in/writereaddata/UpLoadedFiles/PINSA/Vol02_1936_1_Art05.pdf), which is a scalar measure of the multi-variate distance between a point $z$ and a distribution $D$. Distribution $D$ is defined by the mean and the inverse-covariance of the data in $Z$:\n",
    "$$ \\mu_Z = \\rm{mean}(Z) $$\n",
    "$$ \\Sigma_Z = \\rm{cov}(Z) $$\n",
    "$$ \\Sigma_Z^{-1} = \\rm{inv}\\big(\\rm{cov}(Z)\\big) $$\n",
    "Mahalanobis distance is a measure of how many standard deviations away $z$ is from the mean of $D$ along each principal component axis. \n",
    "\n",
    "We'll use the Mahalonobis distance of each point as its anomaly score. We take the top $\\alpha$% of these points to consider as outliers, where $\\alpha$ depends on how sensitive we want our detection to be. For this problem,  we will take the top 1%, i.e. $\\alpha=0.01$. Therefore we calculate the $(1-\\alpha)$-quantile of Distribution $D$ as the threshold for considering a data point anomalous.\n",
    "\n",
    "\n",
    "This method of PCA Anomaly Detection was developed in [A Novel Anomaly Detection Scheme Based on Principal Component Classifier](https://homepages.laas.fr/owe/METROSEC/DOC/FDM03.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Z is the PCA-projected standardized data\n",
    "pca_projected_X_train = pd.DataFrame(np.dot(X_stndrd_train, Vk), index=X_stndrd_train.index)\n",
    "\n",
    "# Calculate Mahalanobis distance for multi-variate deviation\n",
    "Zmean = pca_projected_X_train.mean()\n",
    "covZ = pca_projected_X_train.cov()\n",
    "invcovZ = inv(covZ)\n",
    "M = pca_projected_X_train.apply(lambda x: sp.spatial.distance.mahalanobis(x, Zmean, invcovZ), axis=1)\n",
    "\n",
    "# Threshold the training set's top alpha-%\n",
    "alpha = 0.01\n",
    "threshold = np.quantile(M, 1-alpha)\n",
    "print(threshold)\n",
    "\n",
    "# Plot the density graph for anomaly score and highlight the threshold calculated\n",
    "plt.figure(figsize=[15,5]);\n",
    "M.hist(bins=40, density=True);\n",
    "plt.axvline(threshold, color='red', label='{}%-threshold = {}'.format(int(alpha*100), round(threshold,4)));\n",
    "plt.legend();\n",
    "plt.xlabel(r'Anomaly Score [based on Mahalanobis distance]', fontsize=14);\n",
    "plt.ylabel('Density', fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Unsupervised Anomaly Detection <a name=\"ad\"></a>\n",
    "\n",
    "The above PCA-computed quantities - component matrix $V_K$, projected mean $\\mu_Z$, inverse-covariance $\\Sigma_Z^{-1}$, and threshold - have delivered us an unsupervised anomaly detection method.\n",
    "\n",
    "We create a function below, which transforms the test data according the models fit on. the training data. The function **calcAnomalyScore**() performs the following:\n",
    "* standardizes each test data point according to the training mean and training standard deviation\n",
    "* projects each test data point using the PCs calculated from the training data\n",
    "* measures the Mahalanobis distance of each test data point from the training distribution $D$\n",
    "* a boolean if the test data point's anomaly score exceeds the threshold\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calcAnomalyScore(data, threshold, scaler=scaler, pc=Vk, Zmean=Zmean, invcovZ=invcovZ):\n",
    "    data_stndrd = pd.DataFrame(scaler.transform(data), index=data.index, columns=data.columns)\n",
    "    pc_projected_data = pd.DataFrame(np.dot(data_stndrd, Vk), index=data_stndrd.index)\n",
    "    anomaly_score = pc_projected_data.apply(lambda x: sp.spatial.distance.mahalanobis(x, Zmean, invcovZ), axis=1)\n",
    "    is_anomaly = (anomaly_score>threshold)\n",
    "    y = pd.concat([anomaly_score, is_anomaly], axis=1)\n",
    "    y.columns = ['anomaly_score','is_anomaly']\n",
    "    return y\n",
    "\n",
    "\n",
    "y_test = calcAnomalyScore(X_test, threshold, scaler=scaler, pc=Vk, Zmean=Zmean, invcovZ=invcovZ)\n",
    "print('Fraction of test data flagged as anomalous:', y_test['is_anomaly'].mean())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Understanding Anomaly<a name=\"understandinganomaly\"></a>\n",
    "\n",
    "Data points marked TRUE for \"is_anomaly\" can be passed on for inspection. Given that we now have separated norm data from anomalous data, we can contrast these to see if the differentiating reasons can be identified in the original feature space.\n",
    "\n",
    "We attach the \"is_anomaly\" output as a label to the original claims feature data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#list all claims with anomaly score and anomaly label(True)\n",
    "y_test['anomalous'] = (y_test['is_anomaly']*1.).astype(int)\n",
    "\n",
    "test_claims = claims_features.loc[y_test.index]\n",
    "test_claims = y_test.merge(test_claims, how='outer', left_index=True, right_index=True)\n",
    "test_claims = test_claims.filter([\"anomalous\",\"DRG quintile payment amount code\",\"DRG quintile average payment amount\",\"Inpatient days code\",\"ICD9 primary procedure code\",\"Base DRG code\",\"Beneficiary Age category code\",\"Beneficiary gender code\"])\n",
    "display(test_claims.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(test_claims,hue =\"anomalous\", kind='scatter', plot_kws={'alpha':0.1})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In the above pair plot, look for following patterns\n",
    "1. Plots where orange is asymmetrical with blue. \n",
    "2. Orange appears in patches that doesn't overlap with the blue\n",
    "\n",
    "The above patterns in the pairplot can be used a starting point to target investigation on specific cases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Deploy PCA <a name=\"deployendpoint\"></a>\n",
    "\n",
    "This section is optional, but, in case, you are interested in learning how to do principal component analysis for a given claim record using Amazon SageMaker hosting. Follow the steps below. \n",
    "\n",
    "You may find this step helpful if you want to use principal components of claims data to predict other variables of business significance. Example, find out length of stay based on diagnosis code, gender and age or predict the claims payment amount and quartile based on datapoints in the claims dataset. \n",
    "\n",
    "Here we demonstrate how to deploy PCA model as an endpoint on Amazon Sagemaker for inference. But, to solve the example problems discussed in the above paragraph you will need to collect more data, label them and refactor your training based on the prediction problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#serialize test data to binary format for realtime inference for extracting principal components of claim features\n",
    "X_stndrd_test = scaler.transform(X_test)\n",
    "X_stndrd_test = pd.DataFrame(X_stndrd_test, index=X_test.index, columns=X_test.columns)\n",
    "inference_input = X_stndrd_test.as_matrix().astype('float32')\n",
    "buf = io.BytesIO()\n",
    "smac.write_numpy_to_dense_tensor(buf, inference_input)\n",
    "buf.seek(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print the shape of inference_input matrix \n",
    "inference_input.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Deploy the model using Amaazon SageMaker deploy API. AWS manages the highly avaliable and reliable infrastructure for it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#deploy the Amazon Sagemaker PCA model trained above to create a hosted enpoint for realtime principal component extraction\n",
    "pca_predictor = pca.deploy(initial_instance_count=1,\n",
    "                           instance_type='ml.m4.xlarge')\n",
    "\n",
    "from sagemaker.predictor import csv_serializer, json_deserializer\n",
    "\n",
    "pca_predictor.content_type = 'text/csv'\n",
    "pca_predictor.serializer = csv_serializer\n",
    "pca_predictor.deserializer = json_deserializer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#run inference on first 500 claims. Avoid running it on large number of claims to avoid timeout on connection. \n",
    "#For large dataset use Amazon Sagemaker batch inference\n",
    "result = pca_predictor.predict(inference_input[0:500])\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#normalize above result in json format to more readable columar format with one principal component per column\n",
    "from pandas.io.json import json_normalize\n",
    "#result in json format and components are returned as a list under projections tag\n",
    "result_normalized = json_normalize(result,'projections')\n",
    "# expand df.tags into its own dataframe\n",
    "pca_components = result_normalized['projection'].apply(pd.Series)\n",
    "# rename each variable in pc\n",
    "pca_components = pca_components.rename(columns = lambda x : 'PC_' + str(x))\n",
    "#view the tags dataframe\n",
    "pca_components"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the Endpoint\n",
    "\n",
    "If you're ready to be done with this notebook, please run the delete_endpoint line in the cell below. This will remove the hosted endpoint you created and avoid any charges from a stray instance being left turned on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "sagemaker.Session().delete_endpoint(pca_predictor.endpoint)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_mxnet_p36",
   "language": "python",
   "name": "conda_mxnet_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}