C3_W2_Lab01_PCA_Visualization_Examples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7fa19ee6",
   "metadata": {},
   "source": [
    "# PCA - An example on Exploratory Data Analysis\n",
    "\n",
    "In this notebook you will:\n",
    "\n",
    "- Replicate Andrew's example on PCA\n",
    "- Visualize how PCA works on a 2-dimensional small dataset and that not every projection is \"good\"\n",
    "- Visualize how a 3-dimensional data can also be contained in a 2-dimensional subspace\n",
    "- Use PCA to find hidden patterns in a high-dimensional dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f7908b2",
   "metadata": {},
   "source": [
    "## Importing the libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a7b0864",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.decomposition import PCA\n",
    "from pca_utils import plot_widget\n",
    "from bokeh.io import show, output_notebook\n",
    "from bokeh.plotting import figure\n",
    "import matplotlib.pyplot as plt\n",
    "import plotly.offline as py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b28d534",
   "metadata": {},
   "outputs": [],
   "source": [
    "py.init_notebook_mode()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61ce6848",
   "metadata": {},
   "outputs": [],
   "source": [
    "output_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dec0fa33",
   "metadata": {},
   "source": [
    "## Lecture Example\n",
    "\n",
    "We are going work on the same example that Andrew has shown in the lecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08c5f166",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.array([[ 99,  -1],\n",
    "       [ 98,  -1],\n",
    "       [ 97,  -2],\n",
    "       [101,   1],\n",
    "       [102,   1],\n",
    "       [103,   2]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d45b51d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(X[:,0], X[:,1], 'ro')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f8a9d5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading the PCA algorithm\n",
    "pca_2 = PCA(n_components=2)\n",
    "pca_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4e0e4a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's fit the data. We do not need to scale it, since sklearn's implementation already handles it.\n",
    "pca_2.fit(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5701428a",
   "metadata": {},
   "outputs": [],
   "source": [
    "pca_2.explained_variance_ratio_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcc964ba",
   "metadata": {},
   "source": [
    "The coordinates on the first principal component (first axis) are enough to retain 99.24% of the information (\"explained variance\").  The second principal component adds an additional 0.76% of the information (\"explained variance\") that is not stored in the first principal component coordinates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3b6455c",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_trans_2 = pca_2.transform(X)\n",
    "X_trans_2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1771a36",
   "metadata": {},
   "source": [
    "Think of column 1 as the coordinate along the first principal component (the first new axis) and column 2 as the coordinate along the second principal component (the second new axis).\n",
    "\n",
    "You can probably just choose the first principal component since it retains 99% of the information (explained variance)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc5e7335",
   "metadata": {},
   "outputs": [],
   "source": [
    "pca_1 = PCA(n_components=1)\n",
    "pca_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61054f68",
   "metadata": {},
   "outputs": [],
   "source": [
    "pca_1.fit(X)\n",
    "pca_1.explained_variance_ratio_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf898e90",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_trans_1 = pca_1.transform(X)\n",
    "X_trans_1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f750a0d",
   "metadata": {},
   "source": [
    "Notice how this column is just the first column of `X_trans_2`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f4bfa8b",
   "metadata": {},
   "source": [
    "If you had 2 features (two columns of data) and choose 2 principal components, then you'll keep all the information and the data will end up the same as the original."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79289f34",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_reduced_2 = pca_2.inverse_transform(X_trans_2)\n",
    "X_reduced_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e64d79ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(X_reduced_2[:,0], X_reduced_2[:,1], 'ro')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b27e810",
   "metadata": {},
   "source": [
    "Reduce to 1 dimension instead of 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5c426c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_reduced_1 = pca_1.inverse_transform(X_trans_1)\n",
    "X_reduced_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35ab1633",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(X_reduced_1[:,0], X_reduced_1[:,1], 'ro')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61b5fe66",
   "metadata": {},
   "source": [
    "Notice how the data are now just on a single line (this line is the single principal component that was used to describe the data; and each example had a single \"coordinate\" along that axis to describe its location."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01212aad",
   "metadata": {},
   "source": [
    "## Visualizing the PCA algorithm\n",
    "\n",
    "Let's define $10$ points in the plane and use them as an example to visualize how we can compress this points in 1 dimension. You will see that there are good ways and bad ways."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e53e20f",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.array([[-0.83934975, -0.21160323],\n",
    "       [ 0.67508491,  0.25113527],\n",
    "       [-0.05495253,  0.36339613],\n",
    "       [-0.57524042,  0.24450324],\n",
    "       [ 0.58468572,  0.95337657],\n",
    "       [ 0.5663363 ,  0.07555096],\n",
    "       [-0.50228538, -0.65749982],\n",
    "       [-0.14075593,  0.02713815],\n",
    "       [ 0.2587186 , -0.26890678],\n",
    "       [ 0.02775847, -0.77709049]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c218bdd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "p = figure(title = '10-point scatterplot', x_axis_label = 'x-axis', y_axis_label = 'y-axis') ## Creates the figure object\n",
    "p.scatter(X[:,0],X[:,1],marker = 'o', color = '#C00000', size = 5) ## Add the scatter plot\n",
    "\n",
    "## Some visual adjustments\n",
    "p.grid.visible = False\n",
    "p.grid.visible = False\n",
    "p.outline_line_color = None \n",
    "p.toolbar.logo = None\n",
    "p.toolbar_location = None\n",
    "p.xaxis.axis_line_color = \"#f0f0f0\"\n",
    "p.xaxis.axis_line_width = 5\n",
    "p.yaxis.axis_line_color = \"#f0f0f0\"\n",
    "p.yaxis.axis_line_width = 5\n",
    "\n",
    "## Shows the figure\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65bba887",
   "metadata": {},
   "source": [
    "The next code will generate a widget where you can see how different ways of compressing this data into 1-dimensional datapoints will lead to different ways on how the points are spread in this new space. The line generated by PCA is the line that keeps the points as far as possible from each other. \n",
    "\n",
    "You can use the slider to rotate the black line through its center and see how the points' projection onto the line will change as we rotate the line. \n",
    "\n",
    "You can notice that there are projections that place different points in almost the same point, and there are projections that keep the points as separated as they were in the plane."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "384a5674",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_widget()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60071b9c",
   "metadata": {},
   "source": [
    "## Visualization of a 3-dimensional dataset \n",
    "\n",
    "In this section we will see how some 3 dimensional data can be condensed into a 2 dimensional space. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d91e0c56",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pca_utils import random_point_circle, plot_3d_2d_graphs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "113ac73d",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = random_point_circle(n = 150)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc503f9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "deb = plot_3d_2d_graphs(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a3b252f",
   "metadata": {},
   "outputs": [],
   "source": [
    "deb.update_layout(yaxis2 = dict(title_text = 'test', visible=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bd6577b",
   "metadata": {},
   "source": [
    "## Using PCA in Exploratory Data Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa3c3ca7",
   "metadata": {},
   "source": [
    "Let's load a toy dataset with $500$ samples and $1000$ features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b2dad2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"toy_dataset.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a398b9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "596e20fc",
   "metadata": {},
   "source": [
    "This is a dataset with $1000$ features.\n",
    "\n",
    "Let's try to see if there is a pattern in the data. The following function will randomly sample 100 pairwise tuples (x,y) of features, so we can scatter-plot them.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14f8184c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_pairs(n = 100):\n",
    "    from random import randint\n",
    "    i = 0\n",
    "    tuples = []\n",
    "    while i < 100:\n",
    "        x = df.columns[randint(0,999)]\n",
    "        y = df.columns[randint(0,999)]\n",
    "        while x == y or (x,y) in tuples or (y,x) in tuples:\n",
    "            y = df.columns[randint(0,999)]\n",
    "        tuples.append((x,y))\n",
    "        i+=1\n",
    "    return tuples\n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24a795e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "pairs = get_pairs()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "334f7d4c",
   "metadata": {},
   "source": [
    "Now let's plot them!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6c3bdf6",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(10,10, figsize = (35,35))\n",
    "i = 0\n",
    "for rows in axs:\n",
    "    for ax in rows:\n",
    "        ax.scatter(df[pairs[i][0]],df[pairs[i][1]], color = \"#C00000\")\n",
    "        ax.set_xlabel(pairs[i][0])\n",
    "        ax.set_ylabel(pairs[i][1])\n",
    "        i+=1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e45a6ce6",
   "metadata": {},
   "source": [
    "It looks like there is not much information hidden in pairwise features. Also, it is not possible to check every combination, due to the amount of features. Let's try to see the linear correlation between them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b1eb839",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This may take 1 minute to run\n",
    "corr = df.corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a93d51d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "## This will show all the features that have correlation > 0.5 in absolute value. We remove the features \n",
    "## with correlation == 1 to remove the correlation of a feature with itself\n",
    "\n",
    "mask = (abs(corr) > 0.5) & (abs(corr) != 1)\n",
    "corr.where(mask).stack().sort_values()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f253dac7",
   "metadata": {},
   "source": [
    "The maximum and minimum correlation is around $0.631$ - $0.632$. This does not show too much as well. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9928258f",
   "metadata": {},
   "source": [
    "Let's try PCA decomposition to compress our data into a 2-dimensional subspace (plane) so we can plot it as scatter plot. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7792253",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading the PCA object\n",
    "pca = PCA(n_components = 2) # Here we choose the number of components that we will keep.\n",
    "X_pca = pca.fit_transform(df)\n",
    "df_pca = pd.DataFrame(X_pca, columns = ['principal_component_1','principal_component_2'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03ed2662",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_pca.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e584372",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(df_pca['principal_component_1'],df_pca['principal_component_2'], color = \"#C00000\")\n",
    "plt.xlabel('principal_component_1')\n",
    "plt.ylabel('principal_component_2')\n",
    "plt.title('PCA decomposition')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9def66f",
   "metadata": {},
   "source": [
    "This is great! We can see well defined clusters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa48c38a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pca.explained_variance_ration_ returns a list where it shows the amount of variance explained by each principal component.\n",
    "sum(pca.explained_variance_ratio_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5324318d",
   "metadata": {},
   "source": [
    "And we preserved only around 14.6% of the variance!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80a4be26",
   "metadata": {},
   "source": [
    "Quite impressive! We can clearly see clusters in our data, something that we could not see before. How many clusters can you spot? 8, 10? \n",
    "\n",
    "If we run a PCA to plot 3 dimensions, we will get more information from data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3e49e3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "pca_3 = PCA(n_components = 3).fit(df)\n",
    "X_t = pca_3.transform(df)\n",
    "df_pca_3 = pd.DataFrame(X_t,columns = ['principal_component_1','principal_component_2','principal_component_3'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "428ca989",
   "metadata": {},
   "outputs": [],
   "source": [
    "import plotly.express as px"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb1a8376",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = px.scatter_3d(df_pca_3, x = 'principal_component_1', y = 'principal_component_2', z = 'principal_component_3').update_traces(marker = dict(color = \"#C00000\"))\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "befb75a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "sum(pca_3.explained_variance_ratio_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dae3a319",
   "metadata": {},
   "source": [
    "Now we preserved 19% of the variance and we can clearly see 10 clusters.\n",
    "\n",
    "Congratulations on finishing this notebook!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}