07_k-means.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "xSDP4bMrIL2x"
   },
   "source": [
    "# K-means Model\n",
    "---"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.   **[Introduction to K-means](#1.-Introduction-to-K-means)**\n",
    "2.   **[Foundations of K-means](#2.-Foundations-of-K-means)**\n",
    "3.   **[Model Assumptions](#3.-Model-Assumptions)**\n",
    "4.   **[Model Evaluation](#4.-Model-Evaluation)**\n",
    "5.   **[Exploratory Data Analysis](#5.-Exploratory-Data-Analysis)**\n",
    "6.   **[Model Construction](#6.-Model-Construction)**\n",
    "7.   **[Model Results](#7.-Model-Results)**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"1.-Introduction-to-K-means\"></a>\n",
    "### 1. Introduction to K-means"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.1 Definitions"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**K-means |** An unsupervised learning partitioning algorithm used for clustering unlabeled data\n",
    "\n",
    "**Unsupervised learning |** Used on unlabeled data where the goal is to learn about the data's underlying structure\n",
    "\n",
    "**Centroid |** The center of a cluster determined by the mathematical mean of all the points in that cluster "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"2.-Foundations-of-K-means\"></a>\n",
    "### 2. Foundations of K-means"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**There are 4 steps in the creation of a K-means model:**\n",
    "1. Randomly place centroids in the data space\n",
    "2. Assign each point to its nearest centroid\n",
    "3. Update the location of each centroid to the mean position of all the points assigned to it\n",
    "4. Repeat steps 2 and 3 until the model converges(i.e. all centroid locations remain unchanged with successive iterations)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"3.-Model-Assumptions\"></a>\n",
    "### 3. Model Assumptions"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model assumptions are statements about the data that must be true in order to justify the use of a particular modeling technique\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### 3.1 K-means Assumptions\n",
    "1. **Assumption of Homogeneity of Variance:** K-means assumes that the variance of the distribution of each variable is equal across all clusters.\n",
    "\n",
    "2. **Assumption of Independence:** K-means assumes that the observations in the dataset are independent of each other. This means that the value of one observation does not affect the value of another observation.\n",
    "\n",
    "3. **Assumption of Euclidean Distance:** K-means assumes that the distances between observations are measured using Euclidean distance. This means that the distance between two observations is calculated as the square root of the sum of the squared differences between their corresponding attributes.\n",
    "\n",
    "4. **Assumption of Clustering Structure:** K-means assumes that the data points in each cluster have a similar clustering structure, which means that the clusters are spherical, and have equal variance."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### 3.2 Assumption Checks\n",
    "\n",
    "1. **Homogeneity of Variance:** To check if the variance of each variable is equal across all clusters, you can use statistical tests such as Bartlett's test or Levene's test. If the test shows that the variances are significantly different, then the assumption of homogeneity of variance may not hold.\n",
    "\n",
    "2. **Independence:** To check if the observations are independent of each other, you can examine the dataset for any dependencies or correlations among the variables. If there are dependencies or correlations, the assumption of independence may not hold. Additionally, if the dataset has a time-series structure, then the assumption of independence may not hold, and time-series clustering techniques may be more appropriate.\n",
    "\n",
    "3. **Euclidean Distance:** The Euclidean distance is a fundamental assumption of k-means. To ensure this assumption holds, you can check if the variables are on the same scale, and if not, standardize the variables so that they have a similar range. Additionally, you can use other distance measures such as Manhattan distance or Mahalanobis distance, which can handle variables on different scales or account for correlations between variables.\n",
    "\n",
    "4. **Clustering Structure:** To check if the data points in each cluster have a similar clustering structure, you can use visualization techniques such as scatterplots or boxplots to examine the distribution of the variables within each cluster. If the clusters have different shapes or sizes, then the assumption of clustering structure may not hold.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"4.-Model-Evaluation\"></a>\n",
    "### 4. Model Evaluation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.1 Inertia\n",
    "Sum of the squared distances between each observation and its nearest centroid \n",
    "\n",
    "- Measures intracluster distance\n",
    "- Equal to the sum of the squared distance between each point and the centroid of the cluster that it’s assigned to\n",
    "- Used in elbow plots\n",
    "- All else equal, lower values are generally better\n",
    "\n",
    "Inertia = $\\sum\\limits_{i=1}^{n} (X_1 - C_k)^2$\n",
    "\n",
    "**The elbow method |** The elbow method is a way to help decide which clustering gives the most meaningful model of your data. It uses a line plot to visually compare the inertias of different models. With K-means models, this is done as a comparison between different values of *k.*\n",
    "\n",
    "\n",
    "\n",
    "####  4.2 Silhouette analysis\n",
    "A silhouette analysis is the comparison of different models’ silhouette scores. To calculate a model’s silhouette score, first, a silhouette coefficient is calculated for each instance in the data. \n",
    "An instance’s silhouette coefficient is defined by the following formula, where:\n",
    "\n",
    "- $a$= the mean distance between the instance and each other instance in the same cluster \n",
    "\n",
    "- $b$= the mean distance from the instance to each instance in the nearest other cluster (i.e., excluding the cluster that the instance is assigned to)\n",
    "\n",
    "- $max(a,b)$ = whichever value is greater, $a$ or $b$\n",
    "\n",
    "- $\\text{Silhouette coefficient} = \\frac{(b-a)}{max(a,b)}$\n",
    "\n",
    "A silhouette coefficient can range between -1 and +1. A value closer to +1 means that a point is close to other points in its own cluster and well separated from points in other clusters. As with inertia values, you can plot silhouette scores for different models to compare them against each other. \n",
    "\n",
    "Note that, unlike inertia, silhouette coefficients contain information about both intracluster distance (captured by the variable a) and intercluster distance (captured by the variable b).\n",
    "\n",
    "Silhouette summary:\n",
    "- Measures both intercluster distance and intracluster distance\n",
    "- Equal to the average of all points’ silhouette coefficients\n",
    "- Can be between -1 and +1 (greater values are better)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"5.-Exploratory-Data-Analysis\"></a>\n",
    "### 5. Exploratory Data Analysis"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.1 Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import standard operational packages.\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Important tools for modeling and evaluation.\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.metrics import silhouette_score\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Import visualization packages.\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# Load the dataset into a DataFrame and save in a variable\n",
    "data = pd.read_csv(\"example_file.csv\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.2 Data Exploration\n",
    "After loading the dataset, the next step is to prepare the data to be suitable for clustering. This includes: \n",
    "\n",
    "*   Exploring data\n",
    "*   Checking for missing values\n",
    "*   Encoding categorical data \n",
    "*   Dropping irrelevant columns\n",
    "*   Scaling the features using `StandardScaler`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display the first 10 rows of the data\n",
    "data.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display number of rows, number of columns\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display the data type for each column. NB logistic regression models expect numeric data\n",
    "data.dtypes"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 5.2.1 Check for Missing Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check for missing values.\n",
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop rows with missing values.\n",
    "# Save DataFrame in variable `data_subset`.\n",
    "data_subset = data.dropna(axis=0).reset_index(drop = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check for missing values.\n",
    "data_subset.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# View first 10 rows.\n",
    "data_subset.head(10)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 5.2.2 Encode Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert columns from categorical to numeric.\n",
    "data_subset = pd.get_dummies(data_subset, drop_first= True, columns= ['categorical_column'])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 5.2.3 Drop Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop the island column.\n",
    "data_subset = data_subset.drop(['irrelevant_column'], axis=1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 5.2.4 Scale Features\n",
    "Because K-means uses distance between observations as its measure of similarity, it's important to scale the data before modeling. Use a third-party tool, such as scikit-learn's StandardScaler function. StandardScaler scales each point xᵢ by subtracting the mean observed value for that feature and dividing by the standard deviation:\n",
    "\n",
    "x-scaled = (xᵢ – mean(X)) / σ\n",
    "\n",
    "This ensures that all variables have a mean of 0 and variance/standard deviation of 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a dataframe of the independent variables X by excluding any columns which do not belong\n",
    "X = data_subset.drop(['column'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Scale the features.\n",
    "#Assign the scaled data to variable `X_scaled`.\n",
    "X_scaled = StandardScaler().fit_transform(X)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"6.-Model-Construction\"></a>\n",
    "### 6. Model Construction\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, fit K-means and evaluate inertia for different values of k. Because you may not know how many clusters exist in the data, start by fitting K-means and examining the inertia values for different values of k. To do this, write a function called `kmeans_inertia` that takes in `num_clusters` and `x_vals` (`X_scaled`) and returns a list of each k-value's inertia.\n",
    "\n",
    "When using K-means inside the function, set the `random_state` to `42`. This way, others can reproduce your results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit K-means and evaluate inertia for different values of k.\n",
    "num_clusters = [i for i in range(2, 11)]\n",
    "\n",
    "def kmeans_inertia(num_clusters, x_vals):\n",
    "    \"\"\"\n",
    "    Accepts as arguments list of ints and data array. \n",
    "    Fits a KMeans model where k = each value in the list of ints. \n",
    "    Returns each k-value's inertia appended to a list.\n",
    "    \"\"\"\n",
    "    inertia = []\n",
    "    for num in num_clusters:\n",
    "        kms = KMeans(n_clusters=num, random_state=42)\n",
    "        kms.fit(x_vals)\n",
    "        inertia.append(kms.inertia_)\n",
    "\n",
    "    return inertia"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `kmeans_inertia` function to return a list of inertia for k= 2 to 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return a list of inertia for k=2 to 10.\n",
    "inertia = kmeans_inertia(num_clusters, X_scaled)\n",
    "inertia"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a line plot that shows the relationship between num_clusters and inertia\n",
    "plot = sns.lineplot(x=num_clusters, y=inertia, marker = 'o')\n",
    "plot.set_xlabel(\"Number of clusters\");\n",
    "plot.set_ylabel(\"Inertia\");"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** Where is the elbow in the plot?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"7.-Model-Results\"></a>\n",
    "### 7. Model Results"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 7.1 Silhouette Score\n",
    "\n",
    "Evaluate the silhouette score using the silhouette_score() function. Silhouette scores are used to study the distance between clusters.\n",
    "\n",
    "Then, compare the silhouette score of each value of k, from 2 through 10. To do this, write a function called kmeans_sil that takes in num_clusters and x_vals (X_scaled) and returns a list of each k-value's silhouette score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate silhouette score.\n",
    "# Write a function to return a list of each k-value's score.\n",
    "def kmeans_sil(num_clusters, x_vals):\n",
    "    \"\"\"\n",
    "    Accepts as arguments list of ints and data array. \n",
    "    Fits a KMeans model where k = each value in the list of ints.\n",
    "    Calculates a silhouette score for each k value. \n",
    "    Returns each k-value's silhouette score appended to a list.\n",
    "    \"\"\"\n",
    "    sil_score = []\n",
    "    for num in num_clusters:\n",
    "        kms = KMeans(n_clusters=num, random_state=42)\n",
    "        kms.fit(x_vals)\n",
    "        sil_score.append(silhouette_score(x_vals, kms.labels_))\n",
    "\n",
    "    return sil_score\n",
    "\n",
    "sil_score = kmeans_sil(num_clusters, X_scaled)\n",
    "sil_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a line plot that shows the relationship between num_clusters and sil_score\n",
    "plot = sns.lineplot(x=num_clusters, y=sil_score, marker = 'o')\n",
    "plot.set_xlabel(\"# of clusters\");\n",
    "plot.set_ylabel(\"Silhouette Score\");"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** What does the graph show?\n",
    "\n",
    "Silhouette scores near 1 indicate that samples are far away from neighboring clusters. Scores close to 0 indicate that samples are on or very close to the decision boundary between two neighboring clusters."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 7.2 Optimal k-value\n",
    "To decide on an optimal k-value, fit a n-cluster model to the dataset where n is the number produced by the silhouette score above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To decide on an optimal k-value, fit a n-cluster model to the dataset\n",
    "kmeans6 = KMeans(n_clusters= n, random_state=42)\n",
    "kmeans6.fit(X_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print unique labels.\n",
    "print('Unique labels:', np.unique(kmeans6.labels_))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, create a new column `cluster` that indicates cluster assignment in the DataFrame `data_subset`. It's important to understand the meaning of each cluster's labels, then decide whether the clustering makes sense. \n",
    "\n",
    "**Note:** This task is done using `data_subset` because it is often easier to interpret unscaled data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new column `cluster`.\n",
    "data_subset['cluster'] = kmeans6.labels_\n",
    "data_subset.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `groupby` to verify if any `'cluster'` can be differentiated by `'categorical_column'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify if any `cluster` can be differentiated by `categorical_column1`.\n",
    "data_subset.groupby(by=['cluster', 'categorical_column1']).size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Interpret the groupby outputs using a visualization\n",
    "data_subset.groupby(by=['cluster', 'categorical_column1']).size().plot.bar(title='Clusters differentiated by categorical_column1',\n",
    "                                                                   figsize=(6, 5),\n",
    "                                                                   ylabel='Size',\n",
    "                                                                   xlabel='(Cluster, categorical_column1)');"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `groupby` to verify if each `'cluster'` can be differentiated by `'categorical_column1'` AND `'categorical_column2'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify if each `cluster` can be differentiated by `categorical_column1` AND `categorical_column2`.\n",
    "data_subset.groupby(by=['cluster','categorical_column1', 'categorical_column2']).size().sort_values(ascending= False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** Are the clusters differentiated by `'categorical_column1'` and `'categorical_column2'`?\n",
    "\n",
    "The results of the above evaluation with `groupby` will indicate whether the algorithm clusters produced make sense and how well it performed \n",
    "\n",
    "Finally, interpret the groupby outputs and visualize these results. The graph below shows that each `'cluster'` can be differentiated by `'categorical_column1'` and `'categorical_column2'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_subset.groupby(by=['cluster','categorical_column1','categorical_column2']).size().unstack(level = 'categorical_column1', fill_value=0).plot.bar(title='Clusters differentiated by categorical_column1 and categorical_column2',\n",
    "                                                                                                                      figsize=(6, 5),\n",
    "                                                                                                                      ylabel='Size',\n",
    "                                                                                                                      xlabel='(Cluster, categorical_column2)')\n",
    "plt.legend(bbox_to_anchor=(1.3, 1.0))"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "provenance": [
    {
     "file_id": "1BgN3Lv1fx-AxyKSqB_2kM3dJ4mFBctv_",
     "timestamp": 1662734078308
    },
    {
     "file_id": "1ZYfhIvPRxnw7ghB_BsNQAMUorLXpAZs_",
     "timestamp": 1658889786811
    }
   ]
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}