Practical-06-3. Correlation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "## Considering Correlated Variables (a.k.a. Feature Selection)\n",
    "\n",
    "Depending on the clustering technique, correlated variables can have an unexpected effect on the results by allowing some dimensions to be 'double-weighted' in the results. So we don't want to keep too many correlated variables in the clustering data since that will bias the clustering algorithms and may result in poor 'performance'. \n",
    "\n",
    "<div style=\"padding:5px;margin-top:5px;margin-bottom:5px;border:dotted 1px red;background-color:rgb(255,233,233);color:red\">STOP. Think about _why_ correlation between two variables could lead to 'double-weighting in the clustering results!</div>\n",
    "\n",
    "One way to deal this is to produce a correlation table for all variables and then look to remove problematic variables. For a gentle introduction (that kinds of leaves you hanging at the end) there's a nice-looking blog post on [Medium](https://medium.com/towards-artificial-intelligence/feature-selection-and-dimensionality-reduction-using-covariance-matrix-plot-b4c7498abd07): \n",
    "\n",
    "> Feature selection and dimensionality reduction are important because of three main reasons:\n",
    "> - Prevents Overfitting: A high-dimensional dataset having too many features can sometimes lead to overfitting (model captures both real and random effects).\n",
    "> - Simplicity: An over-complex model having too many features can be hard to interpret especially when features are correlated with each other.\n",
    "> - Computational Efficiency: A model trained on a lower-dimensional dataset is computationally efficient (execution of algorithm requires less computational time).\n",
    "> Dimensionality reduction, therefore, plays a crucial role in data preprocessing.\n",
    "\n",
    "There's also [this post](https://towardsdatascience.com/feature-selection-correlation-and-p-value-da8921bfb3cf) and [this one](https://towardsdatascience.com/why-feature-correlation-matters-a-lot-847e8ba439c4). We could also use Principal Components Analysis (PCA) to perform dimensionality reduction whilst also dealing with correlation between the variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here's an output table which gives you nice, specific \n",
    "# numbers but is hard to read so I'm only showing the \n",
    "# first ten rows and columns... \n",
    "scdf.corr().iloc[1:7,1:7]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding Strong Correlations Visually"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# And here's a correlation heatmap... which is easier to read but has\n",
    "# less detail. What it *does* highlight is high levels of *negative*\n",
    "# correlation as well as positive, so you'll need absolute difference, \n",
    "# not just whether something is more than 0.x correlated.\n",
    "# \n",
    "# From https://seaborn.pydata.org/examples/many_pairwise_correlations.html\n",
    "cdf = scdf.corr()\n",
    "\n",
    "# Generate a mask for the upper triangle\n",
    "mask = np.zeros_like(cdf, dtype=np.bool)\n",
    "mask[np.triu_indices_from(mask)] = True\n",
    "\n",
    "# Set up the matplotlib figure\n",
    "f, ax = plt.subplots(figsize=(10, 10))\n",
    "\n",
    "# Generate a custom diverging colormap\n",
    "cm = sns.diverging_palette(240, 10, as_cmap=True)\n",
    "\n",
    "# Draw the heatmap with the mask and correct aspect ratio\n",
    "sns.heatmap(cdf, mask=mask, cmap=cm, vmax=1.0, vmin=-1.0, center=0,\n",
    "            square=True, linewidths=.1, cbar_kws={\"shrink\": .5})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"padding:5px;margin-top:5px;margin-bottom:5px;border:dotted 1px red;background-color:rgb(255,233,233);color:red\">STOP. Make sure that you understand what the figure above is showing before proceeding to the next stage.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding Strong Correlations Numerically"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate the matrix but capture the output this time\n",
    "cdf = scdf.corr()\n",
    "cdf['name'] = cdf.index # We need a copy of the index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corrh = 0.66 # Specify threshold for highly correlated?\n",
    "print(\"! High correlation threshold is {0}.\".format(corrh))\n",
    "\n",
    "num_corrs = []\n",
    "hi_corrs  = []\n",
    "\n",
    "for c in cdf.name.unique():\n",
    "    if c != 'name':\n",
    "        # Some formatting\n",
    "        print(\"=\" * 10 + f\" {c} \" + \"=\" * 10)\n",
    "        \n",
    "        # Find highly correlated variables\n",
    "        hits = cdf.loc[(abs(cdf[c]) >= corrh), c]\n",
    "        hits.drop(c, inplace=True)\n",
    "        \n",
    "        if hits.size == 0: # No correlations > corrs\n",
    "            print(\"+ Not highly correlated with other variables.\")\n",
    "        else:\n",
    "            num_corrs.append(hits.size)\n",
    "            \n",
    "            print(\"- High correlations ({0}) with other variables:\".format(hits.size))\n",
    "            print(\"    \" + \"\\n    \".join(hits.index.values))\n",
    "            hi_corrs.append(hits.size)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.distplot(hi_corrs, bins=range(0,20), kde=False).set_title(\n",
    "    \"Number of Strong Correlations (> \" + str(corrh) + \")  with Other Variables\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stripping Out 'Redundant' Variables\n",
    "\n",
    "Let's remove any variable that has a '*lot*' of strong correlations correlations with other variables, though we need to define what is 'a lot'. This will reduce the dimensionality of our data and make clustering a bit easier. An alternative approach to dimensionality reduction -- which can be more 'robust' if we ensure that all of the data has unit variance (which we've done using the MinMaxScaler), though harder for many to understand -- would be to apply Principal Components Analysis (PCA) to the data set and to work with the eigenvalues afterwards. PCA is also available in `sklearn`.\n",
    "\n",
    "We'll set our threshold at 5.0 based on a visual inspection of the chart above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corrh     = 0.66 # Specify threshold for highly correlated?\n",
    "maxcorrs  = 4.0 # What's our threshold for too many strong correlations?\n",
    "threshold = 0.5*maxcorrs # What's our threshold for too many strong correlations with columns we keep!\n",
    "\n",
    "print(\"! High correlation threshold is {0}.\".format(corrh))\n",
    "\n",
    "to_drop = [] # Columns to drop\n",
    "to_keep = [] # Columns to keep\n",
    "\n",
    "num_corrs = []\n",
    "hi_corrs  = []\n",
    "\n",
    "for c in cdf.columns:\n",
    "    if c != 'name':\n",
    "        \n",
    "        # Find highly correlated variables, but let's\n",
    "        # keep the focus on *positive* correlation now\n",
    "        hits = cdf.loc[(cdf[c] >= corrh), c]\n",
    "        hits.drop(c, inplace=True)\n",
    "        \n",
    "        multi_vals = False\n",
    "        \n",
    "        # Remove ones with many correlations\n",
    "        if hits.size >= maxcorrs: \n",
    "            print(f\"- {c} exceeds maxcorr ({maxcorrs}) correlation threshold (by {hits.size-threshold}).\")\n",
    "            s1 = set(to_keep)\n",
    "            s2 = set(hits.index.values)\n",
    "            #print(\"Comparing to_keep (\" + \", \".join(s1) + \") to hits (\" + \", \".join(s2) + \")\")\n",
    "            s1 &= s2\n",
    "            #print(\"Column found in 'many correlations' :\" + str(s1))\n",
    "            if len(s1) >= threshold: \n",
    "                multi_vals = True\n",
    "                print(f\"    - Dropping b/c exceed {threshold} correlations with retained cols: \\n        -\" + \"\\n        -\".join(s1))\n",
    "            else:\n",
    "                print(f\"    + Keeping b/c fewer than {threshold} correlations with retained columns.\")\n",
    "        else: \n",
    "            print(f\"+ {c} falls below maxcorr ({maxcorrs}) correlation threshold (by {abs(threshold-hits.size)}).\")\n",
    "            \n",
    "        if multi_vals==True:\n",
    "            to_drop.append(c)\n",
    "        else:\n",
    "            to_keep.append(c)\n",
    "        \n",
    "\n",
    "print(\" \")\n",
    "print(\"To drop ({0}): \".format(len(to_drop)) + \", \".join(to_drop))\n",
    "print(\" \")\n",
    "print(\"To keep ({0}): \".format(len(to_keep)) + \", \".join(to_keep))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_save = scdf.drop(to_drop, axis=1, errors='raise')\n",
    "print(\"Retained variables: \" + \", \".join(to_save.columns.values))\n",
    "to_save.to_pickle(os.path.join('data','LSOA_2Cluster.pickle'))\n",
    "del(to_save)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "GSA2019",
   "language": "python",
   "name": "gsa2019"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}