7-EX-Rescaling-with-cross-validation-v1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creative Commons CC BY 4.0 Lynd Bacon & Associates, Ltd. Not warranted to be suitable for any particular purpose. (You're on your own!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Avoid the Embarrassment of Data Leakage: <br>Rescale or Transform Within CV Folds!\n",
    "\n",
    "A basic cross-validation, \"anti-data leakage\" notion in that test data fold are data that an algorithm hasn't yet \"seen\" when it is learning from a training data fold.   To be consistent with this idea, any rescaling or transformations that depends on the values of the data must be done separately for training data and for test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rescaling Transformations Within Folds\n",
    "\n",
    "In the following example we'll \"standardize\" the patient satisfaction data so that every predictor has mean=0 and SD=1.  In the \"olden days\" of data mining, doing this was sometimes referred to as \"sphering\" the data.  See:\n",
    "\n",
    "[scikit-Learn StandardScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler)\n",
    "\n",
    "There are several other ways of rescaling variables.  Another common method is \"MinMax,\" which rescales a feature's data to be within the range of the minimum and maximum values.\n",
    "\n",
    "[scikit-Learn MinMaxScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html#sklearn.preprocessing.MinMaxScaler)\n",
    "\n",
    "You can find other tools for rescaling at [scikit-learn preprocessing API](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get Some Essential Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn import linear_model  # OLS \n",
    "from sklearn.metrics import mean_squared_error, r2_score # Basic metrics\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn import preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get the PT Satisfaction Data\n",
    "\n",
    "Assuming that they are in the pwd:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['caseID', 'patSat', 'q2', 'q3', 'q4', 'q5', 'q6', 'q7', 'q8', 'q9',\n",
       "       'ptCat'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Input into a DataFrame, check the column names\n",
    "\n",
    "ptSatDF=pd.read_csv('DATA/ML/DECART-patSat.csv')\n",
    "ptSatDF.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dummy Code the pt Categories\n",
    "\n",
    "Just for consistency with things elsewhere, we'll dummy code the categories of `ptCat`, leaving out the first(0) category, medical admission.  (The regular one, not the \"highfalutin\" concierge type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Int64Index([1, 2], dtype='int64')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ptSatDF2=ptSatDF.drop('caseID',axis=1)  # Get rind of caseID\n",
    "ptCats=pd.get_dummies(ptSatDF2.ptCat,drop_first=True) # get 0/1 dummies, drop the 0 category\n",
    "ptCats.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['patSat', 'q2', 'q3', 'q4', 'q5', 'q6', 'q7', 'q8', 'q9', 'ptCat1',\n",
       "       'ptCat2'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ptSatDF3=ptSatDF2.drop('ptCat',axis=1)\n",
    "ptSatDF4=pd.concat([ptSatDF3,ptCats],axis=1,sort=False)\n",
    "ptSatDF4=ptSatDF4.rename(index=str,columns={1:'ptCat1',2:'ptCat2'})\n",
    "ptSatDF4.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1811, 11)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ptSatDF4.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# K-Fold CV with Separate X Train and Test Standardization Within Folds\n",
    "\n",
    "\n",
    "20 folds, using defaults for the `scikit-learn` StandardScaler() method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "kf=KFold(n_splits=20,random_state=99,shuffle=True)\n",
    "X=ptSatDF4.iloc[:,1:].to_numpy()\n",
    "y=ptSatDF4.iloc[:,0].to_numpy()\n",
    "\n",
    "cvres=[]  # Holder list for fold results\n",
    "\n",
    "regr=linear_model.LinearRegression() # define a reg model to use\n",
    "\n",
    "scaler=preprocessing.StandardScaler() # by default, mean=0, sd=1\n",
    "\n",
    "for traindx, testdx in kf.split(X):  # loop over folds\n",
    "    resDict={}                       # Dictionary to hold fold results\n",
    "    XTrainS=scaler.fit_transform(X[traindx])  # Xtrain rescaled\n",
    "    yTrain=y[traindx]\n",
    "    XTestS=scaler.fit_transform(X[testdx])    # Xtest rescaled\n",
    "    yTest=y[testdx]\n",
    "    regModel=regr.fit(XTrainS,yTrain) \n",
    "    trainPred=regModel.predict(XTrainS)\n",
    "    trainR2=r2_score(yTrain,trainPred)\n",
    "    trainMSE=mean_squared_error(yTrain,trainPred)\n",
    "    testPred=regModel.predict(XTestS)\n",
    "    testR2=r2_score(yTest,testPred)\n",
    "    testMSE=mean_squared_error(yTest,testPred)\n",
    "    resDict.update({'trainR2':trainR2,\n",
    "                    'testR2':testR2,\n",
    "                    'trainMSE':trainMSE,\n",
    "                    'testMSE':testMSE})\n",
    "    cvres.append(resDict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rearranging cols to make train vs test comparisons easier\n",
    "\n",
    "cvresDF=pd.DataFrame(cvres)[['trainMSE','testMSE','trainR2','testR2']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>trainMSE</th>\n",
       "      <th>testMSE</th>\n",
       "      <th>trainR2</th>\n",
       "      <th>testR2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>20.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>20.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.905219</td>\n",
       "      <td>2.053023</td>\n",
       "      <td>0.706933</td>\n",
       "      <td>0.675779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.017148</td>\n",
       "      <td>0.309234</td>\n",
       "      <td>0.002749</td>\n",
       "      <td>0.052867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.871845</td>\n",
       "      <td>1.542167</td>\n",
       "      <td>0.700704</td>\n",
       "      <td>0.561533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.894593</td>\n",
       "      <td>1.879348</td>\n",
       "      <td>0.705033</td>\n",
       "      <td>0.650602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.902233</td>\n",
       "      <td>2.088802</td>\n",
       "      <td>0.707270</td>\n",
       "      <td>0.668211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.923043</td>\n",
       "      <td>2.229367</td>\n",
       "      <td>0.708789</td>\n",
       "      <td>0.702608</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.929971</td>\n",
       "      <td>2.599047</td>\n",
       "      <td>0.711835</td>\n",
       "      <td>0.795454</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        trainMSE    testMSE    trainR2     testR2\n",
       "count  20.000000  20.000000  20.000000  20.000000\n",
       "mean    1.905219   2.053023   0.706933   0.675779\n",
       "std     0.017148   0.309234   0.002749   0.052867\n",
       "min     1.871845   1.542167   0.700704   0.561533\n",
       "25%     1.894593   1.879348   0.705033   0.650602\n",
       "50%     1.902233   2.088802   0.707270   0.668211\n",
       "75%     1.923043   2.229367   0.708789   0.702608\n",
       "max     1.929971   2.599047   0.711835   0.795454"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cvresDF.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A UDU: Radon Regression With MinMax Rescaling\n",
    "\n",
    "This can be done essentially like what's above. But instead of\n",
    "\n",
    "`scaler=preprocessing.StandardScaler()`\n",
    "\n",
    "use\n",
    "\n",
    "`scaler=preprocessing.MinMax()`\n",
    "\n",
    "Use the radon data, of course.  Don't forget that there's an observation with a missing value on `hhincome`. "
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": false,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}