04_shallow_regression/04_shallow_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Shallow regression for vector data\n",
    "\n",
    "Reminder: We are within supervised learning (we have labels/targets that are real values) -> Regression\n",
    "\n",
    "Data and goal: In this notebook we read the zip code data produced by **02_vector_preparations** and create different machine learning models for\n",
    "predicting the median income per zip-code area from population and spatial features. We will adjust parameters to improve the performance on a validation dataset and finally assesses the models error metrics with a test dataset.\n",
    "\n",
    "Notebook contents:\n",
    "\n",
    "0. Environment preparation\n",
    "1. Reading the data\n",
    "2. Function defintion\n",
    "3. Lasso regression\n",
    "4. Random Forest\n",
    "5. Task \n",
    "6. Model comparison\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Environment preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "import pandas as pd\n",
    "from math import sqrt\n",
    "# machine learning models\n",
    "from sklearn.linear_model import Lasso\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "# error metrics\n",
    "from sklearn.metrics import mean_squared_error, mean_absolute_error,r2_score\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Reading the data\n",
    "### 1.1 Define input and output file paths "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "username = os.environ.get('USER')\n",
    "base_directory= f'/scratch/project_2002044/{username}/2022/GeoML'\n",
    "\n",
    "# inputs, all datasets created in 02_vector_preparation.ipynb\n",
    "data_directory = os.path.join(base_directory,'data')\n",
    "preprocessed_data_directory = os.path.join(data_directory,'preprocessed_regression')\n",
    "scaled_train_dataset_name = os.path.join(preprocessed_data_directory,'scaled_train_zip_code_data.csv')\n",
    "scaled_test_dataset_name = os.path.join(preprocessed_data_directory,'scaled_test_zip_code_data.csv')\n",
    "scaled_val_dataset_name = os.path.join(preprocessed_data_directory,'scaled_val_zip_code_data.csv')\n",
    "train_label_name = os.path.join(preprocessed_data_directory,'train_income_labels.pkl')\n",
    "test_label_name = os.path.join(preprocessed_data_directory,'test_income_labels.pkl')\n",
    "val_label_name = os.path.join(preprocessed_data_directory,'val_income_labels.pkl')\n",
    "\n",
    "# read also unscaled datasets\n",
    "train_dataset_name = os.path.join(preprocessed_data_directory,'train_zip_code_data.csv')\n",
    "test_dataset_name = os.path.join(preprocessed_data_directory,'test_zip_code_data.csv')\n",
    "val_dataset_name = os.path.join(preprocessed_data_directory,'val_zip_code_data.csv')\n",
    "\n",
    "# outputs\n",
    "results_directory = os.path.join(data_directory,'regression_results')\n",
    "\n",
    "def create_dir(directory_name):\n",
    "    if not os.path.exists(directory_name):\n",
    "        os.makedirs(directory_name)\n",
    "create_dir(results_directory)\n",
    "\n",
    "metrics_filename = os.path.join(results_directory,'shallow_metrics.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for reproducible results when randomness is involved, we can set a random seed\n",
    "random_seed = 42"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Reading the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read train, validation and test datasets\n",
    "scaled_x_train = pd.read_csv(scaled_train_dataset_name)\n",
    "scaled_x_val = pd.read_csv(scaled_val_dataset_name)\n",
    "x_train = pd.read_csv(train_dataset_name)\n",
    "x_val = pd.read_csv(val_dataset_name)\n",
    "y_train = pd.read_pickle(train_label_name)\n",
    "y_val = pd.read_pickle(val_label_name)\n",
    "scaled_x_test = pd.read_csv(scaled_test_dataset_name)\n",
    "x_test = pd.read_csv(test_dataset_name)\n",
    "y_test = pd.read_pickle(test_label_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Function definitions\n",
    "\n",
    "For the regression exercises we will generate the following erroro metrics:\n",
    "Root Mean Square Error (RMSE), Mean Absolute Error (MAE), coefficient of determination (R^2).\n",
    "* RMSE and MAE: related to the income value, the smaller the better, unit is Euros;\n",
    "* RMSE: \"punishes\" larger errors more than smaller\n",
    "* MAE: more intuitive interpretation, \"all errors are equal\"\n",
    "* R^2: determines \"goodness of fit\", not in euros, 1 would be a perfect fit\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculating error metrics for regression predictions\n",
    "\n",
    "def calculate_error_metrics(test_labels,label_predictions, model_name):\n",
    "\n",
    "    #Asessing the performance of the model with root mean squared error, mean absolute error and coefficient of determination r2\n",
    "    rmse = sqrt(mean_squared_error(test_labels, label_predictions))\n",
    "    mae = mean_absolute_error(test_labels, label_predictions)\n",
    "    r2 = r2_score(test_labels, label_predictions)\n",
    "\n",
    "    # store them in a dictionary\n",
    "    metrics_dict = dict(zip(['model','RMSE','MAE','R2'],[model_name,rmse,mae,r2]))\n",
    "\n",
    "    return metrics_dict\n",
    "\n",
    "def print_error_metrics(metrics_dict, model_name, dataset_name):\n",
    "    print(f\"\\nError metrics for {model_name} on the {dataset_name} dataset: \\n\" +\n",
    "            f\"\\t Root mean squared error (RMSE): {round(metrics_dict['RMSE'])} \\n\" +\n",
    "            f\"\\t Mean absolute error (MAE): {round(metrics_dict['MAE'])} \\n\" +\n",
    "            f\"\\t Coefficient of determination (R2): {round(metrics_dict['R2'],4)} \\n\")\n",
    "\n",
    "def store_error_metrics(test_labels, label_predictions, model_name, metrics_collection = None):\n",
    "\n",
    "    metrics_dict = calculate_error_metrics(test_labels, label_predictions, model_name)\n",
    "\n",
    "    # in case these are the first results we want to store in that dataframe, we need to first create it\n",
    "    if metrics_collection is None:\n",
    "        metrics_collection = pd.DataFrame(columns=['model','RMSE','MAE','R2'])\n",
    " \n",
    "    metrics_collection = metrics_collection.append(metrics_dict, ignore_index=True )\n",
    "\n",
    "    #print_error_metrics(metrics_dict,model_name)\n",
    "\n",
    "    return metrics_collection\n",
    "\n",
    "# to time the model training we create a function for model training\n",
    "def train_model(x_train, y_train, model):\n",
    "    start_time = time.time()  \n",
    "    print(model)\n",
    "    model.fit(x_train,y_train)\n",
    "    print('Model training took: ', round((time.time() - start_time), 2), ' seconds')\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Lasso regression \n",
    "\n",
    "One of the simpler approaches for regression with few important features is lasso regression. Compared to any of the following models it is still easy to compute. Here, we use it with default parameters, but you can also check out others following the documentation:\n",
    "\n",
    "* https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lasso = Lasso()\n",
    "lasso_name = \"Lasso Regressor\"\n",
    "lasso.fit(scaled_x_train, y_train)\n",
    "lasso_predictions_val = lasso.predict(scaled_x_val)\n",
    "\n",
    "\n",
    "# then we can get some performance metrics\n",
    "print_error_metrics(calculate_error_metrics(y_val,lasso_predictions_val, lasso_name), lasso_name, 'validation')\n",
    "\n",
    "# and store the results on the test dataset for later model comparison\n",
    "lasso_predictions_test = lasso.predict(scaled_x_test)\n",
    "print_error_metrics(calculate_error_metrics(y_test,lasso_predictions_test, lasso_name), lasso_name, 'test')\n",
    "metrics_collection = store_error_metrics(y_test,lasso_predictions_test, lasso_name)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Random Forest Regressor\n",
    "\n",
    "* https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html\n",
    "* https://scikit-learn.org/stable/modules/ensemble.html#forest\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_forest = RandomForestRegressor(n_estimators=100, max_depth= None, max_features=None, random_state= random_seed, n_jobs=1,verbose=1,)\n",
    "random_forest_name = \"Random Forest Regressor\"\n",
    "\n",
    "random_forest = train_model(x_train, y_train,random_forest)\n",
    "random_forest_predictions = random_forest.predict(x_val)\n",
    "\n",
    "# then we can get some performance metrics\n",
    "print_error_metrics(calculate_error_metrics(y_val,random_forest_predictions, random_forest_name), random_forest_name, 'validation')\n",
    "\n",
    "# and store the results on the test dataset for later model comparison, after we are done optimizing the parameters\n",
    "random_forest_predictions = random_forest.predict(x_test)\n",
    "print_error_metrics(calculate_error_metrics(y_test,random_forest_predictions, random_forest_name), random_forest_name, 'test')\n",
    "metrics_collection = store_error_metrics(y_test,random_forest_predictions, random_forest_name, metrics_collection)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#let's take a look at feature importances\n",
    "# create dataframe of feature importance from random forest model\n",
    "adf = pd.DataFrame(zip(x_train.columns, random_forest.feature_importances_), columns= ['name','importance'])\n",
    "# sort the dataframe to have highest ranking features first\n",
    "adf.sort_values('importance', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Task\n",
    "\n",
    "Study the [scikit-learn documentation](https://scikit-learn.org/stable/supervised_learning.html#supervised-learning) to find other regression models and/or experiment with different hyperparameter values. Can you improve on the performance metrics or make the training faster?\n",
    "Report the best results (on test set, see below) and mark down model and parameters used, so that others can reproduce the results. \n",
    "\n",
    "You can also run below cells twice to check out some other ensemble regressors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load ada_boost.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load bagging.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load extra_trees.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load grad_boost.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Model comparison\n",
    "\n",
    "Let's compare our models performance on the test dataset. Make sure to store your results in metrics_collection (as done for random forest regressor) before running below cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(metrics_collection.sort_values(by=['RMSE'], ascending=False))\n",
    "\n",
    "# store comparison table \n",
    "metrics_collection.to_csv(metrics_filename)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}