t81_558_class09_regularization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 9: Regularization: L1, L2 and Dropout**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), School of Engineering and Applied Science, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module Video Material\n",
    "\n",
    "Main video lecture:\n",
    "\n",
    "* [Part 9.1: L1 and L2 Regularization for Linear Regression](https://www.youtube.com/watch?v=xyymDGReKdY&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=26)\n",
    "* [Part 9.2: L1 and L2 Regularization for Keras](https://www.youtube.com/watch?v=CEFcwpBneFo&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=27)\n",
    "* [Part 9.3: Drop out Regularization for Keras](https://www.youtube.com/watch?v=JPqwyuK7bPg&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=28)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Helpful Functions\n",
    "\n",
    "You will see these at the top of every module.  These are simply a set of reusable functions that we will make use of.  Each of them will be explained as the semester progresses.  They are explained in greater detail as the course progresses.  Class 4 contains a complete overview of these functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import base64\n",
    "import os\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import requests\n",
    "from sklearn import preprocessing\n",
    "\n",
    "\n",
    "# Encode text values to dummy variables(i.e. [1,0,0],[0,1,0],[0,0,1] for red,green,blue)\n",
    "def encode_text_dummy(df, name):\n",
    "    dummies = pd.get_dummies(df[name])\n",
    "    for x in dummies.columns:\n",
    "        dummy_name = f\"{name}-{x}\"\n",
    "        df[dummy_name] = dummies[x]\n",
    "    df.drop(name, axis=1, inplace=True)\n",
    "\n",
    "\n",
    "# Encode text values to a single dummy variable.  The new columns (which do not replace the old) will have a 1\n",
    "# at every location where the original column (name) matches each of the target_values.  One column is added for\n",
    "# each target value.\n",
    "def encode_text_single_dummy(df, name, target_values):\n",
    "    for tv in target_values:\n",
    "        l = list(df[name].astype(str))\n",
    "        l = [1 if str(x) == str(tv) else 0 for x in l]\n",
    "        name2 = f\"{name}-{tv}\"\n",
    "        df[name2] = l\n",
    "\n",
    "\n",
    "# Encode text values to indexes(i.e. [1],[2],[3] for red,green,blue).\n",
    "def encode_text_index(df, name):\n",
    "    le = preprocessing.LabelEncoder()\n",
    "    df[name] = le.fit_transform(df[name])\n",
    "    return le.classes_\n",
    "\n",
    "\n",
    "# Encode a numeric column as zscores\n",
    "def encode_numeric_zscore(df, name, mean=None, sd=None):\n",
    "    if mean is None:\n",
    "        mean = df[name].mean()\n",
    "\n",
    "    if sd is None:\n",
    "        sd = df[name].std()\n",
    "\n",
    "    df[name] = (df[name] - mean) / sd\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the median\n",
    "def missing_median(df, name):\n",
    "    med = df[name].median()\n",
    "    df[name] = df[name].fillna(med)\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the default\n",
    "def missing_default(df, name, default_value):\n",
    "    df[name] = df[name].fillna(default_value)\n",
    "\n",
    "\n",
    "# Convert a Pandas dataframe to the x,y inputs that TensorFlow needs\n",
    "def to_xy(df, target):\n",
    "    result = []\n",
    "    for x in df.columns:\n",
    "        if x != target:\n",
    "            result.append(x)\n",
    "    # find out the type of the target column.  Is it really this hard? :(\n",
    "    target_type = df[target].dtypes\n",
    "    target_type = target_type[0] if hasattr(\n",
    "        target_type, '__iter__') else target_type\n",
    "    # Encode to int for classification, float otherwise. TensorFlow likes 32 bits.\n",
    "    if target_type in (np.int64, np.int32):\n",
    "        # Classification\n",
    "        dummies = pd.get_dummies(df[target])\n",
    "        return df[result].values.astype(np.float32), dummies.values.astype(np.float32)\n",
    "    # Regression\n",
    "    return df[result].values.astype(np.float32), df[[target]].values.astype(np.float32)\n",
    "\n",
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return f\"{h}:{m:>02}:{s:>05.2f}\"\n",
    "\n",
    "\n",
    "# Regression chart.\n",
    "def chart_regression(pred, y, sort=True):\n",
    "    t = pd.DataFrame({'pred': pred, 'y': y.flatten()})\n",
    "    if sort:\n",
    "        t.sort_values(by=['y'], inplace=True)\n",
    "    plt.plot(t['y'].tolist(), label='expected')\n",
    "    plt.plot(t['pred'].tolist(), label='prediction')\n",
    "    plt.ylabel('output')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "# Remove all rows where the specified column is +/- sd standard deviations\n",
    "def remove_outliers(df, name, sd):\n",
    "    drop_rows = df.index[(np.abs(df[name] - df[name].mean())\n",
    "                          >= (sd * df[name].std()))]\n",
    "    df.drop(drop_rows, axis=0, inplace=True)\n",
    "\n",
    "\n",
    "# Encode a column to a range between normalized_low and normalized_high.\n",
    "def encode_numeric_range(df, name, normalized_low=-1, normalized_high=1,\n",
    "                         data_low=None, data_high=None):\n",
    "    if data_low is None:\n",
    "        data_low = min(df[name])\n",
    "        data_high = max(df[name])\n",
    "\n",
    "    df[name] = ((df[name] - data_low) / (data_high - data_low)) \\\n",
    "        * (normalized_high - normalized_low) + normalized_low\n",
    "\n",
    "\n",
    "# This function submits an assignment.  You can submit an assignment as much as you like, only the final\n",
    "# submission counts.  The paramaters are as follows:\n",
    "# data - Pandas dataframe output.\n",
    "# key - Your student key that was emailed to you.\n",
    "# no - The assignment class number, should be 1 through 1.\n",
    "# source_file - The full path to your Python or IPYNB file.  This must have \"_class1\" as part of its name.  \n",
    "# .             The number must match your assignment number.  For example \"_class2\" for class assignment #2.\n",
    "def submit(data,key,no,source_file=None):\n",
    "    if source_file is None and '__file__' not in globals(): raise Exception('Must specify a filename when a Jupyter notebook.')\n",
    "    if source_file is None: source_file = __file__\n",
    "    suffix = '_class{}'.format(no)\n",
    "    if suffix not in source_file: raise Exception('{} must be part of the filename.'.format(suffix))\n",
    "    with open(source_file, \"rb\") as image_file:\n",
    "        encoded_python = base64.b64encode(image_file.read()).decode('ascii')\n",
    "    ext = os.path.splitext(source_file)[-1].lower()\n",
    "    if ext not in ['.ipynb','.py']: raise Exception(\"Source file is {} must be .py or .ipynb\".format(ext))\n",
    "    r = requests.post(\"https://api.heatonresearch.com/assignment-submit\",\n",
    "        headers={'x-api-key':key}, json={'csv':base64.b64encode(data.to_csv(index=False).encode('ascii')).decode(\"ascii\"),\n",
    "        'assignment': no, 'ext':ext, 'py':encoded_python})\n",
    "    if r.status_code == 200:\n",
    "        print(\"Success: {}\".format(r.text))\n",
    "    else: print(\"Failure: {}\".format(r.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Regularization\n",
    "\n",
    "Regularization is a technique that reduces overfitting, which occurs when neural networks attempt to memorize training data, rather than learn from it.  Humans are capable of overfitting as well.  Before we examine the ways that a machine accidentally overfits, we will first explore how humans can suffer from it.\n",
    "\n",
    "Human programmers often take certification exams to show their competence in a given programming language.  To help prepare for these exams, the test makers often make practice exams available.  Consider a programmer who enters a loop of taking the practice exam, studying more, and then taking the practice exam again.  At some point, the programmer has memorized much of the practice exam, rather than learning the techniques necessary to figure out the individual questions.  The programmer has now overfit to the practice exam.  When this programmer takes the real exam, his actual score will likely be lower than what he earned on the practice exam.\n",
    " \n",
    " A computer can overfit as well.  Although a neural network received a high score on its training data, this result does not mean that the same neural network will score high on data that was not inside the training set.  Regularization is one of the techniques that can prevent overfitting. A number of different regularization techniques exist.  Most work by analyzing and potentially modifying the weights of a neural network as it trains.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Data\n",
    "\n",
    "We are going to look at linear regression to see how L1 and L2 regularization work.  The following code sets up the auto-mpg data for this purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LassoCV\n",
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import train_test_split \n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "\n",
    "# create feature vector\n",
    "missing_median(df, 'horsepower')\n",
    "df.drop('name',1,inplace=True)\n",
    "# Do not need zscore standardization for linear regression\n",
    "#encode_numeric_zscore(df, 'horsepower')\n",
    "#encode_numeric_zscore(df, 'weight')\n",
    "#encode_numeric_zscore(df, 'cylinders')\n",
    "#encode_numeric_zscore(df, 'displacement')\n",
    "#encode_numeric_zscore(df, 'acceleration')\n",
    "encode_text_dummy(df, 'origin')\n",
    "\n",
    "# Encode to a 2D matrix for training\n",
    "x,y = to_xy(df,'mpg')\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=45) \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simple function to evaluate the coefficients of a regression\n",
    "%matplotlib inline    \n",
    "from IPython.display import display, HTML    \n",
    "\n",
    "def report_coef(names,coef,intercept):\n",
    "    r = pd.DataFrame( { 'coef': coef, 'positive': coef>=0  }, index = names )\n",
    "    r = r.sort_values(by=['coef'])\n",
    "    display(r)\n",
    "    print(\"Intercept: {}\".format(intercept))\n",
    "    r['coef'].plot(kind='barh', color=r['positive'].map({True: 'b', False: 'r'}))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression\n",
    "\n",
    "To understand L1/L2 regularization, it is good to start with linear regression.  L1/L2 were first introduced for [linear regression](https://en.wikipedia.org/wiki/Linear_regression).  They can also be used for neural networks.  To fully understand L1/L2 we will begin with how they are used with linear regression.\n",
    "\n",
    "The following code uses linear regression to fit the auto-mpg data set.  The RMSE reported will not be as good as a neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 2.9371564388275146\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>positive</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>origin-1</th>\n",
       "      <td>-1.448373</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cylinders</th>\n",
       "      <td>-0.416550</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>horsepower</th>\n",
       "      <td>-0.007785</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>-0.007473</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>displacement</th>\n",
       "      <td>0.024456</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acceleration</th>\n",
       "      <td>0.138122</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin-3</th>\n",
       "      <td>0.629809</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <td>0.801274</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin-2</th>\n",
       "      <td>0.818564</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  coef  positive\n",
       "origin-1     -1.448373     False\n",
       "cylinders    -0.416550     False\n",
       "horsepower   -0.007785     False\n",
       "weight       -0.007473     False\n",
       "displacement  0.024456      True\n",
       "acceleration  0.138122      True\n",
       "origin-3      0.629809      True\n",
       "year          0.801274      True\n",
       "origin-2      0.818564      True"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept: [-18.257513]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAasAAAD8CAYAAADJ7YuWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAGy9JREFUeJzt3Xm0nXV97/H3h6EiBBkkUkTikUFzQSUlGwQZipqLQxGDwEK0Vsaoq47ccJddCi1U6tRb9dqLNEYN3GvRokGQUpk0BAKRnBMyIeLA0MpCPShSgmUwfO4f+xeyOZ5h50z7d3Y+r7XO2s9+nt/ze777OWflk9+zn0G2iYiIqNlWnS4gIiJiJAmriIioXsIqIiKql7CKiIjqJawiIqJ6CauIiKhewioiIqqXsIqIiOolrCIionrbdLqAbrHbbru5p6en02VEREwZfX19D9me3k7bhNU46enpobe3t9NlRERMGZLub7dtDgNGRET1ElYREVG9hFVERFQvYRUREdXLCRYRUQWp0xXEaEzWIxEzsoqIiOolrCIionpTNqwkXSNp5xHaXCBpzmb2+zVJd0taJ+krkrYdW6URETFWUy6s1LSV7TfZ/u1wbW2fZ/uGzdzE14CZwCuA5wJnjrLUiIgYJ1WGlaSzy8hmnaQPSeopo51LgXXAXpLuk7RbaX9uWX6LpMskzS/zF0k6sUzfJ+l8SSslrZU0c7Bt277GBXA78KLJ+dQRETGU6sJK0mzgNOBVwKHAWcAuwH7ARbYPsH1/S/uDgROAA4E3Ao1hun/I9kHAF4H5I9SxLfBO4LvDtJknqVdSb39/fzsfLyIiRqG6sAKOAK6w/Zjt9cBi4EjgftvLB2l/OHCl7cdtPwp8Z5i+F5fXPqBnhDouApbavnmoBrYX2G7Ybkyf3ta9GCMiYhRqDKuhPDYOfTxRXjdQrjGTdK2kVZIWbmwk6a+B6cDZ47DNiIgYoxrD6mZgrqTtJe0AHF/mDWUZ8GZJ20maBhy7ORuz/Xrbs2yfCSDpTOD1wCm2nx7dR4iIiPFU3R0sbK+UtIjmyQ0AC4GHh2m/QtJVwBrgl8Ba4JExlHAxcD9wm5qX1C+2fcEY+ouIiDGSJ+teGRNI0jTb6yVtDywF5tleOZk1NBoN53lWEaOX2y1NTWOJEEl9toc7Ke4Z1Y2sRmmBpP2B7YBLJjuoImLsuuD/zTGBuiKsbL+90zVERMTEqfEEi4iIiGdJWEVERPUSVhERUb2EVUREVC9hFRER1UtYRURE9RJWERFRvYRVRERUL2EVERHVS1hFRET1ElYREVG9rrg3YERMfbnr+tQy2TcezsgqIiKql7CKiIjqJazaJGnrTtcQEbGl6sqwknSBpA+1vL9Q0gclnSNphaQ1ks5vWf5tSX2S7pQ0r2X+ekn/S9Jq4LBJ/hgREVF0ZVgBXwH+AkDSVsDbgF8A+wGHALOA2ZKOKu1Ptz0baAAfkPT8Mn8H4Ae2D7R9y2R+gIiI2KQrzwa0fZ+kX0v6E2B34A7gYOCYMg0wjWZ4LaUZUMeX+XuV+b8GNgDfGmo7ZRQ2D2DGjBkT8EkiIgK6NKyKhcCpwB/THGm9DviE7X9qbSTpaGAOcJjt30laAmxXFj9ue8NQG7C9AFgA0Gg0JvlEzoiILUe3HgYEuAJ4A80R1bXl53RJ0wAk7SnpBcBOwMMlqGYCh3aq4IiIGFzXjqxsPynp+8Bvy+joOkn/DbhNzasP1wN/DnwXeI+ku4C7geWdqjkiIgbXtWFVTqw4FDhp4zzbnwc+P0jzNw7Wh+1pE1NdRERsjq48DChpf+CnwI22f9LpeiIiYmy6cmRl+4fA3p2uIyLaN9n3mouppStHVhER0V0SVhERUb2EVUREVC9hFRER1UtYRURE9RJWERFRvYRVRERUL2EVERHVS1hFRET1ElYREVG9hFVERFSvK+8NGBFTT/PJPTGeuul+ixlZRURE9aZ0WEm6RtLOI7S5QNKczez3y5JWS1oj6Zsbny4cERGdMSXDSk1b2X6T7d8O19b2ebZv2MxNfNj2gbZfCfw78L5RFxsREWNWbVhJOlvSuvLzIUk9ku6WdCmwDthL0n2Sdivtzy3Lb5F0maT5Zf4iSSeW6fsknS9ppaS1kmYOtm3b/1naC3gu0EVHfiMipp4qw0rSbOA04FU0H01/FrALsB9wke0DbN/f0v5g4ATgQJqPqG8M0/1Dtg8CvgjMH6aGrwK/AGYCXxjTB4qIiDGpMqyAI4ArbD9mez2wGDgSuN/28kHaHw5caftx248C3xmm78XltQ/oGaqR7dOAFwJ3AScP1kbSPEm9knr7+/tH+kwRETFKtYbVUB4bhz6eKK8bKKfuS7pW0ipJC1sb2t4AfJ3mqO0P2F5gu2G7MX369HEoLSIiBlNrWN0MzJW0vaQdgOPLvKEsA94sabty5t6xm7Mx26+3Pcv2meXkjX3hme+sjgN+NLqPERER46HKi4Jtr5S0CLi9zFoIPDxM+xWSrgLWAL8E1gKPjHLzAi6R9LwyvRp47yj7ioiIcSB3ySXOkqbZXi9pe2ApMM/2ysnafqPRcG9v72RtLqLr5A4W46/2f94l9dke7oS4Z1Q5shqlBZL2B7YDLpnMoIqIiInVNWFl++2driEiRq/2UUB0Vq0nWERERDwjYRUREdVLWEVERPUSVhERUb2EVUREVC9hFRER1UtYRURE9RJWERFRvYRVRERUL2EVERHVS1hFRET1uubegBExtY3nXddzn8Huk5FVRERUr8qwkrREUlvPOGmjr7nl0SEb318gac549B0REZOjyrDaXJK2HmbxXOCZsLJ9nu0bJr6qiIgYL2MKK0nfltQn6U5J88q8N0haKWm1pBvLvGmSvippraQ1kk4o84+RdFtpf7mkaYNsY9A2ku6T9ClJK4GTJJ0laUXZ7rckbS/p1cBxwGckrZK0j6RFkk4sfbxO0h2lrq9Iek5L3+eXba6VNHMs+ykiIsZmrCOr023PBhrAByTtDnwJOMH2gcBJpd25wCO2X2H7lcD3JO0GfAyYY/sgoBc4u7XzNtr82vZBtr8OLLZ9cNnuXcAZtm8FrgLOsT3L9s9a+t4OWAScbPsVNE82eW9L3w+VbX4RmD/G/RQREWMw1rMBPyDp+DK9FzAPWGr7XgDbvynL5gBv27iS7YclHUvz8NwyNU8D+iPgtgH9HzpCm2+0TL9c0seBnYFpwLUj1P4y4F7bPy7vLwH+Evhceb+4vPYBbx2sgzKanAcwY8aMETYXERGjNeqwknQ0zRA6zPbvJC0BVgHtHjITcL3tU8bQ5rGW6UXAXNurJZ0KHN1mHUN5orxuYIj9ZHsBsACg0WjkZNmIiAkylsOAOwEPl6CaSXMUtB1wlKSXAEjatbS9nuaohTJ/F2A5cLikfcu8HSS9dMA22mmz0Y7Ag5K2Bd7RMv/Rsmygu4GejX0D7wRuauNzR0TEJBtLWH0X2EbSXcAnaQZLP83DYoslrWbTYbqPA7tIWlfmv8Z2P3AqcJmkNTQP7z1rVNZOmxbnAj8AlgE/apn/deCcciLFPi19Pw6cBlwuaS3wNHDxaHZERERMLDmXeo+LRqPh3t7eTpcRMWXlDhZbHkl9ttu6prYrrrOKiIjulnsDRkQVMhqK4WRkFRER1UtYRURE9RJWERFRvYRVRERUL2EVERHVS1hFRET1ElYREVG9hFVERFQvYRUREdVLWEVERPUSVhERUb3cGzAiqjDwruu5V2C0ysgqIiKqt9kjK0l/A6wHngcstX3DZq5/NDDf9rGbu+3JJmku8GPbP+x0LRERW7JRj6xsn7e5QTUFzQX273QRERFburbCStJHJf1Y0i3Ay8q8RZJOLNOflPRDSWsk/X3L8osl9ZZ1/2AkJekQSbeVR87fKmlj31tL+ntJ60qf7y/zZ0u6SVKfpGsl7VHmL5H02bKtuyQdLGmxpJ9I+njL9v5c0u2SVkn6J0lbl/nrJV0oabWk5ZJ2l/Rq4DjgM6X9PmPYzxERMQYjHgaUNBt4GzCrtF8J9LUsfz5wPDDTtiXt3LJ6D3AIsA/wfUn7Duj+R8CRtn8vaQ7wd8AJwLyy7qyybFdJ2wJfAN5iu1/SycCFwOmlrydtNyR9ELgSmA38BviZpM8CLwBOBg63/ZSki4B3AJcCOwDLbX9U0qeBs2x/XNJVwNW2vznSfoqIiInTzndWRwJX2P4dQPkHvNUjwOPAlyVdDVzdsuxfbD8N/ETSPcDMAevuBFwiaT/AwLZl/hzgYtu/B7D9G0kvB14OXK/maUNbAw+29LWxrrXAnbYfLPXeA+wFHEEzwFaU9Z8L/Kqs82RL3X3Af29jvyBpHs1gZcaMGe2sEhERozDmU9fLyOcQ4HXAicD7gNduXDyw+YD3fwt83/bxknqAJcNsSjRD6LAhlj9RXp9umd74fpuy/iW2/2qQdZ+ynzlRdgNt7hfbC4AFAI1GIyfaRkRMkHa+s1oKzJX0XEk7Am9uXShpGrCT7WuADwMHtiw+SdJW5fuevYG7B/S9E/BAmT61Zf71wLslbVO2sWtZd7qkw8q8bSUd0Eb9G90InCjpBRv7lPTiEdZ5FNhxM7YRERETYMSwsr0S+AawGvg3YMWAJjsCV0taA9wCnN2y7N+B28t677H9+IB1Pw18QtIdPHs0s7Csu0bSauDttp+kOXL7VJm3Cnh1W5+y+Tl+CHwMuK7Uej2wxwirfR04p5wAkhMsIiI6RJ6gy8QlLWILOjmh0Wi4t7e302VETFm5g8WWR1Kf7UY7bXMHi4iIqN6E3RvQ9qkT1XdEdJ+MpGI4GVlFRET1ElYREVG9hFVERFQvYRUREdVLWEVERPUSVhERUb2EVUREVC9hFRER1UtYRURE9RJWERFRvYRVRERUL2EVEZ038JbrEQMkrCIionoJq4iIqF5Xh5WkhZL2H6HNIkknDjK/R9LbJ666iIhoV1eHle0zy+PsR6MHSFhFRFRgSoSVpHMkfaBMf1bS98r0ayV9TdIxkm6TtFLS5ZKmleVLJDXK9BmSfizpdklfkvSPLZs4StKtku5pGWV9EjhS0ipJH57EjxsREQNMibACbgaOLNMNYJqkbcu8NcDHgDm2DwJ6gbNbV5b0QuBc4FDgcGDmgP73AI4AjqUZUgAfAW62Pcv2ZwcrStI8Sb2Sevv7+8f4ESMiYihTJaz6gNmSngc8AdxGM7SOBP4L2B9YJmkV8C7gxQPWPwS4yfZvbD8FXD5g+bdtP10OGe7eblG2F9hu2G5Mnz59VB8sIiJGtk2nC2iH7ack3QucCtxKczT1GmBf4F7getunjGETT7RM54KPiIjKTJWRFTQPBc4Hlpbp9wB3AMuBwyXtCyBpB0kvHbDuCuBPJe0iaRvghDa29yiw43gVHxERozfVwmoP4DbbvwQep/mdUj/NEddlktbQPET4rO+kbD8A/B1wO7AMuA94ZITtrQE2SFqdEywiIjpLtjtdw6SQNM32+jKyugL4iu0rxqv/RqPh3t7e8eouYssiwRbyb1FsIqnPdqOdtlNpZDVWf1NOwFhH83uub3e4nojYKEEVI5gSJ1iMB9vzO11DRESMzpY0soqIiCkqYRUREdVLWEVERPUSVhERUb2EVUREVC9hFRER1UtYRURE9RJWERFRvYRVRERUL2EVERHVS1hFROcpj5GL4SWsIiKiegmriIio3riElaQeSevGo6+IiIiBOj6yKg9DrN5UqTMiohuNZ1htLelLku6UdJ2k50qaJWm5pDWSrpC0C4CkJZI+J6kX+KCkkyStK4+QX1rabC3pM5JWlPXfXeYfLWmppH+VdLekiyVtVZadImlt6etTZd5Jkv6hTH9Q0j1lem9Jy8r0bEk3SeqTdK2kPQarcxz3VUREbIbxHC3sB5xi+yxJ/wKcAPxP4P22b5J0AfDXwIdK+z/a+DhjSWuB19t+QNLOZfkZwCO2D5b0HGCZpOvKskOA/YH7ge8Cb5V0K/ApYDbwMHCdpLnAzaUOgCOBX0vas0wvlbQt8AXgLbb7JZ0MXAicPrDOgSTNA+YBzJgxY7T7LSIiRjCeYXWv7VVlug/YB9jZ9k1l3iXA5S3tv9EyvQxYVEJucZl3DPBKSSeW9zvRDMQngdttbxwhXQYcATwFLLHdX+Z/DTjK9rclTZO0I7AX8M/AUTTDajHwMuDlwPVqnj67NfDgEHU+i+0FwAKARqOR53JHREyQ8QyrJ1qmNwA7D9WweGzjhO33SHoV8GdAn6TZgGiOyq5tXUnS0cDAYBgpKG4FTgPupjnSOh04DPgfwAzgTtuHjVRnRER0xkSeYPEI8LCkI8v7dwI3DdZQ0j62f2D7PKCf5gjoWuC95TAdkl4qaYeyyiGSXlK+qzoZuAW4HfhTSbtJ2ho4pWV7NwPzgaXAHcBrgCdsP0IzwKZLOqxsZ1tJB4zfboiIiLGa6DPc3gVcLGl74B6ao5vBfEbSfjRHUzcCq4E1QA+wUs3jc/3A3NJ+BfCPwL7A94ErbD8t6SPlvYB/tX1laX8zzQBcanuDpP8AfgRg+8lyqPF/S9qJ5j75HHDnOO2DiIgYI9lT66uWchhwvu1jO11Lq0aj4d7e3k6XETE1STDF/i2KsZPUN9QJbAN1/DqriIgEVYxkyl3oansJsKTDZURExCTKyCoiIqqXsIqIiOolrCIionoJq4iIqF7CKiIiqpewioiI6iWsIiKiegmriIioXsIqIiKql7CKiIjqJawiIqJ6U+7egBFbrOaTrLtXbmYbw8jIKiIiqjclwkrSovKARCQtlLT/Zq6/fmIqi4iIyTDlDgPaPnMi+y9PJZbtpydyOxER0b6Ojqwk/YWkNZJWS7pC0r2Sti3Lntf6vmWdJZIaZXq9pAvL+ssl7V7mv0TSbZLWSvr4gPXPkbSibPf8Mq9H0t2SLgXWAXuV0dy60seHJ2N/RETE4DoWVpIOAD4GvNb2gcAZNB+q+GelyduAxbafGqabHYDlZf2lwFll/ueBL9p+BfBgyzaPAfYDDgFmAbMlHVUW7wdcZPsAYDdgT9svL318dayfNyIiRq+TI6vXApfbfgjA9m+AhcBpZflpjBwSTwJXl+k+oKdMHw5cVqb/b0v7Y8rPHcBKYCbNkAK43/byMn0PsLekL0h6A/Cfg21c0jxJvZJ6+/v7Ryg1IiJGq6oTLGwvA3okHQ1sbXvdCKs8ZT9zvusGnv0d3GDnwQr4hO1Z5Wdf218uyx5rqeNh4ECaI7330AzRwepdYLthuzF9+vQRSo2IiNHqZFh9DzhJ0vMBJO1a5l8K/DNjO/S2jOZhRIB3tMy/Fjhd0rSyzT0lvWDgypJ2A7ay/S2ahyoPGkMtERExRh0LK9t3AhcCN0laDfxDWfQ1YBc2HcYbjQ8CfylpLbBnyzavoxmEt5Vl3wR2HGT9PYElklYB/w/4qzHUEhERYyRXdtV4uZ7qLbbf2elaNkej0XBvb2+ny4huljtYRJeR1Ge70U7bqq6zkvQF4I3AmzpdS0RE1KOqsLL9/k7XEFGtjDxiC1bV2YARERGDSVhFRET1ElYREVG9hFVERFQvYRUREdVLWEVERPUSVhERUb2EVUREVC9hFRER1UtYRURE9RJWERFRvaruDbjF6va7aUe0I/c+jGFkZBUREdWbsmEl6RpJO4/Q5gJJczaz3/dJ+qkklycGR0REh025w4CSRPOhkSM+88r2eaPYxDLgamDJKNaNiIgJUOXIStLZktaVnw9J6pF0t6RLgXXAXpLu2zjykXRuWX6LpMskzS/zF5UnD1Pany9ppaS1kmYOtm3bd9i+b5I+akREtKG6sJI0GzgNeBVwKHAWsAuwH3CR7QNs39/S/mDgBOBAmk8ZHu4RyQ/ZPgj4IjB/Yj5BRESMt+rCCjgCuML2Y7bXA4uBI4H7bS8fpP3hwJW2H7f9KPCdYfpeXF77gJ6xFippnqReSb39/f1j7S4iIoZQY1gN5bFx6OOJ8rqB8n2dpGslrZK0cHM7s73AdsN2Y/r06eNQXkREDKbGsLoZmCtpe0k7AMeXeUNZBrxZ0naSpgHHbs7GbL/e9izbZ46+5IiImEjVhZXtlcAi4HbgB8BC4OFh2q8ArgLWAP8GrAUeGe32JX1A0s+BFwFrRjPiioiI8SV3wVXjkqbZXi9pe2ApMK+E3qRpNBru7e0d3cq5g0VE7mCxBZLUZ3u4k+KeMeWusxrCAkn7A9sBl0x2UEVExMTqirCy/fZO1zAm+R9lRMSwqvvOKiIiYqCEVUREVC9hFRER1UtYRURE9RJWERFRva64zqoGkvqB+4dpshvw0CSVU7vsi02yLzbJvthkS9kXL7bd1r3qElaTRFJvuxe/dbvsi02yLzbJvtgk++IP5TBgRERUL2EVERHVS1hNngWdLqAi2RebZF9skn2xSfbFAPnOKiIiqpeRVUREVC9hNUEknSTpTklPSxryrB5J90laW55WPMpnjNRtM/bFGyTdLemnkj4ymTVOFkm7Srpe0k/K6y5DtNtQ/iZWSbpqsuucSCP9niU9R9I3yvIfSOqZ/ConRxv74lRJ/S1/C1vsQ2ITVhNnHfBWms/XGslrytOKu/VU1RH3haStgf8DvBHYHzilPPal23wEuNH2fsCN5f1g/qv8TcyyfdzklTex2vw9nwE8bHtf4LPApya3ysmxGX/z32j5W9hiHwabsJogtu+yfXen66hBm/viEOCntu+x/STwdeAtE1/dpHsLcEmZvgSY28FaOqGd33PrPvom8DqpK59QuqX8zY+LhFXnGbhOUp+keZ0upoP2BP6j5f3Py7xus7vtB8v0L4Ddh2i3naReScsldVOgtfN7fqaN7d8DjwDPn5TqJle7f/MnSFoj6ZuS9pqc0urTFQ9f7BRJNwB/PMiij9q+ss1ujrD9gKQXANdL+pHtdg4dVmWc9kVXGG5ftL6xbUlDnY774vJ3sTfwPUlrbf9svGuN6n0HuMz2E5LeTXPE+doO19QRCasxsD1nHPp4oLz+StIVNA8NTLmwGod98QDQ+r/GF5V5U85w+0LSLyXtYftBSXsAvxqij41/F/dIWgL8CdANYdXO73ljm59L2gbYCfj15JQ3qUbcF7ZbP/dC4NOTUFeVchiwgyTtIGnHjdPAMTRPRtgSrQD2k/QSSX8EvA3oqrPgiquAd5XpdwF/MOqUtIuk55Tp3YDDgR9OWoUTq53fc+s+OhH4nrvzgtAR90X5D81GxwF3TWJ9dbGdnwn4AY6neQz6CeCXwLVl/guBa8r03sDq8nMnzUNmHa+9E/uivH8T8GOaI4hu3RfPp3kW4E+AG4Bdy/wGsLBMvxpYW/4u1gJndLrucd4Hf/B7Bi4AjivT2wGXAz8Fbgf27nTNHdwXnyj/NqwGvg/M7HTNnfrJHSwiIqJ6OQwYERHVS1hFRET1ElYREVG9hFVERFQvYRUREdVLWEVERPUSVhERUb2EVUREVO//A5eSW76sKI6HAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import sklearn\n",
    "\n",
    "# Create linear regression\n",
    "regressor = sklearn.linear_model.LinearRegression()\n",
    "\n",
    "# Fit/train linear regression\n",
    "regressor.fit(x_train,y_train)\n",
    "# Predict\n",
    "pred = regressor.predict(x_test)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(\"Final score (RMSE): {}\".format(score))\n",
    "\n",
    "names = list(df.columns.values)\n",
    "names.remove(\"mpg\")\n",
    "report_coef(\n",
    "  names,\n",
    "  regressor.coef_[0,:],\n",
    "  regressor.intercept_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# L1 (Lasso) Regularization\n",
    "\n",
    "L1 Regularization, also called LASSO (Least Absolute Shrinkage and Selection Operator) is should be used to create sparsity in the neural network. In other words, the L1 algorithm will push many weight connections to near 0.  When a weight is near 0, the program drops it from the network.  Dropping weighted connections will create a sparse neural network.\n",
    "\n",
    "Feature selection is a useful byproduct of sparse neural networks. Features are the values that the training set provides to the input neurons.  Once all the weights of an input neuron reach 0, the neural network training determines that the feature is unnecessary.  If your data set has a large number of input features that may not be needed, L1 regularization can help the neural network detect and ignore unnecessary features.\n",
    "\n",
    "L1 is implemented by adding the following error to the objective to minimize:\n",
    "\n",
    "$$ E_1 = \\alpha \\sum_w{ |w| } $$\n",
    "\n",
    "The following code demonstrates lasso regression.  Notice the effect of the coefficients compared to the previous section that used linear regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 3.040905237197876\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>positive</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>origin-1</th>\n",
       "      <td>-1.264473</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>-0.007458</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>horsepower</th>\n",
       "      <td>-0.002797</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cylinders</th>\n",
       "      <td>-0.000000</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin-2</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin-3</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>displacement</th>\n",
       "      <td>0.013005</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acceleration</th>\n",
       "      <td>0.113760</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <td>0.787195</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  coef  positive\n",
       "origin-1     -1.264473     False\n",
       "weight       -0.007458     False\n",
       "horsepower   -0.002797     False\n",
       "cylinders    -0.000000      True\n",
       "origin-2      0.000000      True\n",
       "origin-3      0.000000      True\n",
       "displacement  0.013005      True\n",
       "acceleration  0.113760      True\n",
       "year          0.787195      True"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept: [-17.271273]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAasAAAD8CAYAAADJ7YuWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAGolJREFUeJzt3Xm0XGWd7vHvw6AIQUATaUQwCmgaVGhSIMjQqGkQGzUIXETbFhCirlZULtxlL4VuaGmnvq1e+yLGqIF7bVQ0CCJtwCEEApHUCZkAcWC4LQvtgyJNsBkMz/2j3kBxPEOdsd5zzvNZK6t2vfvd7/7VXpU8eXftqi3bRERE1GyLbhcQERExlIRVRERUL2EVERHVS1hFRET1ElYREVG9hFVERFQvYRUREdVLWEVERPUSVhERUb2tul3AVDFz5kzPnj2722VEREwaPT0999ue1UnfhNUYmT17Ns1ms9tlRERMGpLu6bRvTgNGRET1ElYREVG9hFVERFQvYRUREdXLBRYRMWGkblcQY22ibomYmVVERFQvYRUREdVLWEVERPUSVh2StGW3a4iImK6mZFhJOl/SB9qeXyDp/ZLOlrRK0jpJ57Wt/7akHkm3SlrQ1r5R0v+UtBY4eIJfRkREFFMyrIAvA38NIGkL4C3Ar4C9gAOB/YC5kg4v/U+1PRdoAGdIem5p3w74se19bd/QdyeSFkhqSmr29vaO7yuKiJjGpmRY2b4b+I2kPwOOBG4BDmhbXg3MoRVe0AqotcBKYLe29k3AtwbZz0LbDduNWbM6+i3GiIgYgan8PatFwMnAn9Caab0W+JjtL7R3knQEMA842PbvJS0DtimrH7G9aaIKjoiI/k3JmVVxOfA6WjOqpeXPqZJmAEjaVdLzgB2AB0pQzQEO6lbBERHRvyk7s7L9mKQfAb8rs6NrJP0pcJNaX6PfCPwV8D3g3ZJuB+6gdSowIiIqMmXDqlxYcRBwwuY2258FPttP96P7G8P2jPGpLiIihmNKhpWkvYGrgMtt/6zb9UREy0T9jlxMPVMyrGzfBry423VERMTYmMoXWERExBSRsIqIiOolrCIionoJq4iIqF7CKiIiqpewioiI6iWsIiKiegmriIioXsIqIiKql7CKiIjqJawiIqJ6U/K3ASOiTq2784xOfgx3esrMKiIiqldlWElaJqkxRmPNL7cM2fz8fEnzxmLsiIiYGFWG1XBJ2nKQ1fOBJ8PK9rm2vz/+VUVExFgZVVhJ+rakHkm3SlpQ2l4nabWktZJ+UNpmSPqKpPWS1kk6rrQfKemm0v8ySX90Z96B+ki6W9InJK0GTpB0uqRVZb/fkrStpFcBbwQ+JWmNpD0kLZZ0fBnjtZJuKXV9WdIz28Y+r+xzvaQ5ozlOERExOqOdWZ1qey7QAM6QtDPwReA42/vy1C3lzwEetP1y268AfihpJvARYJ7t/YEmcGb74B30+Y3t/W1/DVhi+4Cy39uBd9q+EbgSONv2frZ/0Tb2NsBi4ETbL6d1scl72sa+v+zz88BZozxOERExCqO9GvAMSceW5d2ABcBy23cB2P5tWTcPeMvmjWw/IOkYWqfnVqh1idAzgJv6jH/QEH2+3rb8MkkfBXYEZgBLh6j9pcBdtn9anl8M/A3wmfJ8SXnsAd7c3wBlNrkAYPfddx9idxERMVIjDitJR9AKoYNt/17SMmAN0OkpMwHX2j5pFH0eblteDMy3vVbSycARHdYxkEfL4yYGOE62FwILARqNRi6ojYgYJ6M5DbgD8EAJqjm0ZkHbAIdLehGApOeUvtfSmrVQ2ncCVgKHSNqztG0n6SV99tFJn822B+6TtDXwtrb2h8q6vu4AZm8eG3g7cF0HrzsiIibYaMLqe8BWkm4HPk4rWHppnRZbImktT52m+yiwk6QNpf3VtnuBk4FLJa2jdXrvabOyTvq0OQf4MbAC+Elb+9eAs8uFFHu0jf0IcApwmaT1wBPARSM5EBERMb7kfB18TDQaDTebzW6XEVG1/IJFtJPUY7uj79ROie9ZRUTE1JbfBoyICZNZUYxUZlYREVG9hFVERFQvYRUREdVLWEVERPUSVhERUb2EVUREVC9hFRER1UtYRURE9RJWERFRvYRVRERUL2EVERHVS1hFxISRxuaX12P6SVhFRET1hv2r65L+HtgIPBtYbvv7w9z+COAs28cMd98TTdJ84Ke2b+t2LRER09mIZ1a2zx1uUE1C84G9u11ERMR011FYSfqwpJ9KugF4aWlbLOn4svxxSbdJWifpn9rWXySpWbb9o5mUpAMl3VRuOX+jpM1jbynpnyRtKGO+r7TPlXSdpB5JSyXtUtqXSfp02dftkg6QtETSzyR9tG1/fyXpZklrJH1B0palfaOkCyStlbRS0s6SXgW8EfhU6b/HKI5zRESMwpCnASXNBd4C7Ff6rwZ62tY/FzgWmGPbknZs23w2cCCwB/AjSXv2Gf4nwGG2/yBpHvCPwHHAgrLtfmXdcyRtDXwOeJPtXkknAhcAp5axHrPdkPR+4ApgLvBb4BeSPg08DzgROMT245IuBN4GXAJsB6y0/WFJnwROt/1RSVcCV9n+5lDHKSIixk8nn1kdBlxu+/cA5R/wdg8CjwBfknQVcFXbum/YfgL4maQ7gTl9tt0BuFjSXoCBrUv7POAi238AsP1bSS8DXgZcq9blRFsC97WNtbmu9cCttu8r9d4J7AYcSivAVpXtnwX8R9nmsba6e4C/6OC4IGkBrWBl991372STiIgYgVHf1r7MfA4EXgscD7wXeM3m1X2793n+D8CPbB8raTawbJBdiVYIHTzA+kfL4xNty5ufb1W2v9j23/az7eP2kzfc3kSHx8X2QmAhQKPRyA27IyLGSSefWS0H5kt6lqTtgTe0r5Q0A9jB9tXAB4F921afIGmL8nnPi4E7+oy9A3BvWT65rf1a4F2Stir7eE7Zdpakg0vb1pL26aD+zX4AHC/peZvHlPTCIbZ5CNh+GPuIiIhxMGRY2V4NfB1YC/wbsKpPl+2BqyStA24Azmxb9/+Am8t277b9SJ9tPwl8TNItPH02s6hsu07SWuCtth+jNXP7RGlbA7yqo1fZeh23AR8Brim1XgvsMsRmXwPOLheA5AKLiIgu0VNnv8Z4YGkx0+jihEaj4Waz2e0yIqq2+dcrxumfnZhkJPXYbnTSN79gERER1Rv1BRYDsX3yeI0dEZNTZlQxUplZRURE9RJWERFRvYRVRERUL2EVERHVS1hFRET1ElYREVG9hFVERFQvYRUREdVLWEVERPUSVhERUb2EVUREVC9hFRER1UtYRURE9SZ1WEm6WtKOQ/Q5X9K8YY77JUlrJa2T9M1yN+SIiOiSSRlWatnC9utt/26wvrbPtf39Ye7ig7b3tf0KWncsfu+Ii42IiFGrNqwknSlpQ/nzAUmzJd0h6RJgA7CbpLslzSz9zynrb5B0qaSzSvtiSceX5bslnSdptaT1kub0t2/b/1n6C3gWkLvwRER0UZVhJWkucArwSuAg4HRgJ2Av4ELb+9i+p63/AcBxwL7A0cBgt0m+3/b+wOeBswap4SvAr4A5wOdG9YIiImJUqgwr4FDgctsP294ILAEOA+6xvbKf/ocAV9h+xPZDwHcGGXtJeewBZg/UyfYpwPOB24ET++sjaYGkpqRmb2/vUK8pIiJGqNawGsjDYzDGo+VxE7AVgKSlktZIWtTe0fYm4Gu0Zm1/xPZC2w3bjVmzZo1BaRER0Z9aw+p6YL6kbSVtBxxb2gayAniDpG3KlXvHDGdnto+yvZ/t08rFG3vCk59ZvRH4ycheRkREjIWtul1Af2yvlrQYuLk0LQIeGKT/KklXAuuAXwPrgQdHuHsBF0t6dlleC7xnhGNFRMQYkD01LnSTNMP2RknbAsuBBbZXT9T+G42Gm83mRO0uImLSk9Rje7AL4p5U5cxqhBZK2hvYBrh4IoMqIiLG15QJK9tv7XYNERExPmq9wCIiIuJJCauIiKhewioiIqqXsIqIiOolrCIionoJq4iIqF7CKiIiqpewioiI6iWsIiKiegmriIioXsIqIiKql7CKiIjqJawiIqJ6kzasJF0tacch+pwvad4wx/2qpDskbZD0ZUlbj67SiIgYrUkXVuW281vYfr3t3w3W1/a5tr8/zF18FZgDvBx4FnDaCEuNiIgxUmVYSTqzzGw2SPqApNlltnMJsAHYTdLdkmaW/ueU9TdIulTSWaV9saTjy/Ldks6TtFrSeklz+tu37atdADcDL5iYVx0REQOpLqwkzQVOAV4JHAScDuwE7AVcaHsf2/e09T8AOA7YFzgaGOwWyffb3h/4PHDWEHVsDbwd+N7IX01ERIyF6sIKOBS43PbDtjcCS4DDgHtsr+yn/yHAFbYfsf0Q8J1Bxl5SHnuA2UPUcSGw3Pb1A3WQtEBSU1Kzt7d3iOEiImKkagyrgTw8BmM8Wh43AVsBSFoqaY2kRZs7Sfo7YBZw5mCD2V5ou2G7MWvWrDEoLyIi+lNjWF0PzJe0raTtgGNL20BWAG+QtI2kGcAxw9mZ7aNs72f7NABJpwFHASfZfmJkLyEiIsbSVt0uoC/bqyUtpnVxA8Ai4IFB+q+SdCWwDvg1sB54cBQlXATcA9wkCWCJ7fNHMV5ERIySWhe9TW6SZtjeKGlbYDmwwPbqiayh0Wi42WxO5C4jIiY1ST22B7so7knVzaxGaKGkvYFtgIsnOqgiImJ8TYmwsv3WbtcQERHjp8YLLCIiIp4mYRUREdVLWEVERPUSVhERUb2EVUREVC9hFRER1UtYRURE9RJWERFRvYRVRERUL2EVERHVS1hFRET1ElYREVG9hFVERFQvYRUREdWbFGElabGk48vyonLvquFsv3F8KouIiIkw6e5nZfu08RxfrXvZy/YT47mfiIjoXFdnVpL+WtI6SWslXS7pLklbl3XPbn/ets0ySY2yvFHSBWX7lZJ2Lu0vknSTpPWSPtpn+7MlrSr7Pa+0zZZ0h6RLgA3AbmU2t6GM8cGJOB4REdG/roWVpH2AjwCvsb0v8E5gGfCXpctbgCW2Hx9kmO2AlWX75cDppf2zwOdtvxy4r22fRwJ7AQcC+wFzJR1eVu8FXGh7H2AmsKvtl5UxvjLAa1ggqSmp2dvbO7wDEBERHevmzOo1wGW27wew/VtgEXBKWX8KA4REm8eAq8pyDzC7LB8CXFqW/09b/yPLn1uA1cAcWiEFcI/tlWX5TuDFkj4n6XXAf/a3c9sLbTdsN2bNmjVEqRERMVJVfWZle0U5JXcEsKXtDUNs8rhtl+VNPP31uJ/+Aj5m+wtPa5RmAw+31fGApH2Bo4B3A/8NOHUYLyUiIsZQN2dWPwROkPRcAEnPKe2XAP/K0LOqwaygdRoR4G1t7UuBUyXNKPvcVdLz+m4saSawhe1v0TpVuf8oaomIiFHqWljZvhW4ALhO0lrgn8uqrwI78dRpvJF4P/A3ktYDu7bt8xpaQXhTWfdNYPt+tt8VWCZpDfB/gb8dRS0RETFKeuosWh3K96neZPvt3a5lOBqNhpvNZrfLiIiYNCT12G500reqz6wkfQ44Gnh9t2uJiIh6VBVWtt/X7RoiIqI+k+LnliIiYnpLWEVERPUSVhERUb2EVUREVC9hFRER1UtYRURE9RJWERFRvYRVRERUL2EVERHVS1hFRET1ElYREVG9hFVERFQvYRUREdUbk7Aqt6If6hb0ERERI9L1mZWkqm5TMpDJUmdExFQ0lmG1paQvSrpV0jWSniVpP0krJa2TdLmknQAkLZP0GUlN4P2STpC0QdJaSctLny0lfUrSqrL9u0r7EZKWS/qupDskXSRpi7LuJEnry1ifKG0nSPrnsvx+SXeW5RdLWlGW50q6TlKPpKWSdumvzjE8VhERMQxjOVvYCzjJ9umSvgEcB/wP4H22r5N0PvB3wAdK/2dsvp2xpPXAUbbvlbRjWf9O4EHbB0h6JrBC0jVl3YHA3sA9wPeAN0u6EfgEMBd4ALhG0nzg+lIHwGHAbyTtWpaXS9oa+BzwJtu9kk4ELgBO7VtnX5IWAAsAdt9995Eet4iIGMJYhtVdtteU5R5gD2BH29eVtouBy9r6f71teQWwuITcktJ2JPAKSceX5zvQCsTHgJttb54hXQocCjwOLLPdW9q/Chxu+9uSZkjaHtgN+FfgcFphtQR4KfAy4FpJAFsC9w1Q59PYXggsBGg0Gh788ERExEiNZVg92ra8CdhxoI7Fw5sXbL9b0iuBvwR6JM0FRGtWtrR9I0lHAH2DYaiguBE4BbiD1kzrVOBg4L8DuwO32j54qDojIqI7xvMCiweBByQdVp6/Hbiuv46S9rD9Y9vnAr20ZkBLgfeU03RIeomk7comB0p6Ufms6kTgBuBm4M8lzZS0JXBS2/6uB84ClgO3AK8GHrX9IK0AmyXp4LKfrSXtM3aHISIiRmu8r3B7B3CRpG2BO2nNbvrzKUl70ZpN/QBYC6wDZgOr1To/1wvML/1XAf8C7An8CLjc9hOSPlSeC/iu7StK/+tpBeBy25sk/TvwEwDbj5VTjf9L0g60jslngFvH6BhERMQoyZ5cH7WU04Bn2T6m27W0azQabjab3S4jImLSkNQz0AVsfXX9e1YRERFDmXRfdLW9DFjW5TIiImICZWYVERHVS1hFRET1ElYREVG9hFVERFQvYRUREdVLWEVERPUSVhERUb2EVUREVC9hFRER1UtYRURE9RJWERFRvYRVREyM1p24I0YkYRUREdWb0mElaZGkvYfos7jcfLFv+2xJbx2/6iIiolNTOqxsn2b7thFuPhtIWEVEVGBShJWksyWdUZY/LemHZfk1kr4q6UhJN0laLekySTPK+mWSGmX5nZJ+KulmSV+U9C9tuzhc0o2S7mybZX0cOEzSGkkfnMCXGxERfUyKsAKuBw4ryw1ghqStS9s64CPAPNv7A03gzPaNJT0fOAc4CDgEmNNn/F2AQ4FjaIUUwIeA623vZ/vTY/6KIiKiY5MlrHqAuZKeDTwK3EQrtA4D/gvYG1ghaQ3wDuCFfbY/ELjO9m9tPw5c1mf9t20/UU4Z7txpUZIWSGpKavb29o7ohUVExNAmxW3tbT8u6S7gZOBGWrOpVwN7AncB19o+aRS7eLRtuePra20vBBYCNBoNj2L/ERExiMkys4LWqcCzgOVl+d3ALcBK4BBJewJI2k7SS/psuwr4c0k7SdoKOK6D/T0EbD9WxUdExMhNtrDaBbjJ9q+BR2h9ptRLa8Z1qaR1tE4RPu0zKdv3Av8I3AysAO4GHhxif+uATZLW5gKLiIjukj09zl5JmmF7Y5lZXQ582fblYzV+o9Fws9kcq+Eiph4Jpsm/N9EZST22G530nUwzq9H6+3IBxgZan3N9u8v1REREhybFBRZjwfZZ3a4hYlrLrCpGYTrNrCIiYpJKWEVERPUSVhERUb2EVUREVC9hFRER1UtYRURE9RJWERFRvYRVRERUL2EVERHVS1hFRET1ElYREVG9afPbgFVTx/d7jJjc8vuAMUKZWUVERPUmbVhJulrSjkP0OV/SvGGO+15JP5dkSTNHV2VERIyFSXcaUJJo3TTy9UP1tX3uCHaxArgKWDaCbSMiYhxUObOSdKakDeXPByTNlnSHpEto3TxxN0l3b575SDqnrL9B0qWSzirtiyUdX5bvlnSepNWS1kua09++bd9i++4JeqkREdGB6sJK0lzgFOCVwEHA6cBOwF7Ahbb3sX1PW/8DgOOAfYGjgcFukXy/7f2BzwO5GWNExCRRXVgBhwKX237Y9kZgCXAYcI/tlf30PwS4wvYjth8CvjPI2EvKYw8we7SFSlogqSmp2dvbO9rhIiJiADWG1UAeHoMxHi2Pmyif10laKmmNpEXDHcz2QtsN241Zs2aNQXkREdGfGsPqemC+pG0lbQccW9oGsgJ4g6RtJM0AjhnOzmwfZXs/26eNvOSIiBhP1YWV7dXAYuBm4MfAIuCBQfqvAq4E1gH/BqwHHhzp/iWdIemXwAuAdSOZcUVExNiSp8A3yiXNsL1R0rbAcmBBCb0J02g03Gw2R7ZxfsEiposp8O9NjB1JPbYHuyjuSZPue1YDWChpb2Ab4OKJDqqIiBhfUyKsbL+12zWMSv63GRExqOo+s4qIiOgrYRUREdVLWEVERPUSVhERUb2EVUREVG9KfM+qBpJ6gXuG7Dj5zATu73YRk0COU2dynDo3HY7VC2139Ft1CasYlKRmp1/am85ynDqT49S5HKuny2nAiIioXsIqIiKql7CKoSzsdgGTRI5TZ3KcOpdj1SafWUVERPUys4qIiOolrOJpJJ0g6VZJT0ga8EokSa+TdIekn0v60ETWWANJz5F0raSflcedBui3qdyJeo2kKye6zm4Z6v0h6ZmSvl7W/1jS7Imvsvs6OE4nS+ptew9N25vEJqyirw3Am2ndF6xfkrYE/jdwNLA3cFK5Rct08iHgB7b3An5Qnvfnv8qdqPez/caJK697Onx/vBN4wPaewKeBT0xsld03jL9HX297D03bm8EmrOJpbN9u+44huh0I/Nz2nbYfA74GvGn8q6vKm4CLy/LFwPwu1lKbTt4f7cfvm8BrpWl3F9L8PRqGhFWMxK7Av7c9/2Vpm052tn1fWf4VsPMA/baR1JS0UtJ0CbRO3h9P9rH9B+BB4LkTUl09Ov17dJykdZK+KWm3iSmtPlPi5osxPJK+D/xJP6s+bPuKia6nVoMdp/Ynti1poMtqX2j7XkkvBn4oab3tX4x1rTFlfQe41Pajkt5Fazb6mi7X1BUJq2nI9rxRDnEv0P4/vBeUtillsOMk6deSdrF9n6RdgP8YYIx7y+OdkpYBfwZM9bDq5P2xuc8vJW0F7AD8ZmLKq8aQx8l2+zFZBHxyAuqqUk4DxkisAvaS9CJJzwDeAkybK92KK4F3lOV3AH80I5W0k6RnluWZwCHAbRNWYfd08v5oP37HAz/09PvS55DHqfxHaLM3ArdPYH1VSVjF00g6VtIvgYOB70paWtqfL+lqePIzhvcCS2n95fmG7Vu7VXOXfBz4C0k/A+aV50hqSNp8xdafAk1Ja4EfAR+3PeXDaqD3h6TzJW2+IvJLwHMl/Rw4k4GvppyyOjxOZ5SvkqwFzgBO7k613ZdfsIiIiOplZhUREdVLWEVERPUSVhERUb2EVUREVC9hFRER1UtYRURE9RJWERFRvYRVRERU7/8DCS/EJZyjdIwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import sklearn\n",
    "from sklearn.linear_model import Lasso\n",
    "\n",
    "# Create linear regression\n",
    "regressor = Lasso(random_state=0,alpha=0.1)\n",
    "\n",
    "# Fit/train LASSO\n",
    "regressor.fit(x_train,y_train)\n",
    "# Predict\n",
    "pred = regressor.predict(x_test)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(\"Final score (RMSE): {}\".format(score))\n",
    "\n",
    "names = list(df.columns.values)\n",
    "names.remove(\"mpg\")\n",
    "report_coef(\n",
    "  names,\n",
    "  regressor.coef_,\n",
    "  regressor.intercept_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jheaton/miniconda3/envs/wustl/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.\n",
      "  ConvergenceWarning)\n",
      "/Users/jheaton/miniconda3/envs/wustl/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.\n",
      "  ConvergenceWarning)\n",
      "/Users/jheaton/miniconda3/envs/wustl/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.\n",
      "  ConvergenceWarning)\n",
      "/Users/jheaton/miniconda3/envs/wustl/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.\n",
      "  ConvergenceWarning)\n",
      "/Users/jheaton/miniconda3/envs/wustl/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.\n",
      "  ConvergenceWarning)\n",
      "/Users/jheaton/miniconda3/envs/wustl/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.\n",
      "  ConvergenceWarning)\n",
      "/Users/jheaton/miniconda3/envs/wustl/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.\n",
      "  ConvergenceWarning)\n",
      "/Users/jheaton/miniconda3/envs/wustl/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.\n",
      "  ConvergenceWarning)\n",
      "/Users/jheaton/miniconda3/envs/wustl/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.\n",
      "  ConvergenceWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(1e-08, 100000000.0)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgMAAAF6CAYAAACEHlvDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xd4m+XV+PHv0fK2leFMO3svOxASEmISSAIps6xQyi5l/ShllLdABy3w8ra0lNJSOqClpS2jUJbDSCBhz+yETLIgdnbiGS+t+/fHoyR25FiP4siOpfO5Ll2WdB5Jx44jHT/3OGKMQSmllFLJy9HeCSillFKqfWkxoJRSSiU5LQaUUkqpJKfFgFJKKZXktBhQSimlkpwWA0oppVSS02JAKaWUSnJaDCillFJJTosBpZRSKsm52juBeOjatavp169fe6ehlFJKtYnFixfvMcbkHunjE7IY6NevH4sWLWrvNJRSSqk2ISJft+bxOkyglFJKJTktBpRSSqkkp8WAUkopleS0GFBKKaWSnBYDSimlVJLTYkAppZRKcloMKKWUUklOiwGllFIqyWkxoJRSSiU5LQaUUkqpJKfFgFJKKZXkErI3QSAAe/c2vc/rBacTamuhri7yMZ06gcNx+HjnziACNTVQXx8Z79LF+rpvHzQ0NI2JWI8HqK4Gn69p3OGwXh+gqgr8/qZxp9PKH6Cy0vr+GnO5ICfHul5RAcFg07jbDdnZ1vXycgiFmsY9HsjKsq6XlYExTeMpKZCZaV0/9OcKkJoKGRnW48rKIuNpaZCebr1uc/H0dOsSDFr5HSojw3qOQMD6/g6VmWnl4Pc3H8/Otr4Hn8/6+R0qJ8f6GTQ0NB/v1Mn6Gfp81jFOp3Vxuax/O5HIxyilVIdijEm4CxxvrI+mg5flq32mosZnfvrzQEQMjNm0xYrffkfz8V3lVvzaGyLjbnfIVNRY8UsuDUbEO3U+GD/73Mh4fp+D8VNOjYyPGHEwPn5CZHz8hOCB+IgRoYj4KacejOf3iYyffe7BeKfOkfFLLj0Yd7sj49feEDAVNT6zq9zX7M/u9jus+KYtzcd/+nMrvnx18/EHH7LiH3/efPyxP/tNRY3PzJ3vbzb+1NNW/KVXm4+/9KoVf+rp5uNz51vxx/7SfPyzRX6zr95vfvdowGRmhkxOTsh07hwyubkh06NHyHy5MWDq/QHz+0eDZtDgkBk6LGRGjgyZMWNCZuzYkNm7N2RCoZD5059C5uSTjTn1VGNmzDBm5kxjzjzTGJ/PGGOM+dvfjPnWt4y59FJjrrjCmKuvNuaGG8wBTz9tzG23GXPHHcbceacxP/qRMb/4xcH4li3G1NcbpVQCAhaZVnxuJuSZgR69Qlx1Q9M/72vwsaUMRoxzctd9zojH7Kn3sa8MCk9ycld2ZHxrpQ+nEyae6qRLr6Zxh8CWMuvP/alnuOg7tOnoiyfFsKXM+nN/5vkuRh7fNJ6RcTB+7iUuJkxtGs/xHozPusrFqWc2jXfNDbGlzDpdcPkNbiormv6p2qPnwfh1t7ipqWkaz+97MH7znW58DU3j/QcejP/PPR5Ch5w5GDIsyJayIMEg3HWfh0ONGG3F6+vh7vubiY+14jU0Hx84JsCWshDBNOHu+wMR8d5DrLgrR/jR/e6IeNc+Vjyje/PxjO5+tpQZuvZx8KP/jfwv4cqx4j0GOLj9xy5CIQgGhVAQAkGoc/rYtNuQ3dPJed9yEwxBKGid6QiFhB019dTvMIRSXQwa7m7y+GAI1u+uJb0Wtle6qfV5CNaFHxuEYEhYvX0fLhcsXZPCZwvcBIMSfm7rDMVtO2sAmD03hdkvWq+///E5nQyzrqlFRLj80lTWrHLw7W8LV18NY8fqWQ2llEWsgiKxjBwz1jz3xnvtnYZS7c6Ygx/4n7zv4pX/uHnnLavgGz3acPfdwiWXtG+OSqnWE5HFxphxR/r4hDwzoJSyNP7Lf9KUAJOmBKiqqOPNYg+vPu9m7eYQ1fWCBNzMmwdnnmnNj1BKJRctBpRKMtleuPgKHxdf4SMUgq/2wBsve7jr+2nk5houvdQaRhgzpr0zVUq1FV1aqFQSc4TfAU4728cf/lFDwQkBHnvMUFAAxx9vml1doZRKPHpmQCmFywUnTwtw8rQA5WXCm6+6WbnMSVUwgLPBw1N/ddGvH5x+unWsUiqx6H9rpVQTnTobvn21tTqmohb2VPn5v19msa3UQY8ehssvt4YRhg9v50SVUkeNDhMopVrkcsHs96t55Ikaho0O8PDDhhEj4OGHE28lklLJSs8MKKWicnvg1JkBTp0ZYO9u4Y1X3AweF6C03MmKBSn88+9Orr4apk+39j5QSnUsemZAKRWTLrmGy6/10W9AiPIaP0vXNDDnLcPMmdC3r+FHP4Ivv2zvLJVSsdBiQCnVKudc6Gf+wioe+nMNA4YGePBBw5QpBr/fGkY4tFeGUurYo8MESqlW86TAaWcGOO3MALt3Cl9vcvDl7iBZKR6mnZjKhAnWpMNTTjm4nFEpdezQ/5ZKqaMqt7th3MQgoRCU7vQx9kQfxbMN06dD//6Ge+6B0tL2zlIp1VhC9iboldfHfPeWO5vcF8zpTaDLAAgFSPnq04jHBDr1IdipLwQaSNmyIDLepT/BnDzEV4undHFkvOsggtk9kYZqPFuXRcT93YYSyuyG1FXg2f5FZLzHCELpXXDU7sW9Y3VE3NdzNCbNi2PfLty71kXGexdiUrJwVm3HtWdDZDzveIwnHWdlKa69myPiDX3GgysFZ/nXuMq3RMb7TQSHC9feTTgrt0bGBxQB4NqzHmfVjqZBh5OGvpOs+K61OGt2Nwkbp4eGPhMAcO9chbO2aZ/kkDuNht7WltueHStw1jfdCSfkyaSh11grvm0pDt++pvGUHBp6jMEAqVsX4Qg0bWIVTOtMQ7cRAKSVLkCCTXtMB9JzCXQbhssheL7+BAkFEREEQMDdqTdpecNxOoTqL+aF77fiIpDVsz9d+41ATJCvF8xFEEQ48Pie/YeRN2gEQV89Kz6cc+D+/TsJDxg+hrwBQ6ndV82i9+dE/OwHjz6enn0GUF1RxtKP50fEhxVOoFvvPlTs3cWKz96PiI8cdxJduvdi785trFr0cUR8zIlT8Hbpxq6tW1i77POI+NiTppHl7cz2LZtY/0Xk/41xU2bicGbxdvFGdpYsp6pSGDnS0LWrEAjAOefMolu3dJYtW8ayZZH/dy699FLcbjcLFy5k1apVEfGrrroKgE8++YQvD5ms4Ha7ufTSSwF4//332by56e9+eno6s2bNAmDevHmUHlKlZGdnc/755wMwZ84cduxo+rvdpUsXzj77bABmz57N3kN6fPfo0YOZM2cC8NJLL1FVVdUknpeXx/Tp0wF4/vnnqa2tbRLv378/U6ZMAeDpp5/Gf0h/8yFDhjBpkvV/6x//+AeHGjlyJCeccAJ+v5+nn346Il5YWEhhYSG1tbU8//zzEfFx48YxatQoKisrefnllyPiEydOZOjQoezZs4fXXnstIn7yySczYMAAduzYwZw5kb+706ZNIz8/n5KSEubPj/zdnTlzJj169GDTpk188MEHEfGzzjqLrl27sm7dOj79NPJ9/bzzziMnJ4eVK1eyaNGiiPisWbNIT0+M371zzjmn4/YmEJGZwO8AJ/BXY8wvD4lfBfwa2P/p8wdjzF+jPW8gZNhdXd/kvs0V5azbVIKTIDM89RGP2VBRxoYNDlLwc0oz8bUVe/kqaEgXHye7I+Mry/dQGgqQLfVMaia+vHw320MNdJZaxjcTX1y+i92hWnId+zjeFRlfULaTMlNNT0cVBc3EP/liB1WmgjxHBaOaiX+wYju1xkM/ZxnDnJHxd5dupQE3g5xlDGom/vbiUoI4Geosp38z8TkLSwAY6aog39EoLhAwDuYvtuJjnJX0POTxDSbAe7ut+HGuKnIdDRz8KIRaY/hol/UrMM5ZQ2dH0w/ralPLZzut+ImuWrKkaby8qoZFO6z4ZFct6dL0DXV3dTXLtm9DEIpc9aTQtDPijqpqlpWWEjIww1OHk1CTeGnZHlauWw/ATE/TN3OAr3ZuY+3iQPh3b19E/PWtm9nwfnX4d+9gobO/qHh165dspYxMh4/xUk347gNFxXsfbKTCU0WGqWNgfc2BImR/fNnCLfhW1TE8J0hmk59s20lNg7EnBNiUGcTnE9xuQ61P2LrFwciRhrPPtjY0Ukq1j3Y7MyAiTuBLYAZQCiwELjHGrG50zFXAOGPM92J5bu1aqOIhZAyhkCEQMgQbXQIhQ9AYgkFDIBQiZAyBYDhuwvFgo+sHLqEmtwOhyOcPhI8LNvO6wUNeJxgyBIIhgoYDz73/+er9Ifbsa2BUr2xunT6E7tmp7f3jBOCLpU7++4yHt2ZbrbUHDDDceKNwxx3tnZlSHUtH7lo4HthgjNkEICLPAecCkefIlToGOERwOAVXB1xHb4xh3pqdPPHhZm5+dinXTO7PaSO6I9Ie5wkOGj02yOixddx5bx3z33RT/IKHTxZAZZ0hO9XFa68J06ZBenq7pqlUwmvPYqA3UNLodikwoZnjLhCRk7HOItxmjClp5hhE5DrgOoCevfOPcqpKdWwiwowRPRiT5+X389fzh3c38NmmvXzvlEF0yUxp7/RIT4ezL/Bz9gV+gkHYshc2r3dy7jmZZGcbLr5YuOoqmDixaVtmpdTR0Z7DBBcCM40x3w3fvhyY0HhIQES6APuMMQ0icj1wsTHm1GjPrcMESh1eyBheW7Gdpz75Co/LwY1TBlI0uGu7nyU4VCgESz538srzHt5+3U1dnTWMMG+e0L8/vPIKPP64tV1y48ujj0KXLvD66/Daawfvdzqtrz/7GaSlwTvvwIIFTR/rdML111vLHxcsgI0bm8Y9noNzG9atg717Dz6vywUpKTBsmBXfvRt8vqav7XZDRkb7/UxV4urIwwRbgcZ/wudxcKIgAMaYxlNz/wr8qg3yUiqhOUQ4p6AXx/Xx8si89fz6rXV8smkvN04ZSE6au73TO8DhgHETg4ybWMfd99fx9utu5s9xs35XHfvchi+3uinZ5iEYFAJBCAasDY5Wb62lmx8+XOjmhf+G4wErHgjCt66rITsLnn3Rw1//6Il43TMuqsXhgEf/5Obf/2j680jPMGzc2oAI/PgeNy8+33TMqFs3w+qNPkSEK6928ebrTVdvDxhoWLYyQLrHhdNxbBVfKrm155kBF9ap/2lYRcBC4NvGmFWNjulpjNkevn4ecKcx5sRoz61nBpSyJxgyvLSklGcWbCEz1cXNpwxifP8u7Z1WmwgGIeAPfw1AMCgEA9C1m/WeuHePUFUpB4qMYACMEUYWWFsqrl/rYPdOx4HHh4LgdMHUGdZqlE8/cLG1VAgG5MDjM7IM517kp3RtGqOHeejbt92+fZVgWntmoF33GRCRM4BHsJYWPmmMeUBE7gMWGWOKReQXwDlAACgDbjTGrI32vFoMKBWbzXv28fDbX/LV3lqmDevGtUUDyEjRDUrjoWyvcMrYLH72M+FnP2vvbFSi6NDFQLxoMaBU7PzBEM8u2MKLS0rpkpnCLdMGU5Dnbe+0EtKV52dAwMmSJTpUoI6O1hYDuh2xUgoAt9PBFRP78asLCvA4HfzklZX85YON1Pu109DRNmWGn6VLhZJm10Yp1fa0GFBKNTG0RxaPXFzI2WN68tqK7dzy3FLWbq+K/kBl2ynheQWzZ7dzIkqFaTGglIqQ6nZy3ckDeeCbowiEDHe+tIKnPvkKfzAU/cEqqn4DQ/TtH+SttxJvmFZ1TFoMKKUOa0yel0cvGcu04d3575JSbn9+GZt2R/ZXULERgT/+s5Z//FuHYNSxQYsBpVSL0j0uvn/qYH565ggq6/z84IXl/GdRCcGQ/lXbGvn9QtQF/dEPVKoNJORqAqdznEnPWNjkvtferaF3vuFvf3Lzh99Ebr/67sJ9eDvBow95ePLPkRuRfL56Hx4P/PLeFP7zr6YbkbjdhgVragC454cpzH6paTzHa3hvkRW/46ZU5s9tumSrV+8Qr79vdbu78cpUPvu4aXzQkBAvvGHFr7wojRVLm250MmZskKdesNryXnRGOhu+bFrjnXhSgD89ZXUKPHNKOtu2No1POz3AQ49Z8anjMqisaDrD+ezz/dz3qwYAxg/PwO9vGr/4cj93/awBnw8mjMg8cP/+X61rbvRx8x0+Ksph6vGZHOrm/2ngmhv9bC0RvlEUuT3bnT9v4NKrAny5xsH5M9Mi4v/7UAPfvCjA0oUOLrsgMv7bP9dz2hlBPnzPyfWXRTboeeLpeiZPDTL3NSe3XJfaJHeAZ4vrGDsuxOyXnTzwkxTcHmsnObfb4HLDb/5SS78BId6Z4+I///JYMY8V93jgth/X07mL4fOPnXz2oQuX29rJzu02uD1wwSU+UtNg7SoHX29yHrh///MfNz6IwwF7dgk1NVbHv8avkRH5I42bqjo/f/5gIx+u38OQ7pncOn0I+Z20ccCRevKxVNIkhf/7v/bORHV0HXkHwrjp2hW+/e2mH1ij+2fSuTOcOgnqKyIfMzwvi4wMOG0KOJop1of0yMLlgm+cClmH1ApOpzCoWxYAZ50G3Ts1jaenH4yfewb0z2sa79TJeSB+wbkwaljTeM+eB+OzzofxxzWN9+vnOhC/ZBZs3940PnSom0HdrALl0kugvLxpfMyYg/ErL4dDWqpzwgkeBnWzvunvXG1twNLY5MlWPBCA717TNCYC06akMKhbCjUZ1lavh5o+OZVB3VLp7IKbboqMT5uUyqBukBGEW74fGZ86wYq7R8Ftt0bGi45PY1A3CBXSbDe8SYVpDMyFk0+AH/6wae4AE0al06cbTBgNF11obTG7/9LQAGP6ZtGnN6zKNATroa7Kut/ngwYf5Hud9OoOr2wQ/vm4RBRTN37XSXYOPDHHyaOPRHZBKtlTj8NpeOhnbv75ZNP/simphjUltRhjuPOWFN541YXbAx6PVUh0zTU893oNBsPWEqF7T4OjFecDs9Pc/PD0YUwcsJs/vbeRW59bxhUT+3J2QS8cx9h2xh3Bpg3Ch/MN990nuBLy3Vh1FAl5ZmDcuHFm0aJF7Z2GUs0yBvz+gwWF12ttvbt798H97BsXG9OmWY9btMjaD3///T6f9Vy33GLFn33WOqbx49PT4bHHYNs2GD3aMO0MHz9+oL5VBcF+ZTU+Hn1nPYu+Lj/mWiN3FPPedHH7dRm89x5MmdLe2aiOTDcdaoYWA0o1ZQzcfTc8+CCcf4mPe35Zd1QKgsatkYFjpjVyR1FbA0VjsvneTcLDD7d3Nqoj002HlFJRicAvfgE//jG89KyHn/9PWsRwz5E9r9Ua+dFLxjK4WyZ/eHcD9722mr37Glr/5EkgPQMmnBSguNiQgH+XqQ5EiwGlkoQI3H+/1cL3lec9PPP3yImyR6p7dir3f3MU1xYNYEVpJd97dikffLmbRDzzeLR941w/YwpDEXN1lGpLOkygVBJ6+mmYcnoD5fX1R/25S8treWTeetbtrOakQV2PudbIx6LsNBd9u0SupFHKLh0mUErF7NJLIa9rChmk8thvUggEjt5z53VK58ELxnDFiX35fNNevvfsEhZs3nv0XiABVdcH+PrrxPvDTHUcWgwolcQ+eieFvzySyl03p+E/ivvfOB3CRePyeXhWAd40N/e/voZH5n1JTcNRrDoSyDN/99C/P+zY0d6ZqGSlxYBSSeyKK+Chh+Ct1zz88KZ0/L6j+/z9u2by8KxCLjo+j3fX7eLm55ayvKSZjT6S3PETAhgjvP56e2eikpUWA0oluR/8AB5+GOa/6eaOG49+QRDRGvnVlfzlfW2N3NjgYSF65YV49dX2zkQlKy0GlFLcdhv8/vewcZ2L8rL47BHQpDXyF9oauTERmDrDz9tvG11VoNqFFgNKKQBuvhlWrxKOH5FKKAS+OGwVoK2RD2/qDD/19cK8ee2diUpGWgwopQ7IyABvuodH7s3klu+m03D0Vx4C2hq5OcefGOShx+o4+eT2zkQlIy0GlFIRJk9y8sn7Lr5/TTr1dfF5DW2N3JTbDaed4yMtU+dSqLanxYBSKsI118CTTwqffeji5u9kUBenggBgfP/O/OGS45g4sAv//uxrfvjickrKk3PgvLoKfv2Q4Ysv2jsTlWy0GFBKNeuqq+Cpp4SFnzj5wfXpcd07f39r5B+ePpTtFfXc+twyXl22lVAC7pDaEhOCe3/q5Jln2jsTlWy0g7ZS6rAuvxycTiEtW3A4iHsznaLBuYzslcOj76znrx9t5rNNe5OqNXK2F46fEKS42MkvfqGdH1Xb0TMDSqkWffvbcN5ZLvp0See9t13UxHmeX+cMD/ecNYLvnzqIjbtruPnZpcxdtSNpmh5NmeFn9Wphw4b2zkQlEy0GlFK2VO52c8cN6dxwWQbVcd4eIJlbI0+dbu0LXVzczomopKLDBEopW/Lz4bnnhFmznNx4WQZ//FcN2Tnxfc39rZFfW7Gdpz75ipueWcLA3Ey86W686R68ae4D13PC1zule3A7O+7fOXl9DcNGBvn6awegQwWqbWgLY6VUTIqL4cILDUNGBPnLv2vI9rbN65aW1/LcwhJ2VdVTUeenotZP3WG2NM7wOK1iId0dLhKaFg6d0g4WEGkeZ9t8AzHw+2Fwz3SyUrX1s7KntS2M9cyAUiom55wDL70kXHCBk3fmuvnmxUex3WEL8jqlc8dpQ5vcV+8PUhkuDCrqfNbXWt+BYqGi1seWslpWlFay7zAdE1NcDqtISPMcPOOQ7g4XD02LiAyPE5H4/7XudlttjTNT3LTByymlxYBSKnZnnQWrVgk98z1s3uMnFKJdPrRS3U5S3U5bqw38wVAzhUPj4sHHzqp61u2oprLOT3PnTF0OOaRAOKSIaFQ4ZKW6cLTih3Ldd1xkp6LLDFWb0GJAKXVEBg0CcFFZksGtt8NDf6qlc5djd9jR7XTQNTOFrpkpUY8NhgxV9U2LhcpwEVEeLiLKan1s2lNDZZ2/2V0THUKTIYqccOHQKd3N0B5ZjOzV8oSLtPQQrxUbGhqElOgpK9UqWgwopVqlbp+LlUsN3704gyeeraFL7rFbENjldAid0j10SvcAGS0eGzKGffWB8FkHq3AoP+SMQ2Wdn60VdVTU+vEFQ6S4HDx77YktTnScOiPAf59O4f334bTTjvI3qNQhtBhQSrXK9Onw+uvCWWc5uObiDP76XA1du3X8gsAuhwjZaW6y09zkd05v8VhjDJ9u2ssv3lzLmu1VjMk7/OzL8ZMCpKUZXn1VtBhQcddx198opY4Zp54Kb74p7Nzm4JpZGezdo7PemiMiFOZ7cQgsK6lo8djUNJh4coDiYhP3nR+V0jMDSqmjYsoUqyB49A+CN877D3Rk6R4XQ3tks7y05WIA4MJLfezaIvj9LjyeNkhOJS0tBpRSR01RERQVOaj3Z7BkXS01dYYePfXP2kMV5uXwn0Ul7KsPkJl6+LfhyacEyEwFj0ffqlV86TCBUuqoS3E5ueumTK65KJPtW3XI4FCFfToRMrBia/SzAzt2B5j9mhZUKr60GFBKHXUi8NCvhaoK4TsXZbK1RAuCxoZ0yyTN7Yw6bwDghac9nHO2UFLSBomppKXFgFIqLsaPh3nzhJpqqyAo/VoLgv1cTgejemfbKgamzrB2Tpw9O95ZqWSmxYBSKm7GjYP584WGOuHeu1pedpdsCvM7sb2ynp1V9S0e139giH4DghQX61CBih8tBpRScXXccfDuu8K//ym4nHp2YL/CfGuPATtnB6bMCPDOO1AV59bRKnlpMaCUirsxY2BQfyd9O2fwm/tT2bxR33ryO6XROcNja4nh1Bl+/H7hvffin5dKTvo/UinVZnbvcPLmKx6umZXBpvXJ/fYjIhTmeVleUkEoyq5CheOCvPFBDeec00bJqaST3P8blVJtqm9feO89weUQrpmVwYZ1yf0WVJDvpao+wOY9NS0e53RCXv8A9f5gG2Wmkk1y/09USrW54cOtgsDjFr57cQZfrknet6H98waW25g3sHO7cNll8PHH8c5KJaPk/V+olGo3Q4fC++8LnTsLNdXO9k6n3XTO8NCnc7qtSYRZ2YbiVxy8+GIbJKaSjhYDSql2MXgwrFktXHJuKh6Xg8ry5FxpUJjvZdW2KnyBUIvHpWfAhJO0cZGKDy0GlFLtxuUCt9PBJ3MzOHtKFquWJ99ZgsJ8L75giDU7oq8bnDojwMaNwpo1bZCYSipaDCil2t0pUxx4c+C6b2fwxdLkKghG9srG6RCWbbGx38B0PwCvvhrvrFSy0WJAKdXu+vWz5hDkdoXrL81g+ZLkKQjSPS6Gds9imY39Brr3NMw4009mpo4TqKNLiwGl1DGhTx9rlUGP7nDDpRns3pk8cwgK871s3LWP6np/1GN/8+darrw20AZZqWSixYBS6piRn28VBA89ZOiTn1zFgAFWlFbaOr6s2s+uXfHNSSUXLQaUUseU3r3hhusd9OuSwdoVLhZ+mvhDBoNjaGkMcOa0FK69Ns5JqaSixYBS6pjkdDh4+H/TuenKDD7/OLELApfTwZi8HFt9CgAKxwV4+21DbW2cE1NJQ4sBpdQxSQRefUUYOAC+d1UGn32Y2AVBQZ6X7ZX17IjS0hisxkV1dcK8eW2QmEoKWgwopY5Z3bpZ7Y+HDIabv5PBx++52juluIlla+LjJwTJzDIUF8c7K5UstBhQSh3TcnPhnXeEYUPhzZc9AJTtFV74t4fFnzupSJCdC/M6pdElw2Nr3oDbA5On+pk92xDU3kXqKEjcMlsplTC6drXOEGRmunE6syleEuL+uw8OG3TpGqL/oBB3/LSOEWNCVFdBQ73QJdcgHaRWEBEK8r0s/KqMkDE4oiR+1Q0NXH8DiHjaKEOVyLQYUEp1CJ067b8mnHOWk6++gjVrYPVqWL3awerVwpDe6fTpHOKvL8Edt7jJ8RoGDA4yYHCIAYODnHuRj+ycdvwmohib7+WdtbvYtLuGQd0yWzx2xJgQGSk+HA4tBlTraTGglOpwHA7o29e6zJy5/14BnICTs04Hz+9h9Wph9Won78118uIzHq65SkhssoJ4AAAgAElEQVRPD/GHRxy8OdtlFQlDggwcHGLAoCC9+7TvmYSCvPC8gdKKqMUAwNJlhv8sNtx9Vwc5/aGOWVoMKKUSztCh1sVifVDu3g25uakADOsPi7sYFnzk5NUXrL+s09INyzbuwxcM8d9n3JTtcTBwcJD+g0Pk9w3haoN3y04ZHvqGWxpfcFxe1OMXfOLiVz8XLroQBg2Kf34qcbXrBEIRmSki60Rkg4jc1Uw8RUT+E45/LiL92j5LpVQiyM09eP3qq2HePGHbNqGsDD7+GP79L2FIjyxG9c5h6cdpPPqrVG69NoNzp2YxYWg2N1+dfuDxiz5zsn6tA1/D0c/TamlcSUMg+szAqTOs7Ytnzz76eajk0m5nBkTECTwGzABKgYUiUmyMWd3osGuAcmPMIBH5FvAgcHHbZ6uUSlSdOsGkSU3ve/UVoboa1q615iSsWSNkZbkZ1jOLhkCIM253UloiOJ2GPv2syYtTpvs571vWh7OvATwpR5ZPYb6XV5dvY8326gPLDQ8nr49h8PAgxcUObrtNhwrUkWvPYYLxwAZjzCYAEXkOOBdoXAycC/w8fP2/wB9ERIwx2rJLKRVXWVlwwgnW5SAHbqeD12bvn7hozUlYvdrBjq+c9PI6qakPMWRwKt26G/oPDs9HGBxk7LggffqHor7uyF45uBzCspKKqMUAwNTpfp78YwplZdC585F/vyq5tWcx0BsoaXS7FJhwuGOMMQERqQS6AHsOfTIRuQ64DqBPnz7xyFcppQAoKLAuBwnGCCIppAr86O79hYLw9Afg8wl33+NnxM0+tm4P8ceHPdx9f/M7DaZ5nAztkWVr8yGAqTMC/OepFFatgqKi1n9vKjklzARCY8zjwOMA48aN0zMHSqk2tX8VQkYG3HPPgXsJBGDTJsjKctOzq5u3XoRn/wE33t6At1Pzb1WF+V6e+XwLVXV+stPcLb7uyIIgH35RzZg+2Ufte1HJpz0nEG4F8hvdzgvf1+wxIuICcoC9bZKdUkodBS4XDBkCPXtat4cPt74uXXj4XguFeeGWxlujtzR2OEAchjpfEB1AVUeqPYuBhcBgEekv1hZa3wIO3Wm7GLgyfP1C4B2dL6CU6sjGjwePx7B04eFPzA7unkW6x35L480bHRSMFt5662hlqZJNuxUDxpgA8D1gLrAGeN4Ys0pE7hORc8KH/Q3oIiIbgNuBiOWHSinVkaSmwrhxsGTB4c8MOB3C6N45LCspt/WcvXqHKC0RbVykjli77jNgjHnDGDPEGDPQGPNA+L57jDHF4ev1xpiLjDGDjDHj9688UEqpjqyoSPhyjbPFfQoK873srGpgR2X0lsYpqTBpSoDiYqNDBeqIaNdCpZRqY//zP7BqQ0OLexHsX1Zod6hg6gw/paXCsmVHI0OVbLQYUEqpNtalC3TtfPhhAoDe3jS6ZnpsDxUUTQsgYnj11aORoUo2WgwopVQ7+PeTLh5+IPWwcRGhMN/LitJKgqHo5/47dzFc9/0GThiv4wQqdi0WAyLiFJGH2ioZpZRKFqtWCS/8y0MgcPhjCvK8VDcE2LR7n63nvOmOBiZPbeEJlTqMFosBY0wQmNxGuSilVNIoKoKaGuHL1Yd/Gy7YP2+g1N68AYAVqwM6b0DFzM4wwVIRKRaRy0Xk/P2XuGemlFIJbHL4z6zFCw6/30CndA/9uqTb3poY4IqLPdx5pw4VqNjYKQZSsXb9OxU4O3w5K55JKaVUosvLg379DEtaKAbAWlWwenuVrZbGAFNmBHj3XaiqOhpZqmQRtRgwxlzdzOU7bZGcUkolsjPOENLTWj6mIN+LP2hYvc3ep/vUGX78fmHu3KOQoEoaUYsBEckTkZdFZFf48qKI5LVFckoplcgeewwee8Lf4jGjwi2Nl9ucN1A4Loi3U0h3I1QxsTNM8HesHgG9wpfZ4fuUUkq1UkaKi1Do8PFUt5NhPbJYanPegNMJJ08LMHeuafF5lWrMTjGQa4z5uzEmEL78A8iNc15KKZUULjrXzb0/bHmsoDDfy6bdNVTWtXwWYb+b7qjnw4UNOHQnGWWTnV+VvSJyWXjPAaeIXIa2EVZKqaMiO0tY9Hm0SYSdAFhhc6igZ2+DK133G1D22SkGvgPMAnYA27FaCV8dz6SUUipZFBVByVcOdu+Uwx4zqFsmGTG0NAZ4vVj49qXauEjZE3UHQuB8Y8w5xphcY0w3Y8w3jTFb2ig/pZRKaEVF1telC6O0NM7LYVlJBcbmp/vuXQ6efUZYs+ZoZKkSnZ0dCC9po1yUUirpFBZCerqd/QY6sau6gR1V0VsaA0yZbs0v0MZFyg47wwQfi8gfRKRIRI7bf4l7ZkoplQTcbrjrLjh+Qstj/IV5sbU07t7TMLIgQHGxjhOo6FouRS2F4a/3NbrPYO1IqJRSqpV++lNh8x7Y18If/b28qeRmpbCspIJvjOpp63mnTg/wx4ed7NgBPXocpWRVQmqxGBARB/AnY8zzbZSPUkolpbIdTirqg3Tr0fxf8iJCYZ6XTzftJRgyOB2Hn3C43ymn+1n4iZvdu51aDKgWRZszEAJ+2Ea5KKVUUqqpgQmFKTz3lKfF4wryvexrCLDRZkvjIcND/OPFGkaPPhpZqkRmZ87APBG5Q0TyRaTz/kvcM1NKqSSRkQHHHQdLFrY8cluQlwMQUxfDQNCwbVeAhoZWpagSnJ1i4GLgJuADYHH4siieSSmlVLIpKhJWLnPia+FD25vuoX/XjJj2G1i9wkHf3k5tXKRaZKdrYf9mLgPaIjmllEoWRUXgaxBWrTj8fgMABXlWS+N6v72WxoOHh0hLRxsXqRbZ6VqYLiI/EZHHw7cHi8hZ8U9NKaWSx0knWV+j7TcwNt9LIGRYvd1eS2O3G06a6mf2bG1cpA7PbtdCHzApfHsr8L9xy0gppZJQbi68+FKIcy70tXjciF7ZuBwS01DB1BkBdu0SFixobZYqUdkpBgYaY34F+AGMMbVA9DUtSimlYnL+eQ7y8lp+e011OxneMzumSYSTp/pxuYwOFajDslMM+EQkDWujIURkIKDzUpVS6ijbuxeefyqF0i0tFwSF+V427bHf0jjbCw88UseVV+luhKp5doqBnwFzgHwReRqYj+49oJRSR11VFfz0Tg8fvetu8bjCfGtr4ljODnzjXD/d8+wVDyr52FlN8DZwPnAV8CwwzhjzXnzTUkqp5NOvH/TubViyoOUVBQNzM8lIcbKs1H4xYAw8/YzhjTdamaRKSHZ6E2CM2Qu8HudclFIqqYlY+w28+74LY6zbzXE6hDG9vQdaGsvhDjzkuX/3sIte3QxnnKHTvlRTdoYJlFJKtZGiIti53cG20ujzBnZXN7C90l5LY4Cp0/189JE1N0GpxrQYUEqpY8jkySBi+HJNy0MF++cNxLLE8JTTAgSDwptvtipFlYAOWww07kPQ3KUtk1RKqWQxahRs2xnilNMCLR7XMyeVbuGWxnaNGBMkt3tIlxiqCC3NGViMtZxQgD5Aefi6F9gC9I97dkoplWQcDuiR62TvNlrcMVBEKMj38smGPbZbGjscMGW6n/WrPRgjh52ToJLPYc8MNOpBMA842xjT1RjTBTgLeKutElRKqWTz8cdww6WZlO1t+dN6bL6XGl/QdktjgDt/Xs+cdxu0EFBN2JkzcKIx5sBiFGPMmxzcmlgppVQcfPSek2ULW543MCbPmjewNIahgpRUqG7wY3T/IdWInWJgW7hRUb/w5cfAtngnppRSyWrcOEhJMVGbFuWkuRnQNSOmzYcAnviTi4ICowWBOsBOMXAJkAu8DLwUvv6teCallFLJLCUFxo8n6uZDYK0qWBNDS2OAzEzDF18IS5e2JkuVSOwUA9OMMbcYY8YaY44zxtwKTI93YkoplcyKioQ1K53U1rR8XEG4pfGqbfZaGgOcPD2AiDYuUgfZKQbutnmfUkqpo+TUU+HESaGokwhH9Iy9pXHnLobCcUGKi3WcQFkOOyAlIt8AzgB6i8jvG4WygZYXwCqllGqVadNg/EkhvtrT8gd2qtvJiF7ZLI+hTwHA1NP8/PYBFyUlkJ/fmkxVImhpdso2YBFwDtaeA/tVA7fFMymllFKQ7nHhawBPSsvHFeZ5+ednX1Ne66NTusfWc087PUB1mR9ouUOiSg4t7TOw3BjzFDDIGPNU+HoxsMEYU95mGSqlVJJ6+DfC1LHZ+KN0Hi4Ib028orTS9nP36R/itp/UkZenQwXK3pyBt0UkO7wF8RLgCRH5bZzzUkqppNevH+yrFtaujN7SODPFxbKS2P5Oa/AZ3pofpLq6FUmqhGCnGMgxxlQB5wP/NMZMAKbFNy2llFKTJ1tfl0bZfMjpEMbk5bCspBITw+YBK5Y4mTnDxZw5rclSJQI7xYBLRHoCs4DX4pyPUkqpsJ49YeDA6JsPgbXfwJ59DWyrsN/SuOD4IN5O2rhI2SsG7gPmYs0VWCgiA4D18U1LKaUUWPsNLF3ojLpb4MGWxvaHCpxOOHlagNdfN1HnJajEFrUYMMa8YIwZY4z5f+Hbm4wxF8Q/NaWUUpdfDjffHiAQ5cO6R3a4pfERLDEsLxc+/rgVSaoOz86ZAaWUUu3k1FPh1lvBHWXFoIgwNt/LF6WVBEP25w1MOjmA22N4443ox6rEpcWAUkod4yrLnHyxNHqfgoJwS+P1u+wvD0jPgGdm7+Pe+0KtSVF1cFoMKKXUMe4Htzr54U3pUY8bk+dFIOYuhkNHhKgP6sayySymYkBEdDWBUkq1saIi2FriYMf2lvsU5KS5GZCbEVOfAoBQCO6+G558sjVZqo4s1jMDveOShVJKqcM6sN+AzSWGa3dUU+ez39LY4YCPPnDwxBO6G2GyirUY0O7XSinVxgoLITPTsGSBjXkDeeGWxtvtb00MMHV6gM8/hx07jjRL1ZEdthgQkcdF5DwRydp/nzHmO22TllJKqf1cLpg4EZYsjH5mYESvbNxOYdmW2IYKTjndjzHC668faZaqI2vpzMDfgALgDRGZLyJ3ikhBG+WllFKqkYceEp74Z0PU41JcTkb0jL2l8eBhIXrl6W6EyaqlroWfG2N+bowpwtqKeAvwAxFZKiJPisisNstSKaWS3JgxMHyIvZHdgnwvX+2tpbzGZ/v5ReDM83x4O+sSw2Rk6zfLGLPXGPOsMeYKY8xY4DFgcHxTU0op1dhLz7mZO9sd9bix+Z0AYj47cPMPG3jwEfsFhEocticQisiM/deNMYuNMQ/EJyWllFLNeervDp75e5StCIH+XTPISnHFvMQQoLrez969R5Kd6shiWU3wYNyyUEopFVVRkbByuZOGKI0JnQ5hTL6X5aUVMbU0Brj/Jx5GjDAE7a9MVAmgXXYgFJHOIvK2iKwPf+10mOOCIrIsfNFpLUqppDZ5Mvh9whfLoi8xLMzzsmefj9KKupheY2RBkF27hAULjjRL1RG1WAyIyN/DkwX/DvQJX39SRFq7T9VdwHxjzGBgfvh2c+qMMYXhyzmtfE2llOrQTjrJ+rrUxhLD/S2NY92aePIpflwuo6sKkky0MwP/AJ4Kfy0PX99/aY1zGz3HU8A3W/l8SimV8Lp0gVGjDNu3trwtMUCPnFR6ZKfGPG8gOweOPzFIcbHuRphMWiwvjTHv778uItWNb7dSd2PM9vD1HUD3wxyXKiKLgADwS2PMK0fp9ZVSqkNauFAoqfRR749+bEG+lw/X7yYYMjgd0QuI/abO8PPgz1xs2ACDBrUiWdVhRD/XdFBM601EZB7Qo5nQjxvfMMYYETlcCdrXGLNVRAYA74jIF8aYjYd5veuA6wD69OkTS6pKKdVhpKZCer2Len/0t+TCfC9zV+1g/c5qhvXMtv0a07/hp2c3B927p7QmVdWB2J5AaIw5MZYnNsZMN8aMaubyKrBTRHoChL/uOsxzbA1/3QS8B4xt4fUeN8aMM8aMy83NjSVVpZTqMOrq4DuXpPDyc9H3GxjTOwcBlsW430D3nobp59STmalDBcmiXVYTAMXAleHrVwKvHnqAiHQSkZTw9a7AScDqNstQKaWOQWlpsHa18PH70U/sZqe5GZibeUT7DezeKfzy1yHdcyBJtFcx8EtghoisB6aHbyMi40Tkr+FjhgOLRGQ58C7WnAEtBpRSSW/yZGHJAhd2thAoOIKWxgDbSh386E4nb7xxhEmqDqVdioHw9sbTjDGDw8MJZeH7Fxljvhu+/okxZrQxpiD89W/tkatSSh1riopgzy4HpV9Hfwsfm+8lGDKs3BZbS+ORBUFyu2vjomRhqxgQkb4iMj18Pa1xW2OllFJtq6jI+rpkQfTNh4b3zMbjdMQ8VOBwwJTpfubMMTREb5aoOrioxYCIXAv8F/hL+K48QJf4KaVUOxk+HE473ZBuY4Kfx+VgRK/smDcfApg6I8C+fcJ77x1BkqpDsXNm4CasyXtVAMaY9UC3eCallFLq8BwOmDtHmHmWvXkAhflevi6rpSyGlsYA4ycFyMwyrFp1JFmqjsROMdBgjDnwGyQiLkDXmyilVDtzhpz4bJzCL8gLb00c4xLD1DSYv7iK730/dCTpqQ7ETjHwvoj8CEgLtzF+AZgd37SUUkq1ZOVKGD0gnffnR19iOCA3g6zUI2tpnJZmtTVWic1OMXAXsBv4ArgeeAP4STyTUkop1bIhQ8DlgiULohcDDhEK8rwsK4m9pXEgAOee5eKBB440U9URROta6AT+ZYx5whhzkTHmwvB1HSZQSql25PHAhAn2igGw5g2U1fgoLY+tpbHLBbW1hhdf1Lf9RNZiMWCMCQJ9RcTTRvkopZSyqahIWLfKQc2+6Mfub2l8JEMFU2b4WbpUKCmJ+aGqg7AzTLAJ+FhEfioit++/xDsxpZRSLZs8GUIhYfni6GcHumen0jMn9pbGAKecFgBgts4WS1h2ioGNwGvhY7MaXZRSSrWjiRPhvv8N0neAvSWGBXlevthaSSAY2+qA/gND9BsQpLhYhwoSVdRy0hhzL4CIZIZv2zghpZRSKt6ysuDHdztYvd3Y6lNQmO9lzqodfLlrHyNiaGkMcPm1PrI8HiD6roeq47GzA+EoEVkKrAJWichiERkZ/9SUUkpFU1UlfDTPg9/GfkJj8qyWxkeyG+FFl/m47JrYNi1SHYedYYLHgduNMX2NMX2BHwBPxDctpZRSdsyfDzdelcbqldH/Ys9KdTOw25G1NAYo3RHg00+P6KHqGGenGMgwxry7/4Yx5j0gI24ZKaWUsm3yZOurnaZFYHUxXLezmlpfIObXeuCnKZx5psGvexAlHFurCcIrCfqFLz/BWmGglFKqnXXvDkOGGJZ8bm+/gYL9LY23VsX8WlNm+CkvFz7+OOaHqmOcnWLgO0Au8BLwItA1fJ9SSqljwOTJ1vLCkI1FAsN7ZONxOWLuUwAw6eQAnhRDcfERJKmOaVGLAWNMuTHm+8aY44wxxxtjbjXGlLdFckoppaIrKoKKcmHT+uh/33lcDkb2zGbpEcwbSM+ACScFKC62t3pBdRx2VhO8LSLeRrc7icjc+KallFLKrnPPhU8X+Rg4xN7+AYX5XkrKatm7z0bLw0NMnRFg40bhyy9jfqg6htkZJuhqjDlQQobPCnSLX0pKKaVi0akTjC1wImLv+P1bEy8vrYz5tU4/y8f8T2sZOjTmh6pjmJ1iICQiffbfEJG+gJ4gUkqpY8inHzn5zf2pto7t1zWD7FQXy0piH/HN9kK3fD+hkH4MJBI7xcCPgY9E5F8i8m/gA+Du+KallFIqFl98AU89nsL2rdFPDzhEKMj3srykMuaWxgDr1zr41rcNO3YcSabqWGRnO+I5InIccGL4rluNMXvim5ZSSqlYHNxvwMWZ50XfCKAw38uH6/dQUl5Hn87pMb1WKAQv/MfBlq+gW3jQ+NRT4dZbrevnnw+BQ7YxOPNMuP568Pvhggsin/PCC+GKK6CqCi67LDJ++eVw0UWwcydce21k/Lrr4Kyz4Kuv4Pvfj4zfcgtMmwZr1sCdd0bG77oLJk2CJUvg5z+PjN97L4wdCx9/DA8+GBn/1a9g2DBrE6jf/S4y/uij0LcvvPYaPP54ZPyJJ6xloi+8AP/6V2T83/+G7Gz45z/hv/9tGnPZW1XaoqhPISInAcuMMa+JyGXAj0Tkd8aYr1v/8koppY6GMWMgO9uwZIHTXjGQt7+lcXnMxcDgYSHO+KaPzRucVG227uu9xc+qbdaExPWbMwgGmp6hWLfFx6ptPvw+2LA5M+I515VY8cqK5uNrSxpYtc3Prh3Chs2R+96tLWmg/zY/X5c62LA58vtZU1JPj20B1h42XkfOtiCrS5xs2JwWEV9dUoene5A1h4uX1hLMDrGmxMWGzZHDNatLa9nnDrGmxM2GzSmR8a017Aka1mxpPr5q6z6y98HaLR42bPY0iTmPQjEg0U4RicgKoAAYA/wd+BswyxgzpfUvHx/jxo0zixYtau80lFKqTX3jG4b1m0K8PN9eP7nr/rWIvE5p3HOWtpvpyERgdJ53sTFm3JE+h505AwFjVQznAo8ZYx5DWxgrpdQxZ/JkIRQUGurtHV+Y72Xl1qqYWxqrxGOnGKgWkbuBy4DXRcQBuOObllJKqVjdfTd8ttRHir1FBRTme6nzB1m3szq+ialjnp1i4GKgAbjGGLMDyAN+HdeslFJKxczhgAyP/QHkMb29OOTIWhqrxGJnO+IdxpiHjTEfhm9vMcb8M/6pKaWUitX/3efk5qvtTQjMTHUxqBUtjVXisHNmQCmlVAcRCgmfvO+irs7e8QV5R97SWCUOLQaUUiqBTJ4Mfr+wcpnT1vFj872EDKzcGvvWxCpxHLYYEJH/EZG8tkxGKaVU65x0EogYlnxub+7AsJ5WS+Mj6WKoEkdLZwZ6AZ+KyIci8v9EJLetklJKKXVkvF4YPRqWLrR3ZsDtdDCqV7ZOIkxyhy0GjDG3AX2AnwCjgRUiMkdErhQR3WdAKaWOUVdeKYw93v7eAYX5XkrK646opbFKDC3OGTCW940xN2ItKfwtcCuwsy2SU0opFbvbb4cf3RNbMQDoqoIkZmsCoYiMBu4DHsPac0C7Fiql1DHMjYu9u6N3MATo2yWDnDQ3y0q1GEhWh51hIiKDgUuwNh0KAs8BpxljNrVRbkoppY7QyZNc9O6Xxm8fr416rEOEgjwvy0sqMMYgYq+IUImjpTMDcwAPcLExZowx5v+0EFBKqY7hhHHCsoVOovSiO6AwP4fyWj9byqIXDyrxtFQMzATmGGNWNr5TRE4SkYHxTUsppVRrTJ4Me/c4+Hqzve1kCnTeQFJr6bfkt0Bzu1BUAY/EJx2llFJHQ1GR9XXJAntLDLtlpdLbm6bFQJJqqRjoboz54tA7w/f1i1tGSimlWm3YMOjSxf7mQ2CdHVi5rRK/tjROOi0VA94WYmlHOxGllFJHjwj88Y/CJVf5bD+mMC+Hen+IL7WlcdJpqRhYJCLXHnqniHwXWBy/lJRSSh0Ns2bBxBPtrwwYnWe1NNatiZNPS+ePbgVeFpFLOfjhPw5rhcF58U5MKaVU6/j98Mm7Hjw5IYYMj37qPzPFxeBuWSwvqeCyCX3bIEN1rGhpO+KdxphJwL3AV+HLvcaYicaYHW2TnlJKqdb47pUuXnrWY/v4wnwvX+6spqZBWxonk6hrTowx7xpjHg1f3mmLpJRSSrWe2w0TJ8KSBbFNIgwZ+EJbGicVewtQlVJKdUiTJwvrVjuorrJ3/LAeWaS4HNrFMMloMaCUUgmsqAiMEZYvtnd2wO10MKp3jvYpSDJaDCilVAI78URwuQzLl9jbfAigMM9LaXkde7SlcdKwP5CklFKqw8nIgJUroSHd/gd7462Jpw/vHq/U1DFEzwwopVSCGzpUSE+xf2agX5d0vGlu3Zo4iWgxoJRSCa60FO67K5XVX9h7yxcRCvIPtjRWiU+LAaWUSnApKfCvv7v49AO37ccU5nmpqPPz9V5taZwMtBhQSqkEl5sLw4YZli60P1SgLY2TixYDSimVBCZPFpYtchGy2ZAwNyvFammsSwyTghYDSimVBIqKoKpS2LDO/tt+Yb6XlVu1pXEy0GJAKaWSQFER5Pcx7N0dWzHQEAixdoe2NE50WgwopVQS6N8fNmw0TDzZfgOi0b1zcAi6NXES0GJAKaWShMflwOUU7K4WzEhxMaR7lk4iTAJaDCilVJKYOxemjs1ka4nYfkxBvpf1u6rZpy2NE5oWA0oplSR69oRdOx0xtTQuzNOWxsmgXYoBEblIRFaJSEhExrVw3EwRWSciG0TkrrbMUSmlEs2oUeD1GpYutF8MDO2RRarboUMFCa69zgysBM4HPjjcASLiBB4DvgGMAC4RkRFtk55SSiUehwNOOklYssD+5kNup4NRvXJ0EmGCa5diwBizxhizLsph44ENxphNxhgf8BxwbvyzU0qpxDV5Mmze4KRsb2zzBrZW1LGruj6Oman2dCzPGegNlDS6XRq+r1kicp2ILBKRRbt37457ckop1RHNnAnXXB/A77P/mLHhrYn17EDiilsxICLzRGRlM5e4/HVvjHncGDPOGDMuNzc3Hi+hlFIdXmEh/P730L2n/W6EfTqn4013s6xEJxEmKvuzSGJkjJneyqfYCuQ3up0Xvk8ppVQruMXJ5g0O+g+yt82wiFCY52VpSQUhY3CI/SEG1TEcy8MEC4HBItJfRDzAt4Dids5JKaU6vPvvFy6YkUltDN2JC/O9VNb5+XpvTfwSU+2mvZYWnicipcBE4HURmRu+v5eIvAFgjAkA3wPmAmuA540xq9ojX6WUSiSTJkEgIKxYoi2NlaW9VhO8bIzJM8akGGO6G2NOD9+/zRhzRqPj3jDGDDHGDDTGPNAeuSqlVKKZNAkcDhPT5kNdM1PI65Sm8wYS1LE8TKCUUioOsrNhzBhi2nwIwi2Nt2lL40SkxYBSSiWhoiJhxWInftQTTW8AAA0BSURBVL/9xxTme/EFQqzdXhW/xFS70GJAKaWS0LXXwpNP+4hlYcD+lsZLdd5Awonb0kKllFLHrtGjoc9AB1vK7D8m3eNiaPcslpdqMZBo9MyAUkolqeWLnbzxijumxxTke9mwax/76rWlcSLRYkAppZLUk39z8ODPUjH2NyOkMN9qabxiq54dSCRaDCilVJIqKoLyMgebN9j/KBjaPYs0t1P3G0gwWgwopVSSKiqyvsbS0tjldDCyV7Y2LUowWgwopVSSGjwYunWLbfMhgLF9vGyrrGdXlbY0ThS6mkAppZKUCEyeLKxYZf/MAEBBnrU18W/nfUnXrJR4pPb/27v3IDvr+o7j7+/uZllzkQKpAZrGyBAQqCWGnVBK1vFClYshVXHaQme4WGOtl3aoHe3Q0Q5OkXZwGMEq0A7aGYu2DZCCtVAoOKlWBpJsgrFJIIQoSQGDDSTkutn99Y9zcttzzmbP5jz77Hme92tmZ3fP7zkn32+enPN88ntuakIrbhtlGJCkErvzTtg+uJedTVx8aNaJk+l98wls3rabV17fl11xGjeGAUkqsenTgdc72fnq6NNARPCFhedkV5SaEgFvu+HYXsNjBiSp5P76i5O4++vdeZehHBkGJKnkVvcH37vfMFBmhgFJKrm+vuDZdR1s9+7EpWUYkKSS6+uDlIJVyz2MrKwMA5JUcvPnw6RJif6nmjvFUMVhGJCkkps8GS69FLo9bKC0nBOSJLF0afDc1v3s2pt3JcqDMwOSJACmdHex3zsTl5JhQJLEnj1w/rnHcffXvLxwGRkGJEn09MC0qXgQYUkZBiRJQOWmRatXdDE4mHclGm+GAUkSULnewOs7gmfWumkoG9e4JAmohAGAlU96olnZGAYkSQDMmgXX/2nizLPdT1A2xj9J0kFfviVY99IQA55iWCrODEiSDhoagp89O4nXtkXepWgcGQYkSQetXQvve8cb+P4jThyXiWFAknTQWWfBiScmVj5lGCgTw4Ak6aCODrjwwqD/SS8+VCaGAUnSEfr6YNPGTn6x1eMGysIwIEk6wsHrDXhp4tJwp5Ak6Qjz5sG9Swc55QzPLywLZwYkSUfo7oYPXN7BtDfmXYnGi2FAklTj+eeDu7/aw87X865E48EwIEmqsWED3Hrzcaxe4d7kMjAMSJJqXHABdHQk+j2IsBQMA5KkGtOmwdy53sGwLAwDkqS6+vqCp/s7GdiXdyXKmmFAklRXXx8E8NPn3VQUnfM/kqS6Fi6En720n5d2DOVdijJm3JMk1dXdDSdM9QDCMjAMSJIauu/eDj525RSGnBwoNMOAJKmhXbvgR//VxcZn3VwUmWtXktTQwZsWeYphoRkGJEkNnXYanHKKFx8qOsOAJKmhCFiwIJwZKDjXriRpRJddBtt3DbFnN/S8Ie9qlAVnBiRJI7r6avjWdwYNAgVmGJAkHdXU47rYvTvvKpQVw4Ak6ag+vriDqxZOzbsMZcQwIEk6qtNPDzas7+TVbZF3KcqAYUCSdFQHrjfgKYbFZBiQJB3V/PnQ3Z3of8qT0IrIMCBJOqqeHujthZVPOjNQREY8SdKoXH99sOnl/XmXoQwYBiRJo/KhD8FLr8HWHXlXolZzN4EkadS2/LSTtWvcdBSNMwOSpFG79ve7mPpLPdx5z668S1EL5RLvIuLDEfGTiBiKiN4RltsUET+OiFURsXw8a5Qk1errC55e2cV+Dx0olLzmetYAHwSWjWLZd6WU5qaUGoYGSdL46OuDnTuDZ/7HXQVFksvaTCmtTSmtz+PPliSN3YIFle8rvKVxoUz0aJeA/4iIFRGxeKQFI2JxRCyPiOVbt24dp/IkqVxmzoTZs734UNFktjYj4lHg5DpDN6SU/nWUL7MgpbQlIt4EPBIR61JKdXctpJTuAu4C6O3tTWMqWpJ0VEuWBAM9e/MuQy2UWRhIKV3UgtfYUv3+84i4H5jP6I4zkCRl5Lzz4IX/6+DVXYN5l6IWmbDzPBExBehIKe2o/vxe4Macy5Kk0tu7F+74SjePPNrN6WccCgRn//ogiz48AMAtN/YwMHDk8+b2DnLJogFSgps/31PzuvN/cz/vuWQ/e3bDrTfVji9493763rWf7a/B395SO/7uiwc4/8JBfrE1uOu242rGL1k0wNzeQV7cEnzzjtrxy68Y4JxzB9m0sYNvf6O7ZvyKq/Yx561DPLO2g3vvqR2/8rp9vPktQ6xZ1cmD906qGb/243s5+dTEyic7efjB2vHFf7yXk6YnnvhBJ48/XDv+yT/bw7Q3wrL/7OKH3z+0+e5swRWicwkDEfEB4Hbgl4F/i4hVKaX3RcSpwN+nlC4FZgD3R8SBOu9JKT2UR72SpEO6u2HZY508tw6eW3doSzS5KzFnRmUj+9hDHezaeeTzph9fGU8JHn6g9pC12adOYs6MxI4d9cfPOaMy/uJQ/fHz51XGY3v98ff0VcZ3/m/98YUXdzFnBrz8bP3x37uiMr6xv/74H1xTGX+6Qf1/8onK+BPbgocfqL0V9F98tovTZsCjL9cfv+nGLk6eAUs3Hzk+qTY3NC1SKt7u9d7e3rR8uZclkCSVQ0SsOJZT8Cf62QSSJCljhgFJkkrOMCBJUskZBiRJKjnDgCRJJWcYkCSp5AwDkiSVnGFAkqSSMwxIklRyhgFJkkrOMCBJUskZBiRJKjnDgCRJJVfIuxZGxA5gfd51ZGQ68EreRWTI/tqb/bWvIvcGxe/vzJTStLE+uauVlUwg64/lVo4TWUQsL2pvYH/tzv7aV5F7g3L0dyzPdzeBJEklZxiQJKnkihoG7sq7gAwVuTewv3Znf+2ryL2B/Y2okAcQSpKk0SvqzIAkSRolw4AkSSVnGJAkqeQMA5IklVypwkBEzIqIpRFxd0R8Lu96Wi0iOiLiryLi9oi4Ou96shARUyJieUS8P+9aWi0ifjsi/i4i/iki3pt3Pcequq7+odrTVXnX02pFW1/1FPz9VujPy2a3d20TBqoN/Twi1gx7/OKIWB8RG0bR8NuAJSml64C3Z1bsGLSov0XATGAA2JxVrWPRov4APgv8czZVjl0r+kspLU0pfRT4Q+B3sqx3rJrs84NU3m8fBS4f92LHoJn+2mF9DTeGf6cT8v3WSJP9TdjPy0aa7K+57V1KqS2+gHcA84A1hz3WCTwHnAZ0A6uBs6t/Cd8d9vUm4CTgceAx4Nq8e8qgv88BH6s+d0nePWXQ328BvwtcA7w/755a3d9hz/syMC/vnlrQ558Dc6vL3JN37a3urx3W1zGuvwn7fmtRfxP287JF/TW1vWubexOklJZFxOxhD88HNqSUNgJExHeARSmlLwE101oR8RngC9XXWgJ8I9uqR69F/W0G9lV/Hcyu2ua1qL93AlOo/EPfHRHfSykNZVn3aLWovwBuBv49pbQy24rHppk+qfxvayawijaZhWymv4hYywRfX8M1uf6mMkHfb4002d8LTNDPy0aa7G+AJrZ3bRMGGvgVKiv0gM3A+SMs/xDwlxFxJbApw7papdn+7gNuj4g+YFmWhbVIU/2llG4AiIhrgFcm+gcTza+/TwEXAcdHxOkppTuyLK6FGvV5G/DViLgMeDCPwlqkUX/tur6Gq9tfSumT0Fbvt0Yarb+v0F6fl4006u8OmtjetXsYaEpKaQ1wRd51ZCWltAv4SN51ZC2l9M28a8hCSuk2KhvQQkgp7QSuzbuOrBRtfTVS4PdboT8vm93etcXU3Qi2AL962O8zq48Vhf21t6L3d0DR+7S/9mZ/o9DuYeApYE5EvCUiuqkc7PJAzjW1kv21t6L3d0DR+7S/9mZ/o5H30ZFNHEX5beBFDp0G8pHq45cCz1A5mvKGvOu0P/srYn9l6dP+7G8if2XZn3ctlCSp5Np9N4EkSTpGhgFJkkrOMCBJUskZBiRJKjnDgCRJJWcYkCSp5AwDkpoSEZsiYvqxLiNp4jAMSJJUcoYBSQ1FxNKIWBERP4mIxcPGZkfEuoj4x4hYGxFLImLyYYt8KiJWRsSPI+Kt1efMj4gfRUR/RPx3RJw5rg1JqsswIGkk16WUzgN6gU9HxEnDxs8EvpZSOgvYDvzRYWOvpJTmAV8HPlN9bB3Ql1J6O/B54KZMq5c0KoYBSSP5dESsBp6gcme0OcPGX0gp/bD687eABYeN3Vf9vgKYXf35eOBfImINcCtwThZFS2qOYUBSXRHxTuAi4IKU0rlAP9AzbLHhNzc5/Pe91e+DQFf15y8Cj6eUfg1YWOf1JOXAMCCpkeOBbSmlXdV9/r9RZ5lZEXFB9ecrgR+M4jUP3Gv9mpZUKemYGQYkNfIQ0BURa4GbqewqGG498InqMidQOT5gJH8DfCki+jk0WyApZ97CWNKYRMRs4LvVKX9JbcyZAUmSSs6ZAUmSSs6ZAUmSSs4wIElSyRkGJEkqOcOAJEklZxiQJKnk/h/NdEM0e5xUbgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.linear_model import LassoCV\n",
    "from sklearn.linear_model import Lasso\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "lasso = Lasso(random_state=42)\n",
    "alphas = np.logspace(-8, 8, 10)\n",
    "\n",
    "scores = list()\n",
    "scores_std = list()\n",
    "\n",
    "n_folds = 3\n",
    "\n",
    "for alpha in alphas:\n",
    "    lasso.alpha = alpha\n",
    "    this_scores = cross_val_score(lasso, x, y, cv=n_folds, n_jobs=1)\n",
    "    scores.append(np.mean(this_scores))\n",
    "    scores_std.append(np.std(this_scores))\n",
    "\n",
    "scores, scores_std = np.array(scores), np.array(scores_std)\n",
    "\n",
    "plt.figure().set_size_inches(8, 6)\n",
    "plt.semilogx(alphas, scores)\n",
    "\n",
    "# plot error lines showing +/- std. errors of the scores\n",
    "std_error = scores_std / np.sqrt(n_folds)\n",
    "\n",
    "plt.semilogx(alphas, scores + std_error, 'b--')\n",
    "plt.semilogx(alphas, scores - std_error, 'b--')\n",
    "\n",
    "# alpha=0.2 controls the translucency of the fill color\n",
    "plt.fill_between(alphas, scores + std_error, scores - std_error, alpha=0.2)\n",
    "\n",
    "plt.ylabel('CV score +/- std error')\n",
    "plt.xlabel('alpha')\n",
    "plt.axhline(np.max(scores), linestyle='--', color='.5')\n",
    "plt.xlim([alphas[0], alphas[-1]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# L2 (Ridge) Regularization\n",
    "\n",
    "You should use Tikhonov/Ridge/L2 regularization when you are less concerned about creating a space network and are more concerned about low weight values.  The lower weight values will typically lead to less overfitting. \n",
    "\n",
    "$$ E_2 = \\alpha \\sum_w{ w^2 } $$\n",
    "\n",
    "Like the L1 algorithm, the $\\alpha$ value determines how important the L2 objective is compared to the neural network’s error.  Typical L2 values are below 0.1 (10%).  The main calculation performed by L2 is the summing of the squares of all of the weights.  The bias values are not summed.\n",
    "\n",
    "The following code uses L2 with linear regression (Ridge regression):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 2.9413130283355713\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jheaton/miniconda3/envs/wustl/lib/python3.6/site-packages/sklearn/linear_model/ridge.py:125: LinAlgWarning: scipy.linalg.solve\n",
      "Ill-conditioned matrix detected. Result is not guaranteed to be accurate.\n",
      "Reciprocal condition number3.879872e-09\n",
      "  overwrite_a=True).T\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>positive</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>origin-1</th>\n",
       "      <td>-1.420853</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cylinders</th>\n",
       "      <td>-0.409656</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>horsepower</th>\n",
       "      <td>-0.007525</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>-0.007473</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>displacement</th>\n",
       "      <td>0.024128</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acceleration</th>\n",
       "      <td>0.137980</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin-3</th>\n",
       "      <td>0.619181</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <td>0.800767</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin-2</th>\n",
       "      <td>0.801894</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  coef  positive\n",
       "origin-1     -1.420853     False\n",
       "cylinders    -0.409656     False\n",
       "horsepower   -0.007525     False\n",
       "weight       -0.007473     False\n",
       "displacement  0.024128      True\n",
       "acceleration  0.137980      True\n",
       "origin-3      0.619181      True\n",
       "year          0.800767      True\n",
       "origin-2      0.801894      True"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept: [-18.230455]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAasAAAD8CAYAAADJ7YuWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAGzRJREFUeJzt3Xm0XGWd7vHvw9AiBBlMpBGJRwbNBZU0KRBkaFQuDo0YBBaibcsYdbUjN9xlL4VuaGmnvq1e+yIdowbutdFGE0GalklDIBDJOSETIg4M3bJQD4o0wWYwPPePeoPF4QyVM9V7Ks9nrVq1a+93v/tX+5ycJ++uXXvLNhERETXbqtMFREREjCRhFRER1UtYRURE9RJWERFRvYRVRERUL2EVERHVS1hFRET1ElYREVG9hFVERFRvm04X0C2mT5/unp6eTpcRETFl9PX1PWh7RjttE1bjpKenh97e3k6XERExZUi6r922OQwYERHVS1hFRET1ElYREVG9hFVERFQvJ1hERBWkTlcQozFZt0TMyCoiIqqXsIqIiOpN2bCSdLWknUdoc4Gkozez369JukvSeklfkbTt2CqNiIixmnJhpaatbL/J9m+Ha2v7PNvXb+YmvgbMAl4BPBc4c5SlRkTEOKkyrCSdXUY26yV9SFJPGe1cCqwH9pR0r6Tppf25ZfnNki6TNL/MXyTpxDJ9r6TzJa2StE7SrMG2bftqF8BtwIsm511HRMRQqgsrSXOA04BXAYcAZwG7APsCF9ne3/Z9Le0PAk4ADgDeCDSG6f5B2wcCXwTmj1DHtsA7ge8O02aepF5Jvf39/e28vYiIGIXqwgo4HFhi+1HbG4DFwBHAfbZXDNL+MOAK24/ZfgT4zjB9Ly7PfUDPCHVcBCyzfdNQDWwvsN2w3Zgxo61rMUZExCjUGFZDeXQc+ni8PG+kfMdM0jWSVktauKmRpL8GZgBnj8M2IyJijGoMq5uAuZK2l7QDcHyZN5TlwJslbSdpGnDs5mzM9uttz7Z9JoCkM4HXA6fYfmp0byEiIsZTdVewsL1K0iKaJzcALAQeGqb9SklXAmuBXwLrgIfHUMLFwH3ArWp+pX6x7QvG0F9ERIyRPFnXyphAkqbZ3iBpe2AZMM/2qsmsodFoOPezihi9XG5pahpLhEjqsz3cSXFPq25kNUoLJO0HbAdcMtlBFRFj1wX/b44J1BVhZfvtna4hIiImTo0nWERERDxDwioiIqqXsIqIiOolrCIionoJq4iIqF7CKiIiqpewioiI6iWsIiKiegmriIioXsIqIiKql7CKiIjqdcW1ASNi6stV16emyboAcUZWERFRvYRVRERUL2HVJklbd7qGiIgtVVeGlaQLJH2o5fWFkj4o6RxJKyWtlXR+y/JvS+qTdIekeS3zN0j6X5LWAIdO8tuIiIiiK8MK+ArwFwCStgLeBvwC2Bc4GJgNzJF0ZGl/uu05QAP4gKTnl/k7AD+wfYDtmyfzDURExB905dmAtu+V9GtJfwLsBtwOHAQcU6YBptEMr2U0A+r4Mn/PMv/XwEbgW0Ntp4zC5gHMnDlzAt5JRERAl4ZVsRA4FfhjmiOt1wGfsP1PrY0kHQUcDRxq+3eSlgLblcWP2d441AZsLwAWADQajUk6gTMiYsvTrYcBAZYAb6A5orqmPE6XNA1A0h6SXgDsBDxUgmoWcEinCo6IiMF17cjK9hOSvg/8toyOrpX034Bb1fz24Qbgz4HvAu+RdCdwF7CiUzVHRMTgujasyokVhwAnbZpn+/PA5wdp/sbB+rA9bWKqi4iIzdGVhwEl7Qf8FLjB9k86XU9ERIxNV46sbP8Q2KvTdURE+ybrGnMxNXXlyCoiIrpLwioiIqqXsIqIiOolrCIionoJq4iIqF7CKiIiqpewioiI6iWsIiKiegmriIioXsIqIiKql7CKiIjqdeW1ASNi6mneuSfGUzddbzEjq4iIqN6UDitJV0vaeYQ2F0g6ejP7/bKkNZLWSvrmprsLR0REZ0zJsFLTVrbfZPu3w7W1fZ7t6zdzEx+2fYDtVwL/Drxv1MVGRMSYVRtWks6WtL48PiSpR9Jdki4F1gN7SrpX0vTS/tyy/GZJl0maX+YvknRimb5X0vmSVklaJ2nWYNu2/Z+lvYDnAl105DciYuqpMqwkzQFOA15F89b0ZwG7APsCF9ne3/Z9Le0PAk4ADqB5i/rGMN0/aPtA4IvA/GFq+CrwC2AW8IUxvaGIiBiTKsMKOBxYYvtR2xuAxcARwH22VwzS/jDgCtuP2X4E+M4wfS8uz31Az1CNbJ8GvBC4Ezh5sDaS5knqldTb398/0nuKiIhRqjWshvLoOPTxeHneSDl1X9I1klZLWtja0PZG4Os0R23PYnuB7YbtxowZM8ahtIiIGEytYXUTMFfS9pJ2AI4v84ayHHizpO3KmXvHbs7GbL/e9mzbZ5aTN/aBpz+zOg740ejeRkREjIcqvxRse5WkRcBtZdZC4KFh2q+UdCWwFvglsA54eJSbF3CJpOeV6TXAe0fZV0REjAO5S77iLGma7Q2StgeWAfNsr5qs7TcaDff29k7W5iK6Tq5gMf5q//Muqc/2cCfEPa3KkdUoLZC0H7AdcMlkBlVEREysrgkr22/vdA0RMXq1jwKis2o9wSIiIuJpCauIiKhewioiIqqXsIqIiOolrCIionoJq4iIqF7CKiIiqpewioiI6iWsIiKiegmriIioXsIqIiKq1zXXBoyIqW08r7qe6wx2n4ysIiKielWGlaSlktq6x0kbfc0ttw7Z9PoCSUePR98RETE5qgyrzSVp62EWzwWeDivb59m+fuKrioiI8TKmsJL0bUl9ku6QNK/Me4OkVZLWSLqhzJsm6auS1klaK+mEMv8YSbeW9pdLmjbINgZtI+leSZ+StAo4SdJZklaW7X5L0vaSXg0cB3xG0mpJe0taJOnE0sfrJN1e6vqKpOe09H1+2eY6SbPGsp8iImJsxjqyOt32HKABfEDSbsCXgBNsHwCcVNqdCzxs+xW2Xwl8T9J04GPA0bYPBHqBs1s7b6PNr20faPvrwGLbB5Xt3gmcYfsW4ErgHNuzbf+spe/tgEXAybZfQfNkk/e29P1g2eYXgflj3E8RETEGYz0b8AOSji/TewLzgGW27wGw/Zuy7GjgbZtWsv2QpGNpHp5bruZpQH8E3Dqg/0NGaPONlumXS/o4sDMwDbhmhNpfBtxj+8fl9SXAXwKfK68Xl+c+4K2DdVBGk/MAZs6cOcLmIiJitEYdVpKOohlCh9r+naSlwGqg3UNmAq6zfcoY2jzaMr0ImGt7jaRTgaParGMoj5fnjQyxn2wvABYANBqNnCwbETFBxnIYcCfgoRJUs2iOgrYDjpT0EgBJu5a219EctVDm7wKsAA6TtE+Zt4Oklw7YRjttNtkReEDStsA7WuY/UpYNdBfQs6lv4J3AjW2874iImGRjCavvAttIuhP4JM1g6ad5WGyxpDX84TDdx4FdJK0v819jux84FbhM0lqah/eeMSprp02Lc4EfAMuBH7XM/zpwTjmRYu+Wvh8DTgMul7QOeAq4eDQ7IiIiJpacr3qPi0aj4d7e3k6XETFl5QoWWx5Jfbbb+k5tV3zPKiIiuluuDRgRVchoKIaTkVVERFQvYRUREdVLWEVERPUSVhERUb2EVUREVC9hFRER1UtYRURE9RJWERFRvYRVRERUL2EVERHVS1hFRET1cm3AiKjCwKuu51qB0Sojq4iIqN5mj6wk/Q2wAXgesMz29Zu5/lHAfNvHbu62J5ukucCPbf+w07VERGzJRj2ysn3e5gbVFDQX2K/TRUREbOnaCitJH5X0Y0k3Ay8r8xZJOrFMf1LSDyWtlfT3LcsvltRb1n3WSErSwZJuLbecv0XSpr63lvT3ktaXPt9f5s+RdKOkPknXSNq9zF8q6bNlW3dKOkjSYkk/kfTxlu39uaTbJK2W9E+Sti7zN0i6UNIaSSsk7Sbp1cBxwGdK+73HsJ8jImIMRjwMKGkO8DZgdmm/CuhrWf584Hhglm1L2rll9R7gYGBv4PuS9hnQ/Y+AI2z/XtLRwN8BJwDzyrqzy7JdJW0LfAF4i+1+SScDFwKnl76esN2Q9EHgCmAO8BvgZ5I+C7wAOBk4zPaTki4C3gFcCuwArLD9UUmfBs6y/XFJVwJX2f7mSPspIiImTjufWR0BLLH9O4DyB7zVw8BjwJclXQVc1bLsX2w/BfxE0t3ArAHr7gRcImlfwMC2Zf7RwMW2fw9g+zeSXg68HLhOzdOGtgYeaOlrU13rgDtsP1DqvRvYEzicZoCtLOs/F/hVWeeJlrr7gP/exn5B0jyawcrMmTPbWSUiIkZhzKeul5HPwcDrgBOB9wGv3bR4YPMBr/8W+L7t4yX1AEuH2ZRohtChQyx/vDw/1TK96fU2Zf1LbP/VIOs+aT99ouxG2twvthcACwAajUZOtI2ImCDtfGa1DJgr6bmSdgTe3LpQ0jRgJ9tXAx8GDmhZfJKkrcrnPXsBdw3oeyfg/jJ9asv864B3S9qmbGPXsu4MSYeWedtK2r+N+je5AThR0gs29SnpxSOs8wiw42ZsIyIiJsCIYWV7FfANYA3wb8DKAU12BK6StBa4GTi7Zdm/A7eV9d5j+7EB634a+ISk23nmaGZhWXetpDXA220/QXPk9qkybzXw6rbeZfN9/BD4GHBtqfU6YPcRVvs6cE45ASQnWEREdIg8QV8Tl7SILejkhEaj4d7e3k6XETFl5QoWWx5JfbYb7bTNFSwiIqJ6E3ZtQNunTlTfEdF9MpKK4WRkFRER1UtYRURE9RJWERFRvYRVRERUL2EVERHVS1hFRET1ElYREVG9hFVERFQvYRUREdVLWEVERPUSVhERUb2EVUR03sBLrkcMkLCKiIjqJawiIqJ6XR1WkhZK2m+ENosknTjI/B5Jb5+46iIiol1dHVa2zyy3sx+NHiBhFRFRgSkRVpLOkfSBMv1ZSd8r06+V9DVJx0i6VdIqSZdLmlaWL5XUKNNnSPqxpNskfUnSP7Zs4khJt0i6u2WU9UngCEmrJX14Et9uREQMMCXCCrgJOKJMN4BpkrYt89YCHwOOtn0g0Auc3bqypBcC5wKHAIcBswb0vztwOHAszZAC+Ahwk+3Ztj87WFGS5knqldTb398/xrcYERFDmSph1QfMkfQ84HHgVpqhdQTwX8B+wHJJq4F3AS8esP7BwI22f2P7SeDyAcu/bfupcshwt3aLsr3AdsN2Y8aMGaN6YxERMbJtOl1AO2w/Keke4FTgFpqjqdcA+wD3ANfZPmUMm3i8ZTpf+IiIqMxUGVlB81DgfGBZmX4PcDuwAjhM0j4AknaQ9NIB664E/lTSLpK2AU5oY3uPADuOV/ERETF6Uy2sdgdutf1L4DGanyn10xxxXSZpLc1DhM/4TMr2/cDfAbcBy4F7gYdH2N5aYKOkNTnBIiKis2S70zVMCknTbG8oI6slwFdsLxmv/huNhnt7e8eru4gtiwRbyN+i+ANJfbYb7bSdSiOrsfqbcgLGepqfc327w/VExCYJqhjBlDjBYjzYnt/pGiIiYnS2pJFVRERMUQmriIioXsIqIiKql7CKiIjqJawiIqJ6CauIiKhewioiIqqXsIqIiOolrCIionoJq4iIqF7CKiI6T7mNXAwvYRUREdVLWEVERPXGJawk9UhaPx59RUREDNTxkVW5GWL1pkqdERHdaDzDamtJX5J0h6RrJT1X0mxJKyStlbRE0i4AkpZK+pykXuCDkk6StL7cQn5ZabO1pM9IWlnWf3eZf5SkZZL+VdJdki6WtFVZdoqkdaWvT5V5J0n6hzL9QUl3l+m9JC0v03Mk3SipT9I1knYfrM5x3FcREbEZxnO0sC9wiu2zJP0LcALwP4H3275R0gXAXwMfKu3/aNPtjCWtA15v+35JO5flZwAP2z5I0nOA5ZKuLcsOBvYD7gO+C7xV0i3Ap4A5wEPAtZLmAjeVOgCOAH4taY8yvUzStsAXgLfY7pd0MnAhcPrAOgeSNA+YBzBz5szR7reIiBjBeIbVPbZXl+k+YG9gZ9s3lnmXAJe3tP9Gy/RyYFEJucVl3jHAKyWdWF7vRDMQnwBus71phHQZcDjwJLDUdn+Z/zXgSNvfljRN0o7AnsA/A0fSDKvFwMuAlwPXqXn67NbAA0PU+Qy2FwALABqNRu7LHRExQcYzrB5vmd4I7DxUw+LRTRO23yPpVcCfAX2S5gCiOSq7pnUlSUcBA4NhpKC4BTgNuIvmSOt04FDgfwAzgTtsHzpSnRER0RkTeYLFw8BDko4or98J3DhYQ0l72/6B7fOAfpojoGuA95bDdEh6qaQdyioHS3pJ+azqZOBm4DbgTyVNl7Q1cErL9m4C5gPLgNuB1wCP236YZoDNkHRo2c62kvYfv90QERFjNdFnuL0LuFjS9sDdNEc3g/mMpH1pjqZuANYAa4EeYJWax+f6gbml/UrgH4F9gO8DS2w/Jekj5bWAf7V9RWl/E80AXGZ7o6T/AH4EYPuJcqjxf0vaieY++Rxwxzjtg4iIGCPZU+ujlnIYcL7tYztdS6tGo+He3t5OlxExNUkwxf4WxdhJ6hvqBLaBOv49q4iIBFWMZMp90dX2UmBph8uIiIhJlJFVRERUL2EVERHVS1hFRET1ElYREVG9hFVERFQvYRUREdVLWEVERPUSVhERUb2EVUREVC9hFRER1UtYRURE9abctQEjtljNO1l3r1zMNoaRkVVERFRvSoSVpEXlBolIWihpv81cf8PEVBYREZNhyh0GtH3mRPZf7kos209N5HYiIqJ9HR1ZSfoLSWslrZG0RNI9krYty57X+rplnaWSGmV6g6QLy/orJO1W5r9E0q2S1kn6+ID1z5G0smz3/DKvR9Jdki4F1gN7ltHc+tLHhydjf0RExOA6FlaS9gc+BrzW9gHAGTRvqvhnpcnbgMW2nxymmx2AFWX9ZcBZZf7ngS/afgXwQMs2jwH2BQ4GZgNzJB1ZFu8LXGR7f2A6sIftl5c+vjrW9xsREaPXyZHVa4HLbT8IYPs3wELgtLL8NEYOiSeAq8p0H9BTpg8DLivT/7el/THlcTuwCphFM6QA7rO9okzfDewl6QuS3gD852AblzRPUq+k3v7+/hFKjYiI0arqBAvby4EeSUcBW9teP8IqT9pPn++6kWd+BjfYebACPmF7dnnsY/vLZdmjLXU8BBxAc6T3HpohOli9C2w3bDdmzJgxQqkRETFanQyr7wEnSXo+gKRdy/xLgX9mbIfeltM8jAjwjpb51wCnS5pWtrmHpBcMXFnSdGAr29+ieajywDHUEhERY9SxsLJ9B3AhcKOkNcA/lEVfA3bhD4fxRuODwF9KWgfs0bLNa2kG4a1l2TeBHQdZfw9gqaTVwP8D/moMtURExBjJlX1rvHyf6i2239npWjZHo9Fwb29vp8uIbpYrWESXkdRnu9FO26q+ZyXpC8AbgTd1upaIiKhHVWFl+/2driGiWhl5xBasqrMBIyIiBpOwioiI6iWsIiKiegmriIioXsIqIiKql7CKiIjqJawiIqJ6CauIiKhewioiIqqXsIqIiOolrCIionpVXRtwi9XtV9OOaEeufRjDyMgqIiKqN2XDStLVknYeoc0Fko7ezH7fJ+mnklzuGBwRER025Q4DShLNm0aOeM8r2+eNYhPLgauApaNYNyIiJkCVIytJZ0taXx4fktQj6S5JlwLrgT0l3btp5CPp3LL8ZkmXSZpf5i8qdx6mtD9f0ipJ6yTNGmzbtm+3fe8kvdWIiGhDdWElaQ5wGvAq4BDgLGAXYF/gItv7276vpf1BwAnAATTvMjzcLZIftH0g8EVg/sS8g4iIGG/VhRVwOLDE9qO2NwCLgSOA+2yvGKT9YcAVth+z/QjwnWH6Xlye+4CesRYqaZ6kXkm9/f39Y+0uIiKGUGNYDeXRcejj8fK8kfJ5naRrJK2WtHBzO7O9wHbDdmPGjBnjUF5ERAymxrC6CZgraXtJOwDHl3lDWQ68WdJ2kqYBx27Oxmy/3vZs22eOvuSIiJhI1YWV7VXAIuA24AfAQuChYdqvBK4E1gL/BqwDHh7t9iV9QNLPgRcBa0cz4oqIiPEld8G3xiVNs71B0vbAMmBeCb1J02g03NvbO7qVcwWLiFzBYgskqc/2cCfFPW3Kfc9qCAsk7QdsB1wy2UEVERETqyvCyvbbO13DmOR/lBERw6ruM6uIiIiBElYREVG9hFVERFQvYRUREdVLWEVERPW64ntWNZDUD9zXMms68GCHyqlR9sezZZ88U/bHs3X7Pnmx7bauVZewmiCSetv9stuWIPvj2bJPnin749myT/4ghwEjIqJ6CauIiKhewmriLOh0AZXJ/ni27JNnyv54tuyTIp9ZRURE9TKyioiI6iWsxomkkyTdIekpSUOevSPpXknryt2JR3lPkfptxv54g6S7JP1U0kcms8bJJmlXSddJ+kl53mWIdhvL78dqSVdOdp0TbaSfuaTnSPpGWf4DST2TX+XkamOfnCqpv+X3You7WWzCavysB95K835aI3lNuTtxN5+SOuL+kLQ18H+ANwL7AaeUW710q48AN9jeF7ihvB7Mf5Xfj9m2j5u88iZemz/zM4CHbO8DfBb41ORWObk249/BN1p+L7a4m8ImrMaJ7Ttt39XpOmrR5v44GPip7bttPwF8HXjLxFfXMW8BLinTlwBzO1hLp7TzM2/dT98EXid19R1Kt7R/B6OSsJp8Bq6V1CdpXqeL6bA9gP9oef3zMq9b7Wb7gTL9C2C3IdptJ6lX0gpJ3RZo7fzMn25j+/fAw8DzJ6W6zmj338EJktZK+qakPSentHp0xc0XJ4uk64E/HmTRR21f0WY3h9u+X9ILgOsk/ch2O4cOqzNO+6OrDLdPWl/YtqShTsV9cfkd2Qv4nqR1tn823rXGlPId4DLbj0t6N82R52s7XNOkSlhtBttHj0Mf95fnX0laQvMQwJQMq3HYH/cDrf9DfFGZN2UNt08k/VLS7rYfkLQ78Ksh+tj0O3K3pKXAnwDdElbt/Mw3tfm5pG2AnYBfT055HTHiPrHd+v4XAp+ehLqqksOAk0jSDpJ23DQNHEPzRIQt1UpgX0kvkfRHwNuArjv7rcWVwLvK9LuAZ40+Je0i6TllejpwGPDDSatw4rXzM2/dTycC33N3fyF0xH1S/nOzyXHAnZNYXx1s5zEOD+B4mseaHwd+CVxT5r8QuLpM7wWsKY87aB4u63jtndof5fWbgB/THDl07f4o7/X5NM8C/AlwPbBrmd8AFpbpVwPryu/IOuCMTtc9AfvhWT9z4ALguDK9HXA58FPgNmCvTtdcwT75RPmbsQb4PjCr0zVP9iNXsIiIiOrlMGBERFQvYRUREdVLWEVERPUSVhERUb2EVUREVC9hFRER1UtYRURE9RJWERFRvf8POwdbvqjxAfAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import sklearn\n",
    "from sklearn.linear_model import Ridge\n",
    "\n",
    "# Create linear regression\n",
    "regressor = Ridge(alpha=1)\n",
    "\n",
    "# Fit/train Ridge\n",
    "regressor.fit(x_train,y_train)\n",
    "# Predict\n",
    "pred = regressor.predict(x_test)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(\"Final score (RMSE): {}\".format(score))\n",
    "\n",
    "names = list(df.columns.values)\n",
    "names.remove(\"mpg\")\n",
    "report_coef(\n",
    "  names,\n",
    "  regressor.coef_[0,:],\n",
    "  regressor.intercept_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 3.031985282897949\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>positive</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>origin-1</th>\n",
       "      <td>-0.938925</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cylinders</th>\n",
       "      <td>-0.257569</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>-0.007462</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>horsepower</th>\n",
       "      <td>-0.002896</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>displacement</th>\n",
       "      <td>0.017533</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acceleration</th>\n",
       "      <td>0.131320</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin-3</th>\n",
       "      <td>0.369088</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin-2</th>\n",
       "      <td>0.458725</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <td>0.788913</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  coef  positive\n",
       "origin-1     -0.938925     False\n",
       "cylinders    -0.257569     False\n",
       "weight       -0.007462     False\n",
       "horsepower   -0.002896     False\n",
       "displacement  0.017533      True\n",
       "acceleration  0.131320      True\n",
       "origin-3      0.369088      True\n",
       "origin-2      0.458725      True\n",
       "year          0.788913      True"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept: [-17.480116]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAasAAAD8CAYAAADJ7YuWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAHThJREFUeJzt3XuYHXWd5/H3h3ATghAkMohgc4lmQCWSA4KAImZBHYQwwAJehnvU9c6GXX0URlgZb7ODrj6IMWpgxgEFE0FEAwghEBJJd8gNMIBcZmRRG0GG4BIwfPaP82vm0PS9+/Spbj6v5+nn1PnVr6q+p3LSn/7VqVMl20RERFTZJq0uICIioj8Jq4iIqLyEVUREVF7CKiIiKi9hFRERlZewioiIyktYRURE5SWsIiKi8hJWERFReZu2uoDxYocddnBbW1ury4iIGDM6OjoetT15IH0TViOkra2N9vb2VpcRETFmSHpooH1zGDAiIiovYRUREZWXsIqIiMpLWEVEROXlBIuIeBGp1RXEWDFat0TMyCoiIiovYRUREZWXsIqIiMpLWA2QpAmtriEi4qVqXIaVpPMlfbLh+QWSPiHpbEnLJa2WdF7D/J9I6pB0p6RZDe3rJf1vSauAA0f5ZURERDEuwwr4HvB3AJI2AU4EfgdMAfYHpgHTJb219D/N9nSgBnxc0itK+9bAr2zvY/vW7huRNEtSu6T2zs7O5r6iiIiXsHEZVrYfBP4o6U3A4cAdwH4N0yuAqdTDC+oBtQpYBuzS0L4R+HEf25lju2a7NnnygK7FGBERQzCev2c1FzgF+CvqI613AF+0/e3GTpIOBWYAB9r+s6RFwJZl9tO2N45WwRER0bNxObIqFgDvpD6iWlh+TpM0EUDSzpJeCWwLPF6CaipwQKsKjoiIno3bkZXtZyTdBPypjI6uk/TXwFLVv56/Hng/8AvgQ5LuBtZRPxQYEREVMm7DqpxYcQBwfFeb7a8DX++h+7t6Woftic2pLiIiBmNchpWkvYBrgAW27211PRFjzWhd7y1ioMZlWNm+C9i91XVERMTIGM8nWERExDiRsIqIiMpLWEVEROUlrCIiovISVhERUXkJq4iIqLyEVUREVF7CKiIiKi9hFRERlZewioiIyktYRURE5Y3LawNGxPDU76ITw5GLAY+sjKwiIqLyxmxYSbpW0nb99Dlf0oxBrvcHktZJWivpe5I2G16lERExXGMurFS3ie132/5TX31tn2v7hkFu4gfAVOANwMuAM4ZYakREjJBKhpWks8rIZq2kT0pqK6OdS4G1wC6SHpS0Q+l/Tpl/q6TLJM0u7fMkHVemH5R0nqQVktZImtrTtm1f6wK4HXj16LzqiIjoTeXCStJ04FTgzdRvS38mMAmYAlxke2/bDzX03w84FtiH+u3pa32s/lHb+wLfAmb3U8dmwAeAXwz91URExEioXFgBB1O/Hf1TttcD84FDgIdsL+uh/0HAVbaftv0k8NM+1j2/PHYAbf3UcRGw2PYtvXWQNEtSu6T2zs7OflYXERFDVcWw6s1TI7CODeVxI+W0fUkLJa2UNLerk6S/ByYDZ/W1MttzbNds1yZPnjwC5UVERE+qGFa3ADMlbSVpa+CY0tabJcB7JG0paSJw5GA2ZvsI29NsnwEg6QzgCOAk288N7SVERMRIqtyXgm2vkDSP+skNAHOBx/vov1zS1cBq4PfAGuCJYZRwMfAQsFT1b0bOt33+MNYXERHDJI+Dr1lLmmh7vaStgMXALNsrRrOGWq3m9vb20dxkRNPkChbDNw5+tTadpA7bfZ0U97zKjayGaI6kvYAtgUtGO6giIqK5xkVY2X5vq2uIGE8yKoiqqeIJFhERES+QsIqIiMpLWEVEROUlrCIiovISVhERUXkJq4iIqLyEVUREVF7CKiIiKi9hFRERlZewioiIyktYRURE5Y2LawNGxMjKVddzfcSqycgqIiIqb0yHlaRrJW3XT5/zJc0Y5Hq/K2mVpNWSrix3II6IiBYZk2Gluk1sv9v2n/rqa/tc2zcMchOfsr2P7TcC/wZ8dMjFRkTEsFU2rCSdJWlt+fmkpDZJ6yRdCqwFdpH0oKQdSv9zyvxbJV0maXZpnyfpuDL9oKTzJK2QtEbS1J62bfs/Sn8BLwNy9DoiooUqGVaSpgOnAm8GDgDOBCYBU4CLbO9t+6GG/vsBxwL7AO8C+rpN8qO29wW+Bczuo4bvA78DpgLfGNYLioiIYalkWAEHAwtsP2V7PTAfOAR4yPayHvofBFxl+2nbTwI/7WPd88tjB9DWWyfbpwKvAu4GTuipj6RZktoltXd2dvb3miIiYoiqGla9eWoE1rGhPG6knLovaaGklZLmNna0vRG4nPqo7UVsz7Fds12bPHnyCJQWERE9qWpY3QLMlLSVpK2BY0pbb5YA75G0ZTlz78jBbMz2Eban2T6jnLyxJzz/mdVRwK+H9jIiImIkVPJLwbZXSJoH3F6a5gKP99F/uaSrgdXA74E1wBND3LyASyS9vEyvAj48xHVFRMQIkMfJ17QlTbS9XtJWwGJglu0Vo7X9Wq3m9vb20dpcRFPlCha5gsVokNRhu68T4p5XyZHVEM2RtBewJXDJaAZVREQ017gJK9vvbXUNEeNFRhVRNVU9wSIiIuJ5CauIiKi8hFVERFRewioiIiovYRUREZWXsIqIiMpLWEVEROUlrCIiovISVhERUXkJq4iIqLyEVUREVN64uTZgRIycZl51PdcdjKHIyCoiIiqvkmElaZGkAd3jZADrmlluHdL1/HxJM0Zi3RERMToqGVaDJWlCH7NnAs+Hle1zbd/Q/KoiImKkDCusJP1EUoekOyXNKm3vlLRC0ipJvyxtEyV9X9IaSaslHVvaD5e0tPS/QtLEHrbRYx9JD0r6sqQVwPGSzpS0vGz3x5K2kvQW4Cjgq5JWStpD0jxJx5V1vEPSHaWu70naomHd55VtrpE0dTj7KSIihme4I6vTbE8HasDHJe0IfAc41vY+wPGl3znAE7bfYPuNwI2SdgA+B8ywvS/QDpzVuPIB9Pmj7X1tXw7Mt71f2e7dwOm2bwOuBs62Pc32bxrWvSUwDzjB9huon2zy4YZ1P1q2+S1g9jD3U0REDMNwzwb8uKRjyvQuwCxgse0HAGw/VubNAE7sWsj245KOpH54bonqpx5tDizttv4D+unzw4bp10v6ArAdMBFY2E/trwMesH1PeX4J8BHga+X5/PLYAfxtTysoo8lZALvuums/m4uIiKEaclhJOpR6CB1o+8+SFgErgYEeMhNwve2ThtHnqYbpecBM26sknQIcOsA6erOhPG6kl/1kew4wB6BWq+WE3IiIJhnOYcBtgcdLUE2lPgraEnirpN0AJG1f+l5PfdRCaZ8ELAMOkrRnadta0mu7bWMgfbpsAzwiaTPgfQ3tT5Z53a0D2rrWDXwAuHkArzsiIkbZcMLqF8Cmku4GvkQ9WDqpHxabL2kV/3mY7gvAJElrS/vbbXcCpwCXSVpN/fDeC0ZlA+nT4BzgV8AS4NcN7ZcDZ5cTKfZoWPfTwKnAFZLWAM8BFw9lR0RERHPJ+Tr5iKjVam5vb291GREjIlewiNEgqcP2gL5TOy6+ZxUREeNbrg0YES+S0U9UTUZWERFReQmriIiovIRVRERUXsIqIiIqL2EVERGVl7CKiIjKS1hFRETlJawiIqLyElYREVF5CauIiKi8hFVERFRerg0YES/SeNX1XCcwqiAjq4iIqLxBj6wkfR5YD7wcWGz7hkEufygw2/aRg932aJM0E7jH9l2triUi4qVsyCMr2+cONqjGoJnAXq0uIiLipW5AYSXps5LukXQr8LrSNk/ScWX6S5LukrRa0j82zL9YUntZ9kUjKUn7S1pabjl/m6SudU+Q9I+S1pZ1fqy0T5d0s6QOSQsl7VTaF0m6sGzrbkn7SZov6V5JX2jY3vsl3S5ppaRvS5pQ2tdLukDSKknLJO0o6S3AUcBXS/89hrGfIyJiGPo9DChpOnAiMK30XwF0NMx/BXAMMNW2JW3XsHgbsD+wB3CTpD27rf7XwCG2/yJpBvAPwLHArLLstDJve0mbAd8AjrbdKekE4ALgtLKuZ2zXJH0CuAqYDjwG/EbShcArgROAg2w/K+ki4H3ApcDWwDLbn5X0FeBM21+QdDVwje0r+9tPERHRPAP5zOoQYIHtPwOUX+CNngCeBr4r6RrgmoZ5P7L9HHCvpPuBqd2W3Ra4RNIUwMBmpX0GcLHtvwDYfkzS64HXA9erfqrSBOCRhnV11bUGuNP2I6Xe+4FdgIOpB9jysvzLgD+UZZ5pqLsD+C8D2C9ImkU9WNl1110HskhERAzBsE9dLyOf/YF3AMcBHwUO65rdvXu35/8LuMn2MZLagEV9bErUQ+jAXuZvKI/PNUx3Pd+0LH+J7c/0sOyz9vMn6G5kgPvF9hxgDkCtVssJvhERTTKQz6wWAzMlvUzSNsB7GmdKmghsa/ta4FPAPg2zj5e0Sfm8Z3dgXbd1bws8XKZPaWi/HvigpE3LNrYvy06WdGBp20zS3gOov8svgeMkvbJrnZJe088yTwLbDGIbERHRBP2Gle0VwA+BVcDPgeXdumwDXCNpNXArcFbDvH8Dbi/Lfcj2092W/QrwRUl38MLRzNyy7GpJq4D32n6G+sjty6VtJfCWAb3K+uu4C/gccF2p9Xpgp34Wuxw4u5wAkhMsIiJaRG7S19MlzeMldHJCrVZze3t7q8uIGBG5gkWMBkkdtmsD6ZsrWEREROU17dqAtk9p1rojorkymoqqycgqIiIqL2EVERGVl7CKiIjKS1hFRETlJawiIqLyElYREVF5CauIiKi8hFVERFRewioiIiovYRUREZWXsIqIiMpLWEVEROUlrCIiovISVhERUXkjElaS2iStHYl1RUREdNfykZWkpt1TaySNlTojIsajkQyrCZK+I+lOSddJepmkaZKWSVotaYGkSQCSFkn6mqR24BOSjpe0VtIqSYtLnwmSvippeVn+g6X9UEmLJf1M0jpJF0vapMw7SdKasq4vl7bjJf1Tmf6EpPvL9O6SlpTp6ZJultQhaaGknXqqcwT3VUREDMJIjhamACfZPlPSj4Bjgf8BfMz2zZLOB/4e+GTpv7ntGoCkNcARth+WtF2ZfzrwhO39JG0BLJF0XZm3P7AX8BDwC+BvJd0GfBmYDjwOXCdpJnBLqQPgEOCPknYu04slbQZ8AzjadqekE4ALgNO619mdpFnALIBdd911qPstIiL6MZJh9YDtlWW6A9gD2M72zaXtEuCKhv4/bJheAswrITe/tB0OvFHSceX5ttQD8RngdttdI6TLgIOBZ4FFtjtL+w+At9r+iaSJkrYBdgH+FXgr9bCaD7wOeD1wvSSACcAjvdT5ArbnAHMAarVabgQeEdEkIxlWGxqmNwLb9daxeKprwvaHJL0Z+BugQ9J0QNRHZQsbF5J0KNA9GPoLituAU4F11EdapwEHAv8d2BW40/aB/dUZERGt0cwTLJ4AHpd0SHn+AeDmnjpK2sP2r2yfC3RSHwEtBD5cDtMh6bWSti6L7C9pt/JZ1QnArcDtwNsk7SBpAnBSw/ZuAWYDi4E7gLcDG2w/QT3AJks6sGxnM0l7j9xuiIiI4Wr2GW4nAxdL2gq4n/ropidflTSF+mjql8AqYDXQBqxQ/fhcJzCz9F8OfBPYE7gJWGD7OUmfLs8F/Mz2VaX/LdQDcLHtjZL+Hfg1gO1nyqHG/yNpW+r75GvAnSO0DyIiYphkj62PWsphwNm2j2x1LY1qtZrb29tbXUZExJghqaO3E9i6a/n3rCIiIvoz5r7oansRsKjFZURExCjKyCoiIiovYRUREZWXsIqIiMpLWEVEROUlrCIiovISVhERUXkJq4iIqLyEVUREVF7CKiIiKi9hFRERlZewiogXqt+ENKJSElYREVF5CauIiKi8cR1WkuZK2qufPvPKzRe7t7dJem/zqouIiIEa12Fl+wzbdw1x8TYgYRURUQFjIqwknS3p42X6Qkk3lunDJP1A0uGSlkpaIekKSRPL/EWSamX6dEn3SLpd0nckfbNhE2+VdJuk+xtGWV8CDpG0UtKnRvHlRkREN2MirIBbgEPKdA2YKGmz0rYa+Bwww/a+QDtwVuPCkl4FnAMcABwETO22/p2Ag4EjqYcUwKeBW2xPs31hT0VJmiWpXVJ7Z2fnMF9iRET0ZqyEVQcwXdLLgQ3AUuqhdQjw/4C9gCWSVgInA6/ptvz+wM22H7P9LHBFt/k/sf1cOWS440CLsj3Hds12bfLkyUN6YRER0b8xcVt7289KegA4BbiN+mjq7cCewAPA9bZPGsYmNjRM50smEREVM1ZGVlA/FDgbWFymPwTcASwDDpK0J4CkrSW9ttuyy4G3SZokaVPg2AFs70lgm5EqPiIihm6shdVOwFLbvweepv6ZUif1EddlklZTP0T4gs+kbD8M/ANwO7AEeBB4op/trQY2SlqVEywiIlpLtltdw6iQNNH2+jKyWgB8z/aCkVp/rVZze3v7SK0uonUkeIn8XojWktRhuzaQvmNpZDVcny8nYKyl/jnXT1pcT0Q1JaiigsbECRYjwfbsVtcQERFD81IaWUVExBiVsIqIiMpLWEVEROUlrCIiovISVhERUXkJq4iIqLyEVUREVF7CKiIiKi9hFRERlZewioiIyktYRURE5b1krg0Y0TIag/fzzMVso2IysoqIiMobE2ElaZ6k48r0XEl7DXL59c2pLCIiRsOYOwxo+4xmrl+SqN+U8rlmbiciIgaupSMrSX8naXW5dfwCSQ9I2qzMe3nj84ZlFkmqlen1ki4oyy+TtGNp303SUklrJH2h2/JnS1petnteaWuTtE7SpdRvzrhLGc2tLevIbe0jIlqoZWElaW/gc8BhtvcBTgcWAX9TupwIzLf9bB+r2RpYVpZfDJxZ2r8OfMv2G4BHGrZ5ODAF2B+YBkyX9NYyewpwke29gR2AnW2/vqzj+8N9vRERMXStHFkdBlxh+1EA248Bc4FTy/xT6T8kngGuKdMdQFuZPgi4rEz/c0P/w8vPHcAKYCr1kAJ4yPayMn0/sLukb0h6J/AfPW1c0ixJ7ZLaOzs7+yk1IiKGqlInWNheArRJOhSYYHttP4s8az9/ju1GXvgZXE/n3gr4ou1p5WdP298t855qqONxYB/qI70PUQ/RnuqdY7tmuzZ58uR+So2IiKFqZVjdCBwv6RUAkrYv7ZcC/8rwDr0toX4YEeB9De0LgdMkTSzb3FnSK7svLGkHYBPbP6Z+qHLfYdQSERHD1LKwsn0ncAFws6RVwD+VWT8AJvGfh/GG4hPARyStAXZu2OZ11INwaZl3JbBND8vvDCyStBL4F+Azw6glIiKGSa7YN9XL96mOtv2BVtcyGLVaze3t7a0uI6ooV7CI6JGkDtu1gfSt1PesJH0DeBfw7lbXEhER1VGpsLL9sVbXEDHiMkqJGLZKnQ0YERHRk4RVRERUXsIqIiIqL2EVERGVl7CKiIjKS1hFRETlJawiIqLyElYREVF5CauIiKi8hFVERFRewioiIiqvUtcGfMkai1fljvEt1zOMisnIKiIiKm/MhpWkayVt10+f8yXNGOR6PyrpPkkudwyOiIgWG3OHASWJ+k0j+73nle1zh7CJJcA1wKIhLBsREU1QyZGVpLMkrS0/n5TUJmmdpEuBtcAukh7sGvlIOqfMv1XSZZJml/Z55c7DlP7nSVohaY2kqT1t2/Ydth8cpZcaEREDULmwkjQdOBV4M3AAcCYwCZgCXGR7b9sPNfTfDzgW2If6XYb7ukXyo7b3Bb4FzG7OK4iIiJFWubACDgYW2H7K9npgPnAI8JDtZT30Pwi4yvbTtp8EftrHuueXxw6gbbiFSpolqV1Se2dn53BXFxERvahiWPXmqRFYx4byuJHyeZ2khZJWSpo72JXZnmO7Zrs2efLkESgvIiJ6UsWwugWYKWkrSVsDx5S23iwB3iNpS0kTgSMHszHbR9ieZvuMoZccERHNVLmwsr0CmAfcDvwKmAs83kf/5cDVwGrg58Aa4Imhbl/SxyX9Fng1sHooI66IiBhZ8jj4prqkibbXS9oKWAzMKqE3amq1mtvb24e2cK5gEVUzDn4vRPVJ6rDd10lxzxtz37PqxRxJewFbApeMdlBFRERzjYuwsv3eVtcwLPkrNiKiT5X7zCoiIqK7hFVERFRewioiIiovYRUREZWXsIqIiMobF9+zqgJJncBDDU07AI+2qJyhSL3NlXqbK/U2V7PqfY3tAV2rLmHVJJLaB/pltypIvc2Vepsr9TZXFerNYcCIiKi8hFVERFRewqp55rS6gEFKvc2Vepsr9TZXy+vNZ1YREVF5GVlFRETlJayGQdLxku6U9JykXs+UkfROSesk3Sfp0w3tu0n6VWn/oaTNm1zv9pKul3RveZzUQ5+3lzsnd/08LWlmmTdP0gMN86a1ut7Sb2NDTVc3tFdx/06TtLS8b1ZLOqFh3qjs397ejw3ztyj7676y/9oa5n2mtK+TdEQz6htCvWdJuqvsz19Kek3DvB7fGy2u9xRJnQ11ndEw7+Ty/rlX0skVqffChlrvkfSnhnmjt39t52eIP8BfA68DFgG1XvpMAH4D7A5sDqwC9irzfgScWKYvBj7c5Hq/Any6TH8a+HI//bcHHgO2Ks/nAceN4v4dUL3A+l7aK7d/gdcCU8r0q4BHgO1Ga//29X5s6PPfgIvL9InAD8v0XqX/FsBuZT0TKlDv2xveox/uqrev90aL6z0F+GYPy24P3F8eJ5XpSa2ut1v/jwHfa8X+zchqGGzfbXtdP932B+6zfb/tZ4DLgaMlCTgMuLL0uwSY2bxqATi6bGeg2zsO+LntPze1qt4Ntt7nVXX/2r7H9r1l+v8CfwAG9KXIEdLj+7Fbn8bXcSXwjrI/jwYut73B9gPAfWV9La3X9k0N79Fl1O/y3SoD2b+9OQK43vZjth8Hrgfe2aQ6uwy23pOAy5pcU48SVs23M/DvDc9/W9peAfzJ9l+6tTfTjrYfKdO/A3bsp/+JvPiNeUE53HKhpC1GvMIXGmi9W0pql7Ss65AlY2D/Stqf+l+zv2lobvb+7e392GOfsv+eoL4/B7LsSBvsNk8Hft7wvKf3RjMNtN5jy7/zlZJ2GeSyI2nA2yyHV3cDbmxoHrX9Oy5uvthMkm4A/qqHWZ+1fdVo19OfvuptfGLbkno9FVTSTsAbgIUNzZ+h/kt4c+qnsv5P4PwK1Psa2w9L2h24UdIa6r9gR9wI799/Bk62/VxpHvH9+1Ii6f1ADXhbQ/OL3hu2f9PzGkbNT4HLbG+Q9EHqo9jDWlzTQJwIXGl7Y0PbqO3fhFU/bM8Y5ioeBnZpeP7q0vZHYDtJm5a/Xrvah6WveiX9XtJOth8pvyz/0Meq/iuwwPazDevuGjVskPR9YHYV6rX9cHm8X9Ii4E3Aj6no/pX0cuBn1P/gWdaw7hHfvz3o7f3YU5/fStoU2Jb6+3Ugy460AW1T0gzqfzC8zfaGrvZe3hvNDKt+67X9x4anc6l/1tm17KHdll004hW+0GD+TU8EPtLYMJr7N4cBm285MEX1M9M2p/4PfrXrn07eRP1zIYCTgWaP1K4u2xnI9l50bLr8Au76PGgmsLYJNTbqt15Jk7oOl0naATgIuKuq+7e8BxYAl9q+stu80di/Pb4fu/VpfB3HATeW/Xk1cGI5W3A3YApwexNqHFS9kt4EfBs4yvYfGtp7fG9UoN6dGp4eBdxdphcCh5e6JwGH88IjGy2pt9Q8lfpJH0sb2kZ3/47WmRzj8Qc4hvox3g3A74GFpf1VwLUN/d4N3EP9L47PNrTvTv0/+33AFcAWTa73FcAvgXuBG4DtS3sNmNvQr436X1ebdFv+RmAN9V+i/wJMbHW9wFtKTavK4+lV3r/A+4FngZUNP9NGc//29H6kfrjxqDK9Zdlf95X9t3vDsp8ty60D3tXM/TmIem8o//+69ufV/b03WlzvF4E7S103AVMblj2t7Pf7gFOrUG95/nngS92WG9X9mytYRERE5eUwYEREVF7CKiIiKi9hFRERlZewioiIyktYRURE5SWsIiKi8hJWERFReQmriIiovP8PgrfMDmV72rgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import sklearn\n",
    "from sklearn.linear_model import ElasticNet\n",
    "\n",
    "# Create linear regression\n",
    "regressor = ElasticNet(alpha=0.1, l1_ratio=0.1)\n",
    "\n",
    "# Fit/train LASSO\n",
    "regressor.fit(x_train,y_train)\n",
    "# Predict\n",
    "pred = regressor.predict(x_test)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(\"Final score (RMSE): {}\".format(score))\n",
    "\n",
    "names = list(df.columns.values)\n",
    "names.remove(\"mpg\")\n",
    "report_coef(\n",
    "  names,\n",
    "  regressor.coef_,\n",
    "  regressor.intercept_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# TensorFlow and L1/L2\n",
    "\n",
    "L1 and L2 regularization are two common regularization techniques that can reduce the effects of overfitting (Ng, 2004).  Both of these algorithms can either work with an objective function or as a part of the backpropagation algorithm.  In both cases the regularization algorithm is attached to the training algorithm by adding an additional objective.  \n",
    "\n",
    "Both of these algorithms work by adding a weight penalty to the neural network training.  This penalty encourages the neural network to keep the weights to small values.  Both L1 and L2 calculate this penalty differently.  For gradient-descent-based algorithms, such as backpropagation, you can add this penalty calculation to the calculated gradients.  For objective-function-based training, such as simulated annealing, the penalty is negatively combined with the objective score.\n",
    "\n",
    "Both L1 and L2 work differently in the way that they penalize the size of a weight.  L2 will force the weights into a pattern similar to a Gaussian distribution; the L1 will force the weights into a pattern similar to a Laplace distribution, as demonstrated the following:\n",
    "\n",
    "![L1 vs L2](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_9_l1_l2.png \"L1 vs L2\")\n",
    "\n",
    "As you can see, L1 algorithm is more tolerant of weights further from 0, whereas the L2 algorithm is less tolerant.  We will highlight other important differences between L1 and L2 in the following sections.  You also need to note that both L1 and L2 count their penalties based only on weights; they do not count penalties on bias values.\n",
    "\n",
    "Tensor flow allows [l1/l2 to be directly added to your network](http://tensorlayer.readthedocs.io/en/stable/modules/cost.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 00260: early stopping\n",
      "Final score (RMSE): 3.0790793895721436\n"
     ]
    }
   ],
   "source": [
    "########################################\n",
    "# TensorFlow with L1/L2 for Regression\n",
    "########################################\n",
    "%matplotlib inline\n",
    "from matplotlib.pyplot import figure, show\n",
    "import tensorflow as tf\n",
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "from keras.callbacks import EarlyStopping\n",
    "from keras.layers import Dense, Dropout\n",
    "from keras import regularizers\n",
    "from keras.models import Sequential\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "# Set the desired TensorFlow output level for this example\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "\n",
    "df.drop('name',1,inplace=True)\n",
    "missing_median(df, 'horsepower')\n",
    "x,y = to_xy(df,\"mpg\")\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=45)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(50, input_dim=x.shape[1], activation='relu'))\n",
    "model.add(Dense(25, activation='relu'))\n",
    "model.add(Dense(10, \n",
    "                kernel_regularizer=regularizers.l2(0.01),\n",
    "                activity_regularizer=regularizers.l1(0.01),activation='relu'))\n",
    "# activity_regularizer can also be used.\n",
    "model.add(Dense(1))\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, verbose=1, mode='auto')\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=0,epochs=1000)\n",
    "pred = model.predict(x_test)\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(\"Final score (RMSE): {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout Regularization\n",
    "\n",
    "Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). [Dropout: a simple way to prevent neural networks from overfitting.](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf) *Journal of Machine Learning Research*, 15(1), 1929-1958.\n",
    "\n",
    "Most neural network frameworks implement dropout as a separate layer.  Dropout layers function as a regular, densely connected neural network layer.  The only difference is that the dropout layers will periodically drop some of their neurons during training.  You can use dropout layers on regular feedforward neural networks. In fact, they can also become layers in convolutional LeNET-5 networks like we studied in class 8.\n",
    "\n",
    "The usual hyper-parameters for a dropout layer are the following:\n",
    "* Neuron Count\n",
    "* Activation Function\n",
    "* Dropout Probability\n",
    "\n",
    "The neuron count and activation function hyper-parameters work exactly the same way as their corresponding parameters in the dense layer type mentioned previously. The neuron count simply specifies the number of neurons in the dropout layer.  The dropout probability indicates the likelihood of a neuron dropping out during the training iteration.  Just as it does for a dense layer, the program specifies an activation function for the dropout layer.\n",
    "\n",
    "![Dropout Regularization](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_9_dropout.png \"Dropout Regularization\")\n",
    "\n",
    "A certain percentage of neurons will be masked during each training step.  All neurons return after training is complete.  To make use of dropout in Keras use the **Dropout** layer and specify a dropout probability.  This is the percent of neurons to be dropped.  Typically, this is a low value, such as 0.1.\n",
    "\n",
    "Animation that shows how [dropout works](https://yusugomori.com/projects/deep-learning/dropout-relu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 00083: early stopping\n",
      "Final score (RMSE): 8.075186729431152\n"
     ]
    }
   ],
   "source": [
    "############################################\n",
    "# TensorFlow with Dropout for Regression\n",
    "############################################\n",
    "%matplotlib inline\n",
    "from matplotlib.pyplot import figure, show\n",
    "import tensorflow as tf\n",
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "from keras.callbacks import EarlyStopping\n",
    "from keras.layers import Dense, Dropout\n",
    "from keras import regularizers\n",
    "from keras.models import Sequential\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "# Set the desired TensorFlow output level for this example\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "\n",
    "df.drop('name',1,inplace=True)\n",
    "missing_median(df, 'horsepower')\n",
    "x,y = to_xy(df,\"mpg\")\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=45)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(50, input_dim=x.shape[1]))\n",
    "model.add(Dropout(0.01))\n",
    "model.add(Dense(25, activation='relu'))\n",
    "model.add(Dense(10, activation='relu'))\n",
    "model.add(Dense(1))\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, verbose=1, mode='auto')\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=0,epochs=1000)\n",
    "pred = model.predict(x_test)\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(\"Final score (RMSE): {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Module 9 Assignment\n",
    "\n",
    "You can find the first assignment here: [assignment 9](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class9.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (wustl)",
   "language": "python",
   "name": "wustl"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}