unit07_optim/demo2_grad_descent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demo:  Gradient Descent Optimization\n",
    "\n",
    "In the [breast cancer demo](../logistic/breast_cancer.ipynb), we used the `sklearn` built-in `LogisticRegression` class to find the weights for the logistic regression problem.   The `fit` routine in that class has an *optimizer* to select the weights to best match the data.  To understand how that optimizer works, in this demo, we will build a very simple gradient descent optimizer from scratch.  You will learn to:\n",
    "* Compute the gradients of a simple loss function and implement the gradient calculations in python\n",
    "* Implement a simple gradient descent optimizer\n",
    "* Visualize the effect of the learning rate in gradient descent\n",
    "* Implement an adaptive learning rate algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the Breast Cancer Data\n",
    "\n",
    "We first load the standard packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "%matplotlib inline\n",
    "from sklearn import preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We next load the data from the [breast cancer demo](../logistic/breast_cancer.ipynb).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>thick</th>\n",
       "      <th>size_unif</th>\n",
       "      <th>shape_unif</th>\n",
       "      <th>marg</th>\n",
       "      <th>cell_size</th>\n",
       "      <th>bare</th>\n",
       "      <th>chrom</th>\n",
       "      <th>normal</th>\n",
       "      <th>mit</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1000025</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1002945</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>10.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1015425</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1016277</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1017023</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id  thick  size_unif  shape_unif  marg  cell_size  bare  chrom  \\\n",
       "0  1000025      5          1           1     1          2   1.0      3   \n",
       "1  1002945      5          4           4     5          7  10.0      3   \n",
       "2  1015425      3          1           1     1          2   2.0      3   \n",
       "3  1016277      6          8           8     1          3   4.0      3   \n",
       "4  1017023      4          1           1     3          2   1.0      3   \n",
       "\n",
       "   normal  mit  class  \n",
       "0       1    1      2  \n",
       "1       2    1      2  \n",
       "2       1    1      2  \n",
       "3       7    1      2  \n",
       "4       1    1      2  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names = ['id','thick','size_unif','shape_unif','marg','cell_size','bare',\n",
    "         'chrom','normal','mit','class']\n",
    "df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/' +\n",
    "                 'breast-cancer-wisconsin/breast-cancer-wisconsin.data',\n",
    "                names=names,na_values='?',header=None)\n",
    "df = df.dropna()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in the breast cancer demo, we create a data matrix `X` of various features of the breast cancer sample.  The response vector `y` is a binary indicating if each sample is benign or malignant.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the predictors\n",
    "xnames = names[1:-1]\n",
    "X = np.array(df[xnames])\n",
    "Xs = preprocessing.scale(X)\n",
    "\n",
    "# Get the response.  Convert to a zero-one indicator \n",
    "yraw = np.array(df['class'])\n",
    "BEN_VAL = 2   # value in the 'class' label for benign samples\n",
    "MAL_VAL = 4   # value in the 'class' label for malignant samples\n",
    "y = (yraw == MAL_VAL).astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to learn the classification rule to predict `y` from `X`.  To this end, we first split the data into training and test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "ntr = 400        # number of training samples\n",
    "Xtr = X[:ntr,:]  # training\n",
    "ytr = y[:ntr]\n",
    "Xts = X[ntr:,:]  # test\n",
    "yts = y[ntr:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing the Gradient and Loss Function\n",
    "\n",
    "Recall that training a logistic function means finding a weight vector `w` for the classification rule:\n",
    "\n",
    "    P(y=1|x,w) = 1/(1+\\exp(-z)), z = w[0] + w[1]*x[0] + ... + w[k]x[k-1]\n",
    "    \n",
    "From the class notes in logistic regression, we saw that the weight vector can be found by minimizing  the negative log likelihood.  The negative log likelihood is called the *loss* function.  For the logistic regression problem, the loss function simplifies to:\n",
    "\n",
    "\n",
    "    f(w) = \\sum_i -y[i]z[i] + ln(1+exp(z[i])),  z[i] = w[0] + w[1]X[i,0] + ... + w[k]X[i,k-1].\n",
    "    \n",
    "We first write a function to compute `f` and its gradient `feval`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def feval(w,X,y):\n",
    "    \"\"\"\n",
    "    Compute the loss and gradient given w,X,y\n",
    "    \"\"\"\n",
    "    # Construct transform matrix\n",
    "    n = X.shape[0]\n",
    "    A = np.column_stack((np.ones(n,), X))\n",
    "\n",
    "    # The loss is the binary cross entropy\n",
    "    z = A.dot(w)\n",
    "    py = 1/(1+np.exp(-z))\n",
    "    f = np.sum((1-y)*z - np.log(py))\n",
    "\n",
    "    # Gradient\n",
    "    df_dz = py-y\n",
    "    fgrad = A.T.dot(df_dz)\n",
    "    return f, fgrad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One issue is that the function `f(w,X,y)` depends on the parameters `X` and `y`.  Most numerical optimizers expect a function that only depends on `w`.  So, we need an alternate way to pass those parameters to the function.  There are two approaches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Method 1.  Use a class method\n",
    "\n",
    "One approach to pass the parameters is to implement the function as a method in a class that contains the parameters.  In the code below, we create a class `LogisticFun` where the constructor takes the parameters `X` and `y`.  Then, the class method `feval` takes a variable `w` and computes the loss and its gradient.  \n",
    "\n",
    "You can see that one benefit of using a class method is that processing that is independent of `w` can be performed in the contructor.  For example, in the code below, constructs `self.A` in the constructor.  Then this step does not need to be repeated every time the function is called."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LogisticFun(object):\n",
    "    def __init__(self,X,y):\n",
    "        \"\"\"\n",
    "        Class for computes the loss and gradient for a logistic regression problem.\n",
    "        \n",
    "        The constructor takes the data matrix `X` and response vector y for training.\n",
    "        \"\"\"\n",
    "        self.X = X\n",
    "        self.y = y\n",
    "        n = X.shape[0]\n",
    "        self.A = np.column_stack((np.ones(n,), X))\n",
    "\n",
    "    def feval(self,w):\n",
    "        \"\"\"\n",
    "        Compute the loss and gradient for a given weight vector\n",
    "        \"\"\"\n",
    "        # The loss is the binary cross entropy\n",
    "        z = self.A.dot(w)\n",
    "        py = 1/(1+np.exp(-z))\n",
    "        f = np.sum((1-self.y)*z - np.log(py))\n",
    "        \n",
    "        # Gradient\n",
    "        df_dz = py-self.y\n",
    "        fgrad = self.A.T.dot(df_dz)\n",
    "        return f, fgrad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have defined the class we can create an *instance* of the class with the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_fun = LogisticFun(Xtr,ytr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can call the method with only the value `w`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Some random point\n",
    "p = X.shape[1]+1\n",
    "w0 = np.random.randn(p)\n",
    "\n",
    "# Call the function\n",
    "f, fgrad = log_fun.feval(w0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Method 2.  Using Lambda functions\n",
    "\n",
    "An alternative method is to use python `lambda` functions.  The code below creates uses the `lambda` construction to fix the parameters `X` and `y`.  A slight disadvantage of this method is that the pre-processing is less easy to separate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a function with X,y fixed\n",
    "feval_param = lambda w: feval(w,Xtr,ytr)\n",
    "\n",
    "# You can now pass a parameter like w0\n",
    "f0, fgrad0 = feval_param(w0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing the gradient\n",
    "\n",
    "Either way you create the function, it is very important to test if the gradient is correct.  This is the number one reason people's code does not work with numerical optimizers.  The simplest method is to take two points `w0` and `w1` that are close to one another and then verify that \n",
    "\n",
    "    f(w1) - f(w0) \\approx fgrad(w0).dot(w1 - w0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual f1-f0    =   2.8742e-04\n",
      "Predicted f1-f0 =   2.8742e-04\n"
     ]
    }
   ],
   "source": [
    "# Take a random initial point\n",
    "p = X.shape[1]+1\n",
    "w0 = np.random.randn(p)\n",
    "\n",
    "# Perturb the point\n",
    "step = 1e-6\n",
    "w1 = w0 + step*np.random.randn(p)\n",
    "\n",
    "# Measure the function and gradient at w0 and w1\n",
    "f0, fgrad0 = log_fun.feval(w0)\n",
    "f1, fgrad1 = log_fun.feval(w1)\n",
    "\n",
    "# Predict the amount the function should have changed based on the gradient\n",
    "df_est = fgrad0.dot(w1-w0)\n",
    "\n",
    "# Print the two values to see if they are close\n",
    "print(\"Actual f1-f0    = %12.4e\" % (f1-f0))\n",
    "print(\"Predicted f1-f0 = %12.4e\" % df_est)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the two agree well.  You would have got the same answer had you used the `feval` constructed in method 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Simple Gradient Descent Optimizer\n",
    "\n",
    "Now, we build a simple gradient descent optimizer function with a fixed learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def grad_opt_simp(feval, winit, lr=1e-3,nit=1000):\n",
    "    \"\"\"\n",
    "    Simple gradient descent optimization\n",
    "    \n",
    "    feval:  A function that returns f, fgrad, the objective\n",
    "            function and its gradient\n",
    "    winit:  Initial estimate\n",
    "    lr:     learning rate\n",
    "    nit:    Number of iterations\n",
    "    \"\"\"\n",
    "    # Initialize\n",
    "    w0 = winit\n",
    "    \n",
    "    # Create history dictionary for tracking progress per iteration.\n",
    "    # This isn't necessary if you just want the final answer, but it \n",
    "    # is useful for debugging\n",
    "    hist = {'w': [], 'f': []}\n",
    "    \n",
    "    # Loop over iterations\n",
    "    for it in range(nit):\n",
    "\n",
    "        # Evaluate the function and gradient\n",
    "        f0, fgrad0 = feval(w0)\n",
    "\n",
    "        # Take a gradient step\n",
    "        w0 = w0 - lr*fgrad0\n",
    "        \n",
    "         # Save history\n",
    "        hist['f'].append(f0)\n",
    "        hist['w'].append(w0)\n",
    "\n",
    "    # Convert to numpy arrays\n",
    "    for elem in ('f', 'w'):\n",
    "        hist[elem] = np.array(hist[elem])\n",
    "    return w0, f0, hist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now run the gradient descent starting from a random initial condition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEACAYAAAC6d6FnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4VOX99/H3N/u+wxCykLAJyCaERUCNa9UquFalVaq21tbaWqv+ap/2sf21ffzZq9ZfXYrSqmCtUqVYsMVaRIMCArIoICi7WdgNBAKELffzxxwQWZRkkswk5/O6rlyZuc99Zr65vlfyybnPzBlzziEiIv4TFe4CREQkPBQAIiI+pQAQEfEpBYCIiE8pAEREfEoBICLiUwoAERGf+tIAMLNnzGyLmS07aizLzKab2Srve6Y3bmb2qJmtNrMlZjbgqH3GePNXmdmY5vlxRETkVJ3KEcB44OJjxn4CzHDOdQNmePcBLgG6eV+3AWMhGBjAA8AQYDDwwOHQEBGR8PjSAHDOvQ1UHzM8Cpjg3Z4AXHHU+HMuaC6QYWa5wFeA6c65aufcdmA6x4eKiIi0oJhG7hdwzm0EcM5tNLP23ngeUHHUvEpv7GTjxzGz2wgePZCYmDiwoKCgkSVCfX09UVE6zRFJ1JPIo55EplD6snLlym3OuXZfNq+xAXAydoIx9wXjxw86Nw4YB1BSUuIWLFjQ6GLKysooLS1t9P7S9NSTyKOeRKZQ+mJmn5zKvMbG/mZvaQfv+xZvvBI4+l/2fGDDF4yLiEiYNDYApgKHX8kzBphy1PhN3quBhgI13lLR68BFZpbpnfy9yBsTEZEw+dIlIDN7ESgFcsyskuCref4HeMnMbgXKgWu96dOAS4HVwB7gZgDnXLWZ/Qp4z5v33865Y08si4hIC/rSAHDO3XCSTeefYK4D7jjJ4zwDPNOg6kREpNno1L+IiE8pAEREfEoBICLiUwoAERGfUgCIiPiUAkBExKcUACIiPqUAEBHxKQWAiIhPKQBERHxKASAi4lMKABERn1IAiIj4lAJARMSnFAAiIj6lABAR8SkFgIiITykARER8SgEgIuJTCgAREZ9SAIiI+JQCQETEpxQAIiI+pQAQEfEpBYCIiE8pAEREfEoBICLiUwoAERGfUgCIiPiUAkBExKcUACIiPqUAEBHxKQWAiIhPKQBERHxKASAi4lMhBYCZ/cjMPjSzZWb2opklmFmxmc0zs1Vm9jczi/Pmxnv3V3vbi5riBxARkcZpdACYWR7wA6DEOdcbiAauBx4CHnHOdQO2A7d6u9wKbHfOdQUe8eaJiEiYhLoEFAMkmlkMkARsBM4DJnnbJwBXeLdHeffxtp9vZhbi84uISCM1OgCcc1XA74Bygn/4a4CFwA7n3EFvWiWQ593OAyq8fQ9687Mb+/wiIhKamMbuaGaZBP+rLwZ2AC8Dl5xgqju8yxdsO/pxbwNuAwgEApSVlTW2RGpra0PaX5qeehJ51JPI1BJ9aXQAABcA65xzWwHMbDIwDMgwsxjvv/x8YIM3vxIoACq9JaN0oPrYB3XOjQPGAZSUlLjS0tJGF1hWVkYo+0vTU08ij3oSmVqiL6GcAygHhppZkreWfz6wHHgLuMabMwaY4t2e6t3H2/6mc+64IwAREWkZoZwDmEfwZO4iYKn3WOOA/wLuNrPVBNf4n/Z2eRrI9sbvBn4SQt0iIhKiUJaAcM49ADxwzPBaYPAJ5tYB14byfCIi0nT0TmAREZ9SAIiI+JQCQETEpxQAIiI+pQAQEfEpBYCIiE8pAEREfEoBICLiUwoAERGfUgCIiPiUAkBExKcUACIiPqUAEBHxKQWAiIhPKQBERHxKASAi4lMKABERn1IAiIj4lAJARMSnFAAiIj6lABAR8SkFgIiITykARER8SgEgIuJTCgAREZ9SAIiI+JQCQETEpxQAIiI+pQAQEfEpBYCIiE8pAEREfEoBICLiUwoAERGfUgCIiPiUAkBExKdCCgAzyzCzSWb2kZmtMLMzzSzLzKab2Srve6Y318zsUTNbbWZLzGxA0/wIIiLSGKEeAfwB+LdzrgfQD1gB/ASY4ZzrBszw7gNcAnTzvm4Dxob43CIiEoJGB4CZpQFnA08DOOf2O+d2AKOACd60CcAV3u1RwHMuaC6QYWa5ja5cRERCEsoRQGdgK/CsmS02sz+bWTIQcM5tBPC+t/fm5wEVR+1f6Y2JiEgYxIS47wDgTufcPDP7A58t95yInWDMHTfJ7DaCS0QEAgHKysoaXWBtbW1I+0vTU08ij3oSmVqiL6EEQCVQ6Zyb592fRDAANptZrnNuo7fEs+Wo+QVH7Z8PbDj2QZ1z44BxACUlJa60tLTRBZaVlRHK/tL01JPIo55EppboS6OXgJxzm4AKMzvNGzofWA5MBcZ4Y2OAKd7tqcBN3quBhgI1h5eKRESk5YVyBABwJ/BXM4sD1gI3EwyVl8zsVqAcuNabOw24FFgN7PHmiohImIQUAM6594GSE2w6/wRzHXBHQx6/onoPd01cfOS+2edPIxx3UuGogYTYaPrF1Tfk6UREfCXUI4BmtWf/IRZX7ADAHXO62B1z/vjY7Z/W7ucf7hC53bZydvd2zVmmiEirFNEBcFqHVGbee26j9i3/dA+jx5bxzWfnc/8lPfnWWcXHHUGIiPhZm70WUGF2Ej8bmshXTu/Ab6at4O6XPqDuwKFwlyUiEjHabAAAJMQYT4wewN0XdueVxVVc++S7bNixN9xliYhEhDYdAABRUcYPzu/Gn24qYd223Yx8fBYL1leHuywRkbBr8wFw2IW9ArzyvWGkxMdww5/m8uL88nCXJCISVr4JAIBugVSm3DGCM7vkcP/kpfzsH0vZf1AvFRURf/JVAACkJ8Xy7DcH8Z2zO/P83HK+8fQ8ttXuC3dZIiItzncBABAdZdx/aU/+cH1/PqjYwcjHZrGsqibcZYmItChfBsBho/rnMen2YQBc8+QcprxfFeaKRERajq8DAKBPfjpT7xxB37wMfjjxfR58bQWH6o+7SrWISJvj+wAAyEmJ5/lvDeEbQwt5auZabhn/HjV7DoS7LBGRZqUA8MTFRPHrK/rw/67sw5w127jij7NZvWVXuMsSEWk2CoBjjB5SyAvfHsquugNc8cQc3li+OdwliYg0CwXACQwqymLq90dQnJPMt/+ygMffXIU79nKjIiKtnALgJDpmJPLy7Wcyql9HfvefldzxwiJ27zsY7rJERJqMAuALJMRG88h1/fnppT3497JNXD12DhXVe8JdlohIk1AAfAkz47azu/DszYPZsGMvIx+fxZzV28JdlohIyBQAp+ic7u2Y+v0R5KTEc+Mz83l29jqdFxCRVk0B0ABFOcm8csdwzuvRnl++upx7Jy3Rh8yISKulAGiglPgYnvrGQH5wfjcmLazk+nFz2byzLtxliYg0mAKgEaKijLsv7M6T3xjAys27uPyxWSwq3x7uskREGkQBEIKLe+cy+XvDSIiN5vqn5vLSgopwlyQicsoUACHq0SGNqd8fzuDiLO6btIRfTP2QA4f0ITMiEvkUAE0gIymO8TcP4tYRxYyfs54r/zibSQsrdYJYRCKaAqCJxERH8fPLevGH6/uzZ98h7nn5Awb/5g1+MfVDPt6ki8qJSOSJCXcBbc2o/nmM7NeReeuqeXF+OS/MK2f8nPUMKMxg9JBOfLVPLolx0eEuU0REAdAczIyhnbMZ2jmbBy7fz+RFlbwwv5x7Xv6AX776IVedkccNQwrp0SEt3KWKiI8pAJpZVnIc3zqrM7eOKGa+d1Tw4nsVTHj3E84ozGD04EIu69tRRwUi0uIUAC3EzBjSOZshnbN5YPd+Ji+u4oV5n3DvpCX89z+Xc+UZedwwuJCeuToqEJGWoQAIg8zkOG4dUcwtw4t4b/12XpxfzsT3Knju3U/oX5DB6CGFXNY3l6Q4tUdEmo/+woSRmTG4OIvBxVk8cHkvJi+q4oX55dw3aQm/enU5F/QKMMTbXpyTjJmFu2QRaUMUABEiIymOW0YUc/PwIhZ+sp0X5pfz9sqtvLK4Cgh+cP3hMBhcnMVpgVSiohQIItJ4CoAIY2aUFGVRUpSFc46123Yzf10189dVM2/tp/xr6UYA0hJiGFycxaCiYCD0zksnNlpv6xCRU6cAiGBmRpd2KXRpl8INgwsBqNy+50ggzF9XzRsrtgCQGBvNwE6Z9CtIp09eOqd3TCc/M1HLRiJyUgqAViY/M4n8zCSuGpAPwJZddSxYv/1IIDw1cy0H64MfVJOZFEvvvPTgV8dgMBRkKRREJCjkADCzaGABUOWcu8zMioGJQBawCLjRObffzOKB54CBwKfAdc659aE+v9+1T03g0j65XNonF4C6A4f4aNMullbV8GFVDUuravjzO2s5cCgYCumJsfTOS6N3x2Aw9MlLp1N2kkJBxIea4gjgh8AK4PAL2B8CHnHOTTSzJ4FbgbHe9+3Oua5mdr0377omeH45SkJsNP0LMuhfkHFkbN/BQ3y8aRfLqnaytKqGZVU1PDt7Pfu9q5amJsRwesc0+uR9FgpF2ck6ySzSxoUUAGaWD3wV+A1wtwX/jTwPGO1NmQD8gmAAjPJuA0wCHjczc/pg3WYXHxNN3/wM+uZ/Fgr7D9azcvMulnlHCcuqapjw7ifsPxgMhZT4GHp5oRAMhjSKc1KIViiItBkWyt9fM5sEPAikAvcA3wTmOue6etsLgNecc73NbBlwsXOu0tu2BhjinNt2zGPeBtwGEAgEBk6cOLHR9dXW1pKSktLo/f3mYL1jQ20963d6XzX1VOyq54D38Qbx0dApLYquGdF0y4yiS0Y0aXENCwT1JPKoJ5EplL6ce+65C51zJV82r9FHAGZ2GbDFObfQzEoPD59gqjuFbZ8NODcOGAdQUlLiSktLj51yysrKyghlf4GDh+pZvbWWpZXBo4T3K2uYXl7DtHXB1hXnJDOwUyaDijIZ1iWHgqykL3w89STyqCeRqSX6EsoS0HBgpJldCiQQPAfwv0CGmcU45w4C+cAGb34lUABUmlkMkA5Uh/D80gJioqPo0SGNHh3SuLakAAieaF5aVcPCT7azYP123vxoC5MWVgJQkJXIsM45DOuazbAuObRLjQ9n+SLyBRodAM65+4H7AbwjgHucc183s5eBawi+EmgMMMXbZap3/11v+5ta/2+dEmKjGVQUfBMa54BzjjVba5m9+lNmr97GtGUb+Zv3+cinBVIZ1jWb4V1yGNI5K8yVi8jRmuN9AP8FTDSzXwOLgae98aeBv5jZaoL/+V/fDM8tYWBmdG2fStf2qYwZVsSheseyqhpmr9nGnNWf8sK8cp6dvZ7oKKNLuvGxreHCXgE6t9O6s0g4NUkAOOfKgDLv9lpg8Anm1AHXNsXzSWSLjjL6FWTQryCD75V2pe7AIRaX72D26m28unAtD772EQ++9hFd2iVzQa8AF/UK0L8gU68wEmlheiewNLuE2GjO7JLNmV2yKYnfSLf+Q3hj+WamL9/M0++s46mZa8lJieP8HgG+0jvAWd3a6bpGIi1AASAtLi8jkTHDihgzrIiddQco+3gr05dvZtrS4LmDzKRYLu2Ty6j+eZR0ytQb0kSaiQJAwiotIZaR/Toysl9H9h+sZ9bqrUx5fwOTF1Xx13nldExP4PL+we29ctN0yQqRJqQAkIgRFxPFeT0CnNcjwO59B3ljxWamvL/hyDJRjw6pXDeogCvPyCMjKS7c5Yq0egoAiUjJ8TGM6p/HqP55VO/ez7+WbuTlBRX88tXlPPjaR1zSuwPXDSpgaHG2lohEGkkBIBEvKzmOG4d24sahnfhwQw0vvVfBK4urmPL+BjplJzF6cCHXDSrQUYFIA+mlFtKqnN4xnV+O6s38/3MBf7i+P4G0BB587SOGPjiD+ycv4aNNO8NdokiroSMAaZUSYqOPLBGt2LiT595dzyuLq3hxfgVDirO4ZUQxF/YMaHlI5AvoCEBavZ65aTx4VV/m3n8+91/Sg8rte/nOXxZy4SMzeWlBxZFLXIvI5ykApM3ISIrjO+d0Yea9pTx6wxnExURz36QlnP3bt/jzO2up3Xcw3CWKRBQFgLQ5MdFRjOzXkWk/GMH4mwdRlJPEr/+1gmEPzuB3r39M9e794S5RJCLoHIC0WWZG6WntKT2tPYvLt/PkzDU8UbaaZ2ev45vDi/j2WZ31yiHxNR0BiC+cUZjJUzeW8J+7zubcHu154q01jHjoLX4/fSU1ew+EuzyRsFAAiK90C6Ty+OgB/PuuszirWw6PzljFiIfe5A9vrNI5AvEdBYD4Uo8OaYz9xkD+9YMRDO2czSNvrOSc377Fc++u58AhvWpI/EEBIL52esd0/nRTCf+4Yzhd26fwf6d8yEWPvM20pRvRB9ZJW6cAEAH6F2Qw8bahPPPNEmKjje/9dRFXjZ3D/HX62GppuxQAIh4z47weAV774dn89uq+bNixl6899S7fmrCAddt2h7s8kSanABA5RnSU8bVBBZTdcy73fuU03l2zjYsemcmD01awq06vGJK2QwEgchKJcdHccW5X3rq3lFH983jq7bWc+7uZvLyggvp6nR+Q1k8BIPIl2qcm8Ltr+zHljuEUZCVy76QlXPnH2Swq3x7u0kRCogAQOUX9CjL4++3D+P3X+rGxpo6r/jiHu//2Plt21oW7NJFGUQCINEBUlHHVgHzeuqeU75V24Z9LNnL+wzMZP3sdh7QsJK2MAkCkEZLjY7jv4h68/qOz6V+YwS9eXc7Ix2exWMtC0oooAERCUJyTzHO3DOaJ0QPYVruPq8bO4aevLGXHHl1xVCKfAkAkRGbGV/vm8sbd53DL8GL+9l4F5z88k0kLK/VuYoloCgCRJpKaEMvPL+vFq98fQafsJO55+QOue2ouH2/aFe7SRE5IASDSxHp1TGPS7cN46Oo+rNyyi68++g4PTlvBbl1tVCKMAkCkGURFGdcNKuTNH5dy9YB8nnp7LRf8fib/XqaLzEnkUACINKOs5DgeuqYvf//umaQnxnL784u4Zfx7lH+6J9yliSgARFrCwE5Z/PPOEfzsqz2Zv66aCx+ZyWMzVrHv4KFwlyY+pgAQaSEx0VF866zOzPhxKRf0DPDw9JVc8r/vMGvVtnCXJj6lABBpYR3SE3ji6wOYcMtg6p3jG0/P484XF+uSEtLiFAAiYXJO93b8+66zueuCbrz+4SbOe3gmz85ex0F9JKW0EAWASBglxEZz1wXd+c9dZzOgUya/fHU5o56YrUtKSItodACYWYGZvWVmK8zsQzP7oTeeZWbTzWyV9z3TGzcze9TMVpvZEjMb0FQ/hEhrV5STzISbB/HHrw/g09r9XDV2DvdP1iUlpHmFcgRwEPixc64nMBS4w8x6AT8BZjjnugEzvPsAlwDdvK/bgLEhPLdIm2NmXNonlzd+fA63Di/mpQUVnPfwTCbOL9eVRqVZNDoAnHMbnXOLvNu7gBVAHjAKmOBNmwBc4d0eBTznguYCGWaW2+jKRdqolPgYfnZZL/555wg65yTzk8lLufyxWcxd+2m4S5M2pknOAZhZEXAGMA8IOOc2QjAkgPbetDyg4qjdKr0xETmBnrlpvHz7mTx2wxnU7D3A9ePm8t3nF1JRrTeRSdOICfUBzCwF+Dtwl3Nup5mddOoJxo47rjWz2wguEREIBCgrK2t0bbW1tSHtL01PPWm4VOCBQca/18fyzxWbmP7hJr5SFMtlXWJJjDnp79spU08iU0v0JaQAMLNYgn/8/+qcm+wNbzazXOfcRm+JZ4s3XgkUHLV7PrDh2Md0zo0DxgGUlJS40tLSRtdXVlZGKPtL01NPGu8i4L6aOn77+kdMXlTFvK1R/Pii7lw7MJ+Y6MYfzKsnkakl+hLKq4AMeBpY4Zz7/VGbpgJjvNtjgClHjd/kvRpoKFBzeKlIRE5Nh/QEfv+1/ky5YzhF2UncP3kpFz7yNv9csoF6nSiWBgrlHMBw4EbgPDN73/u6FPgf4EIzWwVc6N0HmAasBVYDfwK+F8Jzi/hav4IMXr79TP58Uwlx0VF8/4XFXP74LGau3Kqrjcopa/QSkHNuFide1wc4/wTzHXBHY59PRD7PzLigV4Bze7Rn6gdVPPyflYx5Zj5DirO49yunUVKUFe4SJcLpncAirVx0lHHlGfm8+eNSfjnydNZs3c01T77LDePmMmf1Nh0RyEkpAETaiLiYKMYMK+Lt+0r5+WW9WLO1ltF/nsfVY+fw1kdbFARyHAWASBuTFBfDrSOKefu+c/nVFb3ZvHMfN49/j8sem8UriyvZf1AXm5MgBYBIG5UQG82NQztRdm8pv72mL3UHDvGjv33AiIfe5PE3V1G9W9cZ8ruQ3wgmIpEtNjqKr5UUcM2AfN5etZWnZ63jd/9ZyWNvruaqAfn0iNGnkvmVAkDEJ6KijNLT2lN6WntWbt7Fs7PXM3lRJfsO1vP38llcP6iQkf07khKvPwt+oSUgER/qHkjlwav6MP+nF/D1nnHsP1jPT19ZyuDfvMF9kz5g4SfVOmnsA4p6ER9LT4rlwk6x/Pqms3i/YgcT51fw6pINvLSgkvzMRC7v15FR/TvSo0NauEuVZqAAEBHMjDMKMzmjMJOfX96L15dtYuoHGxj39lrGlq2heyCFy/t25KLTO9A9kMIXXPRRWhEFgIh8Tkp8DFcPzOfqgfl8WruPaUs3MvWDDTw8fSUPT19JYVYSF/QMcGGvAIOKMkO6EJ2ElwJARE4qOyWeG88s4sYzi9i8s44ZK7Ywffkmnp/3Cc/MXkd6Yixnd2/HiK7ZDO+aQ35mUrhLlgZQAIjIKQmkJTB6SCGjhxSye99B3lm1lenLt/DOqq28+kHwyu5F2UmM6JbDiK45DCrKIjslPsxVyxdRAIhIgyXHx3Bx71wu7p2Lc45VW2qZtWobs1dv45VFVTw/txyAzjnJDOyUSUlRJgM7ZdGlXbLOH0QQBYCIhMTM6B5IpXsglVtGFHPgUD3vV+xgwfrtLPykmjdWbOblhZUAZCbFMqAwk9556fTJS6d3XjqBtHiFQpgoAESkScVGRzGoKItBRVlAF5xzrNm6m4WfVLNg/XYWV+zgzY+3cPhtBjkp8fTJS6O3Fwg9O6SRn5lIVJRCobkpAESkWZkZXdun0LV9CtcNKgRg976DrNi4k6VVNSytquHDqp3MXLmVwx9qlhgbTfdAypEji+4dUjktkKqjhSamABCRFpccH0NJUdbnPrRm7/5DrNi0k5WbdvHx5l2s3LyLspVbjywfAaQlxNA9kErndsl0bpdCcU4ynXOSKcxOIj4mOhw/SqumABCRiJAYF82AwkwGFGZ+brx6935WeoHw8aZdrNpcy5sfbeWlBZ8FQ5RBXmYinXO8UGiXTHFO8Cs3PZFoLSedkAJARCJaVnIcQztnM7Rz9ufGd9YdYP223azdupu123azbttu1m6t5b311ezZ/9kVTmOijLzMRAqzkijISgp+zwx+L8xKIj0ptqV/pIihABCRViktIZa++Rn0zc/43Lhzji279rFmay3rt+2hYvseKqqDX68t3cj2PQeOeZyYI8FQmJVEflYS+RmJdMxIpGNGAqkJbTcgFAAi0qaYGYG0BAJpCQzrcvz2XXUHqKjeS7kXChXb91BevYePN+9ixkdbjvvEtLSEGDpmJJJ3JBSCwZCfGbzdPjWh1S4xKQBExFdSE2Lp1TGWXh2Pv8JpfX3w6KFqx142HPVVtWMvVTvqWPDJdmr2fv4IIjrK6JCW4AVEAnmZieSmJ5KbHgyh3PQEspLjIvLVSwoAERFPVJTRIT2BDukJDOyUecI5tfsOsnHHXio/FxJ1VO3Yy3vrt/Pqko0cqv/8ZynERUcRSI+nQ1oCHdIT6ZAW731POPJ87VPjiW3hC+spAEREGiAlPoZugVS6BVJPuP1QvWPLrjo21dSxeWcdG2vq2LQzeH9TTR1LKnfwn5o69h2z1GQWfFPc4SOHgDtAaTP/LAoAEZEmFB1l3hJQ4knnOOfYsefAZ8HgBcXmmjo27qyj/NM9uPj6k+7fVBQAIiItzMzITI4jMzmOnrkn/rS1srKyZq9Dn+QgIuJTCgAREZ9SAIiI+JQCQETEpxQAIiI+pQAQEfEpBYCIiE8pAEREfMqcc18+K0zMrAZYdYJN6UDNKYzlANuaobQvc6JaWupxTmWfUOY0ZDySegJN05fm6smpzPui7aH0pbX3pLGP0xS/K83VEwitL52cc+2+dJZzLmK/gHGnOn6SsQWRVHdLPM6p7BPKnNbak6bqS3P15FTmfdH2UPrS2nvSnH0JV09aqi+RvgT0agPGTzY3HJqqlsY8zqnsE8qc1toTaJp6mqsnpzLvi7a31r609t+VVt2TiF4CCpWZLXDOlYS7DvmMehJ51JPI1BJ9ifQjgFCNC3cBchz1JPKoJ5Gp2fvSpo8ARETk5Nr6EYCIiJyEAkBExKcUACIiPuWbADCzZDObYGZ/MrOvh7seCTKzzmb2tJlNCnctEmRmV3i/J1PM7KJw1yNBZtbTzJ40s0lm9t2meMxWHQBm9oyZbTGzZceMX2xmH5vZajP7iTd8FTDJOfdtYGSLF+sjDemLc26tc+7W8FTqHw3syT+835NvAteFoVzfaGBfVjjnbge+BjTJy0NbdQAA44GLjx4ws2jgCeASoBdwg5n1AvKBCm/aoRas0Y/Gc+p9kZYxnob35Gfedmk+42lAX8xsJDALmNEUT96qA8A59zZQfczwYGC195/lfmAiMAqoJBgC0Mp/7kjXwL5IC2hITyzoIeA159yilq7VTxr6u+Kcm+qcGwY0yTJ2W/xDmMdn/+lD8A9/HjAZuNrMxhJZb4X3ixP2xcyyzexJ4Awzuz88pfnWyX5X7gQuAK4xs9vDUZjPnex3pdTMHjWzp4BpTfFEMU3xIBHGTjDmnHO7gZtbuhg54mR9+RTQH5nwOFlPHgUebeli5IiT9aUMKGvKJ2qLRwCVQMFR9/OBDWGqRT6jvkQe9SQytVhf2mIAvAd0M7NiM4sDrgemhrkmUV8ikXoSmVqsL606AMzsReBd4DQzqzSzW51zB4HvA68DK4CXnHMfhrNOv1FfIo8QWvPKAAAARUlEQVR6EpnC3RddDE5ExKda9RGAiIg0ngJARMSnFAAiIj6lABAR8SkFgIiITykARER8SgEgIuJTCgAREZ9SAIiI+NT/B31dmDTWgreSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Initial condition\n",
    "winit = np.random.randn(p)\n",
    "\n",
    "# Parameters\n",
    "feval = log_fun.feval\n",
    "nit = 1000\n",
    "lr = 1e-4\n",
    "\n",
    "# Run the gradient descent\n",
    "w, f0, hist = grad_opt_simp(feval, winit, lr=lr, nit=nit)\n",
    "\n",
    "# Plot the training loss\n",
    "t = np.arange(nit)\n",
    "plt.semilogx(t, hist['f'])\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can measure the accuracy of the final estimate by creating a predict method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy = 0.968198\n"
     ]
    }
   ],
   "source": [
    "def predict(X,w):\n",
    "    z = X.dot(w[1:]) + w[0]\n",
    "    yhat = (z > 0)\n",
    "    return yhat\n",
    "\n",
    "yhat = predict(Xts,w)\n",
    "acc = np.mean(yhat == yts)\n",
    "print(\"Test accuracy = %f\" % acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is not quite as good as the test accuracy for the `sklearn` method, which was about 98.5%.  The reason is that the learning rate was somewhat slow and we didn't yet fully converge.\n",
    "\n",
    "To see the effect of the learning rate, the code below tries different learning rates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lr=    1.00e-05  Test accuracy = 0.879859\n",
      "lr=    1.00e-04  Test accuracy = 0.968198\n",
      "lr=    1.00e-03  Test accuracy = 0.989399\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x131aa65bcc0>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEACAYAAAC6d6FnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd0XOWd//H3M0UzqiONZtSrJVdJ44orBoExGEIzIQQIbVNgNySbhCQLu2d/IdkkG0hIWXaBkAQIoRkCBEwvxjLgXrBluat3q/euub8/7qjZktWb9X2dc89Id+6984wH5qP7VKVpGkIIIWYew2QXQAghxOSQABBCiBlKAkAIIWYoCQAhhJihJACEEGKGkgAQQogZSgJACCFmKAkAIYSYoSQAhBBihpIAEEKIGco02QU4F4fDocXFxY34/MbGRnx9fceuQGLU5DOZmuRzmXpG85ns37+/QtM052DHTekAiIuLY9++fSM+Py0tjdTU1LErkBg1+UymJvlcpp7RfCZKqbyhHCdVQEIIMUNJAAghxAwlASCEEDPUlG4DEEJMvPb2dgoLC2lpaZnsosxoNpuNY8eOnfMYq9VKVFQUZrN5RK8hASCE6KOwsBB/f3/i4uJQSk12cWas+vp6/P39B3xe0zQqKyspLCwkPj5+RK8hVUBCiD5aWloIDg6WL/8pTilFcHDwqO7UpvQdQENbA58Wfgroadebxhm/n/G80WCkXWsf3wIKcZ6SL//pYbSf05QOgLz6PO7dcu+Iz/9y0JdZz/oxLJEQYiL4+fnR0NAwYa+3YcMGdu3axYUXXsjbb7/d7zGtra3ccccd7N+/n+DgYF5++WW6Bqr+6le/4qmnnsJoNPLoo49yxRVXDOv1Bzo/Li4Of39/jEYjJpNpVOOi+jOlA2CWbRYvfeml7t8VZ6TdWb/27Pje1u+R3Zo9nsUTQkygzs5OjEbjuFz7xz/+MU1NTTz55JMDHvPUU08RFBREZmYmmzZt4v777+fll1/m6NGjbNq0iSNHjlBcXMxll13GyZMnh1zWgc7vsnXrVhwOx6jfY3+mdAB4m7xJdiSP6NxFzkXsLtg9xiUSQkyktLQ0fvaznxEeHs7Bgwc5evTouLzOunXrSEtLO+cxb775Jj/96U8BuPHGG/nOd76Dpmm8+eab3HzzzVgsFuLj40lMTGTPnj2sWrWK559/nkcffZS2tjZWrFjB448/flYwDHR+cvLIvvuGY0oHwGi4nC7ez32fsqYyQnxCJrs4QkxLP3vrCEeL68b0mgsiAnjwmqQhH79nzx4yMjK6e7qsXbuW+vr6s4575JFHuOyyy8asnGcqKioiOjoaAJPJhM1mo7KykqKiIlauXNl9XFRUFEVFRRw7doyXX36Z7du3Yzab+fa3v80LL7zAHXfccdZ1+zs/OTkZpRSXX345Sinuuece7r777jF9T+dtAKQ4UgA4XH6YdbHrJrk0QoiRWr58eZ9ujp999tmklOPMjiagN8IOtH/Lli3s37+fCy64AIDm5mZCQs7+Y3Sg8wG2b99OREQEZWVlrF+/nnnz5nHRRReN9q10O28DYH7wfIwYOVRxSAJAiBEazl/q4+XMGTEn6w4gKiqKgoICoqKi6OjooLa2Frvd3r2/S2FhIRERERQXF3PnnXfyq1/9qs91/vGPf/Czn/0MgL/85S8Dng90P4aEhLBx40b27NkzpgEw6DgApVS0UmqrUuqYUuqIUup7nv12pdRHSqlTnscgz36llHpUKZWplEpXSi3pda07PcefUkrdOWbvoh8Wo4UoryjSy9PH82WEEBPss88+4+DBg2dt4/nlD3Dttdfy7LPPAvDqq69y6aWXopTi2muvZdOmTbS2tpKTk8OpU6dYvnw569at49VXX6WsrAyAqqoq8vLy2LhxY3eZly1bNuD5jY2N3UHX2NjIhx9+OObtAkO5A+gAfqhp2gGllD+wXyn1EXAXsEXTtIeUUg8ADwD3A1cCsz3bCuAJYIVSyg48CCwDNM91NmuaVj2m76iXOEsceyr30OHuwGQ4b292hBCjtHbtWo4fP05DQwNRUVE89dRTXHHFFfzkJz/p/pL+xje+we23305iYiJ2u51NmzYBkJSUxE033cSCBQswmUw89thjGI1GFixYwC9+8Qsuv/xy3G43ZrOZxx57jNjY2D6vPdD5ZWVl3H777QB0dHRw6623smHDhrF945qmDWsD3gTWAyeAcM++cOCE5+cngVt6HX/C8/wtwJO99vc5rr9t6dKl2mj85u3faMl/TdaOVR4b1XXE2Nm6detkF0H0o/fncvTo0ckriOhWV1c3pOP6+7yAfdoQvs+HNRWEUioOWAzsBkI1TSvxhEgJ0NW6EQkU9Dqt0LNvoP3jJs4rDkCqgYQQoh9DrhdRSvkBrwHf1zSt7hxDkPt7QjvH/jNf527gboDQ0NBB++aei6XFgp/Bjw8Of0BIiXQFnQoaGhpG9ZmK8dH7c7HZbP02soqJ1dnZOaTPoaWlZcT/Tw0pAJRSZvQv/xc0TXvds/u0Uipc07QSpVQ4UObZXwhE9zo9Cij27E89Y/9ZpdY07U/AnwCWLVumjWaZurS0NJb6LCW/Pl+Wu5siZOnBqan353Ls2LFzzkIpJsZgs4F2sVqtLF68eESvMZReQAp4Cjimadrvej21GejqyXMnettA1/47PL2BVgK1niqiD4DLlVJBnh5Dl3v2jSuX00VObQ61rbXj/VJCCDGtDKUNYA1wO3CpUuqgZ7sKeAhYr5Q6hd4o/JDn+HeBbCAT+DPwbQBN06qAnwN7Pdt/efaNK5fTBUBGRcZ4v5QQQkwrg1YBaZr2Of3X3wOcNcLK0wLd7xSemqY9DTw9nAKOVnJwMgpFekU6ayLXTORLCyHElHbeLwjj5+VHQmCC9AQSYhrx8/Ob0NfbsGEDgYGBXH311QMe09rayle/+lUSExNZsWIFubm53c/96le/IjExkblz5/LBB8Ov2T7X+Z2dnSxevPicZRup8z4AQK8GSi9P73fODSHE9NDZ2Tlu1/7xj3/Mc889d85jek8H/YMf/ID7778f6Dud8/vvv8+3v/3tYZV1sPP/53/+h/nz54/sjQ1iZgSAw0VdWx15dXmTXRQhxDCkpaVxySWXcOutt5KSkjJur7Nu3bpBe9y8+eab3Hmn3u/lxhtvZMuWLeecDhrg+eefZ/ny5SxatIh77rmn32A41/mFhYW88847fPOb3xzjd6ybEfMjpDg9M4NWHCbOFje5hRFiOnnvASg9PLbXDEuBKx8a/DiPmTwd9Pe//31+/etfj9u4jBkRAAm2BHxMPhwqP8Q1CddMdnGEEMMwU6eDfu+99wgJCWHp0qXjNnhyRgSA0WAkxZEiDcFCDNcw/lIfLzN1OuhXX32VzZs38+6779LS0kJdXR233XYbzz///Ji9pxkRAKA3BD+T8QzNHc14m7wnuzhCiBGarDuArumgV61addZ00Lfeeiv33XcfxcXF3dM522w2rrvuOn7wgx8QEhJCVVUV9fX1bNy4kY0bN3Zf19vbu9/zk5OT+e1vfwvobSGPPPLImH75wwwKgBRHCh1aB8cqj7EkdMngJwghZoypOB30RFBTuWvksmXLtH379o34/N7zm1Q0V3DJK5fww6U/5K7ku8amgGLYZC6gqenMuYDGq9uhGLqhzgXU3+ellNqvadqywc6dEd1AARzeDiL9IkmvkHYAIYSAGRQAoI8HkIZgIYTQzawAcLo43XSa0sbSyS6KEEJMuhkXAKAPCBNCiJluRgXAPPs8zAYzh8slAIQQYkYFgJfRi/n2+RwqPzTZRRFCiEk3owIA9Gqgo5VHaXe3T3ZRhBADkOmg9bV+ly9fzsKFC0lKSuLBBx8c9nUHMyMDoKWzhczqzMkuihBiGGbadNAWi4VPPvmEQ4cOcfDgQd5//3127do1qvd5phkXACkOfWZQ6Q4qxNQ3k6eDVkp13wm1t7fT3t6OvkT72JkxU0F0ifSLxG61k16Rzlf56mQXR4gp7eE9D3O86viYXnOefR73L79/yMfP5OmgOzs7Wbp0KZmZmdx7772sWLFiTN/TjAsApVT3CmFCiKlvpk4HDWA0Gjl48CA1NTVs3LiRjIwMkpOTR/tWus24AAB9RHBaQRq1rbXYLLbJLo4QU9Zw/lIfLzN1OujeAgMDSU1N5f3335cAGK3eA8IujLxwkksjhBiOmTIddH5+Pp2dnQQGBtLc3MzHH3/c3fA8VmZkACQ7klEo0svTJQCEEFNyOujS0lJuuOEGOjs7cbvd3HTTTefspjoSM2Y66DPdsPkGQnxC+ONlfxzx9cXwyXTQU5NMBz31yHTQ48jlcHG4/DBuzT3ZRRFCiEkxcwPA6aKurY68urzJLooQQkyKmRsADr0hWLqDCiFmqhkbAPG2eHzNvjI1tBD9mMptg6LHaD+nGRsARoORZEey3AEIcQar1UplZaWEwBSnaRqVlZVYrdYRX2NGdgPt4nK4eDrjaZo7mvE2eU92cYSYEqKioigsLKS8vHyyizKjtbS0DPrlbrVaiYqKGvFrzOgAWOhcSKfWydHKoywNXTrZxRFiSjCbzX2mXhCTIy0tjcWLF4/ra8zYKiCAFKfMDCqEmLlmdADYrXai/KIkAIQQM9KMDgDQxwOkV0gACCFmHgkAp4uypjJKG0snuyhCCDGhJABkQJgQYoaa8QEwzz4PL4OXBIAQYsaZ8QFgNpqZHzxfRgQLIWacGR8AoC8Uf6TyCO3u9skuihBCTJhBA0Ap9bRSqkwpldFr30+VUkVKqYOe7apez/27UipTKXVCKXVFr/0bPPsylVIPjP1bGbmFzoW0drZysvrkZBdlVGTovhBiOIZyB/BXYEM/+3+vadoiz/YugFJqAXAzkOQ553GllFEpZQQeA64EFgC3eI6dErqXiCyfvtVAne5OLn/tcv5+8u+TXRQhxDQxaABomvYpUDXE610HbNI0rVXTtBwgE1ju2TI1TcvWNK0N2OQ5dkoI9w0n2Bo8rRuCixuLKW0s5UTVickuihBimhhNG8B3lFLpniqiIM++SKCg1zGFnn0D7Z8SlFLTfkBYTm0OAFUtQ81qIcRMN9LJ4J4Afg5onsffAl8HVD/HavQfNP1WWCul7gbuBggNDSUtLW2ERYSGhoYhn+9X70deXR7vbHkHX6PviF9zsnxS9wkAOaU5o/o3G2/D+UzExJHPZeqZiM9kRAGgadrprp+VUn8G3vb8WghE9zo0Cij2/DzQ/jOv/SfgT6AvCj+aBcSHswC5b6kvb33wFgFzA1gbtXbErzlZtu3cBtXgtrqn9KLrsij81CSfy9QzEZ/JiKqAlFLhvX7dCHT1ENoM3KyUsiil4oHZwB5gLzBbKRWvlPJCbyjePPJij72k4CQMyjBtq4G6qoCqW6onuSRCiOli0DsApdRLQCrgUEoVAg8CqUqpRejVOLnAPQCaph1RSr0CHAU6gHs1Tev0XOc7wAeAEXha07QjY/5uRsHH7ENiYOK0bQjOrc0FoKa1hk53J0aDcXILJISY8gYNAE3Tbuln91PnOP6XwC/72f8u8O6wSjfBXE4XH+R+gFtzY1DTZ4xcbWstlS2VhPuGU9JYQk1rDcHewZNdLCHEFDd9vuUmgMvhor6tnty63MkuyrB0lXdJ6BJAegIJIYZGAqCXrgFh060aqKv6Z0mIHgDSDiCEGAoJgF7ibfH4mf2m3YjgnNocTMrUHWByByCEGAoJgF4MykCyI3na9QTKrcslOiAap7cTkAAQQgyNBMAZXE4XJ6tP0tTeNNlFGbKc2hziAuIItASiUFS3ShWQEGJwEgBnWOhciFtzc6RySvVSHVCHu4P8+nzibfEYDUYCLYFUNcsdgBBicBIAZ0hxpABMmwViihqK6HB3EG+LByDIGiR3AEKIIZEAOEOQNYho/+hp0xOoawRwXEAcAHarncrmykkskRBiupAA6IfL6SK9PH1aLLDSFQByByCEGC4JgH64HC7Km8s53XR68IMnWW5dLnarHZvFBuh3ANILSAgxFBIA/ejqT3+o/NAkl2RwXT2Autitdmpba+lwd0xeoYQQ04IEQD/mBs3Fy+A1LdoBcmtzu6t/QA8A0CeFE0KIc5EA6IfZaGZB8IIp3xOouqWa6tbqPgEQZNUXZ5NqICHEYCQABuByujhaeZT2zvbJLsqAuiaB6+8OQAJACDEYCYABpDhTaO1s5WT1yckuyoC6JoE7sw0AZEI4IcTgJAAGsNCxEJjaDcE5tTmYDWYi/CK698kdgBBiqCQABhDmG4bT2zml2wFy6nKIDYjFZOhZ18dmsWFQBgkAIcSgJAAGoJQixZEypXsC5dbm9qn+AX1G00BLoASAEGJQEgDn4HK6yK/Pn5L16e2d7RTUF/RpAO5it9qnZJmFEFOLBMA5dA0Im4rVQAUNBXRqncTZ4s56TkYDCyGGQgLgHJKCkzAow5SsBuqeAyjg7DuAIGuQ3AEIIQYlAXAOPmYfZgfOnpIB0N0FdIA7gMoWmRFUCHFuEgCDcDldHK44jFtzT3ZR+sipzcHh7cDfy/+s54KsQdS31U/pQWxCiMknATAIl9NFQ3tD91/cU0VOXU6/DcAAwdZgAJkWWghxThIAg3A5pt7MoJqm9dsFtEvXfEDSDiCEOBcJgEHE2eLwN/uTXjF12gGqWqqoa6sb8A6gazSwtAMIIc5FAmAQBmUgxZnC4fKp0xW0v0ngepM7ACHEUEgADIHL6eJUzSma2psmuyjA2esAn6mrDUDGAgghzkUCYAhSHCm4NTdHKo9MdlEAPQAsRgvhvuH9Pu/v5Y9RGeUOQAhxThIAQ9DVEDxVxgPk1uUSGxCL0WDs93mDMhBkDZI7ACHEOUkADEGgNZDYgNgpEwBnrgPcHwkAIcRgJACGKMWRQnpFOpqmTWo52jrbKGooGrABuIvMBySEGIwEwBC5nC4qmisoaSyZ1HLk1+Xj1tz9TgHRm90iM4IKIc5NAmCIumYGnezxADl1nkngBrsD8JY7ACHEuUkADNGcoDlYjJZJbwfompKiv1lAewuyBNHQ3kBbZ9sElEoIMR1JAAyR2WBmQfCCSQ+AnNocQnxC8DH7nPM4u7esDSyEODcJgGFwOVwcqzw2qbNs5tblDlr9A3obAMhoYCHEwCQAhsHldNHmbuNE9YlJeX1N04bUBRTkDkAIMTgJgGHoagierJlBK5oraGhvGNIdQJBFnw9IAkAIMZBBA0Ap9bRSqkwpldFrn10p9ZFS6pTnMcizXymlHlVKZSql0pVSS3qdc6fn+FNKqTvH5+2Mr1CfUEK8QyatHWCwSeB667oDkCogIcRAhnIH8Fdgwxn7HgC2aJo2G9ji+R3gSmC2Z7sbeAL0wAAeBFYAy4EHu0JjOlFKda8QNhnOtQ7wmfzN/pgMJrkDEEIMaNAA0DTtU+DMb5HrgGc9Pz8LXN9r/9803S4gUCkVDlwBfKRpWpWmadXAR5wdKtNCijOFgvqCSflizanNwdvkTahv6KDHKqX0wWCyKpgQYgAjbQMI1TStBMDzGOLZHwkU9Dqu0LNvoP3TTtfEcJOxPkBOXQ6xAbEY1NA+tiBrEFXNcgcghOifaYyvp/rZp51j/9kXUOpu9OojQkNDSUtLG3FhGhoaRnV+f1rdrRgw8Nb+t9CyJnZeoGOlx4izxA35PalmRW5j7pj/G4zGeHwmYvTkc5l6JuIzGWkAnFZKhWuaVuKp4inz7C8EonsdFwUUe/annrE/rb8La5r2J+BPAMuWLdNSU1P7O2xI0tLSGM35A3nqraeotdSOy7UH0tLRQvUL1dw8+2ZSFw3tdd/79D3Sy9MntJyDGa/PRIyOfC5Tz0R8JiOtAtoMdPXkuRN4s9f+Ozy9gVYCtZ4qog+Ay5VSQZ7G38s9+6alFEcKGRUZuDX3hL1mXl0eGtqgk8D1ZrdKG4AQYmBD6Qb6ErATmKuUKlRKfQN4CFivlDoFrPf8DvAukA1kAn8Gvg2gaVoV8HNgr2f7L8++acnldNHQ3tDdK2ciDKcLaBe71U5jeyOtna3jVCohxHQ2aBWQpmm3DPDUun6O1YB7B7jO08DTwyrdFNU9M2h5OgmBCRPyml1hExsQO+Rzei8OH+YbNi7lEkJMXzISeARiA2Lx9/Kf0BHBObU5hPuG423yHvI5dqs+GKyypXK8iiWEmMYkAEbAoAy4HK4JXRtgqJPA9dYVADIaWAjRn7HuBjqmWto7OVpcN6JzlYL6Ng1N01Cqv16oo+Nyungy/Uka2xvxNfuO+fV70zSN3Npcrk+8fvCDe+kKABkNLIToz5QOgFNlDVz16GejusZ/7PiQ2GAfYoN9ibX7EBfs2/17iL8Fg2Fk4eByunBrbo5UHGF5+PJRlXEwZU1lNHU0DfsOoHcbgBBCnGlKB0Cs3YdHb1s6onPdmsZn+zMwB4WTV9nEkaJaPsgopcPdM3jLajYQY+8Jh1iHL3HBPsTafYkItGIyDlxDluJIAfQlIsc7AIa6DOSZ/Mx+mA1maQMQQvRrSgdAgLeZDckj773iU3mC1NTk7t87Ot0U17SQV9VIbmUTeRWN5FU1kVfZyKcny2nt6OnXbzIoooK89XDofQfh8CEqyAebxUZcQNyEzAza1QNoKOsA9KaUIsgaJHcAQoh+TekAGGsmo4GYYB9ign1YO7vvc263Rll9K3mVjeRVNvWERGUjB/KrqW/p6D5WKQgPsGIMDWdH/QEe35pJnKMnKPwsY/vPmlubi4/JhxCfkMEPPkOwNVjaAIQQ/ZpRAXAuBoMizGYlzGZlxazgPs9pmkZ1U3tPOHiC4VBdHDXGnfzmk51o7fbu40MDLCyKDmRxTBCLowNJibLh4zXyf+qc2hzibHEjasyWOwAhxEAkAIZAKYXd1wu7rxeLY3qWMTha6cVX336J390exBy/C7vD4URpHQcLavjgyGkAjAbFvDB/FscEsjg6iMUxgcQ7fIf8hZ5bl8uS0CWDH9gPu9VOXl3eiM4VQpzfJABGYXbQbKxGKydrjrBxztUkRdj6PF/V2MbBgmq+yK/hi/wa3viimOd35QNg8zazOCaw+05hUVQgNh/zWa/R1N5ESWPJkBaB6U+QNUiqgIQQ/ZIAGAWzwcyC4AUDDgiz+3px6bxQLp2nL+DS6dbIKm/gi/yeUNh28hSap2NSgtNXrzby3CnMCfXr/ut9OJPA9SmD1U5zRzPNHc3DGkUshDj/SQCMksvp4oVjL9DW2YaX0eucxxoNijmh/swJ9eerF8QAUN/STnphLV/kV3OwoIZPjpfx6v5CAPytJuYmZAIQYBjZ+jm9RwN7+0kACCF6SACMksvp4q9H/sqJqhOkOFOGfb6/1cyaRAdrEh2A3uBcUNXMgfxqdmZV8lHJR2j+ilueyGReaBUXz3Fy8Vwny2LteJkGn8mj92jgCL+IYZdPCHH+kgAYpa4lItMr0kcUAGdSSnV3Vb1+cSTubYoDp8P57oYU0k6U8/T2HJ78NBtfLyOrEhykznVy8Rwn0Xaffq/XNRpY2gGEEGeSABilUN9QQnxCOFR+iK/N/9qYXz+3Lpc59lncc3EC91ycQENrBzuzKtl2soy0E+V8fEzvaTTL6UvqnBDWLwhlebwdo2eKC5kPSAgxEAmAMbDQuXBcFol3a25ya3NZFrqse5+fxcT6BaGsXxCKpmlkVzSy7UQ5aSfLeX53Hk9vz8HhZ2FDcihfSokgKUrmAxJC9E8CYAy4HC4+yvuIyuZKgr2DBz9hiE43nqals2XAOYCUUiQ4/Uhw+vH1C+Npautg6/Fy3j1cwqv7C3l+Vz7Bfl4Yos1klBbRuUDrvjMQQggJgDHQtULY4YrDpEanjtl1u+YAGuokcD5eJr7kCudLrvA+YbCt2Zd3jpxi264tbEgO5aqUcFbEB0sYCDHDSQCMgfnB8zEqI+nl6WMbACOcBRT6hsFX3gpH6zAS2mTntf1FPL8rn3CblY2LI/ny0igSnH5jVmYhxPQhATAGvE3ezAmaM+YrhOXU5uBv9ifYOrpqpWBvOzUtNTz2tSU0tXWw5VgZrx8o5I/bsng8LYvFMYF8eUkU17gi+h2NLIQ4P0kAjBGX08Xb2W/T6e7EaDCOyTVza3NHPAlcb8HWYLJrsgH9zuCahRFcszCCsroW3jhYxGv7i/jPNzL4r7ePsn5+KF9eGslFs53nXA9BCDH9SQCMEZfTxcsnXia7NpvZQbMHP2EIcupyWBm+ctTXCbLoM4KeuTxmSICVuy9K4FtrZ3GkuI5X9xfy5sEi3jlcQrjNyk3LovnqBdFEBMoIYiHORxIAY6RrQNjhisNjEgCN7Y2UNZWNqP7/THZvOy2dLTR3NONjPnvAmFKK5EgbyZE2/uOq+Xxy/DQv7ing0U9O8b+fnOKSuSHcuiKG1Lkh0nAsxHlEAmCMxAbEEuAVQHp5OjfMvmHU18utzQWGvwpYf4IsPaOB+wuA3rxMBjYkh7MhOZyCqiY27c3nlX2FbHl2H+E2K1+9IJpbl8cQEmAddbmEEJNLKnnHiFKKFGcKh8oPjcn1RtMD6ExdYxOGOxo42u7Dj6+Yx44HLuWPty0hMcSPP3x8ijUPf8L3Nn3BF/kyuEyI6UzuAMbQQsdCnih6goa2Bvy8Rte1MrsmG4MyEO0fPepydd0BjHQ0sNnYc1eQU9HI33bm8uq+Qt48WMzCKBt3ro7jS65wLKaxafwWQkwMCYAx5HK60NA4UnmEFeErhnROh7uDvLo8Tlaf7LOVNpYSFxA36BTTQ2H3Hrv5gOIdvjx4TRI/vHwurx8o5Nkdudz3yiH++91j3Lo8hq+tjCVUqoeEmBYkAMZQsiMZgPTy9H4DoKqlSv+Cr+r5os+qyaLN3QaASZmIs8WxJGQJc4LmcFHURWNSrt5tAGPFz2LijlVx3L4yls8zK/jr9lz+d2smj6dlcWVKOHetjmNJTOCou7AKIcaPBMAYsllsxAXEcaDsACeqTnCy+iSnqk9xolr/uaK5ovtYh7eDOUFzuHX+rcwJmsOcoDnE2+LH5C/+M/mYffA2eY/LhHBKKdbOdrJ2tpO8ykb+tjOPV/YV8NahYlxRNu5aHcfVroghrV0ghJhYEgBjzOWF5/NdAAAgAElEQVR0sTlrM58XfQ7oy0YmBiayOmJ19xf9nKA5Yzpp3FAEWcZ/beDYYF/+39ULuG/9HF7/ooi/bs/hvlcO8av3jnPbili+tjJmXF9fCDE8EgBj7BvJ3yDCL4JZtlnMCZpDTEAMZsPkT69gt9qpap2YNQF8LSZuXxnLbSti+OxUBU9vz+H3H5/ksa2ZLA8z4JxTS1KEbULKIoQYmATAGJsVOIt7F9072cU4S5A1qE8V1ERQSnHRHCcXzXGSWdbAsztyeXlvHl969HNWxNv5+oXxXDY/VAaXCTFJpGJ2hrBb7VS3Tly//bbONp4/+jztne0AJIb48fPrk/l9qg//cdU8Cqubuee5/aQ+spW/fJZNXUv7hJVNCKGTAJgh7FY7Vc1VaJo2Ia/3Sf4nPLz3YbYXb++z39esuPuiBLb9OJUnvraE8ABvfvHOMVb99xYefDODnIrGCSmfEGKqVwF1tkFt4cjOVUaYoC+76cButdPmbqOpowlfs++4v97hCn2JzKyarH7XSDAZDVyZEs6VKeFkFNXy9PYcXtyTz9925XHJ3BC+viaeNYnB0o1UiHE0tQPg9BH4fdKIT19hDQPtdki+EULmjWHBpp8gq2csQHPVhAZAdm32oMcmR9r43U2LeODKebywK58Xdudx21O7mRPqxz+tief6RZF4e8koYyHG2tQOgMAYuPZnIzu3rYnm3S/h/dlv4dPfQGgKpNwIyV+GwNFPrzDd2K2e0cCtVUQzvu+/3d3O0cqjQM+ylkMR4m/lB+vn8C+pCbx1qJhntufy768f5uH3j3PL8hhuXxkrU1MLMYamdgD4BMOSO0Z8enrLPFKXzocj/4DDf4ePH9S3mNV6GCy4Hnwntj/+ZOkOgObx7wqaWZ1Ja2crDm8H2bXZZ61DMBir2chXlkVz49Io9uRU8fT2HJ7clsWT27K4eI6Tm5fHcOm8EMyyYI0QozK1A2As+IfCyn/Wt6psyHgN0v8O79wH7/0bJFwKKV+BuVeB5fxdG7erCmgiegJ1Vf9cM+sanjnyDKebThPmGzbs6yilWDErmBWzgimoauLlvQW8sq+Ae57bj9PfwleWRnHzBTHEBJ97imshRP9G9SeUUipXKXVYKXVQKbXPs8+ulPpIKXXK8xjk2a+UUo8qpTKVUulKqSVj8QaGxT4LLvox3Lsb/vlzWPUdKDsGr38LfpMIr34dTrwHHW0TXrTx1t0GMM6jgUGfCynIEsTaqLUA3ctRjka03YcfXTGXHQ9cyp/vWIYr0sYft2Vx0W+28rW/7OKtQ8W0dnSO+nWEmEnG4g7gEk3Teo8wegDYomnaQ0qpBzy/3w9cCcz2bCuAJzyPE08pCEvRt3UPQsFuvYro6Bv6HYI1EBZcp98ZxK4Bw/SvavA2eeNt8p6QAMioyCDFmcIs2yxAbwheHbl6TK5tMhpYvyCU9QtCKalt5u/7Cnl5bwHffekLgnzMfMkVznWLIlkaE4RBBpgJcU7jUQV0HZDq+flZIA09AK4D/qbpHdF3KaUClVLhmqaVjEMZhs5ggNhV+nblw5CdpofB4VfhwLPgHwHJN+htBuGL9PCYpuxW+7gHQENbA9m12VwRfwV2qx2bxUZWbda4vFa4zZt/XTeb71ySyOeZFbyyr4BX9xfy/K58ImxWrlkYwbWLIlgQHiDdSYXox2gDQAM+VEppwJOapv0JCO36Utc0rUQpFeI5NhIo6HVuoWff5AZAb0YzzF6vb21NcPI9PQh2Pwk7/w+CE/W7guQbwZE42aUdNrvVPi4zgvZ2pPIIGhouhwulFAm2hDGpAjoXg6FnyonG1g4+OnqazYeKeerzHJ78NJsEpy/XLozkypQwZof4SRgI4THaAFijaVqx50v+I6XU8XMc29//dWeN1FJK3Q3cDRAaGkpaWtqIC9fQ0DCq8yEYwu/B5LgVZ/lOQso+JTDtIVTar6jzT6Qs5CLKQi6kzTI9ehJpjRr5tfmj/Dc5tw9rPwSg5ngNaafSsDZZSW9K737N0X8mgwsE7oiDGyK82Vvawa6SZv7w8Ul+//FJnN6KRSFGFoeYmBNkwCTVRMDEfC5ieCbiMxlVAGiaVux5LFNK/QNYDpzuqtpRSoUDZZ7DC6FPB/QooLifa/4J+BPAsmXLtNTU1BGXLy0tjdGc39c1+kNdMWS8TkDGqwRkPU1i1jMQd6F+Z7DgWvAOGqPXG3tbtm9hR/GOMfw3Odvrn7xOrDuWL637EgAFRwvYsXcHrpUu7Fb7GH8mg7va83i6roUtx8r4+NhptmVW8FFeC/4WExfPdZI6N4Q1icGE22buGIOJ/lzE4CbiMxlxACilfAGDpmn1np8vB/4L2AzcCTzkeXzTc8pm4DtKqU3ojb+1k17/PxIBEbD6O/pWkQkZr+ptBm/9K7zzQ736aP41egOzYw6YLJNd4m5dbQDD7Zc/VJqmcbjiMCvDV3bv624IrsnGHmYf89ccqtAAK7euiOHWFTE0tXXw+akKPj52mk+Ol/F2uv6f4SyHL6sTg1mT4GDlrGCCfMd+cR4hppLR3AGEAv/wfJGYgBc1TXtfKbUXeEUp9Q0gH/iK5/h3gauATKAJ+KdRvPbU4EiE1Afg4vuh5JAeBBmvwYl39eeVEYITIGSBZ5uvP9rjwTDxUxvYrXY63B00tDfg7+U/5tc/3XSaiuYKUhwp3ft69wRaFrasz/Gd7k6KGoqICZjYhWJ8vExcnhTG5UlhuN0ax0vr2ZFVweeZFbx+oIjnd+XrZXf6siQmSN9iA5kd4i9TV4vzyogDQNO0bGBhP/srgXX97NeAqTdR/lhQCiIW6dv6/4KKk/r4grKj+mNpOhx9k+4mD5MVnHP7hkLIfAiIHNdeRt2jgVuqxiUA0svTAfoEQJhvGD4mn37nBNp0YhOP7H2Ej77yEQ5vx5iXZygMBsWCiAAWRATwzbWzaOtwc6iwht3ZlRzIr2HLsdO8ul+fkNDPYmJRdCBJEQHMC/dnfngACU4/GZEspq3zfyTwRDMYPV/q84Ebeva3NUH58b7BkJ0Gh17qOcZi6zk3ZIF+9xAQAf7hYLWNOhy6RwO3VBMbEDuqa/UnoyIDs8HMXPvc7n1KKeJt8WTVnN0V9JP8T+jQOkgvT+fSmEvHvDwj4WUycEGcnQvi9LDUNI3cyiYO5FVzIL+agwU1PLM9l7ZONwBmoyIxxJ/54f7MDwtgltOXeIcv0XYfCQYx5UkATBQvH4hcom+9NVXpwXD6iCccjsGR12H/M32PM/voQRAQAf5hvX7utc8vDEwD11t33QFUtlSO9bsDIL0inXn2eWctbJ8QmMCukl199jW0NXDg9AFAD46pEgBnUkoR79C/1L+8NAqA9k432eWNHC+t42hJHcdK6vn8lF591MVoUEQHeXvO9SPO4UNUkDcRgd5EBnrjb538ZUKFkACYbD52iF2tb100DepLoCpHf6wrhvpSqC+GuhJ95HJ9qb5ewpm87eAXAr5OffMLAV8H+IZgN+lfOtVVmRC2Sg+lMdLh7uBo5VE2Jm4867l4WzybszbT0NbQvW9nyU46tA4sRkv3zKHThdloYG6YP3PD/LluUWT3/urGNnIqG8kpbyS3spHsCv3nXdlVNLf3naYiwGoiMsiHyEBvIgOtRHrCITTASqi/lZAAC1azTIEtxpcEwFSklP5XfUDEwMdoGjRVegKipCccGsugoQwaK6DkoP7YWgdAkALiYqja+gt449/Ay687HHoHhR4czr4/WwPPWQWVVZNFc0czKc6Us55LsCUAfdcG+LTwU/y9/Lkk+hK2FW4bt55JEynI14sgXy+WxPTtCqxpGuX1rRTWNFNc00xRdTNFnsfC6iZ2Z1dS39px1vUCfcyE+lsJtVkJ9bfo4dD75wArDj8vTFLVJEZIAmC6Usrzhe3Qu5yeS3szNJZjaSzHd+s9VM9eB7YUPRwayqCxXJ8ptWC3vu/s8XlgMHvuKhyesAjp9bOTww2ZAKR4vux7mxXY0xMokEDcmpvPiz5nTcQaFjoXsjlrM4UNhUT7n5/rNCilCAmwEhJgPSscutQ2t3O6roXS2hZO13Vtrd0/nyytp7yhlU5338/GoMDh1xMIoQEWwgKshAd6E2HTH8NtVrmbEP2SAJgJzN764jqBMdh9QqgMCIG1P+z/WHenfmfRWN5zJ9HoCYmG8p6fy47rP3uqoTIcdmw+3sT870r9bsEWpfdqskUS6R+OWRnJztvG6raVHCvPoKK5grVRa0kM1KfUOFJ55LwNgKGweZuxeZuZEzpw76xOt0ZlQ2t3MJTWtVDmCYrSuhYKq5vYn1dFdVP7Wefafb0It1kJt3kTEXj245nBImYGCYAZJsgadO75gAxG/a96vxAI7bsc5+Hywzxz5Bkui/kn1seux2ww6dVLjRWkb72XZIMVNfubUFcEtUVQVwiFezE1VxEXGUb2ybe47/TT/PFkIMoWwJrtfyHAnoBZGThy8i02+MRCUJweWOIsRkPPnUQKtgGPa2nvpKS2hZKaZop7P9Y2U1DVxO6cSupb+lY5GRVE799KtN2H2GAfYu2+xAT7EGPXN1+LfFWcj+RTnWHsVjvFDWfNwDGoXSW7+NdP/pX2znY+yvuI3+z7DTfNuYmvzP0KPgHhZDWVsM51NyzqZ6hHWxOz0u7jSPUJjgd+lc+0z0nubCG4qQYOv8Y8u5Uj2R/Cjuf14/3DISheHzDnmA3O+fq4icDY82Jq7vFmNRu7ey4NpKG1ozsYimua2X7wOJqfjfzKJtILS6ht7nsX4fCzEBvsQ4LTl8QQPxJD/Ehw+hEV5COD46YxCYAZxm61c6TiyLDO2ZK3hR9/+mNiA2L542V/5GT1SV44/gKPH3qcPx3+E0tCluDW3H0GgPXh5UNCyEI+LNnB0cilHC56i39Z9C+w8F9A00ja/iBv5b6H+4b/xlCdq7dHVOdA5sdw8IWe65i89UAI8QSCc56+BcVNysjq6czPYmJ2qD+zPVVO4U3ZpKb2dFGubWonr6qRvMom8quayK9sIqeykU+Ol/HKvsLu4ywmA/GOnlBIDPFjXpg/8Q4/CYZpQAJghumaEnqovW7ezHyTn+z4CcnByTx+2ePYLDZCfUNZG7WW3NpcNp3YxBuZb2A2mAcOACA+MB4NjW3129DQuCjqIv0JpUgKW8KmrH+QG7OUWS595pD6tnq+9u7X+F7SI6yzhunjI8pP6GMmcrdD+ss9Fzf7QGgyhLsgzKU/hiyYUvMwTTc2HzMun0BcUYFnPVfT1EZWeQOZZfqWVd5IemEt7xwuQfM0JVhMBuaF6aOlu7Z54f4EyPiHKUUCYIYJsgTRoXVQ11aHzTJwPTLA80ef5+G9D7MifAWPXvIoPua+4wbibHE8sPwBvrv4u1Q0V3SPNO5P15xAO+p3EGwNZr59fvdzycHJgN4Q3NVjaHPWZnJqc3i3aBvrUn8L0cv7XrClDipOQfkxKM3Qp9tIfwX2/kV/3mDS7w66AiHMBeELz+t1nydKoI8XS2PtLI3tO7lfS3snWeUNHC+p51hJHcdK6/jgSCmb9vYsAxJt92Z+WACuKBsLowNxRQZi85FQmCwSADOM3btnPqCBAkDTNJ449ARPHHqCdTHr+PVFvz5rdG9vvmZffM0D1zcDxAXEYVAGWrQWNkRtwKB66vLjbfF4m7zJqMjgmoRrcGtuNh3fBMDe0r24NXef4wGwBkDUUn3r4nZDTS6UpOuBUJIOWVvg0Iv688oAIUkQtQyiLtC34ERpVxgjVrORpAgbSRE9/11pmsbpulaOleijpo+W1HGsuI4Pj57uPibe4ctCTyAsjA5kQXiAdFudIBIAM4zdogdARXMFdqudhvYGGtoaaGxvpKFdf9xdspvXTr3GdQnX8dPVP8VkGP1/Jl5GL6L9o8mry2Nt5No+zxkNRubb55NRmQHoDc65dbmsCl/FzpKdnKo+1Wd+oQEZDGCfpW9J1/fsrz+tz9ZatA8K90JGr6k2rDaI7BUIUUun9JoO041SijCblTCblUvmhXTvr21uJ6OoloMFNRwqqGFHViVvHNQ7J5gMinnh/iyNCWKZZ16mMJt1st7CeU0CYIYJ9tZXL/v6B18/53G3L7idHy370dl/eY/CLNssCuoKWBWx6qznkh3JvHziZdrd7bx0/CXsVjv/b+X/46p/XMXukt1DC4CB+IeC/+Uw53L9d7cbKk/pYVC4Fwr3wae/Bk2f4I2QBRC7BuLW6I9+IQNfW4yIzdvMmkQHaxJ7ZoEtrW3hUKEeCAcLanhlXyHP7swDICrIm+Vxdk8gBJHg9MMgjcyjJgEww8wOms19S++jrbOtu+rGz8tPfzT74Wf2I8ASMC7TM9+ZdCchjSH9TkWd7Ejmb0f/xqeFn7KtYBvfTPkm0QHRxAXEsbt0N3ck3TF2BTEYPL2I5sLi2/R9rfVQ/IU+GjpvBxx8Efb+WX8ueLYnDC7U52yyRQ58bTFi+p1CGFckhQH6pHvHSurYm1vNvtwqPj1Vwetf6BPuBfqYWRar3yGsTggmKcImvY5GQAJghjEoA/+UPDlr8SwNXUq9f32/zyUF64POHtrzEAZl4Ka5NwGwPGw5b2e/Tbu7HbNhHBsLLf4Qf5G+AXS269VGedv1XkcZr8P+v+rPBcXpYRC3Rj/eFjV+5ZrBzEYDrii9J9I3LoxH0zTyKpvYm1vFvtxq9uZV8fExfcVZf6uJFfF2ViU4WDUrmHlh/nKHMAQSAGJKiPaPJsArgNLGUtbHrifMV/8rcEX4Cl45+QpHKo6wKGTRxBXIaPY0Fi+DNd/Tp8g4naGHQd52OPEOHPQMXAtOhFmp+hZ3obQhjBOlFHEOX+IcvnxlmT5tSFl9C7uyq9iZVcHOrMruQAjyMbNyVjCrE4JZlRBMgtNv2k82OB4kAMSUoJQiKTiJnSU7uWXeLd37Lwi7AIA9pXu6A+DP6X/GbDBzV/JdE1dAg1HvRhq+EFZ9W29HKDsKOdv0hX0OvqR3QVUGCF/UEwjRK8AsDZjjJcTfyrULI7h2oT5zbnFNMzuzKtmRVcnOrAreyygFwOlvYZUnENYkOoi2j91U6NOZBICYMq5JuAZ/L3+WhfasHRxkDWJu0Fx2l+zmbtfdfFb4GY9+8ShWo5Ub59yIn9ck9es3GCAsWd9W3QsdbVC0Xw+D7DTY/j/w+e/05T9jVkL8xXoghC+UUcvjKCLQmy8vjeLLS6PQNI38qqaeQMiuZPMhvadRVJA3axIcrE4MZtWsYEICZmZISwCIKeOahGu4JuGas/avCF/BpuObKG0s5Sc7foLT20l5cznv577PjXNunISS9sPkBbGr9O2Sf9cblfN29ATClp/pmzUQ4tfqYRCfqi/7KVUT40IpRWywL7HBvty8PAZN08gsa2BHViXbMyt4L6OEl/fpg9Rmh/ixOiGY1YkOVsYHz5jBaRIAYspbEb6Cvx39G9/68FvUtNaw6UubuP/T+3kj842pEwBnsvjDnCv0DfSxCDmf9gTCsbf0/QGRnsbniz0NytLDaLwopbrnP7pzdRydbo2jxXVsz6pgR1Zld7dTpSA5wsbqxGBWJzi4IC4IH6/z86vy/HxX4ryyJGQJRmUkty6X7y/5PnPtc7k+8Xp+u/+3ZNdkd08fMaX5h4LrK/qmafqEd9lpeiic/AAOvaQfF5zYEwbxF+lLhopxYTQoUqJspETZ+OeLE2jrcHOwoIbtmXqD8tOf5/DktmzMRsXi6CBWedoPFkUH4mU6P0aPSwCIKc/Py48V4SvocHdwV9JdAFydcDV/OPAH3sh6g/uW3seOoh08sv8RfrHmFywIXjDmZShvKqe1s5Uo/yjq2upQqH7HMwyJUnrVT3ACXPANvUH5dIYeBjnb9Inu9j0FKH21t/iL9CqjmFUyl9E48jIZWB5vZ3m8nR+sh6a2DvbmVrMjq4IdmZU8+skp/mfLKbzNRi6I18cfTPcxCBIAYlp4fN3jaGgYPQ2oDm8Ha6PW8lbWWywJWcIP035Im7uNh/Y8xLMbnh2TLn9vZb1FiE8IK8JX8O+f/zvlTeW8ef2b3Lf1PowGI0+uf5L08nTibHEEeAWM/IUMBn3CunAXrP6OPgah6ICnh9E22PMn2Pl/+gR3kcv0wWgxq/QJ8rzPnq1TjA0fLxMXz3Fy8RwnoE+RvTNb7120I6uSh947DkCA1cTKWfrdweqEYBJDpk+XUwkAMS0Y++k5szFxI2kFaXz3k++yIHgBG+I28Lv9v+ODvA/YELeBkoYS3st9j1vn3YrVNHgvj+aOZl489iK3zLsFDY2f7fwZsQGxPHflc+w/vZ8Odwe5tbnsP70fo8FIZXMld753J7cvuJ1/XfKv5Nfnd896Oro3a4aYFfp28b9BWxMU7NLDIPcz2PGo3sMIpa/aFrNSD4SYlTIobRzZfMxsSA5jQ7I+RqWsroWd2XqD8vbMyu4J7hx+FpbFBrEkNpClsUEkRdim7OR2EgBi2lobtZYI3wiCvYP54/o/4mvy5Z3sd/jD/j8Q7hvOD7b+gPLmcoobivnPlf9Jh7uD//vi/1gSuoSLoi7Crbl5eM/DLAldwhVxV/DS8Zf4w4E/oKER4RtBa2crJ6tP8o/Mf9Dh1pdQ/PPhP9OhddDR2cELx16gQ+vgQNkBXjnxCr/e+2ve//L71LbWEuYbNuh020Pm5QMJl+obQFuj3uU0byfk74RDm3qmwbbF6MERuQwil+hVSLLE5rgICbBy3aJIrlukN9znVzaxI6uCXdmV7M+v5v0j+hgEL6OBpMgAlsQEsTQ2iCUxQVNmcjsJADFtmQ1mXrv2NbxN3t13CD+64Ed868Nvcfu7txPqG8pV8Vfx8omXWRW+ii35W3gr+y1ePP4ir1z9Cp8Xfc6Lx1/kzaw3SQpO4rmjzwHw4rEXmWefR4BXAHVtdTz2xWNYjBa8jF68nf02BmXoM2X10cqjWI1WOrVOPi38lN/s/Q3XJ17PqohV5Nflc8v8W8isziTJkdT/1NbD5eV7xrQVHXobQv4uPRByPoPDf9efM5j0FdQiluiBELFE/904M7o5TqSYYB9igmO4eXkMoI9SPpBXwxf51RzIr+a5XXk89XkOAGEBVpIjbSRHBpAcoTdEh/hbJrzqSAJATGtnDgRbGb6Sq2ddTX59Pr9P/T2BlkByanO4b9t9uDU3ty+4nc1Zm/n+1u9T2FDIkpAlZFRkcNf7d1HRXMFdSXfx1yN/pbyonNvm38aBsgMcrTzK6ojVWIwWthZsZaFzIaebTlPaWIrFaKG1s5XdpbsBeDrjaVo6W9hTuodD5YfIrMmkvr2evxz+Cz9f83N+sesX/D719zyT8Qz/tvzf+KzwM26ccyOVLZUjrz4ymiBikb6t/Gd9X12x3o5QfEB/PPoGHHhWf85k1RfLCU3q3sxtNSP9CMQAQvytfaqM2jrcHC2pY39eNYcLa8gormPL8dPdq6g5/CzdgZAUEUBdo3vcy6i0rlefgpYtW6bt27dvxOenpaWRmpo6dgUSozYRn8mZy13m1uZy1/t3ccPsG/ju4u+ytWAr39v6PQK8Anjjujd49eSrPH7ocebb57Pp6k3c8OYNZNVm8dyVz7G3dC+PfvEoP1r2I4zKyMN7H+Ybyd+gqKGI93Pf55Z5t/DScb0Lp4/Jh6aOprPK07U/2BpMZUslEb4RFDcWExcQR25dLimOFPLq8lgSsoRQ31A63B1snL2RxrZGXE4XFpNl9BPhdXU9Lf5C304f0bfGsp5jfEN6QiE4sWfzD5PBauOksbWDYyV1ZBTVklGsP54qa6DTrREXYCDtP64c0XWVUvs1TVs22HFyByDOO2feRsfZ4vjkpk+6q14ujbmUX174S6L9o3H6OPl6ytcpbCjkxjk3YlAG7lt2H+9kv4PL6SLMN4ydJTu5PPZy2t3tPPrFo1wScwkZFRm8n/s+V8VfxaeFn1LaWMpX532VZzKeIcw3jNLG0u7X7wqFypZKAIob9ekIcutyAThccRiAtMI0FAoNjbez36atsw2nt5Mo/yg6tA4ujrqY2YGz8TX7EmeLw+ntHHqVQe+upym9Bs81lEPZETK3v0mif5telbT3L9DR0nOM2ddzbq9QCIyBwGjwD5epLUbB12JimWedgy4t7Z2cOt3Ajj0j/+N3qCQAxIxwZr37tQnXdv9sMVr45YW/7P79oqiLuhetD/MN4+krnu5+bs/X9gAwO3A2AV4BLHQu5Iq4KyhqKCI1KpVnMp7h9vm38/ihx7EYLUT5RZFekd49fYVRGenUOgcsp4Z+R97a2QpAWXMZZc36X+np5elnHT/LNguX00VcQBwup4sY/xhsFtuQej0B4OcEv1QK8yGx687M7Ya6IqjM7LsVe6qStF5VEwaTPpo5MAZs0XooBMZAQIQeDn6h+uyocgcxZFazkZQoG5WZ4x+sEgBCjICP2ad73qIfLP0BoFc9PbT2IdbFrKO+vR5fky8dWgdFDUXcmXQnj+x7hGsTruUfmf9gvn0+x6qO4e/lT31bffdf/sOVXZtNdm12v8+tCFtBkiOJxMBEZgXOIsY/ZmiD1wwGzxd5NCRc0ve5jlaozoWaAqjN9zwW6I/ZaVBfAme+D6NFDwL/UM9jGPiF6b/7OsEnuGez2iQsJpAEgBBjRCnFl2Z9CYB7F90LQKe7k1vn3YrJYGKufS7z7fOJ8IsgNTqVx754jDWRa3ju6HP6cpn1BeTV5WE1WWlobxh1eXaX7u5unO7NpEwsCV3CQudCkhxJzLLNos3dNrSLmiw9q6n1p6NNv3uoL/Fsp6GhtOexMktfT6G5uv/zlbFvIPjYe372deh3E1abZwvs+dnLV4JjBCQAhBhHRoMRH4M+9/zK8JUA/PNCvafO/677XwBunncz9W31dLg7OFF9AofVwebszdgtdvLr83nt1Gu4tbHrEdKhdbCndA97Svf0fcLTSSguII41kWtYELyAhMVvGI0AAAd4SURBVMAE4gPi8TEPcf58kxfY4/XtnIVohYbT0FgBTVXQVOnZKnr9XAUVJ/Vjmqv6Vj2dyWA6Oxi8A/sPC4t/z+blB5YA/WeT19De43lEAkCIKaCraqYrJO5bel/3cz9Z9RNaO1vpcHdwqvoUbs1NVUsVx6uOk1+Xz+mm0xwoOzBmZcmty+1uoO5PhG8EF0dfTIItAZfTRZR/1PDnRTJZPA3JMUM73u2Glhr9zqGlVv+5pVbfmmv631db6DmnBtztg7+G0atXMPj3DQqLn+cxwBMavfd7AsTLV3/Oy1e/1jS4I5EAEGIasBgtWIyWPstiXhZ7WZ9jNE2job2BsqYyqlqqqGqpoqihiPy6fHLrcsmpzaGqpWrUZSluLO7u+joQb5M3y0KXsdC5kHn2ecyyzSLML2zk3VkNBk910AhmR9U0vVdTV1C0NejrNXRtbQ3QWuf5/YznGkqh8lTPcx3NQyyvqW8gnPVz1+8DPe+Hd1Ph8N/rMEkACHGeUEqfodTfy58EEs55bFtnG3VtdVS1VFHeVM7WA1vxjfClpLGEnNocjlcdH1VZmjua+azoMz4r+mxIx1uNVi6MvJDVkav1sPANw261YzFaRj9yWil9OgyzNwSEj+5ane29QqO+V2jU6VN0tDXqz3X/fMbvdcVnPzdA4/98/9lw1W2jK+8gJACEmIG8jF44vB04vB3MCZpD+6l2UpemDni8pmm0dLZQ3VJNdWs1BfUF1LTUcKr6FPn1+eTU5nC66fSIy9PS2cLH+R/zcf7HQz7HbrUTGxBLUnASgZZAQnxC8PPyw2q0YjVZ8TH5YDaasRqteBm9MBvMmAwmzAYzRoMRo9K3YU2/YDSP/E6kP5oG7c1nBEUDtDWQlXGCxWPzKgOSABBCDEophbfJG28/byL8IkgKThrW+Zqm0djeSHVrNeVN5ZQ0llDcUExuXS7ZNdkcrz7ePeHeUHVVc31R9sWwzjtTVxB0hYJSCqMyYlAGFKp7n0EZurfufRgwGAz6Y6/nu66hUH2P73XM/2/vfl7jKOM4jr8/benFqOAPctBSFETsTRAPnjyoxEsVFX/gRSmWCnpX8A/Qo4VijVriSZEgWqHiQQgieGjxpBaxeDFYEBUKLWybnfl6mGRN0kyz0+w+8+zO5wVLkmd+fZMPs19mdrPP5m3WjidVj5Jy+hqApDngXWA38GFEvJ26BjNLSxIze2eY2TvDvhv3jWSfZZT0+j327NrD+UvnB1cpvX6PXtHjcv8yvaLHleIKK+XK4IX0oizoR/W1iKIai2LwcxklZZQEQREFETEYX/s+IigpB+tueFBSltXXtfVXypUN6xRREMRV2wZR7TtKbinGPxtc0gYgaTdwDHgUWAZOSzoZEb+krMPMJt8u7Rq8PXX/Tftbrmb0lpaWxn6M1BNbPgici4jfI+IK8CnwROIazMyM9A3gDuCPdT8vr46ZmVliqV8D2Orl9g3vgZJ0GDgMMDs7u6PLoIsXLya5jLLhOZM8OZf8pMgkdQNYBta/AnQn8Of6FSJiHpiHaj6AnXx2vOcDyI8zyZNzyU+KTFLfAjoN3CPpLkl7geeBk4lrMDMzEl8BRERf0mvAN1RvAz0RET+nrMHMzCrJ/w8gIk4Bp1If18zMNkp9C8jMzDKR9aTwki4Av22x6GbgwhBjtwF/j6G07WxVS6r9DLvNduvVLW8ynlMm0F4u487kWstyz8XnSv3YTjLZHxG3b7tWRGT7AOaHHa8ZO5NT3Sn2M+w2263X5G8/CZm0mcu4M5nkXHyutJtJ7reAvmowXrduG0ZVy/XsZ9httluvyd++bjynTKC9XMadybWW5Z6Lz5VmtYxU1reAdkrSmYh4oO067H/OJE/OJT8pMsn9CmCn5tsuwK7iTPLkXPIz9kym+grAzMzqTfsVgJmZ1XADMDPrKDcAM7OO6kwDkHSDpI8lfSDpxbbrsYqkuyV9JGmx7VqsIunJ1fPkS0mPtV2PVSTdJ+m4pEVJr45inxPdACSdkPSXpJ82jc9J+lXSOUlvrA4/BSxGxCvAweTFdkiTXKKaHe5QO5V2R8NMvlg9T14Cnmuh3M5omMvZiDgCPAuM5O2hE90AgAVgbv3AunmHHwcOAC9IOkA198DabGRFwhq7aIHhc7E0FmieyVury218FmiQi6SDwPfAt6M4+EQ3gIj4Dvh303DdvMPLVE0AJvz3zl3DXCyBJpmo8g7wdUT8mLrWLml6rkTEyYh4CBjJbexpfCKsm3f4c+BpSe+R17/Cd8WWuUi6VdJx4H5Jb7ZTWmfVnSuvA48Az0g60kZhHVd3rjws6aik9xnRR+onnw8ggS3nHY6IS8DLqYuxgbpc/gH8JNOOukyOAkdTF2MDdbksAUujPNA0XgFsO++wtcK55MeZ5ClZLtPYADzvcJ6cS36cSZ6S5TLRDUDSJ8APwL2SliUdiog+sDbv8Fngs/C8w0k5l/w4kzy1nYs/DM7MrKMm+grAzMyunxuAmVlHuQGYmXWUG4CZWUe5AZiZdZQbgJlZR7kBmJl1lBuAmVlHuQGYmXXUfztCHReVwZwUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Learning rate values to test\n",
    "lr_test = [1e-5,1e-4,1e-3]\n",
    "ntest = len(lr_test)\n",
    "\n",
    "# Strings for the legend\n",
    "leg_str = []\n",
    "\n",
    "for i, lr in enumerate(lr_test):\n",
    "    # Run the optimizer\n",
    "    w, f0, hist = grad_opt_simp(feval, winit, lr=lr, nit=nit)    \n",
    "    \n",
    "    # Plot the results\n",
    "    plt.semilogx(t, hist['f'])\n",
    "    leg_str.append(\"lr=%12.2e\" % lr)\n",
    "    \n",
    "    # Measure the test accuracy\n",
    "    yhat = predict(Xts,w)\n",
    "    acc = np.mean(yhat == yts)\n",
    "    print(\"lr=%12.2e  Test accuracy = %f\" % (lr, acc))\n",
    "    \n",
    "plt.grid()\n",
    "plt.legend(leg_str, loc='upper right')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that increasing the learning rate, speeds the convergence time, but the optimization is beginning to go unstable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adaptive Step Size\n",
    "\n",
    "The above example shows that gradient descent is sensitive to the step size.  We now consider a slight variant of gradient descent with an adaptive step-size using the Armijo rule in class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def grad_opt_adapt(feval, winit, nit=1000, lr_init=1e-3):\n",
    "    \"\"\"\n",
    "    Gradient descent optimization with adaptive step size\n",
    "    \n",
    "    feval:  A function that returns f, fgrad, the objective\n",
    "            function and its gradient\n",
    "    winit:  Initial estimate\n",
    "    nit:    Number of iterations\n",
    "    lr:     Initial learning rate\n",
    "    \"\"\"\n",
    "    \n",
    "    # Set initial point\n",
    "    w0 = winit\n",
    "    f0, fgrad0 = feval(w0)\n",
    "    lr = lr_init\n",
    "    \n",
    "    # Create history dictionary for tracking progress per iteration.\n",
    "    # This isn't necessary if you just want the final answer, but it \n",
    "    # is useful for debugging\n",
    "    hist = {'lr': [], 'w': [], 'f': []}\n",
    "\n",
    "    for it in range(nit):\n",
    "\n",
    "        # Take a gradient step\n",
    "        w1 = w0 - lr*fgrad0\n",
    "\n",
    "        # Evaluate the test point by computing the objective function, f1,\n",
    "        # at the test point and the predicted decrease, df_est\n",
    "        f1, fgrad1 = feval(w1)\n",
    "        df_est = fgrad0.dot(w1-w0)\n",
    "        \n",
    "        # Check if test point passes the Armijo condition\n",
    "        alpha = 0.5\n",
    "        if (f1-f0 < alpha*df_est) and (f1 < f0):\n",
    "            # If descent is sufficient, accept the point and increase the\n",
    "            # learning rate\n",
    "            lr = lr*2\n",
    "            f0 = f1\n",
    "            fgrad0 = fgrad1\n",
    "            w0 = w1\n",
    "        else:\n",
    "            # Otherwise, decrease the learning rate\n",
    "            lr = lr/2            \n",
    "            \n",
    "        # Save history\n",
    "        hist['f'].append(f0)\n",
    "        hist['lr'].append(lr)\n",
    "        hist['w'].append(w0)\n",
    "\n",
    "    # Convert to numpy arrays\n",
    "    for elem in ('f', 'lr', 'w'):\n",
    "        hist[elem] = np.array(hist[elem])\n",
    "    return w0, f0, hist\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now run the new optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsnXmYXFWZ/z+n9q7e0+l09nT2kIQlpA0JawIiy4ARRzYXFhHEBXHUUWZ+44wb6jjjKOCCKIigiAgqWwAFbNkCIQlJyEr2Pen0vlTXfn5/3Lq3blVXdVfv1cn7eZ5+qurce26drtNd3/su5z1Ka40gCIIg5BuO4R6AIAiCIGRCBEoQBEHIS0SgBEEQhLxEBEoQBEHIS0SgBEEQhLxEBEoQBEHIS0SgBEEQhLxEBEoQBEHIS0SgBEEQhLzENdwDGAxGjx6tq6ur+9y/o6ODwsLCgRuQMCTIvI08ZM5GJv2dtzVr1tRrrSt7Ou+4FKjq6mpWr17d5/61tbUsXbp04AYkDAkybyMPmbORSX/nTSm1N5fzxMUnCIIg5CUiUIIgCEJeIgIlCIIg5CUiUIIgCEJeIgIlCIIg5CUiUIIgCEJeIgIlCIIg5CUiUIIgCEJeIgIlCIIg5CUiUIIgCEJeIgIlCIIg5CUiUIIgCEJeIgIlCIIg5CUiUIIgCEJeMuQCpZSarZRaZ/tpVUp9USn1DaXUQVv7pbY+/6aU2qGU2qaUumioxywIgiAMPUO+H5TWehtwGoBSygkcBP4M3Aj8SGv9v/bzlVJzgWuAecB44EWl1CytdWxIBy4IgiAMKcPt4rsA2Km17m7zquXAo1rrkNZ6N7ADWDQkoxMEQRCGjeEWqGuA39tef14ptUEp9YBSqjzRNgHYbzvnQKJNEARBOI5RWuvheWOlPMAhYJ7W+qhSqgqoBzTwbWCc1vqTSqmfAiu11r9N9LsfWKG1fiLtercAtwBUVVUtfPTRR/s8tvb2doqKivrcXxgeZN5GHjJnI5P+ztuyZcvWaK1rejpvyGNQNi4B1mqtjwKYjwBKqV8CzyReHgAm2fpNxBC2FLTW9wH3AdTU1OilS5f2eWC1tbX0p78wPMi8jTxkzkYmQzVvw+niuxabe08pNc527ApgY+L5U8A1SimvUmoqMBNYNWSjFARBEIaFYbGglFJ+4ELg07bmHyilTsNw8e0xj2mtNymlHgM2A1Hgc5LBJwiCcPwzLAKltQ4AFWltn+jm/DuBOwd7XIIgCEL+MNxZfIIgCIKQEREoQRAEIS8RgRIEQRDyEhEoQRAEIS8RgRIEQRDyEhEoQRAEIS8RgRIEQRDyEhEoQRAEIS8RgRIEQRDyEhEoQRAEIS8RgRIEQRDyEhEoQRAEIS8RgRIEQRDyEhEoQRAEIS8RgRIEQRDykn7tB6WUmg4c0FqHlFJLgVOAh7TWzQMxuL7S2hnh+Y2H+9x/45Eowaz9VZ+v2xNqkC7tdTlYPK0Cn9s5OG8gCIIwCPR3w8IngBql1Azgfozt2R8BLu3vwPrD3sYAt/52bf8usq6f/fOMYq+Ly08bz5ULJ3LapDLUYKmhIAjCANFfgYprraNKqSuAH2ut71FKvdNTJ6XUHqANiAFRrXWNUmoU8AegGmPL96u01k3K+Ca9C0P0AsANWutu1WPmmCL+ePs5ff6lVq9eTU1NTZd2rft8yR7RDN7FG9rD/OWdg/xp7QEeeWsfM8cUcWXNRK5YMJHKYu+gva8gCEJ/6K9ARZRS1wLXA5cn2tw59l2mta63vb4DeElr/X2l1B2J118DLgFmJn7OAH6eeMyKz+3kpHEluf8WaRwtdvSrfz5y7qxKvrl8Hs9sOMwfV+/nuyu28t/Pb2PZ7DFcWTOR8+eMwe2UkKQgCPlDfwXqRuBW4E6t9W6l1FTgt3281nJgaeL5b4BaDIFajhHX0sCbSqkypdQ4rXXfg0wnKMU+N9cumsy1iyazo66dP67Zz5/WHuTFLUepKPRwxYIJXFkzidlji4d7qIIgCCg9QH4rpVQ5MElrvSGHc3cDTYAGfqG1vk8p1ay1LrOd06S1LldKPQN8X2v9WqL9JeBrWuvVade8BbgFoKqqauGjjz7a59+lvb2doqKiPvcfScTimnfrY7x6MMq6uhgxDVNLHJwz0cUZ41wUukdOrOpEmrfjBZmzkUl/523ZsmVrtNZd4yhp9DeLrxb4YOI664BjSql/aK2/1EPXs7TWh5RSY4C/KaW2dvc2Gdq6qKrW+j7gPoCamhq9dOnSHH6DzNTW1tKf/iONC4AvAg3tIf6y7hB/XL2fhza38cSOGNedWc3N50xjVKFnuIfZIyfavB0PyJyNTIZq3vobdCjVWrcCHwZ+rbVeCLy/p05a60OJxzrgz8Ai4KhSahxA4rEucfoBYJKt+0TgUD/HLWSgosjLTWdP5bnbz+Gpz5/F+SdVce8/dnL2f7/M95/bSkN7aLiHKAjCCUR/BcqVEJOrgGdy6aCUKlRKFZvPgQ8AGzFS1K9PnHY98GTi+VPAdcpgMdAi8afBRSnFKRPLuOfaBfz1i+dywUlV/OKVnZzzg7/zvee2iFAJgjAk9DdJ4lvAC8DrWuu3lVLTgO099KkC/pxYh+MCHtFaP6+Ueht4TCl1E7APuDJx/gqMFPMdGGnmN/ZzzEIvmFlVzD3XLuD2C2Zwz8s7uO+VXTz0xl6uWzKFm8+dxugiSVMXBGFw6JdAaa3/CPzR9noX8M899NkFnJqhvQEjHJLeroHP9WecQv+ZMaaYu65ZwG3nz+QnL2/nl6/u4qGVe/nEkincIkIlCMIg0N8kiYnAPcBZGIkLrwG3a60PDMDYhDxkxpgifnzNAm67YCY/eXkHv3p1Fw+t3EN1RWG3/Qo8Tq5bMoXlp07A4Rg5mYGCIAwf/XXx/RqjtJHpjvt4ou3Cfl5XyHOmVxbxo6tP47bzZ/DA67s51tZ9XGpPfYB/+cN6fvGPXXz14tksmz1Gyi0JgtAt/RWoSq31r22vH1RKfbGf1xRGENMqi/jOh07u8bx4XPPsu4f54V+38ckHV/O+6nK+dvEcaqpHDcEoBUEYifQ3i69eKfVxpZQz8fNxoGEgBiYcXzgcistPHc/fvnQe3/nQfPY0BPjIvSv51G/eZuuR1uEeniAIeUh/LahPAj8BfoQRg3oDybITusHtdPDxxVP48OkTePCNPfy8dieX3PUq5f7cFwI7FFx68ji+fOFsSv25ln4UBGGk0d8svn0YlSQsEi6+H/fnusLxj9/j4rNLZ/DRRZN5eOVe6nqIYdlp7ozw2zf38syGw9xx8Rw+snCiJF4IwnFIfy2oTHwJESghR8r8Hm67YGav+9163jT+68lNfPWJDfz+7X18e/n8QRidIAjDyWDsryC3ssKgM298KX+8dQk/vPJU9jcGuPwnr/Ho1jChaGy4hyYIwgAxGAI1iNv6CUISpRT/vHAiL315KdcumszzeyJc8dM32FHXNtxDEwRhAOiTQCml2pRSrRl+2oDxAzxGQeiW0gI3373iZG4/3cuR1iD/dPdrPLxyDwO1lYwgCMNDn2JQWmvZ0U7IOxaMcfGxSxbzlT9u4OtPbuLxtQepLMqeHTimxMenz53GlB6qYAiCMDwMRpKEIAwbY4p9PHjD+3ho5R7+9M5BDrcEs5772o56Hnt7Px89YzK3nT+TymKpJygI+YQIlHDc4XAobjhrKjecNbXb8462Brn7pe387q19PL7mAP960WxuOLNaSjAJQp4wGEkSgjAiqCrxcecVJ/Pil85j8bQKvvn0Zm797RpaOiPDPTRBEBALShCYOrqQ+6+v4f7XdvP957Zy2T2vctv5M/G6ut6/zRlbwuyxEoIVhKFABEoQMFLWP3XONE6fUs5tj7zDVx/fkPXc9580hs8tm8GCyeVDOEJBOPEQgRIEG6dPLuelL5/HoebOLsfiWrPi3SM88PpurvjZG3z49Al8e/l8Cr3ybyQIg8GQ/2cppSYBDwFjgThwn9b6LqXUN4CbgWOJU/9da70i0effgJuAGPAFrfULQz1u4cTB53YyrbIo47EvXFDMTWdP5ee1O/lZ7Q7e2dfMnR+aT2WxF4dDUV1RiFPqAgrCgDAct35R4Mta67VKqWJgjVLqb4ljP9Ja/6/9ZKXUXOAaYB7GIuAXlVKztNZS00YYFgq9Lr5y0WzOnjmaLz66jo/+6i3r2LzxJXzzg/NknytBGACGXKC01oeBw4nnbUqpLcCEbrosBx7VWoeA3UqpHcAiYOWgD1YQumHxtApe+OK5vL6znrjWNHWE+VntTj5y70rmjC3G5UxaUiU+N59YPIWL5o2VyuuCkCNqOMvBKKWqgVeA+RhV0G8AWoHVGFZWk1LqJ8CbWuvfJvrcDzyntX487Vq3ALcAVFVVLXz00Uf7PK729naKijK7eIT8JR/mLRTVPL8nwq6WeEr74Y44dQHNxCLF1bM9nFwpcSvIjzkTek9/523ZsmVrtNY1PZ03bAKllCoC/gHcqbX+k1KqCqjHKDb7bWCc1vqTSqmfAivTBGqF1vqJbNeuqanRq1ev7vPYamtrWbp0aZ/7C8NDPs9bNBZPbHn/HvsaA5w+uYxRhckyTNPHFHHN+yYzdfSJVXYpn+dMyE5/500plZNADcttnFLKDTwB/E5r/ScArfVR2/FfAs8kXh4AJtm6TwQODdFQBWFAcDkdLD9tAhfPH8uvX9/DincPW2WY4hr+vu0Yv/jHLv7plHH8y/tnMrrIS7HPLQkXwgnNcGTxKeB+YIvW+v9s7eMS8SmAK4CNiedPAY8opf4PI0liJrBqCIcsCAOG1+Xk1vOmc+t501Pa61qDPPzmXn716m6e3WD8G4wv9fHJs6cysbwAALfTwaKpoyj2yTb3wonBcFhQZwGfAN5VSq1LtP07cK1S6jQMF98e4NMAWutNSqnHgM0YGYCfkww+4XhjTImPL39gNh87Ywp/23yEUDTO8xuP8J1nt6Sc53M7+OCp4/ns0hlUn2DuQOHEYziy+F4j8667K7rpcydw56ANShDyhLGlPj6xpBqAm86eyp6GAMGIcT/WHIjw9IZDPLHmAH9+5yD/edlcLpo/1urrdTop9Yt1JRw/SCqRIOQpSqkuSRNLplfwxQtm8tUnjD2vvv7kppTjC6eUs/y08Sw/bQKlBSJWwshGBEoQRhhjSnw8cP37eH7TERo7wlZ7Y0eYZzcc5j+f3MQ9L+/gyoUTcSWSLIp9bs6dVcmsqiLZTkQYMYhACcIIxOFQXHryuC7tX7hgJu/sa+I/n9zEz/+x02rXGu5csYWpowv5z8vmsmzOmKEcriD0CREoQTjOWDC5nKdvOzul7UhLkL9vq+OB13Zz44Nv8++XzuHkCWXW8dFFHqZXFkmVCyGvEIEShBOAsaU+rl00mQ+fPoFPP7yG767Y2uWccr+bD546ni++fxbltkXEgjBciEAJwgmE1+Xkl9fVsH5/M5GYUUVGoznQ1MnrO+r53Vv7+Ovmo9amjAqYUlHIwinlXDi3Cp/bOYyjF040RKAE4QTD7XRkrLZ+Vc0krj+zmh88v5WmRPJFNK55a3cjD76xh8mj/Hz49AmcM3M0XpchVBPLCyjzi7UlDA4iUIIgWJw+uZxHb1mS0haLa1557xhff3IjP35xOz9+cbt1zOVQnDqpjCsXTuTq902SDEFhQBGBEgShW5wOxbI5Y3htzvnUtQbZcKAFjbHD8Pr9zdRuO8Ydf3qXX722m6LE7sKjCj0snFLOwinljCv14XE58LqclBW4JRFDyBkRKEEQcmZMiY/3z/VZry+aN5avfGA2j6zax982H0UDWmt2HWvn5a11Xfq7ncbi48mjCvG6HDTWB3mh8V3K/G7KCtyU+z0UeJz43E6KvC4mlhcwusiL1+UQYTsBEYESBKFfOByKjy+ewscXT0lpb+mMsHJnAx2hKOFYnGAkRl1biE2HWjnY3Ek4GqO1Pc72tiM0ByJE491v/eN1OfB7nJQUuBlV6KHE56bI66K80E2Jz02h10Whx0mRz02xz0Wx10WRz0Wh10WB20mB24nf67TiZ0L+IwIlCMKgUFrg5mJbrcBMmPsKaa0JhGM0BcJ0hmMEI3FagxH2NQZoCoQJRgyB6wzHaOmM0NARojkQZn9TgKaOMK3BKLEeBM7E7VQJMXNRnBAwU9wK3E4KEiJY4nNT4HZYFp3fY5zj97rw284t9LjwuR0SfxsERKAEQRh2lFKWUNg5K8f+WmvCsTjtwSgdoRhtoQjtwSjtIeMnGIkRCBs/7aFo4rzk8ZZAmMPNMToTItgajFhp+LmNH/zuVPHymVabJylkBR7jtd8meuZxe3uBeczrxO924nI6evFpHj+IQAmCMOJRSuF1OfEWOakYgB3ktdYEI3E6I7GEuEUtgQuEDREMRgxBC4RjBELG8Y5wjI6EIJpid6Q1YvULhIz2ntyZ6XhcDsN6s4mcacEZz5NCZ7abbk2v20hQ8bkd+BLC6bO1eV1OvC7jWL5tkCkCJQiCkIZSyviS9wxOvCoSixMIGwJmip8paOZjwHasIxylMxyjI5R6fkN7mP3hKMFInEA4mhDUeJ/H5XIofG5DsLwuB26XA7fT+PE4lfW8pjTK0oH7OLKPZwjeQxAEQbDhdjooLXAMypYo8bi2LD9TsIKRGKFo8nk4GicUjae0haLGYzCSbI/G40RicSIxnXiME4lqemkA9pkRIVBKqYuBuwAn8Cut9feHeUiCIAh5icOROZ43kNTW1g7ate3kfeRNKeUEfgpcAszF2Bp+7vCOShAEQRhs8l6ggEXADq31Lq11GHgUWD7MYxIEQRAGmZHg4psA7Le9PgCc0V2HNWvW1Cul9mY4VAq05NA2Gqjv5TgHmkzjGupr9aZfT+f29Xhv2mXeetcvl/O6O6e3x2TOBqbf8fC/NqXnUzDSKfP5B7gSI+5kvv4EcE+G824BVid+bslyrftybFudB793l3EN9bV606+nc/t6vDftMm+965fLed2d09tjMmcD0+9E+l8bCRbUAWCS7fVE4FD6SVrr+4D7erjW0zm25QMDOa6+Xqs3/Xo6t6/He9s+3IykecvlvO7O6e0xmbOB6XfC/K+phBrmLUopF/AecAFwEHgb+KjWetMgvudqrXXNYF1fGBxk3kYeMmcjk6Gat7y3oLTWUaXU54EXMNLMHxhMcUrQkyUm5CcybyMPmbORyZDMW95bUIIgCMKJyUhIMxcEQRBOQESgBEEQhLxEBEoQBEHIS0SgBEEQhLxEBCoHlFKFSqnfKKV+qZT62HCPR+gZpdQ0pdT9SqnHh3ssQu4opT6U+D97Uin1geEej9AzSqmTlFL3KqUeV0p9ZiCvfcIKlFLqAaVUnVJqY1r7xUqpbUqpHUqpOxLNHwYe11rfDHxwyAcrAL2bM23UbrxpeEYq2OnlvP0l8X92A3D1MAxXoNdztkVrfStwFTCga6NOWIECHgQutjd0Uzl9Isl6gLEhHKOQyoPkPmdC/vAgvZ+3/0gcF4aHB+nFnCmlPgi8Brw0kIM4YQVKa/0K0JjWnK1y+gEMkYIT+DMbbno5Z0Ke0Jt5Uwb/DTyntV471GMVDHr7v6a1fkprfSYwoCEQ+bJNJVPl9AnAn4B/Vkr9nPytJ3aiknHOlFIVSql7gQVKqX8bnqEJ3ZDtf+024P3AR5RStw7HwISsZPtfW6qUulsp9QtgxUC+Yd6XOhpiVIY2rbXuAG4c6sEIOZFtzhoA+YLLX7LN293A3UM9GCEnss1ZLVA7GG8oFlQqOVVOF/IKmbORiczbyGPI50wEKpW3gZlKqalKKQ9wDfDUMI9J6B6Zs5GJzNvIY8jn7IQVKKXU74GVwGyl1AGl1E1a6yhgVk7fAjw2BJXThRyRORuZyLyNPPJlzqSauSAIgpCXnLAWlCAIgpDfiEAJgiAIeYkIlCAIgpCXiEAJgiAIeYkIlCAIgpCXiEAJgiAIeYkIlCAIgpCXiEAJgiAIeYkIlCAIgpCXiEAJgiAIeYkIlCAIgpCXiEAJgiAIeYkIlCAIgpCXiEAJgiAIeYkIlCAIgpCXuIZ7AAOJUupy4PLi4uKbZ82a1efrdHR0UFhYOHADE4YEmbeRh8zZyKS/87ZmzZp6rXVlT+cdlxsW1tTU6NWrV/e5f21tLUuXLh24AQlDgszbyEPmbGTS33lTSq3RWtf0dJ64+ARBEIS8RARKEARByEtEoARBEIQeuerelVTf8SwtgciQvacIlCAIgtAjq/Y0AlD7Xt2QvacIlCAIgpCXiEAJgiAIeYkIlCAIgpCXiEAJgiAIeYkIlCAIgpCXiEAJgiAIeUneC5RS6kNKqV8qpZ5USn1guMcjCIJwIqOUGrL3GlSBUko9oJSqU0ptTGu/WCm1TSm1Qyl1R3fX0Fr/RWt9M3ADcPUgDlcQBEHogaGs3zrY1cwfBH4CPGQ2KKWcwE+BC4EDwNtKqacAJ/C9tP6f1Fqbq8L+I9FPEARBOAEYVIHSWr+ilKpOa14E7NBa7wJQSj0KLNdafw+4LP0ayrAnvw88p7Vem+29lFK3ALcAVFVVUVtb2+dxt7e396u/MDzIvI08ZM5GHls2b2F+SXBI5m049oOaAOy3vT4AnNHN+bcB7wdKlVIztNb3ZjpJa30fcB8Y2230pxS8bAEwMpF5G3nInI0gnn8WgJPmnkRR8/YhmbceBUop5Qe+DEzWWt+slJoJzNZaP9PH98wUYcvq1NRa3w3cndOFExsWzpgxo49DEwRBELoj35Ikfg2EgCWJ1weA7/TjPQ8Ak2yvJwKH+nE9C63101rrW0pLSwficoIg5EgwEuNYW2i4hyH0kbq2IK3BCKt2N/LGjnq2HmnNem4wHBuyceXi4puutb5aKXUtgNa6U/VPQt8GZiqlpgIHgWuAj/bjeoIgDDM/+/sO/rzuIK9+9fzhHorQBxbd+VKXtic+cyYLp5R3af/qExt48OK+b/feG3KxoMJKqQISbjil1HQMi6pHlFK/B1YCs5VSB5RSN2mto8DngReALcBjWutNfRp91/e7XCl1X0tLy0BcThCEHDncEuRoi1hQxxM7j7UP9xBysqC+ATwPTFJK/Q44C7gxl4trra/N0r4CWJHjGHNGa/008HRNTc3NA31tQRCyEwjHCMfiRGNxXM68X/8v2BjKdU29pUeB0lr/VSm1BliMkeBwu9a6ftBH1gckSUIQhodAOApAZyRGsQjUiCKP9alnF59S6iWtdYPW+lmt9TNa63qlVFeHZR4gSRKCMDwEEoHzzsjQBdCFgSGP9Sm7BaWU8gF+YLRSqpxkengJMH4IxiYIwgjBEqghzPASBoaR6uL7NPBFDDFaQ1KgWpGSQ4Ig2DBdfAERqBFH/spTNwKltb4LuEspdZvW+p4hHFOfkRiUIAwP4uIbueSxAZVTksQ9Sqn5wFzAZ2t/KHuv4UGy+ARheBAX38hF57ENlUupo/8ClmII1ArgEuA1bBXKBUE4sekUgRqx5LMFlUs+6EeAC4AjWusbgVMB76COqo/IQl1BGHoisTjhWByAgLj4RhwjXaA6tdZxIKqUKgHqgGmDO6y+IWnmgjD02BMjOhPJEsLIYUS7+IDVSqky4JcY2XztwKpBHZUgCCOGzhSBEgtqpJHPFlS3ApUoCvs9rXUzcK9S6nmgRGu9YUhGJwhC3tNhs5rExTfyyGN96l6gtNZaKfUXYGHi9Z6hGJQgCIPHgaYAz288wqfO6eqpD0Zi/Lx2J59ZOh2f25ly7OGVe6go8tIeinJVTXLHHLvVZN+KIdu1/vD2PhZMLmdWVXG34wxGYtz7j53cel5qf3v7weZO3tzVwMfOmNKlv9aan9Xu5NpFkxlV6AHg5a1H8bqcnDVjdNb3/eumI5QUuFk8raLb8Q0mr7x3jL+sO8gNZ1azancjWw63MWNMEXGticU10bimdlsdY0t8TCz389LWo+xtCKRco8DtpKrESySmOdjc2a/xRBIxRpNofGhkLRcX35tKqfdprd8e9NH0E1kHJQg989T6Q/zg+W1csWACFUWp+U6rdjdy10vbqaku55yZlVZ7KBrj608mNx2wC5Q9BmV/vmZvE3e9tJ1FU0elCMLX/7KJ65ZM4T8um9vtONfubeLHL25nUfUozrT1t7e/vrOen9Xu5KOLJnfZSG9vQ4D/eWEbVSU+PrJwIgB3vbidkgJ3twL1oxe3M6GsYFgF6roHjCjKn9Ye7Pa8DWRPCOuMxNiTJlq9wqZBj63en3LopX1R3t/3K+dMLkkSy4CVSqmdSqkNSql3lVJ56eKTJAlB6JmWQASA1mDXhIbWoHGsPe1Y+uu47Q46m4sv09qoWFwTjsUJRnt2BYai8ZRHqz1xNx+OxQlH42id+Y7evOu33/1HYpporPu7/2gsTiwe7/acEwF78kQokvp5RHr4DAeKXCyoSwZ9FIIgDBktnQmBSjzaaUsIUVuaIHWEUgWlMxKj0Gt8fWRz8ZlVJezVJUIJYQpGehaAbAIVjprCo60vymhMk+aRtB1L9o/G48R6cE+ZLrQTnXxInsilksTeoRiIIAhDQ3PCgmrJIFCmaLWF0iyotNcd4aglUKal5HM7Ulx8wQzlj0xhShedTIQzWED21/b1V+FYnAJSFSoaTwpZsk0T6cE6imlNPB++nYeZfPgEZOMWQTjBsCyoYHYLKt2l15G2vikQsrvyjGMVhd4UMTKfhyKpiRP2x+4wLaVwNLtARbOImNGWsKBsghSN6R4tqGgObsATgXzQaBEoQTjBSLr4usag2swYVChVvDJZUCam1TS6yJO6JiqDi88UppwsqGjSOrITiWrruClCmQTKFC+7uy4ai/cYP4nFexaxE4F8sCKPK4GSUkeC0DOmQGVy8VkWVLqLL82iypS5N6rQQyCSPC9Zny8pHqaLLxcLyoxXpVtQSdeftp5nsnhMYYqmufh6SoCISgwKGCEuPqVUm1KqNe1nv1Lqz0qpvCp5JFl8gtAzrd24+FqzJklkfx0IRSlwO/F7XakJE6Y7z5axZz4P9cPFl2yPEcliZUHSqkpNkujZfReLx/PCehh28uAzyCWL7/+AQ8AjGJsWXgOMBbYBD2BUOhcEYQQQjcWtBIjMWXymi6/7JIkUCypPcwsZAAAgAElEQVQSw+9xUuB2ZnbxZRCtfrn4bBZUplRyE1OIImkuvp6so1xETBgacnHxXay1/oXWuk1r3aq1vg+4VGv9B6B8kMcnCMIAYl/71K2Lr4c0c7sF1RmO4fc68XucqUkS4a4JEaFeuPhMYUoXs4gtc8+KQUUzufiyWVDdi2NcYlDACHHxAXGl1FVKKUfi5yrbsXz4HQRByBG7KHW7ULeLBZUqZnYLqiMUxe92UeBxplY2z5Cxl8zi64UF1SUGlUyMsOJRGeJKyQQKuwXVc3zJiEHJQt088PDlJFAfAz6Bsc3G0cTzjyulCoDPD+LYBEEYYEyBUqp3C3Xb0y0oWxZfZyRGQcLFF4omF8IGM2XxmTGoXlSSyBaDisTiSRdfBpehZUHZ08zjPbv4YnGNGFBGLcOsqOyHBpJcFuruAi7Pcvi1gR2OIAiDiSlQY0t8XQRKa21ZTm1pCRTpSRKp66BiFCZcfGAIU6HXZUszz5TFl/tC3XAsVcwyx6AylTpKzeKLJ4SnOxef1losqAT5oNG5bPleCdwMVNvP11p/cvCGJQjCYGAK1KRRfnYda085FgjHiMU1HqeD9lAUrbVVgLVLFp/NguoIRSn3+ylI1BoyBMuVMQZlz+zr9g6dZLwq20LdcDRuxZ66TZKwFuymPmbCPBSTJInuXXxD9PHkksX3JPAq8CIgm70IQh7QEYpy+T2vcecVJ7Nketeq2//vz+/i9zj5f/+UWjHcEqhyP+v2NaeIkOnWG1fmY29DgM5IDL/H+IroksUXSo01+T1OChLnBtMsp1W7G1l054us+n/vtywnrZPC8dO/7+C9o23cdc0Cqu94FoD7r69JWlAJgfrIz9+gzO+mstiowB6Jxa3Y0+bDrdz4oLHhwg1nVvPgG3u4ZP5YAPY1dljXBUO47K+V6vplfKglmHLOici3ntnMt57ZnPHYUMl3LgLl11p/bdBHMgDIdhvCicK+xgC76jtYt785o0C9tbvRqpVnpyUQBmDyKL9RVTwSpyDhmjPdeuNKDYFqD0ZTBOq8WZX80ynjuLd2Z5dKEnYXXyCD5VTXFurSZsaj1u9vZsuR1hTX269f30Oh17ieKVSr9zYB8OHTJwCpMain1x+y+j74xh4Antt4BIC39zSlfAbpaev5kAwgZCaXJIlnlFKXDvpIBgBZqCucKNS3G1/4DYnHdBo7wjQnxMhOcyCC3+NkdLGxgZ99sa6Z1Te+tABILRjbEYpSWuDmqppJFPlcGRbqumwuPuNYpu3f7Yt2Q7bCsaFIvItwZK/Fl3TddefiEwaPodL0XATqdgyR6kxUkWhTSrUO9sAEQcjOsYRF0tDRVYRicU1TIExjhmPNnRHKCtyU+NxAaiafKVbjynxA6lqo9lByew2/x0lHQny01smFugkLKlMNPjCSFEIpCRPJjL5wLN5lz6FkkkTq16G9ekR3SRLC4DFUVmcuWXzd78ssCMKQY1pQ9RksqJbOCFobMaVoLI7LmbwPbQ5EKPV7KC1wW+eaWDGohAXVnmZBFSVcboUeF0fbgoBh/WgNfq/TsqA6M2yzYZ6bsmjXSjk3xCnbvk/haJYsvmjcVotPLKihZKjS8LMKlFJqjtZ6q1Lq9EzHtdZrB29YgiB0h2lB1bd3tZIaO5Ki1dwZYbRtW/dW04JKCJTdxWfGoCaUJVx8CcGKxuJ0RmIUeY0+fq+LQL0hGqarz+9OjUHF4rqLay4YiaVl9CVcfJE4oWisy9qo7At1u1pQ6VaWMLgM1e1AdxbUl4BbgB9mOKaB8wdlRIIg9IgpTJliUI0dSdFpDoRTBKq5M8y00UWU+Ix/ffuWG/YsPkhaUKY7r9CyoJxWkoSZEOH3uFJcfKb1VOJzWbGtYDSWsv4pZFu0G9ddyymFstTiS12oKzGo4WDYXXxa61sSj8uGZiiCIOSK6dpr7AgTj2scjuTSfrsF1RRIXXDbHIhQ5ndncfFFcDoUY4p91mtIWklFVgzKZaWZm0KU7uLrtG3BYQpUZziWWtk8bXfd9OrqPW1YGIwkq1aIQA0tw+7is6OUOpOuC3UfGqQxCYLQA6aLLxrXtAYjlPk91jG7BdVkS5TQWtPcGaHU76Y4Q5JEWzBKsc9lCZGZJGEKlJkkUeg1LCitddLF53FaKemdNldemd8DDQHAEJRgJIbX5TDiTtEYDpIClV5eKVupI9Nqstf9E4EaWvJmHZRS6mFgOrCO5EJdDYhACcIwUd8eosjroj0Upb49lCZQdgsqKVDBSJxwNE5ZgQePy0GB25kWgzIEyuNy4HU5kmWPMlhQcW0IiGkpFbhdeF1GMkYgnHTxjSpMjst08ZX53RxtDRGMxPGT3BsqvfSSfaGuvbq4KUYB21osyeIbWoZqv6xcLKgaYK7uqS6JIAhDQjQWp6EjzOmTy1mzt4n69jAzxiSPN3ZEcDsVkZhOcfE1dxpiVeY3rKfSAndqdfPOCMWJRIhin8sSJsvF50taUGZ7wBafcjhUYk+oqCVc5TbhDIYNy6qswJMQqJghUJYFlcXFZ0uGsLcHMqyzEoaGoVKDXNZBbcTYoFAQhDygMRBGa5gz1lgB0pCWydfYEWJsqQ+3U6VYUM0JsSpLxJ9KClxdkiSKEyJU5HV1dfF5khYUGAJhJkuYGXzmnlCmBVWeEEMwLKhQNG7Fv4wUdW2LQSXHolRSiELR1EW85vNAWvklYejIpxjUaGCzUmoVYPkOtNYfHLRR2VBKnYSxWHg08JLW+udD8b6CkK/UtxmiYwlUR2omX0NHmFGFXoKROM0pGX3G89KEaJT40iyoYISJ5X7AsJZMF5+51Ybp4itMiFGHzVIy6/D53M4UF58pXJCMQZkp7sFIDPva3O5cfPY4lBnf6hALatjIhzRzk2/09eJKqQeAy4A6rfV8W/vFwF2AE/iV1vr72a6htd4C3KqUcgC/7OtYBOF44Vgig29mVTFKdV0L1RQIM6bYRzAcS7GgWkwXX4HhdistcHOkNWgdbwtGrfTzjBZUwrXnTwhVRyiWdPHZLKhgJEYwTbgguQ6q1BKoeIpA2ZMk7HGncCxVoNLT0YWhJy8sKKWUE/i61vr9fbz+g8BPsCVUJK75U+BC4ADwtlLqKQyx+l5a/09qreuUUh8E7khcq0eaAmEeW72/j0OGQ3VRzk5bgS9kZ93+ZqZVFlrlc3rLmr1NTBtdSLktoN5fWoMRth9tY+GUUb3uu68hQFxrqkcX5vQ+u451cNqkMqttf6PRf0pF5v6BcJQth1szji0UjbF+fwuLpo5Ca83KnQ0snlaRkkZen8jgG1vio9zv6VJNorE9zJyxJXSEopbVBDYXn2lBFbh5r67NOt4WjFjWTbHPzYGmTiC5HqrIEq9kzT0zUcFcA2XuqmtaUAXu5P+Qkd2XdPGt2dtIoyspSvaEDdN1aCaC/PmdgynXEYaXQ+1DY0N1K1Ba65hSKqCUKtVat/T24lrrV5RS1WnNi4AdiY0QUUo9CizXWn8Pw9rKdJ2ngKeUUs8Cj2Q6Ryl1C8bCYjxjZ/DVxzf0drgpPLz5eS6c4ubciS787iHaPnIEsrkhxv+8HWRehZMv13itrRty5UhHnH97tZMzxjm59VRfv8bS3t5ObW0tAL/fEuKve6P8eJmfUm/vxvSdNzuJxuEbZxb0eO4T28Os2BXhpxf48bmM9/neW53ENPzH4sz9n9sd4bFtYe4530+RJ3VsrxyI8MDGMD88r4CWkOZbbwb58kIvJ1cm/1Xf2mVYQlvWraJARdi25yC1tQ3W8fq2IO0NR4l2xjncEbc+k7W7jX7vrn6T91yK1oYQDa1Ramtr0VrTFozScPQgtbXH6GgOUd8co7a2ls3vhXEqWPnaqwDsb0tso7F2Pbtb4jgVvPHqKyilCHV0crgD1m80vi6idbuscW3c8h7BSIxjhw/gVPDilrqU333f4eTruiZDON2JxOH/eWFbj3MhDB31gZj1dzWY5OLiCwLvKqX+BnSYjVrrL/TxPScAdvPmAHBGtpOVUkuBDwNeYEW287TW9wH3AZxy2ul6xdf6vr74sb++wVvNhTy6rZFn9sS5qmYSN55VzaRR/j5f83ikoT3EV+96FZ/bycaGGOExJ3HRvN7l0/zLH9ahOcjqo3FOOn0xVSV9F6na2lqWLl2K1pr/eOvvaKLEKmey9PSJOV8jGImx928vENew+Kxz8Lmd3Z7/m92riOljjJ+zgFMmlqG15vZ//I241px33nkZBfu5+g1o9jN5rtHHzqrntwI7mTj7VNxtIXjzHUZNnsnSM6ZY57zavhn/nn1c8v5lPLTrTSKxOEuXngkYVk34+Rc47aTp7G3oYO/moyxduhSAt4Jbce/YxQcuWIpSirXhbby8fwfnnnsegUgM/cILnDx7OkvPnc7LLRvZ0nyIpUuX8nLLRoqPHrKus68hAK//nakz59B2oAX/oQMsW7bM+jwaOsJMmjoeNm/hqkvO5ZpLFfP/6wXGTJiC3rqd2TOm8vze7V02BXQVFAMtnDWjgq2H24AwFaWFNNWlbqwoDA3vfP1CnE7j73dfQ4DL7kluoF7pd1p/D4NJLgL1bOJnoMh0O5vVo6m1rgVqe/MGHpfDCvb2hdOrXHzp6iVsPNjC/a/t5qGVe3jwjd1cNG8snzpnKqdPLu+1pXC8obXmXx/fQHMgwhOfOZOv/HE933p6M+fNquzxS91k57F2nlx3kEtPHstzG4/w8Mq9fOWi2f0e29YjbZZ7qnbbMT7cC4F692CLtaZm06GWHl2E244Yd/rvHW3nlIllNHSErcSDho7UMkMm+xqNhasHmjq7CNShZmPcR1qD1LUarrujrakuvLq2EGMSm/ZVFHnYdCi5uYBZwXyU30NLZ4TmQMTalLA5EKG0wGP97ZYUuNEa2sNRK95kLuAt8rpoC0atbeALbbEkf0qaeTQlEaLA4yTQlFyoW+B24nIolEpWrcj292Fm8ZUVeKwU90x7WgE41NDFQU5Uin0uK8xhZnea5M12G1rr32T66cd7HgAm2V5PBA5lObdXKKUuV0rd19LSa29kRuZPKOVHV5/Ga187n0+fN503djbwzz9fyYd+9gZPrz90QldQfvCNPby8tY5/u3QOJ08s5ZvL53GwuZOf1+7M+Ro/eXkHXpeTby2fzwVzqnhk1b6UYqJ95a+bjqIUnDNzNK9uP5ayyLMn1u5Nbm63bn/3f0ctnREOtRhJBtuPGkK1w3a3v7u+I2O/pEAFuhwzr3e4JWglMNTZEhkAjrUFrV1lRxd5rZgU2ASq0EO53000rq0v+5bOsBV/AqyYYUsgYiUoWGnmPhfRuJEC3h6MWhl8kEw37wgbSRJ28SpwG1u9d0ZiuBwKt9OBUgqfy2ntT+V1O1EZ7lNbO431Wz6300qKKM4iUPb3FAaHfLgJ71GglFIzlVKPK6U2K6V2mT/9eM+3gZlKqalKKQ9wDfBUP65nMVgbFo4t9fG1i+ew8t/O59vL59HaGeG237/DuT/4O/e9sjMlVfdEYPOhVr63YivnzxnDDWdWA7B4WgWXnzqee/+xk/2NXb940zGtp+uWTGF0kZdPnl1NY0eYp9b1/17lb1uOsGBSGR9ZOJGmQIQNB5pz7rt2XxNTKvyMK/Wxfn/3/d47mkww2J4Qpp3HbAJ1rKtAhaIxDrUYVpJp5dkxLajDzZ0cSYjV0TSBMiwowxU6usiwNkxhN/eHKi/0WItkzVTz5kDEWgMFpFQ0NxfJmhaUKQxtwSgd4aiVIAHgcztQyliH1BmOWQkSAAUeh7EOKhy3avOZfZpNC8qV+WunLRjF43TgsR1Pv3M3Ma04AKdj+L9Ij0e6+1SH6hPPJU3t18DPgSiwDCMj7+FcLq6U+j2wEpitlDqglLpJax0FPg+8AGwBHtNab+rL4DO834BaUOn4PS4+saSal750Hr+6roYpFYV8d8VWzvzeS3zjqU3sbch8x3w8EQhHue33aynzu/mfj5yScpf175fOwelQfPuZzT1e556XtuN1Obn53GkALJlWwZyxxTzw+m76U7TkYHMnGw+28oF5Yzl3ZiVKwT/eO5ZTX601a/Y2s3ByOadOLOtR2LYm3Hs1U8otsdpR106B24nbqdiVwYI62NRprcJPF6hYXFuiZLeg0l18x9pClgVVkXAhmpaTWXuvwiZQZqq5WSjWpKQgWdE8kwUFRgaffbNCMO6sCz0uOhILde0uPr/HRSAcpTMSw2cXLrfTyiLM5uILx+J43U6rZBKQYrnZsY/H/v7CwJEHBlROAlWgtX4JUFrrvVrrb5DjVhta62u11uO01m6t9USt9f2J9hVa61la6+la6zv7Pvwu7zckW747HIr3z63i97cs5pnbzuaieWP53Vt7Wfq/tXz64dWs2t3Yry/ZfOZbT29mV30HP7r6NOvL0WRcaQG3nT+Tv24+Su22uixXML7En1p/iOvOnGLFaJRS3HhWNVuPtPHmrsY+j+/FzUcBuHBuFeWFHk6dWEbtttwE6kBTJ/XtIRZMKefUSWXsaQhk3DbdZNuRVop9Ls6dVcmBpk46QlF2Hutg+phCJo/ys7u+a3B/b8K6rCj0dHHxHWsLEU24I4+0Bi2xqmtLWlDBSIy2YDQpUInUfLOahOXiK/JQXmiIkSlQLZ1GDMrEcvF1RqwU7xIrBmU8tgejKZsVmvg9TkOIwjGrsgQYQhSMxAmEo2kWlLPHGBSAN1EH0KQoiwVld/GJu29wGBEuPiCYWCS7XSn1eaXUFcCYnjqdKMyfUMr/JeJUn106nbd2N3LVL1ay/Kev8+S6g8dVleVnNxzm0bf385nzpnPWjNEZz/nk2dVMHV3It57e3KUKtck9LxvW0y3nTEtpX37aBMr9bh54fXefx/i3zUeZVlnI9MoiAM6bVcn6A80pVb2zsSYRf1o4uZxTJxk3OesPZLfGtx1pY3ZVMbOqjIoOO4+1s7OunRmVRUwdXZQxBrUvUdl7yfQKDjR1ptzImK6/McVeDjV3UtcWxOlQ1LeHrb8js4p5ugVV35HcfsPtVBR7XVYBWdNyaQ6kxqBKU1x8hgVlX6gL0BaK0JGWJAGGBWMu1E1PkgBjmw+7QHndyRiUz539a8fjSnPxZbGgUq02saAGm0wxw6EgF4H6IuAHvgAsBD4OXD+Yg+org+3i646qEh//etEcVt5xAd/50Hzag1Fuf3Qd5/7g79z7j520BEZ2nOpAU4A7/rSB0yaV8S8Xzsp6ntfl5L8un8uu+o6MQmO3ntItMJ/byUfPmMyLW45aX+S9oSOieXNXAx+Ym0x1Xzq7Eq3h1R31PfZfu6+JQo+T2WOLOXlCKUqRNQ6ltWbrkTZmjy1mVpUhhuv3N3OwuZPplUVMqyxkT0OAeFqCxt6GAAVuJwsmlxMIx1KKuZrxp5rq8oQoaUv8TGEyrSkzi6/SFKi2pECV+41MPdPF19hhCFxHOJY5BtVpT5JIFosFw4JqD0a7WDKmBRVIi0GZYtHUEU5z8TlSLags33delwOPs2cLyu7683tFoAYbnZa3l09ZfG9rrduBJq31jVrrf9ZavzkEY+s1Q+Xi644Cj5OPL57Ci186jwduqGHq6EK+/9xWlnz/Jf7ryY3syZLZlc9EY3Fuf3QdWsPd1yzA3UOFjaWzx3Dh3Crufmm75aYyuefl7RS4u1pPJp9YXI1TKX6zck+vx7nhWIxoXHPh3Cqr7ZSJZZT73d26HE3W7G3itMllOB2KYp+b6ZVFWeNQh1uCtAWjzBlbzORRfjxOB89vOgLAjDFFTB1dSDgat6wik32NASaP8jOp3FjEa3fzmQJ1+uRyq+3Uicbfspko0dWCSrj4OpIuPnOLi9ICN0oZlpMpDnYLqtjrQilToCK4HMqybkyBspIk0iyZQo9pQUXTsvic1jjsVSR8bqeVFu5zdefic+J122NQmauTpMSg3OLiO17JJYtviVJqM0ZCA0qpU5VSPxv0kY1wHA7F+XOqeOTmxaz4wjlcMn8cj6zax7If1nLzQ6t5a1fDiIlT3f3SdtbsbeLOK+YzuSK39WX/edlconHNd1dssdp21LUZ1tOS6i7Wk8nYUh+XnjyOx97eb5XYyZW1dVFGF3lZYCs75HQozplZySvv1XexZux0hKJsPdKWJg5lrNvfknGezPVPs8eW4HI6mFZZaMXOpicECrqmmu9r7GDSKL+1Ts+eKHGoOUix18XMhNUEWOukzESJuoRAmVl8fo8Tn9thbf1uFyinQ1Fa4KYpELHcfCU2C8rhMFyBrcEorcEIxT6XFXcwBelYe4i47roeye9NWlCZXHwNHaEuMajk865fO+a5nhwtqEKb1dSdBeXJkjEo9I98yuL7MXAR0ACgtV4PnDuYg+orw+ni646540v44VWn8vrXzufzy2awek8jV9/3Jpf/5DX+8s7BrLGafODNXQ3c8/cdfGThRJafNiHnfpNG+bn1vOk8tf4Qb+4yyvDc/dIOCtxObj5nard9bzyrmrZQlCfWHMj5/ULRGO8ei3Hh3DEpdevAiEPVt4fYfLg1S29Yf6CZWFxz+pSkQJ02qZT69pC1NsnOVkugDDGZVVVMLK5xOhRTKvxMyyBQWmv2NQaYUuFnQgYL6mBzJ+PLChhXmqymYcbCTNfesbYQDpXcCFApRUWhNyVJwl7TsNzvoSkQThaK9afWOywpcFsuvmJbLUVTGEwLOF2gCj3GQt5QNJ7RxReMpLb70qypdMyMQq/LgcdmYWVNM88xSaIgx0XjQn6S0+2F1jq98mpeVmvMBxdfd4wp8fHlD8zmjTsu4LtXnExnOMYX/7COc37wMj+r3dFtxthw0NQR5l/+sI7qikK++cF5ve7/mfOmM6GsgG88tYmtR1p5ekP31pPJgsnlnDapjAff2NOt1WNn5c4GgjFS3Hsm586qBLpPN39nn+HKO31SUqBM6yVTHGrbkVbGl/qsRIOZY4w41ORRfrwuJ5XFXgo9TnbZ1kIdazN2kZ1S4ae0wE2Jz5ViQR1u6WR8mc8SKKdDMauqGJdDJTP6WkOMLvKmrP0ZXeSh3nTxBcJWZh8YLr1mmwVlj0FBcssN+15QYLjaPE6H9b6ZsvjMqup2gfBltZrsCRNdv3ZMceySxZd1oW5uSRKmQHnFkhqR5DJr+5VSZwJaKeVRSn2FhLtP6BsFHiMZ4G//ch6/vvF9zBxTzA+e38aS773M1/+ykV3Hhr/2mNaarz2xgfr2EHdfsyBryZnuKPA4+fplc9l6pI1P3L/KiD2dmzn2lM6NZ1Wzu76D2vd6jh0B/HXzUbxOOHN61+zCymIv8yeUdBuHWrO3iRljiqy9kgDmjCvG43SwPkMcykyQMDHdcmb2oFKKqZWFKRaUmWJu1nScWO7v4uIbX1ZAsc9NkddFZZEXt9PBmGKv5eI71p5cA2ViVpOIxuI0ByIp26ybFlR6JXOT0gK3tVA33Vop8rk43GoKVGq/wkQpJCDNguoaj4LsYpVsM76K0rP4sglLruug7FXWIT/W9gi5k4tA3Qp8DqPI6wHgNOCzgzmoEwWHQ7Fs9hh++6kzeO72c7jslHH84e39XPB//+Czv1tjrWkZDn79+h7+uvkoX7vYKGXUVy6aV8U5M0dzrC3E9WdWp3x5dselJ4+jqsTLr1/f0+O58bjmxc1HOXm0M+sam/NmVbJ2X3PGqh9aa97Z18Tpk1Pr4nldTk4aX8K6fakCFYnF2XWsg9ljS6w2M5NvRsKSAqiuSBOoRGbiFEugCiwXX2c4RmNHmPFlhutvbKmPsQlLakyJz3Lx1dnKHJlUFHlo6AhZlRpGZbKgzCSJgnQXn8taqJu+XUqR18VRy8WX+rmmxICyWDMpAuVKfe5Kc8N6XU7r0S5Q2WJI5p5UTofC240bzxyD+dhdgoaQf+SSxVevtf6Y1rpKaz1Ga/1x4LohGFuvydcYVC6cNK6E/7nyVF6/w4hTvbi5jn+6+1XW7O37otW+8uS6g3z72c1cOLeKT57VfbyoJ5RS3Pmhk/noGZP5dI7WE4Db6eCji6bw6vb6HksnrdnXRF1biNOrslt5586sJBbXVjzMzs5j7TQFIiy0xZ9MTplQyqZDrSmuxl3HOgjH4pw0LmlBVVcUct2SKXzw1PFW27TRhRxoClgxxn2NARwKK0HCtKC01la23/gyQ5Q+dfZUrj/TqGBeVeJNyeIb00WgjBiUGYdKt6AaO8K0BMIo1TWmk+ri6ypQphsv3dVmt5SyWU3pJZDAKPLqdioe+/QSFk0dxcwyB/960WzLUkp38U0bXcQVC4zY5ycWJyu6my4+o96fIXbjS31cv2QKXpfD+j3NMZg3LqZ7USmjeoldUMdmqKSffkOV6w3WSMT+u5lucZPJo/zMHWfckJ01o4JPndy9m36g6Gt+5pcwkifyCq3108DTNTU1Nw/3WPpKZbGXL39gNhfNG8tnf7eWq3/xJndcMoebzp46JCu7X9x8lC89tp4zpo7inmsXdEk46AuTK/x894qTe93vIzUT+fFL7/H4mgPdrr16ev0hvC4HC8ZkvzteMLkcn9vByp0NXbYEeWu3cRNwxtSKLv1OnlDKw2/uZW9jwMrM23zYuAEy/2HBsIa/tXx+St+plYXEtSFMM8YUsa+hg3GlBZZVMLG8wFoLdbjZEKDxpYYFdc2iydZ1qkp8vLmrkVhcU98e7mpBFXqIxjV7EqW2RtkSIUYVeuiMxDjaGqK0wN1lPksSLj4jtT7166DY57IK7XZNkshiNXmyuPVsFpJSivkTSnns00sSW6QYiUOQ6uJTCTH70dWn8aOrTyMW1zz85t7Eexrj8Tgd1rKH808awzeXz+eby+fz7Wc2c/9ru7vEoNxOBw4FLoeD39+ymHSq70jduGHt1y/scs5Q8KXH1vGntQf53ytP5St/XA/Anu//07CMBYwbzRW3n2O9Hoq9oCDHJIkMiCd3kJk/oZSnbzubC04aw3ee3cItD68Z9F7hDYUAABFnSURBVKK0K3c28NlH1jJ/fAm/uv59OW+bMVhMKCvg7BmjeXzNgazJEtFYnBXvHuaCk8ZQ4Mr+Z+lxOXhf9Shez7Bgd9XuRiqLvUzJkEI/b4IhQu8eTFrlmw+14nE5LMHKxtTRhrvPdPPtTayBMploy+Qz10CZLj47VSU+WjojHG7pJBbXVoq5iSlYZkX1UUWpLj6A3Q0dXRIkwIhBBcKxlO3eTeyC1cWCyhIDSrGgMsSdslWR8FgWlDPFmrLflDkdClNfzfG4XUmBcjmS1zbbzPcrsFlcLqcDl1O+wkYCfRWokbGAZ4RTWuDm3o8v5OuXzeXvW+u47J5Xebeb0jv9Yf3+Zj71m7eZMsrPgzcuypo9NdRcVTOJg82dvLGzq2sO4M1djdS3h7n8lPEZj9s5a8Zotte1p2xfobVm1e5GFk0dldFCnVVlJEpssgnUlsNtzBlbbO2Vk42pFWaquZH0sq8hkCKC9rVQB5s7UQor7mTHdOltPGikyXe1oIzX7x013sduQZnVJPY2dFDq7+qesotSJhdfpueQmrmXzcVnFy5fmqstHSsG5XZYzz0ZPl9TyMwYmNuZdPHZ41Vmmy8tBuVyKlwOJRXQRwhZ/8OUUm1KqdYMP21Az98GwoCglOKms6fy2K1LiMU0//zzN3h45Z4BXeS7/Wgb1/96FaOKPDx80xkp62iGmwvnVlFa4OYPq9NXOhg8s+EQhR4ny+b0XB7yrESG30pbHOpAUyeHW4KcMTXzxoRup4M544otC0przebDrZxkS5DIRqnfTUWhh931HbSHojR0hFMWOk8clWpBjSn2ZqzSYe4yvOmQMYauMShjvsyK6uWFXS2oo62hjBaUfeFupiw+MFxt6Zly/ixJEg5bNYoUd5/LtGiyCVQii8+23YYnQ0KD+fmYLke3025BJUXHtKas5IjEo9vhsPapymvEBAC6ESitdbHWuiTDT7HWOj9ur9MYyUkSPXH65HKe/cI5nDWjgq8/uYkvPLqu15UWMrG/McDH738Lt9PBb286I+Md/HDiczu5YsEEXth0pMs6sXA0znMbj3Dh3Kqc3JFzx5dQ4nOluPlWJeJP76vOvnPu/AmlbDxoVJSoawvR2BFm7vieBQpg6uhCdh3rsGoL2l18JT43pQVuDjR1cqilM6N7D5ICZYpkpiw+MJI3SnyulC/fcpvVVJrFxWfS1YIyXhd6XF2sy1QLKvWzTxcF+/NsaeNm8oLXnawkkelcs818T3sMyv57my68ZJKEccyZcPGNFAtqZIxy8Mjz24jeke8LdftLeaGH+69/H/960Wye3XCID97zGluPZK+O0BN1rUE+9qu3CEbi/PamM5hS0X1MZbi4smYi4Wicp9anbmb42o5jtHRGuPzU3Ax6p0OxZHoFr+9IlplatbuREp+L2bbyQunMH19KazDK/sZONie2V++NQO2u72BfoxGHmjIq9TM2Us07OdwctBIk0qkqMV18mQVqlN+DUsZ+SulZZnaBSl8DBakWlFnNwcS0qNJTzCEttTxLhl/mGFQPLj5bLb5MKeamCBXZLChT0DK5+MwxuBOi5HIaFlR6mnu+cqIbUseVQJ0IOByKzy2bwSM3L6YtFGX5T17nsSzur+5oDoT5xP2rqG8P8eCN70tZdJpvzBtfyrzxJV1+z6fXH6bE5+KcmZVZenblrBmjOdjcyf5GIynh7T1G/Km7bMX5iUSJjYdarHJJc3L8vKZWFlLXFmLzYcP9ll7LcGJ5AfsbA4kyR5mt19ICNx6Xg/r2MEVeV0rMB8DldFhClC5QdlHK6OLzdWdBpW69Ycee1ZdeTiiTi6+ghyQJe2KEJTgZ3HDGFvI2l51LWdZSJhefeZ7LkRQmI1EizwUqz4c3VIhAjVAWT6tgxRfOoaa6nK8+voGv/HE9neHcKlB1hKLc8Ou32d3Qwa+uq2HB5K7rf/KNq2omsfFgqxWHCUZi/G3zUS6ZP65XBUHPnG6kkr++s566tiC76ju6de+BUW/P5VC8e9AQqMmj/F2+zLNh1uR75b1jlBa4u7jZJpb72XmsnVA0ntXFp5SyrKj0+JOJWd4oXaB8bqdl7WRKkijtLgbVnUB5ki67dHdZZguqpxiUrVisK7sF5Ulk7dkz9zK5+NxpLj67MBlWlHz1jQRklkYwlcVeHvrkGXzhgpk8sfYAH/rp6+yo675MUjAS45aHV/PuwRZ+cu0Czsyy8WC+sfy08XicDv642iggW7utjvZQlMtOHder60yvLGJMsZfXd9Tz9m5jg8JFWRIkTLwuJ7Oqitl4sIUth1pTFuj2RHVCoNYfaM6Yxj6xvMDahiKbQAFUJVLLR2cTqKLMAgVJN1/mJAl7Fl82F19XgTLdepmOpS+OtT/PVsnBikG5HClilY7p0nMmMvFSYlApLr7UJAlTmNyOkeXiO9ERgRrhOB2KL104i9/cuIhj7SGW/+Q1nlx3MOO50VicL/z+HV7f0cD/XnkKH0hbsJrPlPk9fGBeFX9Zd5BQNMbT6w9TUehhybSui2u7QynFWTNGs3JnA6t2N1DgdjJ/Qs8xy5MnlLJufzO7GzqYOy73GGd1Iq6ndWqChImZag5kjUFBMlEiqwWVKMCbKQPTdPNlikEVuJNlh7qUOupGoNJLCGU8lmHRbs8uPqdl/WRKkrBbWG6nwu1SeBLr39x2F1+aQBlWl0rEoSTNfKRwXAnU8ZzF1xPnzqpkxRfO4aRxJdz+6Dr+4y/vEowkXX7xuOarT2zgr5uP8s0PzuOKBROHcbR946qaSTQHIjy57hAvbT3KpSeP63EtUiaWTK+goSPMn985yOlTynJKOZ4/oYS2YBStc0+QAOOLeULCMspmQZlki0EBjEm4+NITJEzMnXUrurOgMgiUUsaeUW6n6iIIxYksvkwuPqdDUWBzH9ox2/ri4jMX56YXjTXx2NY9ma4+012XycXns7n4kuIkC3VHCseVQB3vWXw9MbbUx+9vWcynz5vGb9/cx0fufYN9DQG01nzrmc38ae1BvnzhLK4/s3q4h9onzpoxmvGlPr799GaCkXjO2XvpmHGo1mCURdW5WWDzbFZWb1x8/P/27v+3rrqO4/jz1S9bR+nGRskgjAjEgcxvkBUiCoYYQkABCRKnkhCVoJCAxsQYIP4BGH8SY0T8kvkDQgwqTKNiYjanfNsXMjZwImODrAIbW/e1dIy1b3+457S37bntPd3tvef2vh5J093P+XzO/fS+d/ru53z5fGB0xomsEVS6LtT8jrYp53kbG0FlJ7Gxa1CTE9ji0RV2s/e/cEEnPV2dk24lT0dQFZe8mN8x6Q4+yB5dTX8XX9uk75Ue1O0su4li3PWo9srPQaXXndK5+wp/DarVb99LFDxKlldnexv3XXcRv7itj90DQ3zux//k249tYfUzr3PHledx92c+2Oguzlh7m7hl5TKOvHeCMxd20ZcxuWs1li0+ZXQ0M931p9SKsxbS3iYWdnWMjoiqNZagJt/Gnz4LdfZpC6aca3HpNCOo9BTfku7Jo6TFU5ziK/WhY9I0RzCWmCottdI9v51Tsk7xld1AMVo2YbLWiUanOiq7FjXVNai0TenB3mQmiYznoNKRWzrjRPoMlK9BNQcnqDnq6hVL+dM9V3B+bzdrXnyTVX3ncP9nL6rLhLOz6ZaV5yDB9R8766Qmsr1yeW9pgtkJS2xU0tXZzoVLe/joskW5P8MLlp6KBOefkf2c2Xm93ZNuP59obAb07OSYnh48c2H2XH7zOtoyH9QFOKOni96MRSR7ujqY195G76nZI68l3fMyR31Luudx2injJ6bt6izNsTdxuY9UmjzT62ALM+54TLf3LBir29PVMdqm/M7KtGxJ9zzmd7Rx6vwOero66Um/V1ipt1y6hEojpOuLnbWoq7WXrY+IOfe1cuXKOBlr1649qfZFcuz9E7H+v3vjxPBIo7tSM5vfGIgjx96fVJ4nbgcG34tt/Qdzve/r+45G/4F3c7WJiBg6fiI2vT5Qcfsb+wan3e/IyEg8+9q+GBnJjuPw8Eg899q+zG2Hh45P+bP+78C78ca+wcxt2/oPxtGMzzqi9Hm8dXBoUvnBwePx7zcPZe7r8NDxcWVpzIaHR2LDrv2j5a/uORJ7Dx+btI+3Dw3FrneORkTEjr1jdTbs2h/DZf/HR0ZG4vmd+8e97853jsaeQ0Oxe2Awdg9k/7zp/re/dSgODh7PrFMP5Z/H3sPHYsfeIw3rS5aT/R0JbIoqfpcrajinW1H09fXFpk2bZty+tATAVbXrkNWF49Z8HLPmdLJxk7Q5Ivqmq9fCY0czMysyJygzMyukQs5KPlOSbgBuAA5LejWjyiJg4kNSWWW9wOSV7eorq1/13leedtPVnen2POWOW7521dSbqk7ebY5ZbdrNhWPtA1XVquZC1Vz5Ah6usqyqC3j17mu995Wn3XR1Z7o9T7njlq9dNfWmqpN3m2NWm3atdKy12im+P1ZZVgS17NdM95Wn3XR1Z7o9b3mjNVPcqqk3VZ282xyz2rRrmWNtTt7Fd7IkbYoq7jCxYnHcmo9j1pzqFbdWG0FV6+FGd8BmxHFrPo5Zc6pL3DyCMjOzQvIIyszMCskJyszMCskJyszMCskJyszMCskJqgqSuiX9WtLPJd3a6P7Y9CSdL+mXkh5vdF+sepJuSo6zJyVd0+j+2PQkXSTpIUmPS7qrlvtu2QQl6VeS9kp6aUL5tZJekbRD0r1J8c3A4xFxB3Bj3TtrQL6YRcTOiLi9MT21cjnj9kRynH0VWNWA7hq5Y7Y9Iu4EvgjU9Nmolk1QwGrg2vICSe3AT4DrgBXAlyWtAJYBu5Nqw3Xso423mupjZsWxmvxx+36y3RpjNTliJulG4F/A32vZiZZNUBGxHhiYUHwZsCP56/s48BjweaCfUpKCFv7MGi1nzKwg8sRNJT8A/hIRL9S7r1aS91iLiDUR8UmgppdA/Mt2vLMZGylBKTGdDfwe+IKkn1Lc+cRaVWbMJJ0u6SHgEkn3NaZrNoVKx9o9wNXALZLubETHrKJKx9pVkh6U9DPgz7V8wzm13EYNKKMsImIQ+Fq9O2NVqRSz/YB/wRVXpbg9CDxY785YVSrFbB2wbjbe0COo8fqBc8peLwPebFBfrDqOWXNy3JpP3WPmBDXeRmC5pPMkzQO+BKxpcJ9sao5Zc3Lcmk/dY9ayCUrSo8CzwIWS+iXdHhEngLuBp4DtwG8j4uVG9tPGOGbNyXFrPkWJmWczNzOzQmrZEZSZmRWbE5SZmRWSE5SZmRWSE5SZmRWSE5SZmRWSE5SZmRWSE5RZjUk6mnw/V9JXarzv+ye8fqaW+zcrEicos9lzLpArQSVLGkxlXIJKZpA2m5OcoMxmzwPAlZK2SPqOpHZJP5S0UdJWSd8ESGaDXivpN8C2pOwJSZslvSzpG0nZA8CCZH+PJGXpaE3Jvl+StE3SqrJ9r0tWO/2PpEckZU36aVY4ns3cbPbcC3w3Iq4HSBLNoYi4VNJ84GlJf0vqXgZ8JCJ2Ja+/HhEDkhYAGyX9LiLulXR3RFyc8V43AxcDHwd6kzbrk22XAB+mNLHn08CnKC0uZ1ZoHkGZ1c81wG2StgDPA6cDy5NtG8qSE8C3JL0IPEdpBunlTO0K4NGIGI6IPcA/gEvL9t0fESPAFkqnHs0KzyMos/oRcE9EPDWuULoKGJzw+mrg8oh4V9I6oKuKfVfyXtm/h/Fxb03CIyiz2XME6Cl7/RRwl6ROAEkXSOrOaLcIOJAkpw8Bnyjb9n7afoL1wKrkOtcZwKeBDTX5KcwaxH9Jmc2ercCJ5FTdauBHlE6vvZDcqPAOcFNGu78Cd0raCrxC6TRf6mFgq6QXIuLWsvI/AJcDLwIBfC8i3k4SnFlT8nIbZmZWSD7FZ2ZmheQEZWZmheQEZWZmheQEZWZmheQEZWZmheQEZWZmheQEZWZmhfR/1/WTgOYDA94AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "winit = np.random.randn(p)\n",
    "nit = 1000\n",
    "w, f0, hist = grad_opt_adapt(feval, winit, nit=nit)\n",
    "\n",
    "t = np.arange(nit)\n",
    "plt.subplot(2,1,1)\n",
    "plt.semilogx(t, hist['f'])\n",
    "plt.grid()\n",
    "plt.ylabel('Loss')\n",
    "\n",
    "plt.subplot(2,1,2)\n",
    "plt.loglog(t, hist['lr'])\n",
    "plt.grid()\n",
    "plt.ylabel('Learning rate')\n",
    "plt.xlabel('Iteration')\n",
    "\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we measure the accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy = 0.989399\n"
     ]
    }
   ],
   "source": [
    "yhat = predict(Xts,w)\n",
    "acc = np.mean(yhat == yts)\n",
    "print(\"Test accuracy = %f\" % acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In-Class Exercise\n",
    "\n",
    "Try to a build a simple optimizer to minimize:\n",
    "\n",
    "    f(w) = a[0] + a[1]*w + a[2]*w^2 + ... + a[d]*w^d\n",
    "    \n",
    "for the coefficients `a = [0,0.5,-2,0,1]`.\n",
    "\n",
    "* Plot the function f(w)\n",
    "* Can you see where the minima is?\n",
    "* Write a function that outputs f(w) and its gradient.\n",
    "* Run the optimizer on the function to see if it finds the minima.\n",
    "* Print the funciton value and number of iterations.\n",
    "* Bonus:  Instead of writing the function for a specific coefficient vector `a`, create a class that works for an arbitrary vector `a`.\n",
    "\n",
    "You may wish to use the `poly.polyval(w,a)` method to evaluate the polynomial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy.polynomial.polynomial as poly\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}