05.logistic-regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# ML-Fundamentals - Logistic Regression and Regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Table of Contents\n",
    "* [Introduction](#Introduction)\n",
    "* [Requirements](#Requirements) \n",
    "  * [Knowledge](#Knowledge) \n",
    "  * [Modules](#Python-Modules)\n",
    "* [Exercises - Multivariate Linear Regression](#Exercises---Multivariate-Linear-Regression)\n",
    "  * [Data Generation](#Data-Generation)\n",
    "  * [Logistic Function](#Logistic-Function)\n",
    "  * [Cross-Entropy](#Cross-Entropy)\n",
    "  * [Loss Function](#Loss-Function)\n",
    "  * [Cost Function](#Cost-Function)\n",
    "  * [Gradient Descent](#Gradient-Descent)\n",
    "  * [Training and Evaluation](#Training-and-Evaluation)\n",
    "      * [Plot Data and Decision Boundary](#Plot-Data-and-Decision-Boundary)\n",
    "      * [Accuracy](#Accuracy)\n",
    "  * [Regularization](#Regularization)\n",
    "  * [Proof - Pen&Paper](#Proof---Pen&Paper)\n",
    "* [Summary and Outlook](#Summary-and-Outlook)\n",
    "* [Literature](#Literature) \n",
    "* [Licenses](#Licenses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Introduction\n",
    "\n",
    "In this exercise you will implement the _logistic regression_. Opposed to the _linear regression_, the purpose of this model is not to predict a continuous value (e.g. the temperature tomorrow), but to predict a certain class: For example, whether it will rain tomorrow or not. During this exercise you will:\n",
    "\n",
    "\n",
    "1. Implement the logistic function and plot it\n",
    "3. Implement the hypothesis using the logistic function\n",
    "4. Write a function to calculate the cross-entropy cost\n",
    "5. Implement the loss function using the hypothesis and cost\n",
    "6. Implement the gradient descent algorithm to train your model (optimizer) \n",
    "7. Visualize the decision boundary together with the data\n",
    "8. Calculate the accuracy of your model\n",
    "9. Extend your model with regularization\n",
    "10. Calculate the gradient for the loss function with cross-entropy cost (pen&paper)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Requirements\n",
    "### Knowledge\n",
    "\n",
    "You should have a basic knowledge of:\n",
    "- Logistic regression\n",
    "- Cross-entropy loss\n",
    "- Gradient descent\n",
    "- numpy\n",
    "- matplotlib\n",
    "\n",
    "Suitable sources for acquiring this knowledge are:\n",
    "- [Logistic Regression Notebook](http://christianherta.de/lehre/dataScience/machineLearning/basics/logistic-regression.php) by Christian Herta and corresponding [lecture slides](http://christianherta.de/lehre/dataScience/machineLearning/logisticRegression.pdf) (German)\n",
    "- [Regularization Notebook](http://christianherta.de/lehre/dataScience/machineLearning/basics/regularization.php) by Christian Herta and corresponding [lecture slides](http://christianherta.de/lehre/dataScience/machineLearning/regularization.pdf) (German)\n",
    "- Chapter 5.1 of [Deep Learning](http://www.deeplearningbook.org/contents/ml.html) by Ian Goodfellow \n",
    "- Some parts of chapter 1 and 3 of [Pattern Recognition and Machine Learning](https://www.microsoft.com/en-us/research/people/cmbishop/#!prml-book) by Christopher M. Bishop\n",
    "- [numpy quickstart](https://docs.scipy.org/doc/numpy-1.15.1/user/quickstart.html)\n",
    "- [Matplotlib tutorials](https://matplotlib.org/tutorials/index.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Python Modules\n",
    "\n",
    "By [deep.TEACHING](https://www.deep-teaching.org/) convention, all python modules needed to run the notebook are loaded centrally at the beginning. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# External Modules\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Exercise - Logistic Regression\n",
    "\n",
    "For convenience and visualization, we will only use two features in this notebook, so we are still able to plot them together with the target class. But your implementation should also be capable of handling more (except the plots). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Generation\n",
    "\n",
    "First we will create some artificial data. For each class, we will generate the features with bivariate (2D) normal distribution;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# class 0:\n",
    "# covariance matrix and mean\n",
    "cov0 = np.array([[5,-4],[-4,4]])\n",
    "mean0 = np.array([2.,3])\n",
    "# number of data points\n",
    "m0 = 1000\n",
    "\n",
    "# class 1\n",
    "# covariance matrix\n",
    "cov1 = np.array([[5,-3],[-3,3]])\n",
    "mean1 = np.array([1.,1])\n",
    "# number of data points\n",
    "m1 = 1000\n",
    "\n",
    "# generate m gaussian distributed data points with\n",
    "# mean and cov.\n",
    "r0 = np.random.multivariate_normal(mean0, cov0, m0)\n",
    "r1 = np.random.multivariate_normal(mean1, cov1, m1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO29e3xV13nn/VsSYEkYW0YCBMaC3KZJgznCFhJuEkhLXaeZ2E7dy6QlidMbUww6QmSapsVIshM6zTSxJOJp8jolnTho2qR+OwntvE2axm7yflLAFjdfSjrOxVx8waBYvhEbpPPMH+ss7bXXXvt2bnufc57v57M+R/u29joSrGet5yqICAzDMEz90pD0ABiGYZhkYUHAMAxT57AgYBiGqXNYEDAMw9Q5LAgYhmHqnDlJDyAO7e3ttHLlyqSHwTAMU1UcPnz4PBEt8rteVYJg5cqVmJiYSHoYDMMwVYUQ4mTQdVYNMQzD1DksCBiGYeocFgQMwzB1TtltBEKILwJ4H4DniWhV/txCAF8BsBLAUwB+g4heKPdYGIapDS5duoQzZ87gtddeS3ooqaKpqQnLly/H3LlzYz1XCWPx/wBwL4D7tXMfB/BtIvozIcTH88d/VIGxMAxTA5w5cwYLFizAypUrIYRIejipgIgwOTmJM2fO4A1veEOsZ8uuGiKi7wL4iXH6VgBfyv/8JQDvL/c4ysX4OLByJdDQID/Hx5MeEcPUPq+99hra2tpYCGgIIdDW1lbQLikpG8ESInoWAPKfi/1uFEJsFkJMCCEmzp07V7EBRmF8HNi8GTh5EiCSn7/920B7OwsGhik3LAS8FPo7Sb2xmIjuI6JuIupetMg3HiIRdu4ELlxwn7t0CZicdATD5s0sDBiGSTdJCYKzQoilAJD/fD6hcRTFqVPh91y4IAUGwzC1z/DwMD796U+Xpe/Dhw/j2muvxZvf/GZks1mUspZMUoJgP4Db8z/fDuDrCY2jKDo7o90XRWAwDMMEsWXLFtx333148skn8eSTT+Ib3/hGyfouuyAQQvw1gAMAfkYIcUYI8bsA/gzAjUKIJwHcmD+uOnbvBlpawu+LKjAYhikP5XDquP/++7F69WpkMhl86EMf8lz/whe+gLVr1yKTyeBXf/VXcSGvR/7bv/1brFq1CplMBuvXrwcAPPHEE+jp6UFXVxdWr16NJ5980tXXs88+i5deegk33HADhBD48Ic/jK997WvFf4k8ZXcfJaLf9Lm0sdzvLjebNsnPnTvlqn/hQuDll4GLF517WlqkwGAYJhmUU4ey5ynbHeD8H47LE088gd27d+N73/se2tvb8ZOfmI6RwG233Ybf//3fBwDceeed2Lt3L/r6+nD33Xfjm9/8Jq6++mpMTU0BAD7/+c+jv78fmzZtwsWLFzEzM+Pq6+mnn8by5ctnj5cvX46nn366sMFbSL2xOO1s2gQ89RSQywHnzwNf/CKwYgUghPy8777C/7ExDFM8NqeOYm13Dz74IH7t134N7e3tAICFCxd67nn88cfxrne9C9deey3Gx8fxxBNPAADe8Y534CMf+Qi+8IUvzE74N9xwA/70T/8Un/rUp3Dy5Ek0Nze7+rLZA0rpNcWCoMToguGpp1gIMEzS+NnoirHdEVHoRPyRj3wE9957Lx577DEMDQ3N+vd//vOfxyc/+UmcPn0aXV1dmJycxG/91m9h//79aG5uxk033YQHH3zQ1dfy5ctx5syZ2eMzZ85g2bJlhX8BAxYEDMPUNH42umJsdxs3bsRXv/pVTE5OAoBVNfTyyy9j6dKluHTpEsY1o8QPf/hD9Pb24u6770Z7eztOnz6NH/3oR3jjG9+IbDaLW265BY8++qirr6VLl2LBggU4ePAgiAj3338/br311sK/gAELggThqGSGKT82p45ibXdvf/vbsXPnTmzYsAGZTAY7duzw3POJT3wCvb29uPHGG/HWt7519vwf/uEf4tprr8WqVauwfv16ZDIZfOUrX8GqVavQ1dWF73//+/jwhz/s6e9zn/scfu/3fg9vfvOb8aY3vQm//Mu/XPgXMBCl9EUtN93d3VQrhWlMAxYg/3GyTYFhwjlx4gTe9ra3Rb5/fNxx6ujslEKgVv+f2X43QojDRNTt9wzvCBIiqgErTbuGNI2FYeLAtrtgqqpUZS0RxYBVDre3QknTWBiGKS28I0iIKAascri9FUqaxsIwTGlhQZAQUQxY5XB7K5Q0jYVhmNLCgiAhNm2ShuGg4LNyuL0VSprGwjBMaWFBkCBhBqxyuL0VSprGwjBMaWFBkGKi7BrqcSwMk1bKmYZ6586duOaaa3D55ZeXvG8WBCknTW5vaRoLw8TCjJeqovgpxc0334yHH364LH2zIGAYprYZHgYGBpzJn0geDw8X1W0l01ADwLp167B06dKixuwHCwKGYWoXImBqChgbc4TBwIA8npoqeGeg0lA/+OCDOH78OMbGxjz33HbbbXjkkUdw/PhxvO1tb8PevXsBYDYN9fHjx7F//34AThrqY8eOYWJiwpVyuhJwQBnDMLWLEMDIiPx5bEw2AOjvl+cLTOUcNQ31nXfeiampKbzyyiu46aabADhpqH/jN34Dt912GwCZhnr37t04c+YMbrvtNrzlLW8paFyFwjsCJhVw+gqmbOjCQFGEEAAqn4a63CQqCIQQA0KIJ4QQjwsh/loI0ZTkeJhkUOkrTp6UO3WVvoKFAVMSlDpIR7cZFECl01CXm8QEgRDiagBZAN1EtApAI4APJDUeJjk4fQVTNnSbQH+/dHnr73fbDAogiTTUH/vYx7B8+XJcuHABy5cvx3CRxm6dxNJQ5wXBQQAZAC8B+BqAPUT0T37P1FIaasahocH+/1EI+f+WYUxipaEeHpaGYaUOUsKhtbVoz6E0UlVpqInoaQCfBnAKwLMAXrQJASHEZiHEhBBi4ty5c5UeZiqpNX06p69gysrwsNsmoGwGNSgECiVJ1dBVAG4F8AYAywDMF0J80LyPiO4jom4i6l60aFGlh5k6alGfzukrmLJjGnZLWPi9FkjSWPyLAH5MROeI6BKAvwPwcwmOpyqoRX06p69gCqGaqitWikJ/J0kKglMA1gkhWoT0w9oI4ESC46kKKpUOOo76qRSqKk5fwcShqakJk5OTLAw0iAiTk5NoaorvfJlYQBkRHRJCPADgCIBpAEcB3JfUeKqFzk6pDrKdLxVxqpHZ7v3Qh4DvfQ/4i78o3ZgYRmf58uU4c+YM2G7opqmpqaCoZC5eX2VUouj9ypV2YdPYKFfsevFvv3uFAL78ZV7ZM0waSK3XEFMYldCn+6mZZma8Bmq/e4mq227BMPUE7wgYF+PjwO23y0k/jBUr5KdtRwBwHADDpAXeEehUYU7ySsYMKLVTFCEAyN3Ae9/rf53jABimOqgfQVCmnOTlJHLMQIkEnM01NYiFC4Evfcl+jeMAGKZ6qA9BUKac5OUmUsxAgQLOttMIckG1BXyp8Zg0NnIcAMNUFURUNe3666+ngsnliPr7ieRUKVt/vzyfUoRwD1c1IfI36N9JfRfz2MK+fUQtLe4+W1qI2trs71uxQj6zYoV8tzoOHV8KsX0Phql1AExQwNya+OQepxUlCIjkxKjPWCkWAkRyovKbmGcpQMD59dvWZhcQarLct88tLBoaIowvhGIn5jjP+wlAFgZMrcOCQFGFO4LIE1dMARe0kvebWPftI5o71/6c2ceWLe7v4DdRFzoxqz7V+6I+H0mwMkwNwoKAqGAVShoIXfGWcEegq4HiPGM2NRmHTfR+fTY2+n9fW59+E7v5uwsSXgxTy7AgUAwNuSdINYEODRXeZ9KU0EYQtqr220X4tcbG8Ik6Sp/mDiOKQFI7G/M7+r2PdwRMrRMmCOoroIzInX7WPK5GCiy6MT4uvY/8gsFWrJDJ3xR+qSQKQQWaRe1TT1fhV8RGJyjQTf2KFKVOz8EwaSQsoCzxVX6cVrSxuFYxV/7GcZB6KarnT1QbQZSmq27C1DzmM2E7ArWbCdptsNcQU28gZEdQH3EEtU5A0Y2woLSo1cE2bQL+6q+AtjbnXFsbsGULMG9e9KHqgWZ63qQwVIzD7t3B77v9dtnvwoX262qnE5TuutYqwDFMKEFSIm2NdwTxCfOUKdalct++YHuA2draCjNG63p8v3gHdZ/f7mXevGgeSexiytQaYGNxfRNF9RPmmaS7a6pJXw8yi6sask2sW7aEPxcWyKaan2Bqa/P/PYV9FzYoM9UMC4KUU+5I12J954P0+FH1+1HeH1WgBEVAKwEXdC3udwx7lmGqgTBBwDaCBKlEIfqCC8MTAdDzHZHnlgsXZF6hQjA9eqKW2rxwAXjtNWDuXPt18g5zloYGr95fpd0OS7bHmVSZWiZRQSCEaBVCPCCE+L4Q4oQQ4oYkx1NpKlGIvqBCNloiOzlhE0YwgCEMe26dmSlcGOjEmWhffRW47DL3Od2I7YdZWOeOO6Kl3eZMqkytk/SOYAzAN4jorQAyqLbi9ebyM2g5aqFShehjFYYnms3U+mBmAEoIbMcYWjEF286gsdE7McclqK6BjVdecX5uaZGJZOMIpAsXgM99LnwnUI4KcAyTOoL0RuVsAK4A8GPkq6RFaamyEZQgUjm1uW9yOTpxkzttxQj6Ccj56tDb2qIZfPWmiBNPEGRzKOZ5my2CPYWYWgEpthG8EcA5AH8lhDgqhPhLIcR88yYhxGYhxIQQYuLcuXOVH6UNbdVcTH2DgvX35UYIvOfEiOvUAEYA+EdhT04CX/1q9Fds3Cg/o+rowzh1Klo8QlR4F8DUFUFSopwNQDeAaQC9+eMxAJ8IeiZVO4ISZTNNZX78XI5GEW9HEN6cZzduJNr35RzNn1+6FXxbW7A3Udym5zfSSeXfi2FCQFrdRwF0AHhKO34XgP8d9EyqBAFR1dU3iIQm4NTkPwL3sd8Ebz8mGsLQ7LOXXUazfQ5hqGQT95w5pRMCgIxFMOFgM6ZaCRMEiamGiOg5AKeFED+TP7URwL8lNZ7YEEl1kI5eMrJaEQJobcX3b+rHzmapDhrACEbRjym0QlcPDWEYI5AGZYnNu4jQiilsxxhGMIDXXw83PhfC9HRJupnF5klUCS8vhkmEIClR7gagC8AEgEcBfA3AVUH3p2ZHUK76BiHJ4ypKLmdUJPOu/PWdwtw5QTsH55pqxauayt/a2qq/NCfDEIXvCOorDXUpKTD9M4jsqbAL7S+of/W31Y9jpt0OThXtrO4Vo+j3MSwT9A2oQM5yT2XZsgW4/34ZlxBGY2NepOW818yU3QyTNsLSULMgKAa/Sd0Pv8n+yiuBF1+UXkf9/fK68kJSx1EmcL3/u+4CXnhBnr/qKmBoqCDBEpb/X4CQC53g4wiMyqEm8GJKUnA9A6Ya4HoEaSFMnTQzU5wXkv58NkvU1+f0k83KVoDqKtg/P0d7F4SpfLwqo2Djc+WaUukU+nxjo91QzJ5FTNpAWr2GCmlVLQiIwl1OC/VCUsFtNmFSqGDJ4x/slaN75ziCbN+XnWNzgte9hnThUEqvoUJaW1txgWg22wB7FjFphAVB2vCb7AuNS7DtLPxmrgKNz35pqI//ypBrjPu+LHcI9gne7maq6gvHrYlcilbsO/VU3Gr17xfLUGi0OO8umFLAgiBN+E32+kq+EC8kW7+2VoxHU9C7jeO4q2yi0gaDVapt3Bi9fGchnkW8u2BKRZggSDrpXP1A5DYA53Lyc2wM2LFDGox1w/DIiDxubQ23ZgoB3HOP+1xfn/NzNiubnhKj2O+iv9s4jpsi4447ihtOUjz0EHDpUrR7C0ljzXELTKWYk/QA6oZ8oJZnsgccTx4iZ9JX16O4tBBJYaLz8MNSGAjheA2pMQT0OT4uJ5pTp+TktXu34RETwc110yb5NScno/1qPve5aPelDZsrqR8nTwLt7VIWR/UwqlR2WoZJXN0Tp1W9aogoPGgsblBZkPdRNiuP1X0hffmqIr5ssWOEqLBKkVG0FtucOf6lQE07QGqz0zJVB9hGkBKiTPCFpLbO5dzP5XJEg4NEXV3yM6gffQy5HK3odLt9AtLjZ+8CbUwzM7JvfWbysT24I5PDW70IDr12cpAdgG0ETKlgQZAGokzwhaStMAVALid3Ab294f3ozw4NEWWzNIps3uNHuXcOOqkhzL70FmGnoVa8fpOj8slX98XLJhqe+C5tTRG26i/Ua4i9jRgdFgRJE2eCj+NCGtSvHkBm6yfg3hFkaQR9+Z/7CZihvZcbfUXcEdjwK15jS/u8b1/4hJrWGIWogqAc+Yt4J8GYsCBIA3EneP2+oAk2qN+wfkJcTkeQJSBHLc05OnFT1nuPTfiYQs18Z95esWWLikfIUWOjf+7/cDfUKFHL5u5gxtNHpYWAEOWLOwj6vbFtoX5hQZAWokzwhQSV2fqN2o/5rNZGkaUVnXYhMNnZRSuumZGTWWeOnn9Tr1RH6TuOXuNciN3CpsqIFvDln9nU3C0MYZCOoIuGMOh6NundQ0OD+7jY1TtnSWVMWBCkgSgTcyE2Alu/ep4h9Zwtz1DUIDS9X62Wsb7ivrcxa3+fek43MHd1eQLo9n05Z1VlxLER6CfUuNy7gxk6AjmGI+giYCY1OY8AclVra2srThDwjoAxYUGQNHEm+DheQza1jDpeskQKA9OArPoJsidks0TbtrlnENUXyR3AyKxRWd0y40k+Z7VT+NgW5MTlnYjb2qJ4EgXVOvBeU8LAe29lJ/6w5rcriGIEZhsBY8KCIA3EneCDjs1+e3sdQTAzQ5TJyD9rT497dW7q8C1eQ9TbS7RsmXfC7uiYVekMY2jWfqBPxMMYdD9js1OYeZDyNgPho6IRQsYwuFe4cTObmruFGeM4fUJANXMFH2eCZ68hRocFQVqIGygWtU9d7TM46AgCvem6er8xzMy4dwJdXd7jS5dmV/7mxPv4XEN4RNkRdHURTU8bfTrG3JErh3yT2qnkd3eJIK+h6t0RKEGowyofplBSLwgANAI4CuAfwu6takFQLmy6fnPCNXcDQX319Hhnmq4uol27iPr76fj7B5300/l2TOTfF2QjUAIqP/mbY7x3jhQCahKX3krhKjW7Z1D12QiiTPBsBGYKpRoEwQ4A/5MFQREEeP/EEgR+fV265EzCXV10/P27XNeP/8qg+x3Ka0ipp/IqrMlrMjRyxSAJ5GjvfLcdYt/9jp1h7wJpQPYIFU2Q7Nun7BX+Xj/V4jVkazaVj9+OoLGRVUBMMKkWBACWA/g2gF9gQVAgYd4/cSqT2SZevXV0yM8lS9zne3qkQVm3eQwOOruB/PiOIZOPVg54hxn9bNmhvHplB32ycTDiij4sjiDsuPI7hYYG/wC7efOCn2WjMGMj7YLgAQDXA3i3nyAAsBnABICJzs7OMv2aqhSzjsH0tHeS7uuTE3JQviIiu0rH9B4ym00FNTPjtV3MzNDjc7y2ixFkacU1dgMy5XLucps+zbQPxJnE07tjyNHcudqErmwk+6LVP2CbAWMSJggSq0cghHgfgOeJ6HDQfUR0HxF1E1H3okWLKjS6KmB42F3H4DOfAZYvB86eBTIZYHBQ1iD47GeBgwdlGmqFFLDuY5Wiev58eS6Xk5XrTXp6nJ+PHXN+7u2Vqal37AB+/uflOVUDobERb58+7upmFFkM4B4MnDbSZ+/Y4R1fAFO4EoDAEIYxggEA6lnCCAYwhGGfJwmteAHbMZZ/LodbsR9rcAy3Yj+AHEYwgO0YQyumtH7Li/oely4Rbr8dGN9HMs338DB27oxW/+DkSfnnnDNHfq5cKdOLM4wvQVKinA3AfwVwBsBTAJ4DcAHAvqBnWDWUxzSczsy4V/PbtrlX5brXUJAra5hqCPDuOFQbHHTZEWZ3CL4r+a10BBnnO9x5p/Oc+k49Pf7vyzdp+J0OcSP1qnuG8m6wpprqGXQY46ykIdnr/rqnwfk7iyLGwSqj+gZpVg3NDiJANaQ3FgQaYVHF+jlbNLFfriCbMGhvjz7j2ASTa2LtowNw9P4zDY1Er7/uCIGmJqLOTmdcq1f7vkt5AanJ3hZYpjKout1Ls3QAvfl7sp7x6ceVtxHYA+S88RTxW1sbxxbUKywIahm/PEPmOfMZU4DoQWk23Xx3tz0+wdYGB+MLE5VLoqnJfT7knY35nYCu03dP4l4XUec465n0deGiT8JJCAP395A2g40bS/se3iXUD2GCIBU1i4noX4jofUmPo6ogkrpjne3bZdMZGHDXVBRC2hN0enuBPXvks8quoDMxARx36/h92b8fmJkBDh1yn1+2TNoubKialq+95j5/663SJtHVZX3s8Zs+ipfFlWjFi1A6fZ1R7MAUrsBRdGE7xkBowHaM4Si6Zm0LOkfRhTU4hqPogsA0RtHvsiFUBvJ8D2Uz+Pa3S/smrn/MzBIkJdLWeEeQx6biMT1+9HtUojciqYs3V+eZjFPMRjWfwK/Aplbwqv/eXtmHOjZX/GGtp8f5XnrSOu3zeKMTJKZ/qkhnT8Tz7Mo/Qwew1nXuNJbSEXTRQ1if3wlMu9RL5fceipIyo7Tv9AtG4xQVtQWqYUfAxER5+PT3O0XkR0flyr63V/4sBHDFFbJi+rFj0htnehr49KeB8+fl+elpudo+fty7Gn/kEeCjH5XPzpkTbVzr18tV//nz8rinR773/Hlg3jzvO2y0tQF9ffLnhx+WHk/ZLHDLLUB/P8Z3HMYXF/TjrmO34Oi8XlyYmYdRZLEG0oNpDY7hIHrxwKWbga4uzLz3fXgWHa5XXEAT1uA41uERAMBRyJ3KcjyL7+AdOJbfQRxBN6YgdxyV8R4SmEIrRtGPAYwAEBjACEbRjym0wtzBeMcSf2ydne7j8XH5T+ODH5TeR0Tyc/Nm9jyqaYKkRNoa7wgMbPp/M7pXX0WrNmeO+9i2I9B3Bvoqv5CWzdpTV0TZDShvJiItXbVKIZHNr5a9Bt/ZqmodHZHep3YSB9BLejoKp884QWvFrtzD+ytFZTYhZOCaWv2rc373c3xC9YJqMBZHbSwIYhDkCnrtte7jjObGOT3tvV8JgbY277PNzeEup9PT/oImSHgodVYer9dMziMECLKozomNd8R6n1QD+X8Pv8k9mVKZlVchKSHBVCcsCOqZiNG5BDg2Ab90FWYZLX3puHVr8I5BXdu2ze4JZHMRteRH8q5W7YKgkKbiEfyu2ydYfVfSbz2uhDAIHmPpGu8IqhcWBPWMXhnMnJTNtm2bs7IPCeKK3PRJv73dfdzS4l91RqmRVCCcKoqzwpkE1acek+BpIcZppf45i3bXp2pn0U5ypyCFqTPRThNAWlCaW9hK9VIlXE5tldnK9z42GFcvYYKAjcW1CpF0HdXTQACOoVinqwtYuFAaoDMZmaYCkMbetjb/d2zb5n8tkwFuvtlxGT1/3u2C+ju/A/zu79qf/cEP5OeZMzKVxsAAQITdu4GWplw+dcQQRrAd6/AwRtGHg1jr7cdinH4WHTiIbgDAd/BzGEUWf4E/wPNox2Kcn71PHZ/GCmzAdzGKPkyhFUMYwll04CGsRyumsB17sAH/v+sdB9Hr/3spGXY3U6+bK1VgLEzVEyQl0tZ4RxAD3VisSk0GrfiVKkatxqO4jZor7vZ2tz0gm7XbHNS7Ll0KXrXrkdLZLNHgIE12OkbdI3N76NTCDB2Y8046i2gFjg+hm4BLNIKt9Apa6MfoJJlywl1hTdkMXkEzqd0DMD37HrlbuOSkydCaWcGtHDsBPxuBqrWg31cqe0W5VUPsslo+wKqhOkaVoFS6dr1AjDpvM/T6TeBBTUUHZ7OOzr+nx57BVPUfJGxaWmTMgy1thqrJ7CeQAtqluc10BBl6HdJz6izaaQTbPDEFusrHqzLyFzpKMEhhMKNNxO4Sm8Gf4c1rpJ4xUm6U3oBcTmMx11kuL2GCQMh7qoPu7m6amJhIehjVBZGMKVAMDQHf/KYTbwDIeAQ9mnhmRvr/j415+1u9Gmhuls7lzz3nPr9hg/T9P3RIqp/On/c+r+jqkmqroPsyGalOUvcWS1OTR110FKuxBo9qx12zMQmAkyWVMMd1bjv2+L7mWSzBSazAQazDduzBKLIAgClcBQBoxdRsfMIA7sEIdsweT6EVd/lmTDUhuGMLchjBDmyH83fTYxKKZcUK4Kmniu7GysqV8p9UJd9ZTwghDhNRt+8NQVIibY13BCXAUivAWktYrdxNL59t22TZyqDVvO3aHXd4vYPiRC1XoEm1ykXXuQPo9sQU2HYBtrxFcjXepxmbszSCbbPvsn0Wv3r3NyCbjl9xWrlX51yGs7yAVUMMEbldMf1KQOrqlq4uZ+LOZNxBYcrzSKWusLmEzp8ffZa57DL3cXNz4TOWrcWYAV+FXc10BBk6gmsDnzUzq+rtAHppCLvoCLp87ymFECilS6manFeskIFn5dTf+2VWZZfV0sCCgLHXIDAFgVlZTBcI6pqqiLZ+ffTVfNSspYsWFRe9XIJ2EY1FPQ9M0xHY02aPoM+z+vc+H2fC9pbgLFWQmTnZV0J/zzaC8sKCoN4JS1CnmrpmprFW0b1KmKjjKMZkvziBoBbV8FtI3wHtGUSPnTiAtWRLQ6H6COorSM0UdcL2i2Z+CBtKEuXc1uaegCu1WmevofJRFkEA4MZCniu2sSAoEL8C92aW0mzWKyTU5G9WRIuyI1i4MPrsE7U1NMRTO4W0V+fH34WM4I7ZCOKzaMtnK3V+b0FeRSPY6jqObyMISy/hrcZWyK9GX42z/r76KZcgOFXIc8U2FgRFYK70zcpl2azj/29WL/OrOqbURraEcosWeZPblaoFqJByRfY9jYCsa7OT97X5QvdS7XUay2gIu+gZLAndWZjXpTE5m091nfVZvfurgZx+Sp9eQu0MGn00ZrYdAa/q00mYIPDNLyyE2O93CUBAuCmTOoi8RWx0VBrru+4C1q1zUluPjMjrra3A3Xd7n3vXu+T5f/1X4Prr3ZHDL78s01y3twMf+ABw773y/GWXAa+/Xtz3CXBLLdZJsjFCJO4aPIY1eGz2+CpM4T34RyzFWd9nXkUzXkXL7D3PYgmAHLbjsxhFFndhSIsUJqhvMoRhtOIFDGA0f45m3U11HIsfBcgAACAASURBVBdR51mzr7hMTsoA8JkZ77WWFmD3bve58XGZrvrCBXms0lcDwKZNBQ2BqRR+EgLACwD+I4ANRns3gLNB0iVKA3ANgIcAnADwBID+sGd4R1AAQXWKdQOyutd2HJTJdO1afzVR3EI0ZVrBV6JJY3CwV5Fqr8CxbxxBhoawa9bF9AB6ZoPRpNpHne8lM4LY3BHY6zOXJxPq/PneVT97/qQXFKoaAvCPAH7e59p3gzqN0gAsBXBd/ucFAP4PgJ8NeoYFQYHYvIb6++X5KPcECQJdCPT1ue/r7nbfu3atvY8C26UivXxK1Q6g2zeOwK+piGb9nKqQpiZ6qX7yClm/COJyRxb7NWVPYFtCeilYEMzeYJmcAbw77Lm4DcDXEWKEZkFQBLYiNvrPYbsGla5C/x+ez//jqTWQzcrAM9v5FEzcaWlmCm2/LKimMPBf+Q9WPDW1aitW+O8I2toq+O+csVIKQfA4gD+CVDQ2A/gsgANhz8VpAFYCOAXgCsu1zQAmAEx0dnaW83dV39g8i8JiD1Qyu5kZ9/npaXf0ctCOIkorJiQ2gRYWiRy3mZ5Gyn3VfZsjFPQLlUmHHdzmzfMajdmoXFlKIQjmA7gXwIG8UPhjAA1hz0VtAC4HcBjAbWH38o6gzJieRaaaSJ/Y1Wq/p8eeomL9endsghIEPT3xy1YuWpTsTJZvMyHXX0ILnUW4y6ytn/Co5W2uHcRpLLNM8pUvVhO16buCYoPHWIjEpxSCYB6APwdwDMAPAHwg7JmoDcBcAN8EsCPK/SwIAghS/UR9PmhHsGGDO8pYpaNQLqLKpVNPPzE97fSvVEs2N1TVFi92Zy8tZ2toILrhhkj3Xiyg/wNY6xtlHKeZxnClNnoGHSQL5OjqIbNaWmVsBFFbKYzKHIFcGGGCIEphmkcA/BTAWgDvBPCbQogHIjwXiBBCANgL4AQR3VNsf3WNVrwFgPwcGJDno6DuHxuTmUhzOfk5NibP53JOBtAdO+QzKqPp9LT8VC6d58/Le+fNAz7xCfcYR0eBhgbgqquAbNY7jpUr5VgefdR7rZTMmye/04EDkW4nNAbfsHq159Q6POLKahqFC2gCIIviAMCraPG4sy7GeRxFF/4f/D4O4B0YwXYgf886HMSz6Mi7lgoM4B6Moh9TaEUpso8Wy86d8vPUKft1v/NmH8o9VXHhgtM3UyBBUkIKEnRbzn0o7LkI/b4T8l/wo5C7jWMA3hv0DO8ILMRxDw0izLPItmPwW9mr3YLf+202AzMnUdQcRYU0W42EUrR58yKnvjANwqrJTKbO+FRhHL3JwjrO728EWVciO3cthMGyrvLjNOU9VMyOgD2TCgOca6gOCFPrxOkn7Ng24duapfi8Z6z9/dLryBaHUGJX05K3GFHTZ7HQk3HUnOAvBsRD2GspX7K6rKoKZWlTC+kT/b59RHPnuq/NnRtNvcOxCoURJgi4ZnEtoEcBK1R0cNx+/I6JvNHJn/2soxZRtYl1iOzvaG2Vqqd77gFefFGqnJqa3Pc98UT88Ydx2WWl60upxCKwGD/J11bOYhTb8CpaMB8/xfNoh8AlHEUX5loiml9Fc/55qXY7itWztZSfwTWA5Zk1OAZCI7ZjzFKUxrzf8vcpE0K4I5GD/qkFsXu3jGrWsUU5MzEJkhJpa7wj8KFUO4Io/au8Qx0d8rijQ6pazDrIPT1yV6AHrZl9EkmDcqXSTyfkhvoa5tIRrCYVLTyEOz11D2yqItNtVD5/iY6gi05jWeh79Z2AX8ZSJ+rY64FU6l9FFGNxFI8g9hqKD1g1VOOUykYQhmlD8MtA2tfnVhn5qYh0Bge9/eitXMnrKtTUpH0APQRMe1w8o7ZnsCSv/78YKZLZmfiDM5ZWKjVFS4sscBP3GZ7oi4cFQT0QJYVEKTAndDOQzLQZ9PY69QvMPmwxCra2aJE0wpZ6gi5Hnz7tLK7ynPOrhKZ2BhfRkP+c4zpPcPIT2TKdPoMOV4SxM8H7ZSwtXUGbKM3P2BvUWP9fPCwI6oVi4wgKeV+YF9HgoF1AbdjgrXOQcHWytLQD6KYXscB1Tia0W00voYVegr9X0hFkSMYWEOmrekct5BbcjkdRegPRAPYIKgUsCJjSE7XqmVIdmSor/fzgoCMESlx1zNqqQM2kdgOqvQpZ0/kVNHsS1RGcjKXeCdytFjJTXygPI3Wvfi0tQgDgHUEpYEHAlAddHaULgt5e+6SvmrkTUC2TCXZHjdKamhLNS/Qq5tB0mfoOKrjjjhsYstxiK6vZYTybNfp07AtJCwKzdCYTnzBBwO6jTGEMDzsuqkI40cIHDsjjK6+UEcY33+x+7sorZXSx6e565AiwcCHQ1+c+v3Vr9DG99pqMGE6AC7gMLZgOi0EumCDvyg34DgBZnOYuDFnuaMDXcYvrzFfw6wBkNPIIBrAdewAAR5HBKLLYjjGMYDtGsB1DGC7BNyicyUlZ4GZ8PNFh1DQsCJjC0Z2/VQoJIeRCTsUH/P3fu5/Zv1+WvDJjEnbsAAYHvQ7lX/lK+DgWL3Yf/+f/DDSWa0q204LXMQOBVzGvou99FS1Yg+MYwQ4AhBEM5Cdu0u4itGLKeFJgFH1Yh4ddQmANZJW5UWSxDoewHXvQihe0/sjouzguvzzan4rTSJSZoO1C2hqrhqoIm3upOo5iO5iedo7b24m2un3qZ1t7O9GqVcXpHpq9aRyqqUkvIsdT6AB68yofZSPIzp53ewW51UE219Zn0EEj6HP1dQC9JXMtFcKeSM6vMYUBthEwiWHGByjbgPIa8vMmUudV+oldu6Kloo6aQ8jPYJwSz6Uge0BYG0HWYzhW6S0c4TAzO6G7BUoXAdOh/br7Kv4rr1gh4wv8As30tmVLMv+Uqx0WBEwy+EU7q7gCP3dX83N6OroR2a92co22EWz1xBKYuwDn3j5y6iBnNeEg3UqVMdmW18gmFEptRFbBZmE7g8ZG/39ycSKO6y06mQUBUz7MYDF9ki8m2lmPP8jlnEI2UVw/U7Kqr0RTbqV6k+qfGU+SuxH0uXYBbvURETAdWiJTtaQ9iWzEqVNQjzUNWBAw5UEvVEPk2AQ2bJDHhUY769lI1Q5C5TGKWtmsEoVtUtTM6ma2iGO96fYD/dIQdrnuA6atwqAcO4KozW9HECcraT1mMGVBwJQe3RCshIF5TBQ/2jko/sAWZ1BnE37cFn1F7w48sz1fThtBnOZnI4hTp6AeaxqECYI5SXosMVVKQwNw+DBw/fXSRVT5/3V1yfMNea/kuLmG9XTaY2Pua+vXy7TXmYxMJ/3ww+WvZFbFjGIbBjAK23/xEQzMpqcewnDePRTYjj0Yhawctw6HsA6H8Cpa8AX8DgYwCoCwAd/FIpzDFK5CZaqekes9f/Hf3ceKzk7g5Env0w0N8p9VY6P0Wl6xQoarTE567+3sLN2oqw2OI2AKQwkDHV0IFIoQsk6BycMPSyFw/LhcwOlMT0shVA70GgZRYxMWLiz+91Ak/wlfxRFc7zp3FBkcRE8+WGwAQA6tmMJ27ME6HJoVAtuxBwfRg6PownxcgJomRrADa3AMD+BXtcA1Mj7hcxyfIQznxyn7WtFJviVYbXUKADn5658nTwIvvSSrlerUfU2DoO1CuRuA9wD4dwA/APDxsPtZNZQi/OIETANyVMLSW5vvsR2X22uop8dbQvMyr8E2yXYW7fSSUf3sCFbTWbTlf+6iEfTREHblLwclnPNmLFUupvqzD2HDrPeRfr7QMpmNje5xjaCfWppzdOImebx3QT8J5DzePronkOzDv7W1sdeQ3mJP3qVqABoB/BDAGwHMA3AcwM8GPcOCICVEtRFERRmWzWyku3Z5J17TkKzbFMpVixjw2ic6Ooiu8qaXTkPzT3HdRkMYpCHspLNop4ewPn/JHTvgLmjjrRVxBF2uADZlSzDLZKr7nEfDbQpqAm9rI2pbmKNRQxDdO8edFdXP2ycs3XUt2wNshAmCJPevPQB+QEQ/IqKLAP4GwK0JjoeJSkODk0tIqYMOH5bHKpdQVIiAqSlpE9ixw+n3/HmZpsIsLzlvnvs9IyMyx9G8ecC99wIdHfK+9vbSfV8A+M533MfPPSd1DDptbaV9Z4G04DXr+adxNYAcPobPYDHOYy0mAFzCWXS47nPUMTnciv2ua8+jHWtwDMO4e7Yc5nWYwFF0ucpkPosOrMGxfGoL2ZeT/sIfpcKZnAR++prAv/+BOyfVtmm99KZ/6okwfX892wOsBEmJcjYAvwbgL7XjDwG413LfZgATACY6OzvLIi2ZAvGLI4iLX22DoBW+Xvksl5O7ir6+6C6mNdouooDKL5AqJeCSpo7pm/USsgWVqeaszmes11V2U7VriFfnIFqdBNvqPihtRa3HDNhAilVDv24RBJ8NeoZVQzVMzp0Pn2ZmiK6+Wv5sqofmz/cKnaEhKRxmZojWri1+Um2yqFeam4kuXoz2vFnDuYLtCLy5l17xURepdgBrXSmpD6CHTmPZbLpqv2ZWOXOPY7VxrNc/CGvB5TV1YeDn/69sBoCjcqoHe4CNMEGQpPvoGQDXaMfLATyT0FiYJCHyZiMdGACuvhp4+mnpOnr8uHPt7W93u6ISSfXSnj3y/MGDxWcfXbUKmJhwn/vpT73uJjaWLAHOni3u/UWwBo+7jl9GExb4qIsU6/AI1uERAEADcliHhwEAo9iKDfge1uDY7L3Pox1L8BxG8FFsxxg24DtYg2Oz6iFnHG733utwGNEdFQWm0IpR9M+6uspPYAqtUOqhIG+fTZtkYyIQJCXK2QDMAfAjAG+AYyx+e9AzvCOoQYLSUfT2El3rjpqdPW8LVguqfRyl6ekpOoJXwqHP662tzX2sdhslrpY2XaBa6BXYs6+qXYL/TkBGHv8YnYbqx1sIx7aSj7ozMI8bG+vH26dUIK3GYiKaBrANwDcBnADwVSJ6IqnxMAkhBNDaCvT3O4VulAGYCHjsMe8zhw4B27fL63o/ZvzB9LSMPTAx6xcozp+XhXG6uqQxuL9fWi9tfejv1Z/PZIDmZvc9rxmr8VWrpDFbja+nx7//GDubRh/f/RmfwK8cgFfRhPn4qW+fU7gCF9DkOvc82jGFKzCCj2INjuFreD++jpu11Xsu3xyOokuLX7CP0x5vKIx7BL70JVl/6KmneMVfMoKkRNoa7whqGNsKX88zpFomI20BZs6iQgzOtjYzQ7R+veMGOzPjtVHo7Y473MdqvF1dMnPq4sXu8ehuqPPny/Ht2iWv2VJmXHed3V6hWoRE/heuCLZX+LmbHsGqWUOxMiabK3138rppGsLg7DNHkCE9X5HXnTR+27hR/rnrLXtosSCtxuJCGguCGsYUBLa6xvqEqt9vqpf0Z+fPD55ATXVOV5czWau+/PqwPa/6UEJk2TL/Z9XEn8kQdXc755ubwyOifN5vqodeQQsN40/ouLDnZTIT1tmaEgLSYOuOpxjBtvxnHx3BanoJ813PuGMKdoW9KrQ1NtrTVau4ARYKdlgQMOnHL1OpWpmbk+zgYHgfZoSy2kmo4yVLnIlYFb9R92ez7nv9mpp9TGHQ1yffn836C4LFi4kuXfJ+v6gCQGsX8iv6i7A/a0tXHac5Kay9rrmvoMXjHWS6m7qjjotvDQ3B1+vRPTQMFgRMugkyFvuVtPSraWCeGxyUhmVTtbNkibym6h7MzDiqKJUq26yuBsidgbkUbWwkWrrUfa6nxxEkfmql9vZ4RXdiNjMV9XQJ+5ZCIbqLbhJZSgtNKV2rKicWBEz68atmpheo0e8Lq2lg9j1jBDvt2uUufKP3q+43V+qm50+Uls3KVX8Fi+WM4g460tgdfN+8edbzzyPadxxBH436eBMR/HYE4cKgra2wX7OtFZJCopYL1rAgYKoDM6DMLFmp3xe3X1PIBO00zPxF+nOZTLDh2FzdK/VQhYQAAVLdVchsms+b9Arm0zNYTKOwG9kfnxPw/bVm5h0KEwZqwt23j2ju3ODubRszs8XZEeiBZ8X2lVZYEDDpx29HEHfSN9En9bBJXn+fngRPvycsfYUpJHp6ypvyopRJ75qbibZulbswm8oqRlK/59HuyW46hCHXbXPnSnllU8Hs2+dvnxfCkXPKnGImmIuzig9KRaG/s9phQcCkm2LrG/uhJnPdDqAfmzaAKF5LPT3hqa7NSmrKfVQZkJWBevFieRxHUCxYIPtvzgeAdXdLtVPYEjpqy2SI7rzT+d5XX+3eBbW3h6bdfh3SkuveAeRcE3ao7j2X03YG7pKajQ3uXYUqel+oXj9oJ6Aa7whS1lgQ1CiF1jf2I0jd098vV7xBOxCbMNJVPGZyu74+b7xDNivvufpqeV3ZInp7pRBQnk9hcQr6JK3GoPpW0c/Fluxsb3cEVlcXUWen+319fcGxDMBsvQOCNFT7uYqGTqr5fwttC3M0hKHZEplyR2HfXRQzUYelq2YbQQobC4IaplhbgK0/22RvCgW/HYhNOPX2OuktVLZTFdxmqpF0+4CKe1DvUcemgLFN0GpyVs+Z92cy9jQcZh9RWr6fyU75vgPoocfnxi/2cwA92uTtvhyoZsk5xWdsNZLVsWlv8Ms+GmWXEGYbqAUhQMSCgKlnbBlNieRqXI9H8NuB2ISTGcimex0FCR39vClc1IQOeA29KkJZCRybW6tfU0r0GLEJ597USy1NM745hqI0b8ppI1Ooj5Dft4+opdmbelpvThSze8JWz6uJPardoJY9hXRYEDD1iZ+3kG1HoO4P6892HGbjMHcKZj8qfbayXVy65L5/1y5n56AHvUVt7e1E73xn5J3B3svVRJuL9x7PhC2FgFTvyJ9bWoj2fdlf7eeszv3fbe40dG+jQj2JajV2QIcFAVN/2CbjIC+hMMJsGH7X1eQe9l5135132l1WlT0ibm3mlhainTudviOt5mV+IFsUsd7CopWljcBZ3Y/mV/J7FxjqMe1vJlfxwTsCAmjvgqynZnEUo28teP8UCgsCpj6xpZwIWpn7EdWryfyMaovQz6tVu231rgzUHR2RJ3WXB1AU4ZG3ERyBY7y2VSd7BS0EvBZYDU3mGpomVehGv3ZI9NKJX8p6fgcjVwxqcQdZT16j2WYKEQo3+gbtCHRqdXfAgoCpX5RNwGaUtUwmvkSNczCFTxxbRFgtBd2HP656SHkXZbPB1duyWTpxYx+dxtV0AD3emgI+updX4D5/EXPyAqWLRtDn2V3MCgZDQJ64qZ8+OWfQUCv1ecZoUyuFxdBF0fvXsr2ABQFTn+hBYbpBdnDQEQq2Ajd++EU+69fj7BzC+vdrSghkMv6TuhkMplRLyiBt8zIy1E57F2RpGLu8k7DlfafhzrV0AN2BdY5HkPWmqMj/jvbtI1pxjUxQ19iQ8xqtLQI8LBo56sreT73EcQQpaywImEiYrppKEKjjuIIg6o6g0Ahp3XsorK1f76iOenu9k35fn7evvr7gqGAzjNfP42nbNo/a6jwW0uQ14XEQyttHqX72LjAmeLV7MwP/wgQr+U/gbW3x/tl41UtOINzs36lKYUHA1Cdh6paoqqG4kc9Rcybp95sr7SiF75Vbqa1wj96H+jSDznp6/DOf6gF3SrUVliU1JChuBH2kvJFGbSmt1ffR80BFTDroZx+IaxzWBYru7TTr8lpMkGPCpFIQAPhzAN8H8CiA/wWgNcpzLAiYWASpW+Ks7qJGPofFE/hNIsqF1K92wbJl4Xl+9PoKamKOUL3MGpVs1mdQk/SyZd46y83Nckexa5e9/2yWnn9jz6wwEJiZ1fs/A2m7+OkCS3Eg3b5j/o4NSqXS2bJFPZsjPVnelj9w2zFWdOaqzpicVkHwSwDm5H/+FIBPRXmOBQETmbAdQdw8RmETkv4+083TzHZqQwWN2XYGZonLsNV4lB1F2OSvn1dj0MdnFvoxBcr8+c648p9nGzpoCIN0AL2zgsEauKaEQES8Rt6c28gb8e/sFiheN9YTN/VTS7M315GKY0izt1EqBYFrAMCvABiPci8LAiYSpo1An7D041JkONXRDdRhEcXmc36TvT7eri57VTO/NBVRcxCpCVvlYTInZfO7+KmBbEn5MhmrgHEihGe8/ajfofk3DUBNxMMYor0L+mXgmnouokrHZiPQT6zotKfRbmtLv7dRNQiCvwfwwYDrmwFMAJjo7Owsyy+JqUF0dY6aaJXrYTn1vfpkb1vlmu+2JbTTn+vr86ppAG8eIr9mEy5+cQp+AsyMwTAFjMqGqgumvj6ZcM9nXEoIeFxU9e/p9zsL+t0Xkck2bEcwGrHAjmpp8jZKTBAA+GcAj1vardo9O/M2AhGlT94RMLGw5QWyXSvHe+Omt/DzGlKeTbZJuq/PmehtCfwbG71qoqA6y7o3lbkTiTr7Ac54AxLq6WqhyU7jd6PGVsBkHslzy0fN56iYHCFw75x+Jy0GoldbU7/qtJDaHQGA2wEcANAS9RkWBExRRDA8luQdhaS3CMpequ43dxkq/5Cuqmlv907ypoHXllK6t9eJsdDTT+tj37YtmlBQ39siCCavydDey6XwOjKvV7qe7trl/O7UdyrEDVf/G5iCSRFi+A9SMcmAtyFX1y0t/sFsvCMIFwLvAfBvABbFeY4FAVMwpa55EOddYektongb+Rm/VW0Cv5JeQU0V0VEBZ+qdQe/q9qmHrHYupsHctitQuw+1CzJVeDahV4oo8EJcgY1jm1G4GiKS0yoIfgDgNIBj+fb5KM+xIGAKokjdccHvNN9tW93adhD6TmBmxl5i02awjVMfWa+kpttP9PFH7butTRqZ1WSbzcrAN1v9BFNVZZukbc9FifuI8ncuNOgvhHJ5DZWq31QKgkIbCwKmYMo0AUR+Z5AA0lfCugeRvhPYsMHfI6kQQWCbiNWYbUFuNiP2zIyTx6inRz6jUnqsXesIKzWx28pyhr03rpdXlJ1fobuNClPKnQYLAoZRJDEBRJmY1ORvrvzNNAu6F43fxK4bfQHp4RMU9au7aereVWoS9ntWjVcJCCUQTBtCUxPR1q3y561bvfYJmwrM/Bv5TeY6EYzBrt9/hRYExazoS5n7iAUBwxAlsyPQ3+13HNe47LdqVvp2VeSmt9fx4Vcr8SVLvKtyXSevxmF6DZmCQxcUId5Bs03PU6SEgekSa3unuWMhKtzeU2EVYbEr+lKlziBiQcAwydgICh2frdnsCeZkber5lQsnkVNf2bQJmMZaP7dX2wper98ctEMxW3u7FAo2gWSqgPxcbf3+lpZiNx4q6DRQ7IqedwQsCJhSU0mvoUIImkxNe4I+cevfI0igKfuD37NB4zBtE3oUtN/Owa8pX0vTaGyOT43F9jeyCSwlGKP8fcO8g0pEsSt6thGwIGDKQYUmgNj4rcSj1leO8z3i6s/9InxzOXvUctzgsyAVUNB38/NqStGOrxQrevYaYkHA1APmpKWqmpkr8XLvXKKqz0xPJ1Ptc+edcnV+/fX2wDVz8i5k0rYJrCD7QkKkKb4gTBDMAcMwySEE0NoK9PcDIyPyeGgI2LFDnm9ocM5XehwjI/Jaa6vzfiHknPbCC8CePe4+zp8H/uEfgIkJYGAAOHwYaG+X54OwvccPItn32JgzVnWczXr7Nfsjcp8zj0vIpk3yc+dO4NQpoLMT2L3bOZ8qgqRE2hrvCJiaJS1qqyjjsHku2dq2bf4ZUPWMpzaVUBA2W4JZic62IyiDnSjt6acVCNkRNCQtiBiGgXdVWu4dQDHjEAK46irvCnzbNvfxAw8Ajz7qHLe3Oz/PmyefV7uAON93eNi+2j90SO4Scjn5OTYmdwtKNExNuc+pncTUlDyOyfg4sHkzcPKkfPzkSXk8Ph67q+QJkhJpa7wjYBiDpHYStl2BLb21rsNXxXd6etxeTjYjuLlLKFVEcQljSaqp2D1CdgSCCpCESdHd3U0TExNJD4NhkoM0nfbwsNTVj446uvuBAbnKHh4u7xh0Pf1nPgN0dLhtAZkMcPy4c5zLOfaPqSnvmI8dA7q65Er/rrvk9wLkzmNoKNr30n83tmN1rkFThKhxFUBDg30jIYTsNk0IIQ4TUbffdVYNMUy1MDzsVnUog+0NN5RE1REZ07Dc2Ahs2eK+Z8MG9/HAgJwdX3xRjtlUz7z4ovzcvt35Xnv2yJ+3b5fXXnjB/b3M7xim1lLvM8dV4O+qszPe+VQTtF1IW2PVEFO32NwsbQbbSrpM2qJ9w9xDg6qgBUVXxwkY8xtriaPL0+QeGgY4joBhUkpc/b6f/7x+HKWPOO8Mwzah6nmPbBHMZjCYPrkH2RiKncTZa4gFAcOkimISpwUJgqCJsVxpNvzcOW3ZQv0MtmE7glIFjKXFTbfCsCBgmLRRqJoijgrGNuGVM/Fe1PgDvzH4ZSA1E+vF2f0ws4QJAo4sZphKo0ftjo3JBrijek2IvBG1N9wgfecVQRG6hbwz7ncKOlbn/KKXjx1zzt91lxOjoLyGAPd3BeTvoxJR13VAou6jQoj/AuDPIWsXh8Shs/soU2NQTFfG4WHpEaQmPyLpUXPVVY5bJVlcJot5Z5TvEDdlg98z+nk1L+nfc88eb1qJUgmyGie17qNCiGsA3AjgVFJjYJjEUCt8nTBXRjOiVgjpj6/71ocJgRK6T7rcWfX+w2IY/HYP+nk92lhFMps7if7+aPmJqoDxcWDlSimjV65MIDo5SG9UzgbgAQAZAE8BaI/yDNsImJqg3Pr6crzTvG5LkV2J7xB0XKVUwg0VabQRCCFuAfA0ER0XIdJcCLEZwGYA6KzKSA2GMYia6TMt77SppHbsAK680snpU2qbg993CDquUnbuBC5ccJ+7cEGer1Sm0rLZCIQQ/wygw3JpJ4A/AfBLRPSiEOIpAN3ENgKm3ihEv17pd9qM1PrxPffIyGJFsTaHOqQSqSrCbARl2xEQ0S/6DOhaAG8AoHYDywEcEUL0ENFz5RoPw6SOJFa4cd8ZQkOOxAAAB/RJREFU5G10zz1yZ6DDnjyx6eyUmUtt5ytFxY3FRPQYES0mopVEtBLAGQDXsRBgmJSiCwOFEgJqZ2BL/eyHeS1Bz8U0sHs30NLiPtfSIs9XCo4jYBgmGJu3kW4jiGNzsNkbKpExNcWkoZJZ4tlH8zuDUPsAwzAJYNoI9JX/iy/KnYHu5jky4j+hE5W8OEytsGkT8NRT8tf71FOVL2fJOwKGYfwJ8zZqaPDeH9RXOaObmYLhwjQMw4RTSg+nUkc31yIl9ihLbWQxwzBVRKk8nEod3VyLFBqxXQQsCBiGqQxB9gYWBpKE7ChsI2AYpjIkEVFdbSRkR2EbAcMwlaUSEdVJRG2XkhLbUdhGwDBMuih3RHUCOvaSkoAdhQUBwzC1Q7XHKiRkR2EbAcMwtUO1xyokZEdhGwHDMLVHtccqcBwBwzBMEdRCrEKFM9OyIGAYpnbgWIWCYBsBwzC1A8cqFATbCBiGqT2qPY6gxLCNgGGY+qNG6xuXCxYEDMMwdQ4LAoZhmDonMUEghOgTQvy7EOIJIcR/S2ocDMMw9U4iXkNCiJ8HcCuA1UT0uhBicRLjYBiGYZLbEWwB8GdE9DoAENHzCY2DYRim7klKEPwHAO8SQhwSQnxHCLHW70YhxGYhxIQQYuLcuXMVHCLDMEx9UDbVkBDinwF0WC7tzL/3KgDrAKwF8FUhxBvJEtRARPcBuA+QcQTlGi/DMEy9kkhAmRDiG5CqoX/JH/8QwDoiClzyCyHOAThZ/hH60g7gfILvt8Fjik4ax8Vjik4ax1UtY1pBRIv8HkgqxcTXAPwCgH8RQvwHAPMQ4ZcZ9EUqgRBiIig6Lwl4TNFJ47h4TNFJ47hqZUxJCYIvAviiEOJxABcB3G5TCzEMwzDlJxFBQEQXAXwwiXczDMMwbjiyOB73JT0ACzym6KRxXDym6KRxXDUxpqrKPsowDMOUHt4RMAzD1DksCBiGYeocFgQFkNaEeUKI/yKEICFEewrG8udCiO8LIR4VQvwvIURrgmN5T/7v9QMhxMeTGoeOEOIaIcRDQogT+X9H/UmPSSGEaBRCHBVC/EPSYwEAIUSrEOKB/L+nE0KIG1IwpoH83+1xIcRfCyGaEhrHF4UQz+c9MNW5hUKIbwkhnsx/XhXWDwuCmBgJ894O4NMJDwmAnFgA3AjgVNJjyfMtAKuIaDWA/wPgj5MYhBCiEcB/B/DLAH4WwG8KIX42ibEYTAP4KBG9DTLCfmtKxgUA/QBOJD0IjTEA3yCitwLIIOGxCSGuBpAF0E1EqwA0AvhAQsP5HwDeY5z7OIBvE9FbAHw7fxwIC4L4pDVh3giAjwFIhfWfiP6JiKbzhwcBLE9oKD0AfkBEP8q7Lf8NpCBPFCJ6loiO5H9+GXJyuzrZUQFCiOUA/iOAv0x6LAAghLgCwHoAewHpek5EU8mOCoB0vW8WQswB0ALgmSQGQUTfBfAT4/StAL6U//lLAN4f1g8LgvhETphXKYQQtwB4moiOJz0WH34HwD8m9O6rAZzWjs8gBROujhBiJYA1AA4lOxIAwCjkgiKX9EDyvBHAOQB/lVdX/aUQYn6SAyKipyE1AacAPAvgRSL6pyTHZLCEiJ4F5IIDQGia/6Qii1NNqRLmVXBMfwLgl8r5fhtBYyKir+fv2QmpBhmv5Ng0bMVqU7FrAgAhxOUA/l8A24nopYTH8j4AzxPRYSHEu5Mci8YcANcB6COiQ0KIMUhVx66kBpTXud8K4A0ApgD8rRDig0S0L6kxFQsLAgtE9It+14QQWwD8XX7if1gIkYNM8lTWHNl+YxJCXAv5D/K4kAW6lwM4IoToIaLnkhiTNrbbAbwPwMYEU4icAXCNdrwcCW3jTYQQcyGFwDgR/V3S4wHwDgC3CCHeC6AJwBVCiH1ElGQWgDMAzhCR2i09gAg67zLziwB+rJJkCiH+DsDPAUiLIDgrhFhKRM8KIZYCCFVfs2ooPiphHuIkzCsXRPQYES0mopVEtBLyP8515RYCYQgh3gPgjwDcQkQXEhzKIwDeIoR4gxBiHqRRb3+C4wEACCm19wI4QUT3JD0eACCiPyai5fl/Rx8A8GDCQgD5f8enhRA/kz+1EcC/JTgkQKqE1gkhWvJ/x41Il3F9P4Db8z/fDuDrYQ/wjiA+nDAvGvcCuAzAt/I7lYNE9AeVHgQRTQshtgH4JqR3xxeJ6IlKj8PCOwB8CMBjQohj+XN/QkT/X4JjSit9AMbzgvxHAH47ycHkVVQPADgCqfY8ioRSTQgh/hrAuwG0CyHOABgC8GeQKuvfhRRavx7aD89hDMMw9Q2rhhiGYeocFgQMwzB1DgsChmGYOocFAcMwTJ3DgoBhGKbOYUHAMCVCCHF7PuPjk/lgOoapCth9lGFKgBBiIYAJAN2QKSwOA7ieiF5IdGAMEwHeETBMTIQQa/N1FpqEEPOFEE8A2ArgW0T0k/zk/y140wMzTCrhyGKGiQkRPSKE2A/gkwCaIXPMXELKs5wyjB+8I2CYwrgbshBQN4D/hpRnOWWYIFgQMExhLARwOYAFkJk6U5vllGHCYGMxwxRAXjX0N5ApwJcCGIQ0EF+Xv+UIpLHYrB7FMKmDbQQMExMhxIcBTBPR/8zXRP5XAF0APgGZ9hoA7mYhwFQLvCNgGIapc9hGwDAMU+ewIGAYhqlzWBAwDMPUOSwIGIZh6hwWBAzDMHUOCwKGYZg6hwUBwzBMnfN/AcSngkS4pDLnAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n",
      "0.0\n"
     ]
    }
   ],
   "source": [
    "plt.scatter(r0[...,0], r0[...,1], c='b', marker='o', label=\"class 0\")\n",
    "plt.scatter(r1[...,0], r1[...,1], c='r', marker='x', label=\"class 1\")\n",
    "plt.xlabel(\"x0\")\n",
    "plt.ylabel(\"x1\")\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "X = np.concatenate((r0,r1))\n",
    "y = np.zeros(len(r0)+len(r1))\n",
    "y[:len(r0),] = 1\n",
    "\n",
    "for i in y:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Logistic Function\n",
    "\n",
    "For the logistic regression, we want the output of the hypothesis to be in the interval $]0, 1[$. This is done using the _logistic function_. The logistic function is a special case of the _sigmoid function_, though in the domain of machine learning, the term _sigmoid function_ is often used as a synonym for _logistic function_:\n",
    "\n",
    "**Task:**\n",
    "\n",
    "Implement the _logistic function_ and plot it for 1000 points in the interval of $[-10,10]$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAb3ElEQVR4nO3de5hU1Znv8e/LTaKiorTKTcFHdMR4ZiQdb5ijRoxAEhCNCo4xKhO8BHNmNCZkNIniM0nUGGNG1IOCRkaCyBFoCAbNqHGigrYXUERCCxhajBBECQo0l/f8saq1LKrp6u6qWlW7fp/n2U/Vrr2q6u3dxa8Xq/Ze29wdEREpf+1iFyAiIvmhQBcRSQgFuohIQijQRUQSQoEuIpIQHWK9cbdu3bxPnz6x3l5EpCy99NJLf3P3qmzbogV6nz59qK2tjfX2IiJlyczebmqbhlxERBJCgS4ikhAKdBGRhFCgi4gkhAJdRCQhmg10M5tsZmvN7PUmtpuZ/drM6sxssZkNyH+ZIiLSnFx66A8Ag3ezfQjQL7WMAe5ue1kiItJSzR6H7u7PmFmf3TQZDjzoYR7eBWa2n5l1d/d381SjiCSVOzQ0wObNYdmyBbZuhe3bc1+2bQu3O3aE12tcdu5ser0129JrzvwZmrufuf71r8MXv5i//ZiSjxOLegKr09brU4/tEuhmNobQi+eQQw7Jw1uLSDQ7d8L778N774Vl7VpYtw42boQPP/x0aVzfuPHT4G4M7y1bdg2+JDMLtz16lGygW5bHsv6G3H0iMBGgurq6gn6LImVo505YtQqWL4eVK8P9lSvDUl8fwnv79uzP7dwZ9t03LPvsE24POgj23BM+97mwvanbPfaAjh2hQ4eWLe3ahcXs09vGJX29tdssLeosI/Z2t62I8hHo9UDvtPVewJo8vK6IFEtDA7zyCixcCIsXw2uvwZIl8NFHn7bp2BEOPRT69oVjjgkBnblUVYXw7tQp3s9SwfIR6DXAWDObBhwPfKjxc5ES19AAf/oTPP44PPss1NaG4Q+Abt1CYI8eHW6PPDKEePfu0L593Lplt5oNdDP7LXAq0M3M6oGfAB0B3P0eYB4wFKgDPgYuKVSxItIGGzfCzJlQUwNPPAF//3vodQ8YAFdeCSedBCeeGII74rCBtF4uR7mMama7A9/JW0Uikj87dsDvfw9TpsDs2aEX3qsXjBoFQ4fC6afD3nvHrlLyJNr0uSJSQBs3wqRJ8Otfhy8zDzgALr0UvvlNOP549cATSoEukiQffgi/+AXccUcYUjn5ZLj1Vhg2TF9UVgAFukgSbN0aeuM/+xls2ADnngvf/z5UV8euTIpIgS5S7p5+Gq64At58E4YMgf/4Dzj22NhVSQSabVGkXG3cGMbFTzst9NDnzQuLwrxiqYcuUo5eeCEcqbJqFYwbBz/6UTgLUyqaeugi5eauu2DgwHDa/R//GMbNFeaCAl2kfGzbFk4A+s53wlj5okXhKBaRFA25iJSDjz6CESPCGZ7jxoUvPtupPyafpUAXKXUffABf/SosWAD33w8XXxy7IilRCnSRUvb++zBoELz+OkyfDuecE7siKWEKdJFS9dFHoWe+ZEmYh2XIkNgVSYlToIuUooaG0Bt/4QWYMUNhLjlRoIuUGnf4l3+B+fPDBFsjRsSuSMqEviYXKTW33x6mu73xxnAmqEiOFOgipeTxx+Haa8Nwy/XXx65GyowCXaRU/OUvMHIkHH00PPCAjjOXFtMnRqQU7NgBF14YzgZ99FFdRUhaRV+KipSCn/0M/ud/4MEH4fDDY1cjZUo9dJHYFi6EG26ACy4IvXSRVlKgi8S0dStccgn07BlmUdS1PqUNNOQiEtPPfw5Ll4YLU+y7b+xqpMyphy4Sy9Kl8NOfhgtV6ExQyQMFukgM7nDZZeFoll/9KnY1khAachGJ4eGHw1Et994LBx4YuxpJCPXQRYpt82b4/vfDxZwvuSR2NZIg6qGLFNttt8Hq1WG+lvbtY1cjCaIeukgxrVkTTiI65xw45ZTY1UjCKNBFiummm8Lp/bfcErsSSSAFukixrFwJ990X5jo/7LDY1UgCKdBFiuWmm8KY+XXXxa5EEiqnQDezwWa2zMzqzGxclu2HmNlTZvaKmS02s6H5L1WkjC1fHibeuuKKcJq/SAE0G+hm1h6YAAwB+gOjzKx/RrPrgenufiwwErgr34WKlLXx42GPPWDcLv0hkbzJpYd+HFDn7ivcvQGYBgzPaOPAPqn7+wJr8leiSJlbsQKmTg2984MOil2NJFgugd4TWJ22Xp96LN0NwIVmVg/MA67K9kJmNsbMas2sdt26da0oV6QM3XYbdOgAV18duxJJuFwCPdt8np6xPgp4wN17AUOBKWa2y2u7+0R3r3b36qqqqpZXK1Ju1q6FyZPhm9+EHj1iVyMJl0ug1wO909Z7seuQymhgOoC7Pw90Brrlo0CRsvaf/xnmPL/22tiVSAXIJdBfBPqZWV8z60T40rMmo81fgNMBzOwoQqBrTEUq26ZNMGECnHUWHHlk7GqkAjQb6O6+HRgLzAeWEo5mWWJm481sWKrZNcC3zWwR8FvgYnfPHJYRqSy/+Q1s2BAm4hIpAouVu9XV1V5bWxvlvUUKzh3694cuXeCFF2JXIwliZi+5e3W2bZptUaQQ/vAHePPNcDKRSJHo1H+RQrjzTqiqgvPOi12JVBAFuki+rVwJc+bAmDHh7FCRIlGgi+TbXXdBu3Zw+eWxK5EKo0AXyactW2DSJBgxAnr1il2NVBgFukg+PfpoOFRRvXOJQIEukk+TJkHfvnDaabErkQqkQBfJl7fegiefhEsvDWPoIkWmT51Ivtx/fwjyiy+OXYlUKAW6SD5s3x4CffBgfRkq0SjQRfJh/nxYswZGj45diVQwBbpIPkyeHM4M/drXYlciFUyBLtJW778Pc+fCBRdAp06xq5EKpkAXaasZM6ChIVyVSCQiBbpIWz30EPzDP8CAAbErkQqnQBdpi7ffhmeegQsvBMt2+V2R4lGgi7TF1Knh9oIL4tYhggJdpPXcYcoUGDgwnO4vEpkCXaS1Xn0Vli4Nwy0iJUCBLtJaDz0EHTvCuefGrkQEUKCLtM7OnTBtWjjV/4ADYlcjAijQRVrn+efhnXfg/PNjVyLyCQW6SGs88ki4XujXvx67EpFPKNBFWmrnzhDogwfDPvvErkbkEwp0kZZ6/vkws+J558WuROQzFOgiLaXhFilRCnSRlmgcbhkyBLp0iV2NyGco0EVaonG4RceeSwlSoIu0xPTpGm6RkqVAF8nVzp1h7nMNt0iJyinQzWywmS0zszozG9dEm/PM7A0zW2JmU/NbpkgJeO45Hd0iJa1Dcw3MrD0wATgDqAdeNLMad38jrU0/4IfAQHffYGYHFqpgkWgefTQMt+i6oVKicumhHwfUufsKd28ApgHDM9p8G5jg7hsA3H1tfssUicwdZs2CQYM03CIlK5dA7wmsTluvTz2W7gjgCDN71swWmNngbC9kZmPMrNbMatetW9e6ikVieO01WLkSzjordiUiTcol0LNdV8sz1jsA/YBTgVHAfWa23y5Pcp/o7tXuXl1VVdXSWkXimTUrXGJOR7dICcsl0OuB3mnrvYA1WdrMdvdt7r4SWEYIeJFkmDULTjoJDjoodiUiTcol0F8E+plZXzPrBIwEajLazAJOAzCzboQhmBX5LFQkmrffhlde0XCLlLxmA93dtwNjgfnAUmC6uy8xs/FmNizVbD6w3szeAJ4CrnX39YUqWqSoZs8Ot8MzjwUQKS3mnjkcXhzV1dVeW1sb5b1FWuTLX4a1a+H112NXIoKZveTu1dm26UxRkd1Zvx6eeUbDLVIWFOgiu/O738GOHQp0KQsKdJHdmTkTevaEL3whdiUizVKgizTl449h/vzQO7dsp2OIlBYFukhTnngCNm/WcIuUDQW6SFNmzYJ994VTToldiUhOFOgi2WzfDnPmhJkVO3aMXY1IThToItk8+2w4ZFHDLVJGFOgi2cyaFeY+P/PM2JWI5EyBLpJJc59LmVKgi2RavBhWrdJwi5QdBbpIJs19LmVKgS6SSXOfS5lSoIukW7UKXn1Vwy1SlhToIuk097mUMQW6SLpZs+Doo6GfrqAo5UeBLtKoce5z9c6lTCnQRRrNnQs7d8KIEbErEWkVBbpIo1mzNPe5lDUFugho7nNJBAW6CGjuc0kEBboIaO5zSQQFuojmPpeEUKCLaO5zSQgFuojmPpeEUKBLZdPc55IgCnSpbJr7XBJEgS6VTXOfS4Io0KWyae5zSRAFulQuzX0uCaNAl8pVUxNuNbuiJEROgW5mg81smZnVmdm43bT7hpm5mVXnr0SRAtHc55IwzQa6mbUHJgBDgP7AKDPrn6VdF+C7wMJ8FymSd5r7XBIolx76cUCdu69w9wZgGpDtX8FNwC3AljzWJ1IYNTWwY4fGzyVRcgn0nsDqtPX61GOfMLNjgd7uPnd3L2RmY8ys1sxq161b1+JiRfJmxgw49FCo1uigJEcugZ5tcmj/ZKNZO+B24JrmXsjdJ7p7tbtXV1VV5V6lSD598EGYLvcb39Dc55IouQR6PdA7bb0XsCZtvQvweeBpM1sFnADU6ItRKVlz5sC2bSHQRRIkl0B/EehnZn3NrBMwEqhp3OjuH7p7N3fv4+59gAXAMHevLUjFIm31yCPQuzccf3zsSkTyqtlAd/ftwFhgPrAUmO7uS8xsvJkNK3SBInm1cWO41Nw552i4RRKnQy6N3H0eMC/jsR830fbUtpclUiBz50JDg4ZbJJF0pqhUlhkzoHt3OPHE2JWI5J0CXSrHpk3w2GNhuKWdPvqSPPpUS+WYNw+2bNFwiySWAl0qxyOPhGlyTz45diUiBaFAl8qwaRP87ndw9tnQvn3sakQKQoEulWH2bNi8GS64IHYlIgWjQJfKMHUqHHJIuDqRSEIp0CX51q0LJxONGqWjWyTR9OmW5JsxI0yVq+EWSTgFuiTf1KnhykTHHBO7EpGCUqBLsr39NvzpT6F3rrlbJOEU6JJs06aF21Gj4tYhUgQKdEm2qVPDvC19+8auRKTgFOiSXK+8AosXwz//c+xKRIpCgS7J9cAD0KmThlukYijQJZkaGuChh+Css2D//WNXI1IUCnRJpjlzYP16uOSS2JWIFI0CXZLp/vuhRw8444zYlYgUjQJdkufdd+H3v4eLLtLMilJRFOiSPP/1X+FUfw23SIVRoEuy7NwJ994LAwfCEUfErkakqBTokixPPgnLl8Pll8euRKToFOiSLHffDd266bqhUpEU6JIc77wTrkx06aXQuXPsakSKToEuyXHffWEM/bLLYlciEoUCXZJh2zaYOBHOPBMOOyx2NSJRKNAlGWpqYM0auOKK2JWIRKNAl2S4/fYwRe5Xvxq7EpFoOsQuQKTNFiyAZ5+FO+7QmaFS0dRDl/J3222w337h6BaRCpZToJvZYDNbZmZ1ZjYuy/arzewNM1tsZv9tZofmv1SRLFasgEcfDUe27L137GpEomo20M2sPTABGAL0B0aZWf+MZq8A1e7+v4AZwC35LlQkq1/9KgyzXHVV7EpEosulh34cUOfuK9y9AZgGDE9v4O5PufvHqdUFQK/8limSxdq14djzUaOgZ8/Y1YhEl0ug9wRWp63Xpx5rymjgsWwbzGyMmdWaWe26detyr1Ikm1/8ArZuhX//99iViJSEXALdsjzmWRuaXQhUA7dm2+7uE9292t2rq6qqcq9SJNO6dTBhAowcCUceGbsakZKQy2GL9UDvtPVewJrMRmY2CLgOOMXdt+anPJEm/PKXsHkzXH997EpESkYuPfQXgX5m1tfMOgEjgZr0BmZ2LPB/gWHuvjb/ZYqkWb8e7rwTzj8fjjoqdjUiJaPZQHf37cBYYD6wFJju7kvMbLyZDUs1uxXYG3jEzF41s5omXk6k7X76U/j4Y/XORTLkdKaou88D5mU89uO0+4PyXJdIditXht75xRfD0UfHrkakpOhMUSkv118fjju/8cbYlYiUHAW6lI+XXoKpU+Hf/g166VQHkUwKdCkP7nD11eHycj/4QexqREqSZluU8jBlCjzzTLiIxT77xK5GpCSphy6lb8MG+N734IQTYPTo2NWIlCz10KX0XXddOPb88cehnfogIk3Rvw4pbc8+C/fcA9/9LvzTP8WuRqSkKdCldG3aBBddBH36wPjxsasRKXkacpHSde214USiP/4RunSJXY1IyVMPXUrTY4+FoZZrroEvfSl2NSJlQYEupWf16jDU8vnPw003xa5GpGwo0KW0NDTAueeGC1fMmAGdO8euSKRsaAxdSsvVV8PChSHMdeEKkRZRD11Kx513hqsQXXMNnHNO7GpEyo4CXUrDrFnhWPPhw+Hmm2NXI1KWFOgS33PPwahRcNxxYTbF9u1jVyRSlhToEtdzz8GZZ0Lv3jBnDuy5Z+yKRMqWAl3iaQzz7t3hqaegqip2RSJlTYEuccydC2ecEcL86aehZ8/YFYmUPQW6FN/dd4cvP486Ksxx3qNH7IpEEkGBLsWzZQtceWVYhg4Nc7QcfHDsqkQSQ4EuxbFiBQwcGHrn114LM2fCXnvFrkokUXSmqBTWzp1w110wbhx07AizZ8OwYbGrEkkk9dClcN54A045Ba66Ck4+GRYtUpiLFJACXfLvr3+Fyy+HY46BJUvggQfCdLiHHBK7MpFE05CL5M8778Dtt4d5zLduhbFj4Uc/gm7dYlcmUhEU6NI27vDyy2GcfMoU2LEDzjsPbrwRjjgidnUiFUWBLq3z3nswfTpMmhTGxjt3hm9/O8yUeNhhsasTqUgKdMmNO/z5z+EMz5kzw2n77jBgQJjydtQo6No1dpUiFU2BLtlt3w7LloXgfuqpcHr+u++Gbf/4j/CTn8DZZ4cvPkWkJCjQK507rFkDy5eHHviiRWFMfNEi2Lw5tDn4YDjttLAMGgR9+8atWUSyyinQzWwwcAfQHrjP3X+esX0P4EHgC8B64Hx3X5XfUqXFtm2DDRvCePeaNZ9d3nkHVq6Eujr4+ONPn7PPPnDssXDZZWE45YtfDJeCM4v3c4hITpoNdDNrD0wAzgDqgRfNrMbd30hrNhrY4O6Hm9lI4Gbg/EIUXJbcwxBG47JtW27rDQ0hbDdv3v3tpk0huBuXDz4Itx99lL2e/fcPE2Ideiicfjr06/fp0rs3tNPpCSLlKJce+nFAnbuvADCzacBwID3QhwM3pO7PAO40M3N3z2OtweTJcOut4b77rku2x3N9rBDP37EjLIXQuXO4IMRee4UvJLt2hcMPh/32+3S9a1c46KAQ4D16hOlqO3cuTD0iElUugd4TWJ22Xg8c31Qbd99uZh8CBwB/S29kZmOAMQCHtPaswW7dwhdxjUMAZrsu2R7P9bF8P79Dh0+Xjh1btt6pUwjsz31u19vOndWTFpHPyCXQsw2eZva8c2mDu08EJgJUV1e3rvc+bJjmAxERySKXLl490DttvRewpqk2ZtYB2Bd4Px8FiohIbnIJ9BeBfmbW18w6ASOBmow2NcC3Uve/ATxZkPFzERFpUrNDLqkx8bHAfMJhi5PdfYmZjQdq3b0GmARMMbM6Qs98ZCGLFhGRXeV0HLq7zwPmZTz247T7W4Bz81uaiIi0hA6TEBFJCAW6iEhCKNBFRBJCgS4ikhAW6+hCM1sHvN3Kp3cj4yzUEqG6WkZ1tVyp1qa6WqYtdR3q7lXZNkQL9LYws1p3r45dRybV1TKqq+VKtTbV1TKFqktDLiIiCaFAFxFJiHIN9ImxC2iC6moZ1dVypVqb6mqZgtRVlmPoIiKyq3LtoYuISAYFuohIQpRsoJvZuWa2xMx2mll1xrYfmlmdmS0zszObeH5fM1toZsvN7OHU1L/5rvFhM3s1tawys1ebaLfKzF5LtavNdx1Z3u8GM3snrbahTbQbnNqHdWY2rgh13Wpmb5rZYjObaWb7NdGuKPuruZ/fzPZI/Y7rUp+lPoWqJe09e5vZU2a2NPX5/z9Z2pxqZh+m/X5/nO21ClDbbn8vFvw6tb8Wm9mAItR0ZNp+eNXMNprZv2a0Kdr+MrPJZrbWzF5Pe2x/M3silUVPmFnXJp77rVSb5Wb2rWxtmuXuJbkARwFHAk8D1WmP9wcWAXsAfYG3gPZZnj8dGJm6fw9wRYHrvQ34cRPbVgHdirjvbgC+10yb9ql9dxjQKbVP+xe4rq8AHVL3bwZujrW/cvn5gSuBe1L3RwIPF+F31x0YkLrfBfhzlrpOBeYW6/OU6+8FGAo8RriC2QnAwiLX1x74K+HEmyj7C/jfwADg9bTHbgHGpe6Py/a5B/YHVqRuu6bud23p+5dsD93dl7r7siybhgPT3H2ru68E6ggXsv6EmRnwZcIFqwF+A5xVqFpT73ce8NtCvUcBfHLxb3dvABov/l0w7v64u29PrS4gXP0qllx+/uGEzw6Ez9Lpqd91wbj7u+7+cur+34GlhGv2loPhwIMeLAD2M7PuRXz/04G33L21Z6C3mbs/w65Xa0v/HDWVRWcCT7j7++6+AXgCGNzS9y/ZQN+NbBetzvzAHwB8kBYe2drk05eA99x9eRPbHXjczF5KXSi7GMam/ts7uYn/4uWyHwvpUkJvLpti7K9cfv7PXPwcaLz4eVGkhniOBRZm2XyimS0ys8fM7OgildTc7yX2Z2okTXeqYuyvRge5+7sQ/mADB2Zpk5d9l9MFLgrFzP4AHJxl03XuPrupp2V5rFUXrc5FjjWOYve984HuvsbMDgSeMLM3U3/JW213dQF3AzcRfuabCMNBl2a+RJbntvkY1lz2l5ldB2wHHmriZfK+v7KVmuWxgn2OWsrM9gb+H/Cv7r4xY/PLhGGFTanvR2YB/YpQVnO/l5j7qxMwDPhhls2x9ldL5GXfRQ10dx/UiqflctHqvxH+u9ch1bPK1iYvNVq4KPbZwBd28xprUrdrzWwm4b/7bQqoXPedmd0LzM2yKZf9mPe6Ul/2fA043VODh1leI+/7K4uWXPy83op48XMz60gI84fc/dHM7ekB7+7zzOwuM+vm7gWdhCqH30tBPlM5GgK87O7vZW6Itb/SvGdm3d393dQQ1NosbeoJY/2NehG+P2yRchxyqQFGpo5A6Ev4S/tCeoNUUDxFuGA1hAtYN9Xjb6tBwJvuXp9to5ntZWZdGu8Tvhh8PVvbfMkYtxzRxPvlcvHvfNc1GPgBMMzdP26iTbH2V0le/Dw1Rj8JWOruv2yizcGNY/lmdhzh3/H6AteVy++lBrgodbTLCcCHjUMNRdDk/5Jj7K8M6Z+jprJoPvAVM+uaGiL9SuqxlinGN7+t/LZ4BOGv1lbgPWB+2rbrCEcoLAOGpD0+D+iRun8YIejrgEeAPQpU5wPA5RmP9QDmpdWxKLUsIQw9FHrfTQFeAxanPkzdM+tKrQ8lHEXxVpHqqiOME76aWu7JrKuY+yvbzw+MJ/zBAeic+uzUpT5LhxVhH51M+K/24rT9NBS4vPFzBoxN7ZtFhC+XTypCXVl/Lxl1GTAhtT9fI+3otALXtichoPdNeyzK/iL8UXkX2JbKr9GE713+G1ieut0/1bYauC/tuZemPmt1wCWteX+d+i8ikhDlOOQiIiJZKNBFRBJCgS4ikhAKdBGRhFCgi4gkhAJdRCQhFOgiIgnx/wGa62hXrxFyWQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def logistic_function(x):\n",
    "    \"\"\" Applies the logistic function to x, element-wise. \"\"\"\n",
    "    return 1.0 / (1 + np.exp(-x))\n",
    "\n",
    "### Insert code to plot the logistic function below\n",
    "\n",
    "def plot_logisticFunction(x_points):\n",
    "    fig = plt.figure()\n",
    "    ax = fig.add_subplot(1, 1, 1)\n",
    "    plt.plot(x_points,logistic_function(x_points), 'r')\n",
    "    plt.show()\n",
    "\n",
    "xpoints = np.linspace(-10,10,1000)\n",
    "plot_logisticFunction(xpoints)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic Hypothesis\n",
    "\n",
    "The logistic hypothesis is defined as:\n",
    "\n",
    "$$\n",
    "h_\\theta(\\vec x) = sigmoid(\\vec \\theta^T \\vec x')\n",
    "$$\n",
    "\n",
    "with:\n",
    "\n",
    "$$\n",
    "\\vec x = \\begin{pmatrix} \n",
    "x_1 & x_2 & \\ldots & x_n \\\\\n",
    "\\end{pmatrix}\n",
    "\\text{   and   }\n",
    "\\vec x' = \\begin{pmatrix} \n",
    "1 & x_1 & x_2 & \\ldots & x_n \\\\\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "or for the whole data set $X$ and $X'$\n",
    "\n",
    "$$\n",
    "X = \\begin{pmatrix} \n",
    "x_1^1 & \\ldots & x_n^1 \\\\\n",
    "x_1^2 & \\ldots & x_n^2 \\\\\n",
    "\\vdots &\\vdots &\\vdots \\\\\n",
    "x_1^m & \\ldots & x_n^m \\\\\n",
    "\\end{pmatrix}\n",
    "\\text{   and   }\n",
    "X' = \\begin{pmatrix} \n",
    "1 & x_1^1 & \\ldots & x_n^1 \\\\\n",
    "1 & x_1^2 & \\ldots & x_n^2 \\\\\n",
    "\\vdots &\\vdots &\\vdots &\\vdots \\\\\n",
    "1 & x_1^m & \\ldots & x_n^m \\\\\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "**Task:**\n",
    "\n",
    "Implement the logistic hypothesis using your implementation of the logistic function. `logistic_hypothesis` should return a function which accepts the training data $X$. Example usage:\n",
    "\n",
    "`>> theta = np.array([1.1, 2.0, -.9])`\n",
    "\n",
    "`>> h = logistic_hypothesis(theta) `\n",
    "\n",
    "`>> print(h(X))`\n",
    "\n",
    "**Note:** The training data was sampled with random noise, so the actual values of your h(X) may differ.\n",
    "\n",
    "`array([0.03587382, 0.0299963 , 0.97389774, ...,`\n",
    "\n",
    "**Hint:**\n",
    "\n",
    "You may of course also implement a helper function for transforming $X$ into $X'$ and use it inside the `lamda` function of `logistic_hypothesis`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.99999939 0.99999998 0.99999776 ... 0.99998955 0.99998655 0.99350005]\n"
     ]
    }
   ],
   "source": [
    "def logistic_hypothesis(theta):\n",
    "    return lambda x : logistic_function(np.dot(generateNewX(x), theta.T))\n",
    "\n",
    "def generateNewX(x):\n",
    "    x = np.insert(x, 0, 1, axis=1)\n",
    "    return x \n",
    "    \n",
    "### Uncomment to test your implementation\n",
    "theta = np.array([1.,2.,3.])\n",
    "h = logistic_hypothesis(theta)\n",
    "print(h(X))\n",
    "xxx = h(X)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-entropy\n",
    "\n",
    "The cross-entropy costs are defined with:\n",
    "\n",
    "\\begin{equation}\n",
    "    loss(h_\\theta (x^i), y^i) = -y^i \\cdot log(h_\\theta (x^i)) - (1-y^i) \\cdot log(1-h_\\theta(x^i))\n",
    "\\end{equation}\n",
    "\n",
    "**Task:**\n",
    "\n",
    "Implement the cross-entropy cost. \n",
    "\n",
    "Your python function should return a function, which accepts the vector $\\vec \\theta$.\n",
    "The returned function should return the cost for each feature vector $\\vec x^i$. The length of the returned array of costs therefore has to be the same length as we have feature vectors (and also labels $y$). Example usage:\n",
    "\n",
    "`>> J = cross_entropy_loss(logistic_hypothesis, X, y)`\n",
    "\n",
    "`>> print(J(theta))`\n",
    "\n",
    "**Note:** The training data was sampled with random noise, so the actual values of your h(X) may differ.\n",
    "\n",
    "\n",
    "`array([ 7.3,  9.5, ....`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[6.07677761e-07 1.88853160e-08 2.23830384e-06 ... 1.14686756e+01\n",
      " 1.12163103e+01 5.03596075e+00]\n"
     ]
    }
   ],
   "source": [
    "def cross_entropy_costs(h, X, y):\n",
    "    ''' Implements cross-entropy as a function costs(theta) on given traning data \n",
    "    \n",
    "    Args:\n",
    "        h: the hypothesis as function\n",
    "        x: features as 2D array with shape (m_examples, n_features)  \n",
    "        y: ground truth labels for given features with shape (m_examples)\n",
    "        \n",
    "    Returns:\n",
    "        lambda costs(theta) that models the cross-entropy for each x^i\n",
    "    '''\n",
    "    \n",
    "    y = -y \n",
    "    return lambda thetas :  np.multiply(y,np.log(logistic_hypothesis(thetas)(X))) - np.multiply(y+1,np.log(-logistic_hypothesis(thetas)(X) +1))\n",
    "    \n",
    "### Uncomment to test your implementation\n",
    "theta = np.array([1.,2.,3.])\n",
    "costs = cross_entropy_costs(logistic_hypothesis, X, y)\n",
    "print(costs(theta))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss Function\n",
    "\n",
    "\\begin{equation}\n",
    "    J_D(\\theta)=\\frac{1}{m}\\sum_{i=1}^{m}\\left(loss(h_\\theta (x^i), y^i)\\right)\n",
    "\\end{equation}\n",
    "\n",
    "**Task:**\n",
    "\n",
    "Now implement the loss function $J$, which calculates the mean costs for the whole training data $X$. Your python function should return a function, which accepts the vector $\\vec \\theta$.\n",
    "\n",
    "**Note:** You can ignore the parameter `lambda_reg` for now, it is a hyperparameter for regularization. In a later exercise, you may revisit your implementation and implement regularization if you wish."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.9644023910334787\n"
     ]
    }
   ],
   "source": [
    "\n",
    "def mean_cross_entropy_costs(X, y, hypothesis, cost_func, lambda_reg=0.1):\n",
    "    ''' Implements mean cross-entropy as a function J(theta) on given traning data \n",
    "    \n",
    "    Args:\n",
    "        X: features as 2D array with shape (m_examples, n_features)  \n",
    "        y: ground truth labels for given features with shape (m_examples)\n",
    "        hypothesis: the hypothesis as function\n",
    "        cost_func: cost function\n",
    "        \n",
    "    Returns:\n",
    "        lambda J(theta) that models the mean cross-entropy\n",
    "    '''\n",
    "    m = len(y)\n",
    "    return lambda thetas : (np.sum(cost_func(hypothesis, X, y)(thetas))+ np.sum(np.square(thetas[1:]))*lambda_reg)/m\n",
    "## Uncomment to test your implementation\n",
    "theta = np.array([1.,2.,3.])\n",
    "J = mean_cross_entropy_costs(X,y, logistic_hypothesis, cross_entropy_costs, 0.1)\n",
    "print(J(theta))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "###  Gradient Descent\n",
    "\n",
    "A short recap, the gradient descent algorithm is a first-order iterative optimization for finding a minimum of a function. From the current position in a (cost) function, the algorithm steps proportional to the negative of the gradient and repeats this until it reaches a local or global minimum and determines. Stepping proportional means that it does not go entirely in the direction of the negative gradient, but scaled by a fixed value $\\alpha$ also called the learning rate. Implementing the following formalized update rule is the core of the optimization process:\n",
    "\n",
    "\\begin{equation}\n",
    "    \\theta_{j_{new}} \\leftarrow \\theta_{j_{old}} - \\alpha * \\frac{\\delta}{\\delta\\theta_{j_{old}}} J(\\theta_{old})\n",
    "\\end{equation}\n",
    "\n",
    "**Task:**\n",
    "\n",
    "Implement the function to update all theta values.\n",
    "\n",
    "**Note:** You can ignore the parameter `lambda_reg` for now, it is a hyperparameter for regularization. In a later exercise, you may revisit your implementation and implement regularization if you wish."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-192.57905482    1.85065328    2.81506312]\n"
     ]
    }
   ],
   "source": [
    "def compute_new_theta(X, y, theta, learning_rate, hypothesis, lambda_reg=0.1):\n",
    "    ''' Updates learnable parameters theta \n",
    "\n",
    "    The update is done by calculating the partial derivities of \n",
    "    the cost function including the linear hypothesis. The \n",
    "    gradients scaled by a scalar are subtracted from the given \n",
    "    theta values.\n",
    "    \n",
    "    Args:\n",
    "        X: 2D numpy array of x values\n",
    "        y: array of y values corresponding to x\n",
    "        theta: current theta values\n",
    "        learning_rate: value to scale the negative gradient  \n",
    "        hypothesis: the hypothesis as function\n",
    "\n",
    "        \n",
    "    Returns:\n",
    "        theta: Updated theta_0\n",
    "    '''\n",
    "\n",
    "    h = np.sum(hypothesis(theta)(X))\n",
    "    t_ =np.ones(len(X[0,:])+1)\n",
    "    m = len(X)\n",
    "    t_[0] = theta[0] - learning_rate*np.mean(h-y)\n",
    "    for i in range(1,(len(X[0,:])+1)) :\n",
    "         t_[i] = theta[i]*(1 - learning_rate*(lambda_reg/m)) - (learning_rate/m)*np.mean((h-y)*X[:,(i-1)])\n",
    "     \n",
    "    theta = t_.copy()              \n",
    "    return theta\n",
    "\n",
    "### Uncomment to test your implementation\n",
    "theta = np.array([1.,2.,3.])\n",
    "theta = compute_new_theta(X, y, theta, .1, logistic_hypothesis, .1)\n",
    "print(theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Using the `compute_new_theta` method, you can now implement the gradient descent algorithm. Iterate over the update rule to find the values for $\\theta$ that minimize our cost function $J_D(\\theta)$. This process is often called training of a machine learning model. \n",
    "\n",
    "**Task:**\n",
    "- Implement the function for the gradient descent.\n",
    "- Create a history of all theta and cost values and return them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(X, y, theta, learning_rate, num_iters, lambda_reg=0.1):\n",
    "    ''' Minimize theta values of a logistic model based on cross-entropy cost function\n",
    "    \n",
    "    Args:\n",
    "        X: 2D numpy array of x values\n",
    "        y: array of y values corresponding to x\n",
    "        theta: current theta values\n",
    "        learning_rate: value to scale the negative gradient  \n",
    "        num_iters: number of iterations updating thetas\n",
    "        lambda_reg: regularization strength\n",
    "        \n",
    "    Returns:\n",
    "        history_cost: cost after each iteration\n",
    "        history_theta: Updated theta values after each iteration\n",
    "    '''\n",
    "    history_cost = []\n",
    "    history_theta = []\n",
    "    \n",
    "    #history_cost.append(costs(theta))\n",
    "    #history_theta.append(theta)\n",
    "    \n",
    "    \n",
    "    for _ in range(num_iters-1):\n",
    "        try:\n",
    "            theta =  compute_new_theta(X, y, theta, learning_rate,logistic_hypothesis,lambda_reg)\n",
    "            updatedcost =J(theta)\n",
    "            print(updatedcost)\n",
    "            history_cost.append(updatedcost)\n",
    "            history_theta.append(theta.tolist())\n",
    "        except Exception as e:\n",
    "            print(e)\n",
    "\n",
    "   \n",
    "    return history_cost,history_theta "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training and Evaluation\n",
    "\n",
    "**Task:**\n",
    "\n",
    "Choose an appropriate learning rate, number of iterations and initial theta values and start the training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.925192645292619\n",
      "4.832444232442643\n",
      "4.740191550046077\n",
      "4.648459041058143\n",
      "4.557273618300717\n",
      "4.466664815214257\n",
      "4.376664645967102\n",
      "4.287307219003909\n",
      "4.198628541525295\n",
      "4.110666095123127\n",
      "4.0234583141148414\n",
      "3.9370440775874314\n",
      "3.8514623216551076\n",
      "3.7667515588389993\n",
      "3.6829494306904778\n",
      "3.6000924341614184\n",
      "3.518215523277363\n",
      "3.4373518630461177\n",
      "3.357532500264986\n",
      "3.2787861417993422\n",
      "3.201138881548693\n",
      "3.124614012804104\n",
      "3.049231834502544\n",
      "2.975009546062501\n",
      "2.9019611751683767\n",
      "2.830097574401367\n",
      "2.7594264875442964\n",
      "2.6899526639655194\n",
      "2.621678046449465\n",
      "2.554602013347721\n",
      "2.488721674589749\n",
      "2.424032221472063\n",
      "2.360527316058268\n",
      "2.298199515293056\n",
      "2.237040701042951\n",
      "2.177042510067708\n",
      "2.1181967287095085\n",
      "2.060495635963186\n",
      "2.0039322770267205\n",
      "1.948500651225015\n",
      "1.8941958138343846\n",
      "1.8410138922740995\n",
      "1.7889520287142453\n",
      "1.738008264728819\n",
      "1.6881813841942976\n",
      "1.639470734201327\n",
      "1.5918760370602958\n",
      "1.5453972055949519\n",
      "1.5000341685655703\n",
      "1.4557867090087184\n",
      "1.4126543160783889\n",
      "1.370636049745613\n",
      "1.329730417118775\n",
      "1.2899352602554008\n",
      "1.2512476564524484\n",
      "1.2136638330877236\n",
      "1.177179099850816\n",
      "1.141787801408674\n",
      "1.1074832931010221\n",
      "1.0742579415060882\n",
      "1.0421031501475813\n",
      "1.01100940957491\n",
      "0.9809663691099362\n",
      "0.9519629265822133\n",
      "0.9239873310349354\n",
      "0.8970272928952159\n",
      "0.8710700957051656\n",
      "0.8461027038787624\n",
      "0.8221118615468501\n",
      "0.7990841786050256\n",
      "0.7770062013556809\n",
      "0.7558644663859246\n",
      "0.735645537567176\n",
      "0.7163360271088529\n",
      "0.6979226023497611\n",
      "0.6803919805221292\n",
      "0.6637309139900935\n",
      "0.6479261685735606\n",
      "0.6329644974933537\n",
      "0.6188326133329227\n",
      "0.6055171601586985\n",
      "0.593004687653208\n",
      "0.581281628758697\n",
      "0.5703342819604783\n",
      "0.5601487989371997\n",
      "0.5507111779279515\n",
      "0.5420072628113285\n",
      "0.5340227475945405\n",
      "0.5267431857861711\n",
      "0.5201540039746835\n",
      "0.5142405188522471\n",
      "0.5089879569108089\n",
      "0.5043814760609188\n",
      "0.5004061884871103\n",
      "0.497047184127526\n",
      "0.4942895542504817\n",
      "0.49211841468000833\n",
      "0.4905189283023551\n",
      "0.48947632655563883\n"
     ]
    }
   ],
   "source": [
    "# TODO: Assign sensible values\n",
    "alpha = 0.0001\n",
    "theta = np.array([3, 4, 3])\n",
    "num_iters = 100\n",
    "history_cost, history_theta = gradient_descent(X, y, theta, alpha, num_iters)\n",
    "#print(history_cost)\n",
    "#print(history_theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the training has finished we can visualize our results.\n",
    "\n",
    "**Task:**\n",
    "\n",
    "Plot the costs over the iterations. Your plot should look similar to this one:\n",
    "\n",
    "<img src=\"https://gitlab.com/deep.TEACHING/educational-materials/raw/dev/media/klaus/exercise-multivariate-linear-regression-costs.png\" width=\"512\" alt=\"internet connection needed\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def plot_progress(costs):\n",
    "    \"\"\" Plots the costs over the iterations\n",
    "    \n",
    "    Args:\n",
    "        costs: history of costs\n",
    "    \"\"\"\n",
    "    fig = plt.figure(figsize=(16,16))\n",
    "    ax1 = fig.add_subplot(2, 2, 1)\n",
    "\n",
    "    number_of_iterations = len(costs)\n",
    "    iteration = np.arange(0,number_of_iterations)\n",
    "    costs = np.array(costs)\n",
    "    curve =ax1.plot(iteration,costs)\n",
    "    ax1.title.set_text('Fortschritt')\n",
    "    ax1.set_xlabel('Iterationen', labelpad=3)\n",
    "    ax1.set_ylabel('Kosten', labelpad=3)\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "costs before the training:\t  4.925192645292619\n",
      "costs after the training:\t  0.48947632655563883\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAb4AAAHHCAYAAADAoD3gAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3yV5f3/8dcnmyQkJCFhBQgQtrJFUFAQHChuLVpHXdXWhdZW22qrrW2tX/dWah11W1TcA1FAQZG9ZW8whJmQkH39/jgHf0gZAXJyn3Pu9/PxyCOckXO/Pd7wznWf+74uc84hIiLiFzFeBxAREalPKj4REfEVFZ+IiPiKik9ERHxFxSciIr6i4hMREV9R8YlEATMbZGZrD/JnnjazP4Uqk0i4UvGJHCIzW2lmO81sx25fzQ/xdYaGIuP+OOd+5Zy7O5jhf4rTzO4ys5frO5dIqKn4RA7P6c651N2+1tf2B80sLpTBDrDtWK+2LeI1FZ9IHTOzM8xsvpltM7PxZtZ5t8dWmtltZjYHKDGz14BWwPvBEeOtZpZkZi+b2ebga0w1sybBn880s+fNbL2ZbTWzMXts+xYz22hmG8zs8t3uf8HMnjKzj8ysBBgcvO9vZpYCfAw0323k+nPgj8CI4O3ZoX/nROqHZ79xikQjM+sAvAacBYwHbiZQal2ccxXBp10InAZscs7tNLP+wFXOuc+Dr3ENkA60BMqBHsDO4M++BOwAuga/H7Pb5psGf64FcCIw2szGOOe2Bh//OXAqMBxIAC4GcM6VmNkw4GXnXO4e/y35zrmL6+K9EQkXGvGJHJ4xwVHZtuDoawTwoXNurHOuErgfaMBPC+pR59wa59zOvb0gUAlkESidaufcdOdckZk1A4YBv3LObXXOVTrnJuzxc38N3v8RgWLsuNvj7zrnJjnnapxzZXXzny8SeVR8IofnLOdco+DXWUBzYNWuB51zNcAaAqOwXdYc4DVfAj4FXg8e0vw/M4snMALcstsIbk+bnXNVu90uBVIPYrsivqDiE6lb64HWu26YmREorHW7PWfPJVF+cjs4YvuLc64LgZHicOBSAsWVaWaNDjHb/pZi2dtjWrpFopKKT6RuvQmcZmZDgqO0Wwh8Tjd5Pz9TALTddcPMBpvZkcEzL4sIHMKsds5tIHASypNmlmFm8WZ2XB3lLgCyzCx9j/vyzEz/TkhU0Q4tUoecc4sInDTyGLAJOJ3AJQ8V+/mxe4A7gp8T/pbASSqjCZTeQmACsOt6uksIFOH3wEbgpjrK/T2Bk3KWB3M0B/4bfHizmc2oi+2IhAPTQrQiIuInGvGJiIivqPhERMRXVHwiIuIrKj4REfGVsJmyrHHjxi4vL8/rGCIiEgWmT5++yTmXvbfHwqb48vLymDZtmtcxREQkCpjZqn09pkOdIiLiKyo+ERHxFRWfiIj4iopPRER8RcUnIiK+ouITERFfUfGJiIivhOw6PjNbCRQD1UCVc65PqLYlIiJSW6G+gH2wc25TiLchIiJSazrUKSIivhLK4nPAZ2Y23cyu3tsTzOxqM5tmZtMKCwtDGEVERCQglMV3rHOuFzAMuM7MjtvzCc65Uc65Ps65PtnZe51LVEREpE6FrPicc+uD3zcC7wB9Q7UtERGR2gpJ8ZlZipk13PVn4CRgXii2JSIicjBCdVZnE+AdM9u1jVedc5+EaFsiIiK1FpLic84tB7qH4rX3Z3FBMYXF5Ryb37i+Ny0iIhEiqi5n+P1bc7h19Bwqqmq8jiIiImEqqorvxiHtWbdtJ6Onr/U6ioiIhKmoKr7jO2TTo2UjnvhyqUZ9IiKyV1FVfGbGzSd20KhPRET2KaqKD+C49o3p2UqjPhER2buoKz4z46ahgVHff6ev8TqOiIiEmagrPtht1PeFRn0iIvJTUVl8ZsbNQzuwfnsZb07TqE9ERP6/qCw+gIHtG9Mr+FlfeVW113FERCRMRG3x7TrDc8P2Mt6cpjM8RUQkIGqLD2BAfmP6tM7gSY36REQkKKqLb/dR3xtT9VmfiIhEefEBHNMui6PyMnjiy6WUVWrUJyLid1FffLvO8CwoKuf171Z7HUdERDwW9cUH0L9dFn3bZPLk+GUa9YmI+Jwvim/XqG9jcTmvTtGoT0TEz3xRfBAY9fVvm8VTE5axs0KjPhERv/JN8QHcfGIHCovLeWXKKq+jiIiIR3xVfH3bZDIgvzFPjV9GaUWV13FERMQDvio+gJtPbM/mkgpe+kajPhERP/Jd8fVunclxHbJ5esIydpRr1Cci4je+Kz6Am4e2Z2tpJS9OXul1FBERqWe+LL6erTIY3DGbUROXU1xW6XUcERGpR74sPgic4bl9ZyXPT1rpdRQREalHvi2+brmNGNq5Cf/6ajnbd2rUJyLiF74tPgic4VlcVsW/v1rudRQREaknvi6+rs3TGXZEU56btJKtJRVexxERkXrg6+IDuGloB0oqqhilUZ+IiC/4vvg6Nm3I8G7NeXHySjbtKPc6joiIhJjviw9g5JD2lFVW88yEZV5HERGREFPxAfk5qZzVowUvfbuKjUVlXscREZEQUvEF3TikPZXVjifHa9QnIhLNVHxBeY1TOK9XLq9OWc36bTu9jiMiIiGi4tvNDUPycTge/3Kp11FERCREVHy7yc1I5oKjWvHm1DWs3lzqdRwREQkBFd8erhucT0yM8egXS7yOIiIiIaDi20PT9CQuPro1b89Yy/LCHV7HERGROqbi24tfD2pHYlwsD32uUZ+ISLRR8e1FdsNELjs2jw/mrOf7H4q8jiMiInVIxbcP1xzXltSEOB4au9jrKCIiUodUfPvQKDmBKwe24dP5Bcxdu93rOCIiUkdUfPtx5YA2NEqO54Gxi7yOIiIidUTFtx8Nk+L51fHtGL+okGkrt3gdR0RE6oCK7wAu7d+axqmJPPCZPusTEYkGKr4DSE6I47rB7fhm+WYmLd3kdRwRETlMKr5auLBvK5qnJ3Hfp4twznkdR0REDoOKrxaS4mO5YUh7Zq3ZxriFG72OIyIih0HFV0vn9c6ldVYy93+2iJoajfpERCKViq+W4mNjuHloB77/oZgP5m7wOo6IiBwiFd9BOL17czo0SeXhsYupqq7xOo6IiBwCFd9BiI0xbjmpI8s3lfD2jHVexxERkUOg4jtIJ3VpQvfcdB4Zt4Tyqmqv44iIyEFS8R0ks8Cob922nbz+3Rqv44iIyEFS8R2Cge0bc3SbTB77YimlFVVexxERkYOg4jsEZsatp3Rk045ynp+00us4IiJyEFR8h6h360yGdMrhmQnL2F5a6XUcERGpJRXfYbjlpI4UlVUx6qtlXkcREZFaUvEdhi7N0zi9e3Oe+3olhcXlXscREZFaUPEdppuHtqeiuoYnvlzqdRQREakFFd9hapudyvm9c3l1ymrWbi31Oo6IiByAiq8OjBzaHgwe/nyJ11FEROQAVHx1oFl6Ay7t15q3Z6xlSUGx13FERGQ/VHx15NrB+SQnxHH/Z4u8jiIiIvuh4qsjmSkJXDWwDZ/OL2DWmm1exxERkX1Q8dWhqwa2JTMlgfs+/d7rKCIisg8qvjqUmhjHdYPzmbR0M18v2eR1HBER2QsVXx276OhWtGjUgP/79Hucc17HERGRPaj46lhSfCwjh7ZnztrtfDzvB6/jiIjIHlR8IXBur1za56Ry/6eLqKqu8TqOiIjsRsUXArExxm9P7sjyTSWMnr7W6zgiIrIbFV+InNSlCT1bNeLhz5dQVlntdRwREQlS8YWImXHbKZ34oaiMFyev9DqOiIgEhbT4zCzWzGaa2Qeh3E646tc2i+M7ZPPk+GVs36nFakVEwkGoR3wjgYUh3kZY+93JHdm+s5KnJ2ixWhGRcBCy4jOzXOA04NlQbSMSHNEinTN7NOf5SSsoKCrzOo6IiO+FcsT3MHArsM/z+c3sajObZmbTCgsLQxjFW7ec2JHqGqdli0REwkBIis/MhgMbnXPT9/c859wo51wf51yf7OzsUEQJC62ykrno6Na8OW0NSzfu8DqOiIivhWrEdyxwhpmtBF4HTjCzl0O0rYhw/Qn5JMXFcP+nWrZIRMRLISk+59wfnHO5zrk84ALgC+fcxaHYVqRonJrIL49ryyfzf2DG6q1exxER8S1dx1ePrhrYlsapCdz7sSawFhHxSsiLzzk33jk3PNTbiQSpiXHcOKQ9U1ZsYfyi6D2ZR0QknGnEV88uOKoVrbOS+efH31Ndo1GfiEh9U/HVs4S4GH53ckcWFRTz9gxNYC0iUt9UfB447chmdM9N58GxizWBtYhIPVPxecDM+P2wzmzYXsYLmsBaRKReqfg80r9dFoM7ZvPkl0vZVlrhdRwREd9Q8XnotmGdKC6v4okvl3odRUTEN1R8HurUNI3zeuXy4uRVrNlS6nUcERFfUPF57DcndcAMHvhMU5mJiNQHFZ/HmqU34MoBbRgzaz3z1m33Oo6ISNRT8YWBXw1qR2ZKAv/4aKGmMhMRCTEVXxhIS4rnxhPymbxsM+MXayozEZFQUvGFiZ8f3Zq8rGT++ZGmMhMRCSUVX5hIiIvh1lM6saigmNHT13gdR0Qkaqn4wsiwI5rSs1UjHhy7mNKKKq/jiIhEJRVfGDEzbj+1MwVF5Tz71Qqv44iIRCUVX5jpk5fJsCOa8vSEZWwsLvM6johI1FHxhaHbTulERVUND41d4nUUEZGoo+ILQ3mNU7ikf2vemLqaxQXFXscREYkqKr4wdeMJ7UlJjOOejxZ6HUVEJKqo+MJURkoCN5yQz5eLCvl6ySav44iIRA0VXxi7tH8euRkN+PtHC3VRu4hIHVHxhbGk+FhuO6UTCzcU8db0tV7HERGJCiq+MDe8WzN6tmrE/Z8toqRcF7WLiBwuFV+YMzPuOK0LG4vLeWbicq/jiIhEPBVfBOjdOoPh3ZoxauIyNmzf6XUcEZGIpuKLELed0omaGrj/08VeRxERiWgqvgjRMjOZywfk8daMtcxdq5XaRUQOlYovglw3OJ/MlATu/nCBVmoXETlEKr4IkpYUz29O7MB3K7bwybwfvI4jIhKRVHwR5oKjWtKhSSr3fPw95VXVXscREYk4Kr4IExcbwx2ndWH1llJemLTS6zgiIhFHxReBjuuQzeCO2Tz+xVI27Sj3Oo6ISERR8UWo20/rQmllNQ+N1eUNIiIHQ8UXofJzUrmkX2te+241CzcUeR1HRCRiqPgi2E1D25PWIJ67P9DlDSIitaXii2CNkhO4eWgHJi/bzGcLCryOIyISEVR8Ee6io1vRPieVf3y0UJc3iIjUgoovwsXFxvCn4V1YtbmU53V5g4jIAan4osBxHbIZ2jmHx79YSmGxLm8QEdkfFV+UuP20LpRXVXP/p4u8jiIiEtZUfFGiTeMULjsmjzenr9HqDSIi+6HiiyI3DGlPVkoCd70/X5c3iIjsg4oviqQlxXPryZ2Yvmor785a73UcEZGwpOKLMuf1zqVbbjr3fLyQkvIqr+OIiIQdFV+UiYkx7jy9KwVF5Tw5fqnXcUREwo6KLwr1bp3B2T1b8K+vVrB6c6nXcUREwoqKL0r9flgn4mKMuz9c4HUUEZGwouKLUk3Skrj+hHzGLihgwuJCr+OIiIQNFV8Uu3JAG9o0TuEv782noqrG6zgiImFBxRfFEuNi+fPwLizfVMLzk1Z4HUdEJCyo+KLc4E45DOmUw6PjllBQVOZ1HBERz6n4fOBPw7tQWe3458ffex1FRMRzKj4fyGucwi+Pa8M7M9cxdeUWr+OIiHhKxecT1w3Op3l6En9+dz5V1TrRRUT8S8XnE8kJcdwxvAsLNxTxypTVXscREfGMis9Hhh3RlAH5jbn/s0Vs2qEFa0XEn1R8PmJm3HVGV8oqq7lXJ7qIiE+p+HwmPyeVKwa04b/T1zJ91Vav44iI1DsVnw/deEJ7mqYl8ed351FdowVrRcRfVHw+lJIYxx3DOzN/fRGvTFnldRwRkXql4vOp045sxoD8xtz36SIKi3Wii4j4h4rPp8yMv57ZlfLKGu75aKHXcURE6o2Kz8faZqdy9XFteXvmOr5dvtnrOCIi9ULF53PXDc4nN6MBfxozj0rN6CIiPqDi87kGCbH85YyuLNm4g+e+1tJFIhL9VHzCkM5NGNq5CY+MW8K6bTu9jiMiElIqPgHgrjO64Bz85b35XkcREQkpFZ8AkJuRzMih7flsQQFjFxR4HUdEJGRUfPKjKwe0oUOTVO56bz6lFVVexxERCQkVn/woPjaGv599JOu27eSRcUu8jiMiEhIqPvmJo/Iy+VmfXP791QoW/VDsdRwRkTqn4pP/8fthnWmYFMft78ylRpNYi0iUCUnxmVmSmX1nZrPNbL6Z/SUU25HQyExJ4I+ndmbaqq28MW2N13FEROpUqEZ85cAJzrnuQA/gFDPrF6JtSQic1zuXfm0zueejhWwsLvM6johInQlJ8bmAHcGb8cEvHTOLIGbG388+krLKGv72gSaxFpHoEbLP+Mws1sxmARuBsc65KXt5ztVmNs3MphUWFoYqihyidtmpXDu4He/NXs+Exfr/IyLRIWTF55yrds71AHKBvmZ2xF6eM8o518c51yc7OztUUeQw/HpQO9pmp3DHmLnsrKj2Oo6IyGEL+VmdzrltwHjglFBvS+peYlws/zj7SNZs0bV9IhIdQnVWZ7aZNQr+uQEwFPg+FNuS0OvXNosRfVryr6+WM3/9dq/jiIgcllCN+JoBX5rZHGAqgc/4PgjRtqQe/OHUTmQkx/OHt+dSrWv7RCSCheqszjnOuZ7OuW7OuSOcc38NxXak/jRKTuDO07syZ+12Xpi80us4IiKHTDO3SK0N79aMEzrl8MBni1i7tdTrOCIih0TFJ7VmZtx91hEYcMeYeTinQ54iEnlUfHJQWjRqwG9P7sj4RYW8N3u913FERA6aik8O2qX98+jZqhF3vTefzTvKvY4jInJQal18ZtbCzI4xs+N2fYUymISv2Bjj3nO7saO8ir9+sMDrOCIiByWuNk8ys3uBEcACYNf0HQ6YGKJcEuY6NGnI9YPb89Dnizmje3OGdG7idSQRkVqpVfEBZwEdnXM6riU/+vWgdnw0dwN3jJlH3zaZNEyK9zqSiMgB1fZQ53ICKyyI/CghLoZ7z+tGQVEZ//xYE/OISGSo7YivFJhlZuMIrLUHgHPuxpCkkojRo2Ujrji2Dc9+vYLh3ZrTv12W15FERPartiO+94C7gcnA9N2+RLjlpI60zkrmtrfmUFpR5XUcEZH9qlXxOedeBN4EvnXOvbjrK7TRJFI0SIjl3nO7sXpLKQ98ttjrOCIi+1Wr4jOz04FZwCfB2z3M7L1QBpPI0q9tFpf0a81zk1YwfdVWr+OIiOxTbQ913gX0BbYBOOdmAW1ClEki1G3DOtE8vQG3jp5NWaUWrRWR8FTb4qtyzu25EJsmapSfSE2M455zjmRZYQkPf65Fa0UkPNW2+OaZ2c+BWDNrb2aPETjRReQnjuuQzYg+LRk1cRmz1mzzOo6IyP+obfHdAHQlcCnDq8B2YGSoQklku314Z5qkJXHLm7N0yFNEwk5ti+8059ztzrmjgl93AGeEMphErrSkeP55bjeWFZbw0Oc6y1NEwktti+8PtbxPBIDjO2RzwVEt+dfE5cxYrbM8RSR87HfmFjMbBpwKtDCzR3d7KA3QlcqyX7ef1pmJiwv53X9n8+GNA0mKj/U6kojIAUd864FpQBk/nbHlPeDk0EaTSNcwKZ57zwsc8nzgs0VexxERAQ4w4nPOzQZmm9mrzrlKADPLAFo653T8Sg5oYPtsLjq6Fc9+vYITuzSlb5tMryOJiM/V9jO+sWaWZmaZwGzgeTN7MIS5JIr88dTO5GY04Lf/nU1JuY6Qi4i3alt86c65IuAc4HnnXG9gaOhiSTRJSYzj/vO6s2ZrKfd8vNDrOCLic7Utvjgzawb8DPgghHkkSh3dNosrj23Dy9+uZuLiQq/jiIiP1bb4/gp8Cixzzk01s7aA5qSSg/LbkzuSn5PKraPnsH1npddxRMSnarss0X+dc92cc78O3l7unDs3tNEk2iTFx/Lgz7pTuKOcO9+d53UcEfGp2i5LlGtm75jZRjMrMLO3zCw31OEk+nTLbcQNJ+QzZtZ6Ppyzwes4IuJDtT3U+TyBa/eaAy2A94P3iRy06wbn0z03ndvHzKWgqMzrOCLiM7Utvmzn3PPOuarg1wtAdghzSRSLj43hwRE9KKus5nej5+CcVrgSkfpT2+LbZGYXm1ls8OtiYHMog0l0a5edyh9PDUxp9vKU1V7HEREfqW3xXUHgUoYfgA3AecDloQol/nBJv9Yc1yGbv3+4gGWFO7yOIyI+sd/i23UCi3NutXPuDOdctnMuxzl3FtC9XhJK1DIz7juvG0nxsdz0+iwqq2u8jiQiPnCgEd84M8vb804zuxx4OBSBxF+apCXxz3OOZO667TystftEpB4cqPhuJjBPZ/tdd5jZH4DfAMeHMpj4xylHNONnfXJ5cvwyvluxxes4IhLl9lt8zrmPgF8BH5vZEWb2MDAcOM45t7Y+Aoo/3Hl6V1plJnPzG7MoKtOsLiISOgc8ucU5Nw64DBgPtAWGaEkiqWspiXE8NKIHPxSV8ecxmtVFRELnQCe3FJtZEfAxgVXXhwAbd7tfpM70apXBjSe0Z8ys9YyZuc7rOCISpQ50qLOhcy4t+D3BOZey2+20+gop/nHd4HYclZfBHWPmsXpzqddxRCQK1fY6PpF6ERcbw0MjemAGI9+YSZUucRCROqbik7CTm5HM388+kpmrt/HoOK1+JSJ1S8UnYemM7s05t1cuj3+5lCnLNTueiNQdFZ+Erb+cGbjE4aY3ZrG1pMLrOCISJVR8ErZSE+N47MJebNpRzq1vaRUHEakbKj4Ja0fmpnPbKZ0Yu6CAl75d5XUcEYkCKj4Je1cOaMPgjtn87cOFLFivy0dF5PCo+CTsmRn3n9+dRg3iuf61GZRWVHkdSUQimIpPIkJWaiIPX9CDFZtK+NOY+V7HEZEIpuKTiHFMu8bccEJ73pqxltHTNUe6iBwaFZ9ElJFD2tOvbSZ/GjOPJQXFXscRkQik4pOIEhtjPHJBT5ITYrn+1ZnsrKj2OpKIRBgVn0ScJmlJPDSiB4s3FvOX9/V5n4gcHBWfRKTjOmRz7aB2vD51DW/P0Od9IlJ7Kj6JWDcP7cDRbTK5/Z15LNbnfSJSSyo+iVhxsTE8dmFPUhLjuPaVGZSU6/o+ETkwFZ9EtJy0JB69sAfLC3dw+ztzNZ+niByQik8i3jHtGnPz0A6MmbWe175b43UcEQlzKj6JCtcNzuf4Dtnc9d585qzd5nUcEQljKj6JCjExxsMjepDdMJFfvzyDbaVav09E9k7FJ1EjIyWBJy/qRWFxOTe9MYuaGn3eJyL/S8UnUaV7y0bceUYXxi8q5LEvlnodR0TCkIpPos7P+7binF4teHjcYsYv2uh1HBEJMyo+iTpmxt/POpKOTRoy8vVZrNlS6nUkEQkjKj6JSg0SYnnmkt4457jmpemazFpEfqTik6jVOiuFRy7sycIfinRxu4j8SMUnUW1wxxxuHtqBt2eu48XJK72OIyJhQMUnUe/6wfkM7ZzD3z5cyHcrtngdR0Q8puKTqBcTYzw4ogetMpO59pXprN+20+tIIuIhFZ/4QlpSPKMu7U1ZZQ3XvDSdskqd7CLiVyo+8Y38nIY8NKIHc9dt549v62QXEb9S8YmvnNilyY8nuzw3aaXXcUTEAyo+8Z0bTsjnpC5N+MdHC/l6ySav44hIPVPxie/sOtmlXXYK1706gxWbSryOJCL1KCTFZ2YtzexLM1toZvPNbGQotiNyqFIT43j20qOIMfjlf6ZRVFbpdSQRqSehGvFVAbc45zoD/YDrzKxLiLYlckhaZSXz5EW9WbmphJGvzaRayxiJ+EJIis85t8E5NyP452JgIdAiFNsSORz922Vx5xld+XJRIf/3yfdexxGRehAX6g2YWR7QE5iyl8euBq4GaNWqVaijiOzVJf1as+iHIp6ZuJz8nFTO79PS60giEkIhPbnFzFKBt4CbnHNFez7unBvlnOvjnOuTnZ0dyigi+3Xn6V05Nj+LP74zl6krNa2ZSDQLWfGZWTyB0nvFOfd2qLYjUhfiY2N48ue9aZmRzDUvTWf1Zq3hJxKtQnVWpwH/BhY65x4MxTZE6lp6cjz/vuwoqmscV744lWKd6SkSlUI14jsWuAQ4wcxmBb9ODdG2ROpMm8YpPHVRL1ZsKuH6V2dSVV3jdSQRqWOhOqvza+ecOee6Oed6BL8+CsW2ROraMfmNufusI5iwuJC73p+vOT1FokzIz+oUiUQX9m3Fyk0lPDNxOXlZKVw1sK3XkUSkjqj4RPbhtlM6sWpzKX//aCGtMpM5qWtTryOJSB3QXJ0i+xATYzw0ogfdWqQz8vVZzF273etIIlIHVHwi+9EgIZZ//aIPmSkJXPHiVNZs0WUOIpFOxSdyADkNk3jh8qMoq6zm8hemsr1UlzmIRDIVn0gttG/SkFGX9GHV5hKueXka5VXVXkcSkUOk4hOppf7tsrj//O58u3wLt46eQ41WcxCJSDqrU+QgnNmjBWu37uS+TxfRLL0Bvx/WyetIInKQVHwiB+naQe1Yv20nT09YRtO0RC47to3XkUTkIKj4RA6SmfHXM4+gsLicv3ywgJy0JE49spnXsUSklvQZn8ghiI0xHr2wJ71aZXDTG7OYsnyz15FEpJZUfCKHKCk+lmcv7UPLjAZc9Z9pLNzwP0tOikgYUvGJHIaMlARevKIvKQlx/OK573SBu0gEUPGJHKbcjGRevKIvZZXVXPrcd2zaUe51JBHZDxWfSB3o2LQhz112FBu27+Ty56eyo7zK60gisg8qPpE60icvkycv6sWCDUVc/Z9plFVqdheRcKTiE6lDJ3Rqwn3ndWPyss3c+JpWcBcJRyo+kTp2Tq9c7jq9C58tKOC2t+ZqajORMKML2EVC4LJj27B9ZxUPfb6YtAZx/Hl4F8zM61gigopPJGRuHJLP9p2VPDdpBQ2T4vnNiR28jiQiqPhEQsbMuOO0zuwor+TRcUtISYjlmlgZGs8AABTnSURBVOPbeR1LxPdUfCIhFBNj3HNON0orqrnn4+9JTojlkv55XscS8TUVn0iIxcYYD43oQVllNX96dz4NEuI4r3eu17FEfEtndYrUg/jYGB7/eS8Gtm/MraNn8/7s9V5HEvEtFZ9IPUmKj+WZS3rTJy+Tm96YxSfzNngdScSXVHwi9Sg5IY7nLjuK7rnpXP/qTD5fUOB1JBHfUfGJ1LPUxDheuKIvXZunce0rM/hy0UavI4n4iopPxANpSfH854qjad8klWtems6ExYVeRxLxDRWfiEfSk+N5+cqjyc9O5Zf/mabyE6knKj4RD2WkJPDKVSo/kfqk4hPxmMpPpH6p+ETCwE/K78VpfPG9zvYUCRUVn0iYyEhJ4NVfHk3Hpg255qXpfDb/B68jiUQlFZ9IGGmUnMDLVx1N1+bpXPvKDD6aq4vcReqaik8kzKQ3iOelK/vSo2UjbnhtJu/OWud1JJGoouITCUMNk+J58Yq+9A1Ob/bad6u9jiQSNVR8ImEqJTGO5y8/ikEdsvnD23P599crvI4kEhVUfCJhLDCxdR+GHdGUuz9YwKPjluCc8zqWSERT8YmEuYS4GB67sCfn9GzBg2MX84+PFqr8RA6DFqIViQBxsTHcf353GibF8a+vVrB9ZyX/OPtI4mL1u6vIwVLxiUSImBjjrjO6kp6cwKPjllC0s4pHLuxBYlys19FEIop+XRSJIGbGb07swJ+Hd+GT+T9wxQtT2VFe5XUskYii4hOJQFcMaMODP+vOt8u3cMGobygsLvc6kkjEUPGJRKhzeuXy7KV9WLaxhPOensyqzSVeRxKJCCo+kQg2uFMOr/zyaLbvrOTcpyYzb912ryOJhD0Vn0iE69Uqg9G/OobEuFhGPPONljUSOQAVn0gUyM9J5e1rj6FVVgpXvDCVN6et8TqSSNhS8YlEiSZpSbx5TT+OaZfFraPn8PDni3Whu8heqPhEokjDpHieu+wozuudy8OfL+F3o+dQUVXjdSyRsKIL2EWiTHxsDPed143cjAY8/PkS1m3dydMX9yY9Od7raCJhQSM+kShkZtw0tAMPjejO9FVbOfupSbrcQSRIxScSxc7umctLV/ZlS0kFZz85makrt3gdScRzKj6RKHd02yzeufZY0hvE8/N/fct/dcan+JyKT8QH2jROYcy1x9K3TSa/Gz2Hf3y0kOoanfEp/qTiE/GJ9OR4Xri8L5f0a82oicu5+j/TKC6r9DqWSL1T8Yn4SHxsDHefdQR3n9mV8YsLOeuJSSwv3OF1LJF6peIT8aFL+ufx8pVHs7W0kjOfmMSXizZ6HUmk3qj4RHyqf7ss3rv+WFpmJHPFC1N5cvxSzfQivqDiE/Gx3Ixk3vr1MQzv1pz/+2QRv3p5uj73k6in4hPxuQYJsTx6QQ/uOK0zny/cyJlPTGLpxmKvY4mEjIpPRDAzrhrYlleuOpqinZWc+fgkPpyzwetYIiGh4hORH/Vrm8X7NwygQ9OGXPfqDO56b74muZaoo+ITkZ9olt6AN67uzxXHtuGFySs5/5lvWLu11OtYInVGxSci/yMhLoY/n96Fpy7qxfKNOxj+2NeMW1jgdSyROqHiE5F9GnZkM96/YQAtGjXgyhen8df3F+jQp0Q8FZ+I7Fde4xTevvYYLjsmj+cmreC8pydriSOJaCo+ETmgxLhY7jqjK09f3JuVm0o47dGvGTNzndexRA6Jik9Eau2UI5ry0ciBdGrakJvemMVNr8+kSBe8S4RR8YnIQcnNSOb1q/tx89AOvD9nA6c+8hXTV2mBW4kcKj4ROWhxsTGMHNqeN6/pjxmc//Q33P/pIp34IhFBxScih6x36ww+unEg5/bK5fEvl3L2k5NYUqDpziS8qfhE5LA0TIrnvvO7M+qS3vywvYzTHvuaZ79arhXeJWyFpPjM7Dkz22hm80Lx+iISfk7q2pRPbz6O4ztk87cPFzLimW+0yK2EpVCN+F4ATgnRa4tImGqcmsioS3rz0IjuLC4oZtgjX2n0J2EnJMXnnJsI6DQvER8yM87umcvY3xzPgPzG/O3DhfzsmW+01JGEDU8/4zOzq81smplNKyws9DKKiNSxJmlJPPuLPjz4s+4sK9zBqY98zWPjllBZrTM/xVueFp9zbpRzro9zrk92draXUUQkBMyMc3rlMvbm4zmpaxMeGLuY0x/7mpmrt3odTXxMZ3WKSMhlN0zk8Z/34l+X9mFbaSXnPDWZP42Zp1lfxBMqPhGpNyd2acLntxzPZcfk8cqUVQx9YAIfzFmPczr5RepPqC5neA34BuhoZmvN7MpQbEdEIk9qYhx3nt6Vd68bQE5aIte/OpNLn/uOZbr0QeqJhctvWn369HHTpk3zOoaI1KPqGscrU1Zx36eLKKus5qqBbbnhhHySE+K8jiYRzsymO+f67O0xHeoUEc/ExhiX9s/ji1sGcUb3Fjw1fhlDHpjAu7PW6fCnhIyKT0Q8l90wkQd+1p3Rv+pPVmoCI1+fxflPf8Pctdu9jiZRSMUnImGjT14m7143gH+ecyQrNpVwxhNf87v/zqagqMzraBJFVHwiElZiY4wL+rbiy98N4qoBbRgzax2D7hvPQ2MXU1pR5XU8iQIqPhEJS2lJ8dx+WhfG/WYQJ3TK4ZFxSxh033he+241VZr9RQ6Dik9EwlqrrGSeuKgXb/26P7kZDfjD23M56eGJfDJvg06AkUOi4hORiNC7dSZv/foYnrmkNzFm/OrlGZz95GS+XrJJBSgHRcUnIhHDzDi5a1M+GTmQe889ko1FZVz87ylcMOpbpq7UgjBSO7qAXUQiVlllNa9/t5rHv1zGph3lHNchm5FD2tO7dYbX0cRj+7uAXcUnIhFvZ0U1//lmJc9MXM6WkgoGtm/MyCHt6ZOX6XU08YiKT0R8obSiipe/XcWoicvZtKOC/m2zuHZwOwbkN8bMvI4n9UjFJyK+srOimlemrOJfXy2noKicbrnpXDuoHSd1aUpMjArQD1R8IuJL5VXVvDNjHU9NWMaqzaW0aZzCVQPbcG6vXJLiY72OJyGk4hMRX6uqruGT+T8wauJy5qzdTlZKApf0b83F/VrTODXR63gSAio+ERHAOceUFVsYNXE5X3y/kYTYGM7o0ZzLj82ja/N0r+NJHdpf8WnRKxHxDTOjX9ss+rXNYunGHbw4eSWjp69l9PS19G2TyaX9W3Ny16bEx+oS52imEZ+I+Nr20kpen7qal75dxdqtO8lumMiFfVtxYd+WNEtv4HU8OUQ61CkicgDVNY4Jizfy0jerGL+4EAMGd8zhwr6tGNQxmziNAiOKik9E5CCs3lzKG9NW8+a0tRQWl9M0LYlzerXgvN65tM1O9Tqe1IKKT0TkEFRW1/DF9xt5/bvVTFhcSI2D3q0zOK93Lqce2Yz0BvFeR5R9UPGJiBymjUVlvD1zHf+dtoZlhSUkxMUwpFMOZ/VswaCO2STG6brAcKLiExGpI845Zq/dzpiZ63h/9no2l1SQlhTHyV2bMrx7c45pl6WzQsOAik9EJASqqmv4aukm3p+9nrHzCyguryIjOZ5TjmjKyV2bcky7xiTEqQS9oOITEQmxsspqJi4u5IM5Gxi3sICSimoaJsUxtHMTTurShIEdsklN1KXT9UUXsIuIhFhSfCwndW3KSV2bUlZZzaSlm/hk3g+MXVjAOzPXkRAbQ792WQztnMPgjjm0zEz2OrJvacQnIhJCVdU1TF+1lc8XFvD5wo2s2FQCQLvsFAZ1zGFQx2yOysvUpNl1TIc6RUTCxLLCHYxfVMj4RRuZsmILFVU1JMbFcFReJgPaN+bYdo3p0jyNWC2fdFhUfCIiYai0ooopy7fw1ZJNfL20kMUFOwBIS4qjb5ss+rfL4ug2mXRq2lAzxxwkfcYnIhKGkhPiGNwph8GdcgAoKCrjm2Wb+Xb5Zr5ZvpnPFxYAkJoYR89WjejTOpNerRvRvWUj0pJ08fyh0ohPRCRMrd+2k6krtzBt5VamrtzCooJinAMzaJ+TSo+WjeiW24huuel0bNpQF9HvRoc6RUSiQFFZJXPWbGfm6q3MXLONmau3srW0EoD4WKNDk4Z0aZZGl+ZpdGmWRqemaaQn+3NkqOITEYlCzjnWbdvJ3LXbmb12O/PXb2fB+iI2l1T8+JymaUl0aNqQDjmp5Oek0i4nlfzsVDJSEjxMHnr6jE9EJAqZGbkZyeRmJDPsyGZAoAwLi8uZv6GIxT8Us6igmMUFxbz07WbKq2p+/NmM5HjyGqeQl5VC66xkWmcl0zL4WjkNE4mJ4rNKVXwiIlHEzMhJSyInLYnBHXN+vL+6xrF+206WFu5g2cYdLN9UwqrNJXy3YgtjZq1j94N/CbExNGuURLP0JJqnN6BZoySapCWR0zAx8NoNE2mcmhix1x6q+EREfCA2xmiZmUzLzOSfFCIEpltbt20na7aUsnbrTtZsLWX9tjI2bNvJlBVb+KGojOqa//1YLCUhlszUBDJTEslIjqdRg3gaJSeQ3iCehklxpCUFvqckxpGSGEtyQhzJCbEkxceSFBdLYnwMiXExmNXv6FLFJyLic0nxsbTLTqXdPhbZra5xbC4pZ2NROYXF5WwsLmNzSQWbd1SweUc5m0sq2FJSwbLCHWwrqaS4vOqgth8fa8THxhAXYwzulMMjF/Ssi/+sfVLxiYjIfsXGGDkNk8hpmFSr51fXOHaUV1FcVklxWRWlFVWUlFf/+L2sqpqyyhrKKqspr6qhqrqGyuoaKqsd+TmhX+FexSciInUqNsZIbxAftivUaw4cERHxFRWfiIj4iopPRER8RcUnIiK+ouITERFfUfGJiIivqPhERMRXVHwiIuIrKj4REfEVFZ+IiPiKik9ERHxFxSciIr6i4hMREV9R8YmIiK+o+ERExFdUfCIi4ivmnPM6AwBmVgisqoOXagxsqoPX8QO9VwdH71ft6b2qPb1XtXcw71Vr51z23h4Im+KrK2Y2zTnXx+sckUDv1cHR+1V7eq9qT+9V7dXVe6VDnSIi4isqPhER8ZVoLL5RXgeIIHqvDo7er9rTe1V7eq9qr07eq6j7jE9ERGR/onHEJyIisk8qPhER8ZWoKj4zO8XMFpnZUjP7vdd5womZtTSzL81soZnNN7ORwfszzWysmS0Jfs/wOmu4MLNYM5tpZh8Eb7cxsynB9+oNM0vwOmM4MLNGZjbazL4P7l/9tV/tm5ndHPw7OM/MXjOzJO1bAWb2nJltNLN5u923133JAh4N/ns/x8x61XY7UVN8ZhYLPAEMA7oAF5pZF29ThZUq4BbnXGegH3Bd8P35PTDOOdceGBe8LQEjgYW73b4XeCj4Xm0FrvQkVfh5BPjEOdcJ6E7gPdN+tRdm1gK4EejjnDsCiAUuQPvWLi8Ap+xx3772pWFA++DX1cBTtd1I1BQf0BdY6pxb7pyrAF4HzvQ4U9hwzm1wzs0I/rmYwD9OLQi8Ry8Gn/YicJY3CcOLmeUCpwHPBm8bcAIwOvgUvVeAmaUBxwH/BnDOVTjntqH9an/igAZmFgckAxvQvgWAc24isGWPu/e1L50J/McFfAs0MrNmtdlONBVfC2DNbrfXBu+TPZhZHtATmAI0cc5tgEA5AjneJQsrDwO3AjXB21nANudcVfC29q+AtkAh8HzwsPCzZpaC9qu9cs6tA+4HVhMovO3AdLRv7c++9qVD/jc/morP9nKfrtXYg5mlAm8BNznnirzOE47MbDiw0Tk3ffe79/JU7V+B0Usv4CnnXE+gBB3W3Kfg51NnAm2A5kAKgUN2e9K+dWCH/HcymopvLdByt9u5wHqPsoQlM4snUHqvOOfeDt5dsOvwQPD7Rq/yhZFjgTPMbCWBQ+YnEBgBNgoengLtX7usBdY656YEb48mUITar/ZuKLDCOVfonKsE3gaOQfvW/uxrXzrkf/OjqfimAu2DZ0clEPjA+D2PM4WN4GdU/wYWOuce3O2h94BfBP/8C+Dd+s4Wbpxzf3DO5Trn8gjsR1845y4CvgTOCz5N7xXgnPsBWGNmHYN3DQEWoP1qX1YD/cwsOfh3ctf7pX1r3/a1L70HXBo8u7MfsH3XIdEDiaqZW8zsVAK/mccCzznn/u5xpLBhZgOAr4C5/P/Prf5I4HO+N4FWBP5Snu+c2/PDZd8ys0HAb51zw82sLYERYCYwE7jYOVfuZb5wYGY9CJwElAAsBy4n8Eu19qu9MLO/ACMInGk9E7iKwGdTvt+3zOw1YBCB5YcKgDuBMexlXwr+4vA4gbNAS4HLnXPTarWdaCo+ERGRA4mmQ50iIiIHpOITERFfUfGJiIivqPhERMRXVHwidcDMdgS/55nZz+v4tf+4x+3Jdfn6In6j4hOpW3nAQRVfcIL1/flJ8TnnjjnITCKyGxWfSN36JzDQzGYFl5+JNbP7zGxqcOmUayBwfWBwmahXCVxbiZmNMbPpwSVrrg7e908CExrPMrNXgvftGl1a8LXnmdlcMxux22uP322poFeC1zxhZr3NbEJwO5/uNiPGeDO718y+M7PFZjawnt83kXoTd+CniMhB+D3BC94BggW23Tl3lJklApPM7LPgc/sCRzjnVgRvXxG8MLcBMNXM3nLO/d7MrnfO9djLts4BehBYCqhx8GcmBh/rCXQlMIXTJOBYM5sCPAac6ZwrDBbl34Ergj8T55zrG5wI4k4C02uJRB0Vn0honQR0M7Nd01GlE1g/rAL4brfSA7jRzM4O/rll8Hmb9/PaA4DXnHPVBOYznAAcBRQFX3stgJnNInAIdhtwBDA2OACMJbBCwC675m+dHny+SFRS8YmElgE3OOc+/cmdganQSva4PRTo75wrNbPxQFItXntfdp/uqprA33UD5jvn+h/gZ3Y9XyQq6TM+kbpVDDTc7fanwK+DK2NgZh2C69XtKR3YGiy9TkC/3R6r3PXze5gIjAh+jphNYEHY7/aTbRGQbWb9g1nizaxrrf/LRKKEik+kbs0BqsxstpndTGDy5gXADDObBzzD3kdTnwBxZjYHuBv4drfHRgFzdp3cspt3gtubDXwB3BpcLWGvnHMVBFYAuNfMZgOzCCyJI+IrmqRaRER8RSM+ERHxFRWfiIj4iopPRER8RcUnIiK+ouITERFfUfGJiIivqPhERMRX/h/IMs9a1sAD/AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1152x1152 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_progress(history_cost)\n",
    "print(\"costs before the training:\\t \", history_cost[0])\n",
    "print(\"costs after the training:\\t \", history_cost[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot Data and Decision Boundary \n",
    "\n",
    "**Task:**\n",
    "\n",
    "Now plot the deicision boundary (a straight line in this case) together with the data.         "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZyNdeP/8ddnLGmRZCnSXbTdc2bFWIbsQkSWElKI5EaFsqVEIluWshQRSSQSkj1blmGGWc6cqchPCUkpSkny+f4xx/2bWzMZzJnrzDnv5+NxHme5rjnX2+cc77nmOtd8xlhrERGRwBTidAAREfEdlbyISABTyYuIBDCVvIhIAFPJi4gEsPxOB8ioePHi9tZbb3U6hohInpKQkPCDtbZEZsv8quRvvfVW4uPjnY4hIpKnGGO+zmqZDteIiAQwlbyISABTyYuIBDCVvIhIAFPJi4gEMJW8iEgAU8mLiASwgCj5o0eP0rt3b44fP+50FBERvxIQJb9u3Tpee+01XC4Xy5YtczqOiIjfCIiSb9OmDdu3b6dYsWI0a9aMdu3acfToUadjiYg4LiBKHqBSpUrEx8czdOhQFi5cSGhoKO+99x76y1ciEswCpuQBChYsyODBg9m9eze33347Dz/8ME2bNuXAgQNORxMRcURAlfw5YWFhbNmyhXHjxvHpp58SFhbGm2++ydmzZ52OJiKSqwKy5AHy5ctH7969cbvdVKpUiW7dulG3bl327NnjdDQRkVwTsCV/Trly5Vi7di1vvfUWiYmJREZGMnbsWM6cOeN0NBERnwv4kgcwxtC5c2c8Hg8NGzakb9++xMbGkpyc7HQ0ERGfCoqSP6d06dIsXryY999/n6+//pqKFSsyePBg/vjjD6ejiYj4RFCVPKTv1bdu3Zq0tDTatGnDsGHDqFChAtu3b3c6mohIjgu6kj+nWLFizJkzh+XLl3PixAmqVatG7969OXnypNPRRERyTNCW/DmNGzcmNTWVbt26MWHCBCIiIli3bp3TsUREckTQlzzAtddey5QpU9i4cSP58+enfv36dOnShZ9//tnpaCIil0Uln0HNmjVJSkqif//+zJo1C5fLxUcffeR0LBGRS6aSP8+VV17JyJEjiYuLo2TJkrRo0YLWrVtz5MgRp6OJiFw0lXwWKlasyM6dO3n55ZdZsmQJLpeLOXPmaMIzEclTVPL/oECBAgwaNIjExETuuusuHn30UZo0acI333zjdDQRkWxRyWdDaGgomzdvZuLEiWzcuJGwsDCmTJmiCc9ExO+p5LMpX758PPXUU7jdbmJjY+nRowe1a9fmyy+/dDqaiEiWVPIXqWzZsqxatYq3336blJQUIiMjGTVqlCY8ExG/pJK/BMYYOnbsiMfjoXHjxgwYMIAqVaqQmJjodDQRkf+hkr8MpUqV4sMPP2ThwoUcPHiQmJgYBg0axKlTp5yOJiICqORzRKtWrfB4PLRv354RI0ZQvnx5tm7d6nQsERGVfE65/vrrmTVrFitXruS3337j7rvv5qmnnuLXX391OpqIBDGVfA5r2LAhbrebHj16MGnSJMLDw1m9erXTsUQkSOVIyRtjZhpjvjfGuDM8dr0xZo0xZo/3umhObCsvKFy4MK+//jqbNm2iUKFCNGzYkE6dOnHs2DGno4lIkMmpPflZQKPzHhsArLPW3gGs894PKnfffTeJiYkMHDiQOXPm4HK5WLRokdOxRCSI5EjJW2s3Aefvpt4PzPbeng00z4lt5TWFChVixIgR7Ny5k1KlSvHAAw/wwAMP8N133zkdTUSCgC+Pyd9grT0M4L0umdlKxpiuxph4Y0z80aNHfRjHWeXLl2fHjh2MGDGCjz/+GJfLxaxZszThmYj4lOMfvFprp1lrY6y1MSVKlHA6jk8VKFCAgQMHkpiYiMvlolOnTjRq1Ij9+/c7HU1EApQvS/6IMaYUgPf6ex9uK0/597//zaZNm5g0aRJbt24lPDyc119/XROeiUiO82XJLwU6eG93AJb4cFt5TkhICD169MDtdv/3nPqaNWvy+eefOx1NRAJITp1COQ/YBtxljPnWGNMZGAncY4zZA9zjvS/nueWWW1ixYgWzZ8/G4/EQFRXFiBEj+PPPP52OJiIBwPjTB38xMTE2Pj7e6RiOOXLkCD179mThwoVER0czY8YMKlSo4HQsEfFzxpgEa21MZssc/+BV/r8bbriBDz74gA8//JDvvvuOypUrM3DgQH7//Xeno4lIHqWS90MtWrTA4/HQoUMHRo4cSXR0NJ999pnTsUQkD1LJ+6miRYsyY8YM1qxZw+nTp6lRowY9e/bkl19+cTqaiOQhKnk/V79+fVJSUnj66aeZMmUKYWFhrFixwulYIpJHqOTzgGuuuYYJEyawZcsWrrnmGho3bsyjjz7Kjz/+6HQ0EfFzKvk8JDY2lt27d/P8888zb948XC4XH3zwgaZGEJEsqeTzmCuuuIJhw4YRHx/PzTffTOvWrWnZsiWHDx92OpqI+CGVfB4VFRXF9u3bGT16NCtXriQ0NJSZM2dqr15E/odKPg/Lnz8/ffv2JSkpiaioKDp37kyDBg3Yt2+f09FExE+o5APAnXfeyfr165k6dSpxcXFEREQwYcIE/vrrL6ejiYjDVPIBIiQkhG7dupGamkqtWrXo3bs3d999Nx6Px+loIuIglXyAufnmm1m+fDnvvvsue/bsoXz58gwbNozTp087HU1EHKCSD0DGGB5++GE8Hg8tW7Zk8ODBVKpUiWCe/E0kWKnkA1jJkiWZN28eS5Ys4YcffqBKlSr069dPE56JBBGVfBBo1qwZqampdO7cmTFjxhAZGcnGjRudjiUiuUAlHySuu+46pk2bxrp16zh79iy1a9fmP//5DydOnHA6moj4kEo+yNStW5fk5GT69OnDtGnTCAsLY/ny5U7HEhEfUckHoauvvppXX32VrVu3UqRIEe677z7at2/PDz/84HQ0EclhKvkgVqVKFXbt2sWLL77IggULCA0NZf78+ZoaQSSAqOSDXMGCBRkyZAgJCQmULVuWtm3b0rx5cw4ePOh0NBHJASp5ASAiIoJt27YxduxY1qxZg8vlYvr06dqrF8njVPLyX/ny5eOZZ54hOTmZChUq0LVrV+rVq8dXX33ldDQRuUQqefmb22+/nXXr1vHmm2+SkJBAREQE48aN04RnInmQSl4yFRISQteuXUlNTaVevXo888wzVKtWDbfb7XQ0EbkIKnn5R2XKlGHp0qXMmzePffv2UaFCBYYOHaoJz0TyCJW8XJAxhjZt2pCWlsaDDz7IkCFDqFixIjt27HA6mohcgEpesq148eLMnTuXZcuW8dNPPxEbG8szzzzDb7/95nQ0EcmCSl4u2n333UdqaiqPP/4448aNIyIigvXr1zsdS0QyoZKXS1KkSBHeeOMN1q9fT0hICHXr1qVr164cP37c6WgikoFKXi5L7dq1SUpKom/fvsyYMQOXy8WyZcucjiUiXip5uWxXXXUVo0ePJi4ujmLFitGsWTPatm3L0aNHnY4mEvRU8pJjYmJiiI+P56WXXmLRokWEhoYyd+5cTY0g4iCVvOSoggUL8sILL7B7925uv/122rdvT9OmTTlw4IDT0USCkkpefCIsLIwtW7Ywfvx41q9fT1hYGG+88QZnz551OppIUFHJi8/ky5ePXr16kZKSQuXKlfnPf/5D3bp12bNnj9PRRIKGSl58rly5cqxZs4YZM2aQmJhIZGQkY8aM4cyZM05HEwl4Pi95Y8x+Y0yKMSbRGBPv6+2JfzLG8Nhjj+HxeGjYsCH9+vUjNjaW5ORkp6OJBLTc2pOvY62NttbG5NL2xE+VLl2axYsXs2DBAr755hsqVqzI4MGD+eOPP5yOJhKQdLhGcp0xhgcffBCPx0Pbtm0ZNmwY5cuXZ9u2bU5HEwk4uVHyFlhtjEkwxnQ9f6ExpqsxJt4YE69fngkuxYoV45133uGTTz7h119/pXr16vTq1YuTJ086HU0kYORGyVe31lYA7gV6GGNqZlxorZ1mrY2x1saUKFEiF+KIv7n33ntJTU2le/fuTJw4kfDwcNauXet0LJGA4POSt9Ye8l5/DywGKvt6m5L3FC5cmEmTJrFp0yYKFCjAPffcQ+fOnfn555+djiaSp/m05I0xVxtjCp+7DTQA9PfjJEs1atQgKSmJAQMGMHv2bFwuFx999JHTsUTyLF/vyd8AfGaMSQJ2AMuttSt9vE3J46688kpeeeUV4uLiKFmyJC1atKB169YcOXLE6WgieY5PS95au89aG+W9hFlrh/tyexJYKlasyM6dOxk+fDhLliwhNDSUd955RxOeiVwEnUIpfq1AgQI899xzJCYmEhoaSocOHWjcuDHffPON09FE8gSVvOQJoaGhbN68mddee43NmzcTFhbG5MmTNeGZyAWo5CXPCAkJ4cknn8TtdhMbG0vPnj2pVasWX3zxhdPRRPyWSl7ynFtvvZVVq1bx9ttv43a7iYqKYuTIkfz5559ORxPxOyp5yZOMMXTs2JG0tDSaNGnCwIEDqVKlCrt373Y6mohfUclLnnbjjTeyaNEiFi5cyKFDh6hUqRKDBg3i1KlTTkcT8QsqeQkIrVq1wuPx8MgjjzBixAiio6PZsmWL07FEHKeSl4Bx/fXX8/bbb7Nq1SpOnTpFjRo1eOqpp/j111+djibiGJW8BJwGDRrgdrvp2bMnkyZNIjw8nNWrVzsdS8QRKnkJSNdcc81/z6kvVKgQDRs2pFOnThw7dszpaCK5SiUvAa169eokJiby3HPPMWfOHFwuF4sWLXI6lkiuUclLwCtUqBDDhw8nPj6e0qVL88ADD9CqVSsOHz7sdDQRn1PJS9CIjo5mx44djBw5kuXLl+NyuZg1a5YmPJOAppKXoJI/f3769+9PUlIS4eHhdOrUiYYNG7J//36no4n4hEpegtJdd93Fxo0bmTx5Mtu2bSM8PJzXX39dE55JwFHJS9AKCQmhe/fuuN3u/55TX6NGDdLS0pyOJpJjVPIS9G655RY++eQT3nnnHT7//HOio6MZPny4JjyTgKCSFyF9wrNHHnkEj8dD8+bNef7556lUqRK7du1yOprIZVHJi2Rwww038P7777N48WKOHDlC5cqVGTBgAL///rvT0UQuiUpeJBPNmzfH4/HQsWNHRo0aRXR0NJs3b3Y6lshFU8mLZKFo0aK89dZbrFmzhtOnT1OzZk169OjBiRMnnI4mkm0qeZELqF+/Pm63m169ejF16lTCw8NZsWKF07FEskUlL5INV199NePHj2fLli0ULlyYxo0b8+ijj/Ljjz86HU3kH6nkRS5CbGwsu3bt4oUXXmDevHmEhoayYMECTY0gfkslL3KRrrjiCl566SUSEhL417/+xUMPPUTLli05dOiQ09FE/kYlL3KJIiMj2b59O6NHj2blypW4XC5mzJihvXrxKyp5kcuQP39++vbtS3JyMlFRUXTp0oV77rmHffv2OR1NBFDJi+SIO+64g/Xr1zN16lR27NhBREQEEyZM4K+//nI6mgQ5lbxIDgkJCaFbt26kpqZSp04devfuTfXq1UlNTXU6mgQxlbxIDrv55ptZtmwZc+fOZe/evZQvX55hw4Zx+vRpp6NJEFLJi/iAMYZ27dqRlpZGq1atGDx4MDExMezcudPpaBJkVPIiPlSiRAnmzZvHkiVL+PHHH6latSr9+vXjt99+czqaBAmVvEguaNasGR6Ph86dOzNmzBiioqLYsGGD07EkCKjkRXJJkSJFmDZtGuvWrePs2bPUqVOHbt26cfz4caejSQBTyYvksrp165KSksIzzzzD9OnTCQsLY/ny5U7HkgDl85I3xjQyxnxhjNlrjBng6+2J5AVXXXUVY8eOZdu2bRQtWpT77ruPhx9+mKNHjzodTQKMT0veGJMPmAzcC7iAtsYYly+3KZKXVK5cmYSEBIYMGcIHH3yAy+Vi/vz5mhpBcoyv9+QrA3uttfustaeB+cD9Pt6mSJ5SsGBBXnzxRXbt2kW5cuVo27Yt999/PwcPHnQ6mgQAX5f8TcCBDPe/9T4mIucJDw9n69atvPrqq6xduxaXy8W0adM4e/as09EkD/N1yZtMHvufn0ONMV2NMfHGmHgdj5Rgly9fPvr06UNKSgoVK1bkiSeeoF69euzdu9fpaJJH+brkvwVuznC/DPA/k25ba6dZa2OstTElSpTwcRyRvOG2225j3bp1TJ8+nV27dhEZGcmrr76qCc/kovm65HcCdxhjyhpjCgJtgKU+3qZIQDDG0KVLFzweD/Xr1+fZZ58lNjYWt9vtdDTJQ3xa8tbaM0BPYBWQBiyw1mpKPpGLcNNNN7FkyRLmz5/P/v37qVChAkOGDNGEZ5ItPj9P3lr7ibX2Tmvtbdba4b7enkggMsbw0EMP4fF4aN26NUOHDqVChQrExcU5HU38nH7jVSQPKV68OO+++y4ff/wxx48fJzY2lj59+nDy5Emno4mfUsmL5EFNmjQhNTWVbt26MX78eCIjI/n000+djiV+SCUvkkdde+21TJkyhQ0bNhASEkK9evV4/PHH+fnnn52OJn5EJS+Sx9WqVYvk5GT69evHzJkzCQsLY+lSncQm6VTyIgHgyiuvZNSoUcTFxVGsWDHuv/9+2rRpw/fff+90NHGYSl4kgMTExBAfH8+wYcNYvHgxLpeLuXPnasKzIKaSFwkwBQsW5Pnnn2f37t3ccccdtG/fnvvuu48DBw5c+Isl4KjkRQKUy+Xis88+Y8KECWzYsIGwsDCmTp2qCc+CjEpeJIDly5ePp59+GrfbTZUqVejevTt16tRhz549TkeTXKKSFwkCZcuWZfXq1cyYMYOkpCQiIyMZPXo0Z86ccTqa+JhKXiRIGGN47LHH8Hg8NGrUiP79+1O1alWSkpKcjiY+pJIXCTKlS5fmww8/ZMGCBRw4cICYmBheeOEF/vjjD6ejiQ+o5EWCkDGGBx98EI/HQ7t27Xj55ZcpX74827Ztczqa5DCVvEgQK1asGLNnz2bFihWcPHmS6tWr06tXL3799Veno0kOUcmLCI0aNcLtdtO9e3cmTpxIREQEa9ascTqW5ACVvIgAULhwYSZNmsSmTZsoWLAgDRo0oHPnzvz0009OR5PLoJIXkf9Ro0YNkpKSGDBgALNnz8blcrF48WKnY8klUsmLyN8UKlSIV155hR07dnDjjTfSsmVLWrduzZEjR5yOJhdJJS8iWapQoQI7duxgxIgRLF26lNDQUN555x1NeJaHqORF5B8VKFCAgQMHkpiYSGhoKB06dODee+/l66+/djqaZINKXkSy5d///jebN2/m9ddf57PPPiM8PJzJkydrwjM/p5IXkWwLCQmhZ8+euN1uqlWrRs+ePalVqxZffPGF09EkCyp5Eblot956KytXrmTWrFmkpqYSFRXFyJEj+fPPP52OJudRyYvIJTHG0KFDBzweD02bNmXgwIFUqVKF3bt3Ox1NMlDJi8hlufHGG/nggw9YtGgRhw4dolKlSjz33HOcOnXK6WiCSl5EckjLli1JS0vj0Ucf5ZVXXiE6OpotW7Y4HSvoqeRFJMcULVqUmTNnsmrVKk6dOkWNGjV48skn+eWXX5yOFrRU8iKS4xo0aIDb7ebJJ59k8uTJhIeHs2rVKqdjBSWVvIj4xDXXXMPEiRP57LPPuOqqq2jUqBEdO3bk2LFjTkcLKip5EfGpatWqsXv3bgYNGsTcuXMJDQ1l4cKFTscKGip5EfG5QoUK8fLLL7Nz507KlCnDgw8+SKtWrTh8+LDT0QKeSl5Eck10dDRxcXGMHDmS5cuX43K5ePvttzXhmQ+p5EUkV+XPn5/+/fuTnJxMREQEjz32GA0bNmT//v1ORwtIKnkRccSdd97Jhg0bmDx5Mtu2bSM8PJzXXnuNv/76y+loAUUlLyKOCQkJoXv37qSmplKzZk2efvppatSoQVpamtPRAoZKXkQc969//Yvly5czZ84cvvjiC6Kjoxk+fLgmPMsBPit5Y8wQY8xBY0yi99LYV9sSkbzPGEP79u1JS0ujefPmPP/888TExJCQkOB0tDzN13vy46210d7LJz7elogEgJIlS/L++++zePFijh49SpUqVRgwYAC///6709HyJB2uERG/1Lx5czweDx07dmTUqFFERUWxadMmp2PlOb4u+Z7GmGRjzExjTNHMVjDGdDXGxBtj4o8ePerjOCKSl1x33XW89dZbrF27ljNnzlCrVi169OjBiRMnnI6WZ5jL+SUEY8xa4MZMFg0CtgM/ABYYBpSy1j72T88XExNj4+PjLzmPiASukydP8sILLzBhwgTKlCnDG2+8QePG+qgPwBiTYK2NyWzZZe3JW2vrW2vDM7kssdYesdb+Za09C0wHKl/OtkQkuF199dWMGzeOrVu3UrhwYZo0acIjjzzCDz/84HQ0v+bLs2tKZbjbAnD7alsiEjyqVq3Krl27GDx4MPPnz8flcrFgwQJNjZAFXx6TH22MSTHGJAN1gN4+3JaIBJErrriCoUOHkpCQwC233MJDDz1EixYtOHTokNPR/I7PSt5a+4i1NsJaG2mtbWat1XRzIpKjIiMj2bZtG2PGjGHVqlW4XC5mzJihvfoMdAqliORp+fPn59lnnyUlJYXo6Gi6dOlC/fr12bdvn9PR/IJKXkQCwu23386nn37Km2++yc6dOwkPD2f8+PFBP+GZSl5EAkZISAhdu3bF4/FQt25d+vTpQ/Xq1UlNTXU6mmNU8iIScMqUKcOyZct47733+OqrryhfvjwvvfQSp0+fdjparlPJi0hAMsbQtm1bPB4PDzzwAC+++CIxMTHs3LnT6Wi5SiUvIgGtRIkSvPfeeyxdupRjx45RtWpV+vbty2+//eZ0tFyhkheRoNC0aVNSU1N5/PHHGTt2LJGRkWzYsMHpWD6nkheRoFGkSBHeeOMNPv30UwDq1KnDE088wfHjxx1O5jsqeREJOnXq1CE5OZlnn32Wt956i7CwMD7++GOnY/mESl5EgtJVV13FmDFj2LZtG0WLFqVp06a0a9eOQJvyXCUvIkGtcuXKJCQkMHToUBYuXIjL5WLevHkBMzWCSl5Egl7BggUZPHgwu3fv5rbbbqNdu3Y0a9aMb7/91ulol00lLyLiFRYWxpYtWxg3bhzr1q0jLCyMadOmcfbsWaejXTKVvIhIBvny5aN379643W5iYmJ44oknqFevHnv37nU62iVRyYuIZKJcuXKsXbuW6dOns2vXLiIiIhg7dixnzpxxOtpFUcmLiGTBGEOXLl3weDw0aNCAvn37Uq1aNVJSUpyOlm0qeRGRC7jpppv46KOPmD9/Pvv376dChQq8+OKL/PHHH05HuyCVvIhINhhjeOihh/B4PLRp04aXXnqJihUrEhcX53S0f6SSFxG5CMWLF2fOnDksX76c48ePExsbS58+fTh58qTT0TKlkhcRuQSNGzcmNTWVbt26MX78eCIiIli3bp3Tsf5GJS8icomuvfZapkyZwsaNG8mfPz/169fn8ccf5+eff3Y62n+p5EVELlPNmjVJSkqiX79+zJw5E5fLxZIlS5yOBajkRURyxJVXXsmoUaOIi4ujRIkSNG/enDZt2vD99987mkslLyKSg2JiYoiPj+fll19m8eLFhIaG8u677zo24ZlKXkQkhxUoUIBBgwaRmJjIXXfdxSOPPEKTJk345ptvcj2LSl5ExEdCQ0PZvHkzEydOZOPGjYSFhTF16tRcnfBMJS8i4kP58uXjqaeewu12U7VqVbp3707t2rX58ssvc2X7KnkRkVxQtmxZVq9ezcyZM0lJSSEqKorRo0f7fMIzlbyISC4xxtCpUyc8Hg/33nsv/fv3p0qVKiQlJflsmyp5EZFcVqpUKT788EMWLlzIwYMHiYmJYcKECT7ZlkpeRMQhrVq1wuPx8PDDD3Pbbbf5ZBv5ffKsIiKSLddffz2zZs3y2fNrT15EJICp5EVEAphKXkQkgKnkRUQC2GWVvDHmQWNMqjHmrDEm5rxlA40xe40xXxhjGl5eTBERuRSXe3aNG2gJvJnxQWOMC2gDhAGlgbXGmDuttX9d5vZEROQiXNaevLU2zVr7RSaL7gfmW2v/sNb+P2AvUPlytiUiIhfPV8fkbwIOZLj/rfexvzHGdDXGxBtj4o8ePeqjOCIiwemCh2uMMWuBGzNZNMham9XftzKZPJbpjPnW2mnANO+2jhpjTgI/XCiXHyqOcuemvJob8m525c5dF5P7lqwWXLDkrbX1s5sog2+BmzPcLwMcysa2Shhj4q21MRda198od+7Kq7kh72ZX7tyVU7l9dbhmKdDGGHOFMaYscAeww0fbEhGRLFzuKZQtjDHfArHAcmPMKgBrbSqwAPAAK4EeOrNGRCT3XdYplNbaxcDiLJYNB4ZfwtNOu5xMDlLu3JVXc0Peza7cuStHchun/oK4iIj4nqY1EBEJYCp5EZEA5njJG2PeN8Ykei/7jTGJWay33xiT4l0vPrdzZpJniDHmYIbsjbNYr5F3/p69xpgBuZ0zkzxjjDGfG2OSjTGLjTHXZbGeX4z3hcbPewbX+97lccaYW3M/5d8y3WyMWW+MSfPO7fR0JuvUNsYcz/D+GexE1vNd6HU36V7zjneyMaaCEznPy3RXhnFMNMacMMb0Om8dvxlvY8xMY8z3xhh3hseuN8asMcbs8V4XzeJrO3jX2WOM6ZCtDVpr/eYCvAoMzmLZfqC40xkz5BkCPHuBdfIBXwHlgIJAEuByOHcDIL/39ihglL+Od3bGD+gOvOG93QZ43w/eG6WACt7bhYEvM8ldG/jY6awX+7oDjYEVpP/CY1UgzunMmbxnvgNu8dfxBmoCFQB3hsdGAwO8twdk9v8SuB7Y570u6r1d9ELbc3xP/hxjjAFaA/OczpKDKgN7rbX7rLWngfmkz+vjGGvtamvtGe/d7aT/opq/ys743Q/M9t5eCNTzvpccY609bK3d5b39C5BGFtN65EH3A+/YdNuB64wxpZwOlUE94Ctr7ddOB8mKtXYTcOy8hzO+j2cDzTP50obAGmvtMWvtT8AaoNGFtuc3JQ/UAI5Ya/dksdwCqyxW+2sAAAMJSURBVI0xCcaYrrmY65/09P7IOjOLH6+yPYePQx4jfa8sM/4w3tkZv/+u4/3mdRwolivpssF7+Kg8EJfJ4lhjTJIxZoUxJixXg2XtQq+7v7+n25D1jqI/jvc5N1hrD0P6TgJQMpN1Lmnsc+UPeWdz/pu2/PNefHVr7SFjTElgjTHmc+93RJ/5p9zAVGAY6f8phpF+qOmx858ik6/1+Tmr2RlvY8wg4AwwN4unyfXxzkR2xs+RMc4OY8w1wCKgl7X2xHmLd5F+SOFX7+c5H5H+m+FOu9Dr7s/jXRBoBgzMZLG/jvfFuKSxz5WStxeY/8YYk5/0eekr/sNzHPJef2+MWUz6j/I+LZ0L5T7HGDMd+DiTRZc0h8/lysZ4dwDuA+pZ78G+TJ4j18c7E9kZv3PrfOt9HxXh7z8K5zpjTAHSC36utfbD85dnLH1r7SfGmCnGmOLWWkcn0srG6+7Iezqb7gV2WWuPnL/AX8c7gyPGmFLW2sPew1/fZ7LOt6R/tnBOGWDDhZ7YXw7X1Ac+t9Z+m9lCY8zVxpjC526T/uGhO7N1c8t5xyFbkHmencAdxpiy3r2MNqTP6+MYY0wjoD/QzFr7Wxbr+Mt4Z2f8lgLnzjJ4APg0q29cucX7mcAMIM1aOy6LdW4899mBMaYy6f8Xf8y9lJlmys7rvhR41HuWTVXg+LnDDH4gy6MB/jje58n4Pu4AZDbD7yqggTGmqPfwcAPvY//M6U+avf8fZwHdznusNPCJ93Y50s+sSAJSST/s4HTmOUAKkOx9gUqdn9t7vzHpZ1d85Se595J+XC/Rezl3Zopfjndm4we8RPo3KYBCwAfef9cOoJwfjPHdpP8YnZxhnBsD3c69z4Ge3rFNIv0D8Gp+kDvT1/283AaY7H09UoAYp3N7c11FemkXyfCYX4436d+IDgN/kr533pn0z5HWAXu819d7140B3srwtY953+t7gU7Z2Z6mNRARCWD+crhGRER8QCUvIhLAVPIiIgFMJS8iEsBU8iIiAUwlLyISwFTyIiIB7P8AmHptSdsE4UgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_decision_boundary(thetas, data_set):\n",
    "    minx = np.min(data_set[:, 0])\n",
    "    maxx = np.max(data_set[:, 1])\n",
    "\n",
    "    y1 =  -(thetas[0] + thetas[1]*minx) / thetas[2]\n",
    "    y2 =  -(thetas[0] + thetas[1]*maxx) / thetas[2]\n",
    "    \n",
    "    plt.plot([minx, maxx], [y1, y2], c='black')\n",
    "\n",
    "plot_decision_boundary(history_theta[-1], X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy\n",
    "\n",
    "The logistic hypothesis outputs a value in the interval $]0;1[$. We want to map this value to one specific class i.e. $0$ or $1$, so we apply a threshold known as the decision boundary: If the predicted value is < 0.5, the class is 0, otherwise it is 1.\n",
    "\n",
    "**Task:**\n",
    "\n",
    "1. Calculate the accuracy of your final classifier. The accuracy is the proportion of the correctly classified data.\n",
    "2. Why will the accuracy never reach 100% using this model and this data set?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ammount of non accurate data: 323\n",
      "Ammount of accurate data: 1677\n",
      "Accuracy in percentage: 83.85%\n"
     ]
    }
   ],
   "source": [
    "def print_accuracy(thetas,data_set):\n",
    "\n",
    "    data = logistic_hypothesis(np.asarray(thetas))(data_set)\n",
    "    counter = 0\n",
    "    for i in range(len(data)):\n",
    "\n",
    "        if((y[i] == 1.0) and (data[i] > 0.5)):\n",
    "                pass\n",
    "        elif((y[i] == 1.0) and (data[i] < 0.5)):\n",
    "                counter = counter + 1\n",
    "        elif((y[i] == 0.0) and (data[i] < 0.5)):\n",
    "                pass\n",
    "        elif((y[i] == 0.0) and (data[i] > 0.5)):\n",
    "                counter = counter + 1\n",
    "\n",
    "    accuracy = 100 - (counter/2000 * 100)\n",
    "\n",
    "    print(\"Ammount of non accurate data: \" + str(counter))\n",
    "    print(\"Ammount of accurate data: \" + str(len(X) - counter))\n",
    "    print(\"Accuracy in percentage: \" + str(accuracy) + \"%\")\n",
    "    \n",
    "print_accuracy(history_theta[-1], X)\n",
    "\n",
    "###Exercise 2\n",
    "\n",
    "# Our model wont be 100% accurate because of overfitting. To improve this we need to apply regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regularization\n",
    "\n",
    "**Task:**\n",
    "    \n",
    "Extend your implementation with a regularization term $\\lambda$ by adding it as argument to the functions `mean_cross_entropy_costs`, `compute_new_theta` and `gradient_descent`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Proof - Pen&Paper\n",
    "\n",
    "The sigmoid activation function is defined as $\\sigma (z) = \\frac{1}{1+\\exp(-z)}$ \n",
    "\n",
    "**Task:**\n",
    "\n",
    "Show that:\n",
    "$$\n",
    "\\frac{d \\sigma(z)}{d z} = \\sigma(z)(1-\\sigma(z))\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "**Task:**\n",
    "\n",
    "Now show that:\n",
    "$$\n",
    "\\frac{\\partial \\sigma(z)}{\\partial \\theta_1} = \\sigma(z)(1-\\sigma(z)) \\cdot x_1\n",
    "$$\n",
    "with:\n",
    "$$\n",
    "z = \\theta_0 x_0 + \\theta_1 x_1\n",
    "$$\n",
    "\n",
    "Note that in general (because of symmetry) holds:\n",
    "\n",
    "$$\n",
    "z = \\theta_0 x_0 + \\theta_1 x_1 + \\dots\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\sigma(z)}{\\partial \\theta_j} = \\sigma(z)(1-\\sigma(z)) \\cdot x_j\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "**Task:**\n",
    "\n",
    "Show from\n",
    "$$\n",
    "    \\frac{\\partial}{\\partial \\theta_j}  J(\\theta)  =  \n",
    "    \\frac{\\partial}{\\partial \\theta_j}  \\left( - \\frac{1}{m}  \\sum_{i=1}^{m} \n",
    "    \\left[  y^{(i)} \\log h_\\theta({\\vec x}^{(i)})+\n",
    "      (1 - y^{(i)}) \\log \\left( 1- h_\\theta({\\vec x}^{(i)})\\right) \\right] \\right)\n",
    "$$  \n",
    "that\n",
    "$$\n",
    "\\frac{\\partial}{\\partial \\theta_j}  J(\\theta)  =   \\frac{1}{m}\n",
    "     \\sum_{i=1}^{m} \\left( h_\\theta({\\vec x}^{(i)})- y^{(i)}\\right) x_j^{(i)}\n",
    "$$\n",
    "\n",
    "with the sigmoid function as hypothesis $h_\\theta(\\vec x^{(i)})$\n",
    "\n",
    "**Hint:**\n",
    "\n",
    "Make use of your knowlede, that:\n",
    "$$\n",
    "\\frac{\\partial \\sigma(z)}{\\partial \\theta_j} = \\sigma(z)(1-\\sigma(z)) \\cdot x_j\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Summary and Outlook\n",
    "\n",
    "During this exercise you learned about logistic regression and used it to perform binary classification on multidimensional data. You should be able to answer the following questions:\n",
    "* How can you interpret the output of the logistic function?\n",
    "* For which type of problem do you use linear regression and for which type of problem do you use logistic regression?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Licenses\n",
    "\n",
    "### Notebook License (CC-BY-SA 4.0)\n",
    "\n",
    "*The following license applies to the complete notebook, including code cells. It does however not apply to any referenced external media (e.g., images).*\n",
    "\n",
    "Exercise: Logistic Regression and Regularization <br/>\n",
    "by Christian Herta, Klaus Strohmenger <br/>\n",
    "is licensed under a [Creative Commons Attribution-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-sa/4.0/).<br/>\n",
    "Based on a work at https://gitlab.com/deep.TEACHING.\n",
    "\n",
    "\n",
    "### Code License (MIT)\n",
    "\n",
    "*The following license only applies to code cells of the notebook.*\n",
    "\n",
    "Copyright 2018 Christian Herta, Klaus Strohmenger\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}