04-01-Logistic Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resources:\n",
    "\n",
    "[Logistic Regression Tutorial for Machine Learning](http://machinelearningmastery.com/logistic-regression-tutorial-for-machine-learning/)\n",
    "\n",
    "[Logistic Regression for Machine Learning](http://machinelearningmastery.com/logistic-regression-for-machine-learning/)\n",
    "\n",
    "[How To Implement Logistic Regression With Stochastic Gradient Descent From Scratch With Python](http://machinelearningmastery.com/implement-logistic-regression-stochastic-gradient-descent-scratch-python/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Logistic regression is the go-to linear classification algorithm for two-class problems. It is easy to implement, easy to understand and gets great results on a wide variety of problems, even when the expectations the method has for your data are violated.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Description\n",
    "\n",
    "#### Logistic Regression\n",
    "\n",
    "Logistic regression is named for the function used at the core of the method, the [logistic function](https://en.wikipedia.org/wiki/Logistic_function).\n",
    "\n",
    "The logistic function, also called the **Sigmoid function** was developed by statisticians to describe properties of population growth in ecology, rising quickly and maxing out at the carrying capacity of the environment. It’s an S-shaped curve that can take any real-valued number and map it into a value between 0 and 1, but never exactly at those limits."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\frac{1}{1 + e^{-x}}$$\n",
    "\n",
    "$e$ is the base of the natural logarithms and $x$ is value that you want to transform via the logistic function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.16.5\n",
      "0.25.1\n",
      "3.7.4 (default, Aug 13 2019, 15:17:50) \n",
      "[Clang 4.0.1 (tags/RELEASE_401/final)]\n",
      "0.21.3\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn\n",
    "%matplotlib inline\n",
    "\n",
    "print(np.__version__)\n",
    "print(pd.__version__)\n",
    "import sys\n",
    "print(sys.version)\n",
    "import sklearn\n",
    "print(sklearn.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsYAAAHiCAYAAADrvQoIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xl4VNXBx/HfyWSyL0ASwpKwBBI2QZBdRVFBQa12USu2KLWuVduqfa221lq72VprtbXvq7Vad8Udd1BxX9h3SICwZIGQhezbLOf9I4HGCBIgyZ3l+3meeWbu3JO5v/A8zPw4nLnXWGsFAAAAhLsIpwMAAAAAgYBiDAAAAIhiDAAAAEiiGAMAAACSKMYAAACAJIoxAAAAIIliDCDMGWO+Z4xZGGjHNca8b4y5rDszHQ5jzDRjTK7TOQCgM1GMAYQ8Y8yJxphPjTFVxpgKY8wnxpiJkmStfdJae3p3Zzqa4xpjbjfGeIwxtW1uN3V2xnbHtMaYofu2rbUfWWuHdeUxAaC7RTodAAC6kjEmSdJrkq6WNF9SlKRpkpqczNUJnrXWft/pEAAQSpgxBhDqciTJWvu0tdZnrW2w1i601q6RJGPMPGPMx/sGG2NON8bkts4u/9MY88G+JQ2tYz8xxtxjjKk0xuQbY45vfb7AGLPHGHNJm9dKNsY8ZowpNcbsMMbcaoyJOMhxZxpjNrUe9x+SzJH8ssaY7caYGW22bzfGPNH6eFDrzO8lxpidxpgyY8wv24x1GWN+YYzZaoypMcYsN8ZkGmM+bB2yunV2+rvGmOnGmMI2PzuidflHpTFmvTHmnDb7/mOMud8Y83rr635hjBlyJL8fAHQlijGAUJcnyWeMedQYM9sY0/NgA40xqZKel3SLpBRJuZKObzdssqQ1rfufkvSMpImShkr6vqR/GGMSWsf+XVKypCxJJ0u6WNIPDnLcFyTdKilV0lZJJxzJL9tBJ0oaJuk0SbcZY0a0Pn+DpDmSzpSUJOlSSfXW2pNa9x9rrU2w1j7bLr9b0quSFkrqLek6SU8aY9outZgj6TeSekraIun3XfGLAcDRoBgDCGnW2mq1FEEr6V+SSo0xC4wx6QcYfqak9dbaF621Xkn3Sdrdbsw2a+0j1lqfpGclZUq6w1rbZK1dKKlZ0lBjjEvSdyXdYq2tsdZul3S3pLkHOe4Ga+3z1lqPpL8d4LjtXdA6O7vv1u/Qfxr7/aZ15ny1pNWSjm19/jJJt1prc22L1dba8g683hRJCZLutNY2W2vfU8vylTltxrxorV3S+uf6pKSxh5EXALoFxRhAyLPWbrTWzrPWZkg6RlI/tZTP9vpJKmjzc1ZSYbsxJW0eN7SOa/9cglpmfqMk7Wizb4ek/h08bsEBxrU131rbo82t+BDj22pbuutb80otJX/rYbzOPv0kFVhr/W2ea/+7HuyYABAwKMYAwoq1dpOk/6ilILe3S1LGvg1jjGm7fZjKJHkkDWzz3ABJRQc5bma742YeYFxH1EmKa7Pd5zB+tkDSkaz9LZaUuW/9dKuD/a4AELAoxgBCmjFmuDHmRmNMRut2plr+i//zAwx/XdJoY8w3jTGRkq7R4RXL/VqXWsyX9HtjTKIxZqBa1vA+cZDjjjLGfLv1uD8+0uNKWiXpQmOM2xgzQdJ5h/GzD0n6rTEm27QYY4xJad1Xopa10gfyhVoK+U2tx50u6RtqWX8NAEGDYgwg1NWo5QtzXxhj6tRSiNdJurH9QGttmaTzJf1ZUrmkkZKW6chP7XadWgpjvqSP1fJlvYe/5rh3th43W9InR3jMX6ll1nevWr7s9tRh/Oxf1VLmF0qqlvRvSbGt+26X9GjreuYL2uVvlnSOpNlqmSn/p6SLW2fnASBomJalbACA9lqXBhRK+p61drHTeQAAXYsZYwBowxhzhjGmhzEmWtIv1HI+4QMtuwAAhBiKMQB82VS1nJmhTC3rZL9prW1wNhIAoDuwlAIAAAAQM8YAAACAJIoxAAAAIEmKdOrAqampdtCgQU4dHgAAAGFi+fLlZdbatEONc6wYDxo0SMuWLXPq8AAAAAgTxpgdHRnHUgoAAABAFGMAAABAEsUYAAAAkEQxBgAAACRRjAEAAABJFGMAAABAEsUYAAAAkEQxBgAAACRRjAEAAABJFGMAAABAEsUYAAAAkEQxBgAAACRRjAEAAABJHSjGxpiHjTF7jDHrDrLfGGPuM8ZsMcasMcYc1/kxAQAAgK7VkRnj/0ia9TX7Z0vKbr1dIel/jz4WAAAA0L0OWYyttR9KqviaIedKesy2+FxSD2NM384KCAAAAHSHzlhj3F9SQZvtwtbnAAAAEMZ8fqtGj0+1TV5V1jc7HeeQIjvhNcwBnrMHHGjMFWpZbqEBAwZ0wqEBAADQnt9v1eT1q8HjU4PHp0aPTw3Nrfcenxo9Lfsam31q9PrU5PGr2edXs/e/9559296D72va97jNWI/Pqtnnl9fnl79NI3RFGG39w5nO/aF0QGcU40JJmW22MyQVH2igtfZBSQ9K0oQJEw5YngEAAMKJ329V1+xVXVPLzGpd662mzePaJl/rvXf/mH33DR6/Gj3/Lb0NzT41ef1HnMftMopyRcgdGaEoV4Si2t63Pna7IhQXF/nlfa33bleE3JFG7oiWx5EuI7fLKDIiQtZaGXOgOdXA0BnFeIGka40xz0iaLKnKWrurE14XAAAgaDR5faqoa1Z5bbOqGzyqanOrbPO47b7Keo9qGj1fmlk9GFeEUXyUSwnRkYpvvSVERyolwaVYt0sx7oiW+yiXYiJdio367/Mx7n2Pv/p8jNul6NZCG+WKUERE4BbXrnbIYmyMeVrSdEmpxphCSb+W5JYka+3/SXpD0pmStkiql/SDrgoLAADQXay1qqz3qKSmUeW1zSqrbVJ5bXNL+a1rUtm+x63P1zR5D/pakRFGybHullucW73iozQ4NX7/c4kxkUqIdis+uqX4JrQpvvvuY9wRAT3bGgoOWYyttXMOsd9KuqbTEgEAAHSxZq9fu6saVVLT2HJf3XK/u7pRe6qbtLu65bkDLUlwRRj1jItSakKUUhKiNCajh3rF79uOVs84t5Jjo5Qc61aPuJbiGxflotQGgc5YSgEAABBQvD6/dlU1qnBvgwr21qtwb4MKK+r3b++ubpRtt3whxh2hPkkxSk+K0djMHuqT3PI4PSlaaQnRSkmIUkp8tJJj3WG93CCUUYwBAEBQ8vutdlU3Kr+0VvmlddpWVqetpbXaXl6n4spG+dos3DVG6psUo4xecZo6JEWZPePUv0es+iTHtBTgxBglxUYyqxvmKMYAACCg+fxW28vrlLu7Rpt212jrntr9BbjR89+lDvFRLg1Oi9fYzJ4659hYZfaMU0bPOGX2ilXf5FhFRXbG5RsQyijGAAAgYJTXNmnDrur9JTh3d43ySmr2r/WNMFJmrzhlpcbrhKGpykqL1+DUeA1JS1DvxGhmfHFUKMYAAMARVfUerS2q0pqiSq0trNKawioVVTbs35+aEK0RfRM1d8pADeuTqOF9kpSdnqAYt8vB1AhlFGMAANDlvD6/Nu2u0ZJtFVpZUKm1hZXaXl6/f//AlDiNG9BDlxw/UMf0S9awPolKSYh2MDHCEcUYAAB0uoZmn1YW7NWy7Xu1dHuFVuzYq7pmnySpb3KMxmQk6/wJmRqTkazR/ZPVIy7K4cQAxRgAAHSCZq9fK3fu1cdbyvTxljKtLayS129ljDQsPVHfPi5DEwb11MRBvdSvR6zTcYEDohgDAIDDZq1VXkmtPtpcqk+2lOmLbRWqb/bJFWE0JiNZV5yUpYmDeum4AT2VHOd2Oi7QIRRjAADQIY0enz7LL9e7G0v07sY92lXVKEnKSo3Xd47L0InZqZo6JEVJMRRhBCeKMQAAOKjy2ia9t2mP3t24Rx9uLlV9s0+xbpdOyknVT07L1rScNPVnaQRCBMUYAAB8SXltk95ct1uvrSnWF9sqZK3UJylG3xrXXzNGpmtqVgqnTENIohgDAABV1jfr7fW79dqaXfp0a7l8fqshafG67tRsnT4yXaP6JXHxDIQ8ijEAAGHK4/Pr/dxSPbesQItz98jjsxqYEqerTs7S2WP6aXifRMowwgrFGACAMJO7u0bPLSvQy6uKVFbbrNSEaM07fpDOOba/junPzDDCF8UYAIAwUN/s1SurivX0kp1aU1ilyAij00b01vnjM3XysDS5XRFORwQcRzEGACCEbSur0+Of7dBzywtU0+jV8D6J+tXZI/XNsf245DLQDsUYAIAQ4/dbLc7do0c/26EP80oVGWE0e3RfXTx1oCYM7MlSCeAgKMYAAISIJq9Pr6ws1gMfbtXW0jqlJ0Xr+hk5mjMpU72TYpyOBwQ8ijEAAEGuutGjp7/YqYc/2aaS6iaN6Jukey8cqzNH92XtMHAYKMYAAASpyvpmPfTRNj366XbVNHl1wtAU3XXesZqWncpyCeAIUIwBAAgyVQ0e/fvjbXrk422qafLqzNF9dPXJQzU6I9npaEBQoxgDABAkqhs9euTj7Xro43zVNHo1+5g++smMbA3vk+R0NCAkUIwBAAhwzV6/nvh8h+57b7Mq6z2aOTJdP52RrVH9mCEGOhPFGACAAGWt1VvrduvOtzZpR3m9Thiaop/PGq4xGT2cjgaEJIoxAAABaOXOvfr96xu1bMdeZfdO0CM/mKjpOWl8qQ7oQhRjAAACyJ6aRt35xia9uLJIqQnR+uO3R+v88RmK5LRrQJejGAMAEAC8vpZ1xHcvzFOT169rThmiq6cPVUI0H9VAd+FvGwAADluxc69ufWmdNuyq1rTsVP3mnFHKSktwOhYQdijGAAA4pLbJqz+9uUmPf75DfZJidP9Fx+nM0X1YRww4hGIMAIADPsgr1S9eXKviqgZdesJg3XB6DssmAIfxNxAAgG5UVe/R717foOeWF2po7wQ9f9XxGj+wp9OxAIhiDABAt1mcu0c/f36Nyuuadc0pQ3TdqdmKcbucjgWgFcUYAIAu1ujx6Y9vbNSjn+3QsPREPTxvoo7pz1XrgEBDMQYAoAutL67ST59Zpc17anXpCYN106xhzBIDAYpiDABAF/D7rf798Tbd9XaukuPceuzSSTopJ83pWAC+BsUYAIBOVlnfrBvnr9a7m/Zo5sh0/ek7Y9QrPsrpWAAOgWIMAEAnWlNYqaufWKE9NY26/RsjdcnxgzgvMRAkKMYAAHQCa62e+GKnfvvqBqUmRGn+lVM1bgCnYQOCCcUYAICj1NDs080vrtErq4o1fVia7rlgrHqydAIIOhRjAACOQnFlg654fJnWF1frxpk5uuaUoYqIYOkEEIwoxgAAHKHlO/bqyseXq9Hj00MXT9BpI9KdjgTgKFCMAQA4As8vL9QvXlyrvj1i9PTlk5Wdnuh0JABHiWIMAMBh8Put/vTWJj3wYb6OH5Ki+y86jvXEQIigGAMA0EGNHp9ufG61Xl+zS3OnDNRt3xgptyvC6VgAOgnFGACADqiq9+jyx5dpybYK/fLMEbps2mDOTwyEGIoxAACHULi3XvMeWaqd5fW6b844nXNsP6cjAegCFGMAAL7GhuJqzXtkiRo8Pj166SRNHZLidCQAXYRiDADAQSzfUaF5jyxVQnSkXrj6eOVw5gkgpFGMAQA4gI83l+nyx5apb3KMnrhssvr1iHU6EoAuRjEGAKCdRRtKdM2TK5SVFq/HfzhZaYnRTkcC0A0oxgAAtPHKqiLdMH+1jumfrEd/MFE94jhHMRAuKMYAALSav7RAP39xjSYP7qWHLpmohGg+JoFwwt94AAAkzV/WUoqnZafpwbnjFeN2OR0JQDfjcj0AgLD34opC/fyFNTpxaCqlGAhjFGMAQFh7ZVWRfvbcak3NStG/Lp5AKQbCGMUYABC2Xl1drOufXaVJg3vp35dMpBQDYY5iDAAIS2+t262fPrtKEwa2lOLYKEoxEO4oxgCAsPPpljL9+OmVGpORrId/MFHxnH0CgCjGAIAws7qgUpc/tkyDU+P1yDxOyQbgvyjGAICwsWVPreY9skS9EqL02A8ncfEOAF9CMQYAhIWiygbN/fcXckVE6PFLJys9KcbpSAACDMUYABDyKuqaNfffX6i2yavHLp2kQanxTkcCEIAoxgCAkNbo8emyR5eqaG+D/n3JRI3sl+R0JAABim8cAABClt9vdeP81VpZUKl/XnScJg3u5XQkAAGMGWMAQMj609ub9PraXfrF7BGaPbqv03EABDiKMQAgJD3x+Q498EG+5k4ZqMumDXY6DoAgQDEGAIScxZv26LZX1unU4b3162+MlDHG6UgAggDFGAAQUjbuqtY1T63QiL5J+vuccYp08VEHoGN4twAAhIyKumZd9ugyJcZE6uF5XOoZwOHpUDE2xswyxuQaY7YYY24+wP4BxpjFxpiVxpg1xpgzOz8qAAAH5/H59aMnl6u0tkkPzp3ABTwAHLZDFmNjjEvS/ZJmSxopaY4xZmS7YbdKmm+tHSfpQkn/7OygAAB8nTte3aDP8yv0p++M1rGZPZyOAyAIdWTGeJKkLdbafGtts6RnJJ3bboyVtO+M6cmSijsvIgAAX++pL3bq8c936MqTsvStcRlOxwEQpDqy+Kq/pII224WSJrcbc7ukhcaY6yTFS5rRKekAADiEJdsqdNsr6zR9WJpumjXc6TgAglhHZowPdI4b2257jqT/WGszJJ0p6XFjzFde2xhzhTFmmTFmWWlp6eGnBQCgjV1VDbr6ieUakBKney8cJ1cEp2UDcOQ6UowLJWW22c7QV5dK/FDSfEmy1n4mKUZSavsXstY+aK2dYK2dkJaWdmSJAQCQ1Oz160dPrlCjx6cH505Qcqzb6UgAglxHivFSSdnGmMHGmCi1fLluQbsxOyWdJknGmBFqKcZMCQMAuswf3tiolTsrddf5x2po7wSn4wAIAYcsxtZar6RrJb0taaNazj6x3hhzhzHmnNZhN0q63BizWtLTkuZZa9svtwAAoFMsWF2s/3y6XZeeMFhnju7rdBwAIaJDZz631r4h6Y12z93W5vEGSSd0bjQAAL5qy54a3fzCGk0Y2FO3nMmX7QB0Hq58BwAIGnVNXl31xArFRbn0j4uOk5vLPQPoRFwrEwAQFKy1uuXFtcovrdUTP5ysPslc2Q5A5+Kf2gCAoPDs0gItWF2sG2bm6PihXznxEQAcNYoxACDgbS6p0e2vrtcJQ1P0o+lDnY4DIERRjAEAAa3R49N1T69UfFSk7rlgrCK4iAeALsIaYwBAQPvDGxu1aXeNHvnBRPVOYl0xgK7DjDEAIGC9vX63Hvtshy47cbBOGdbb6TgAQhzFGAAQkIorG3TT82s0un+ybprF+YoBdD2KMQAg4Pj8Vj99dpW8Pr/umzNOUZF8XAHoeqwxBgAEnIc+yteSbRW667wxGpwa73QcAGGCf4IDAALKxl3VunthnmaN6qPzxmc4HQdAGKEYAwACRpPXp+ufXaWkWLf+8O3RMoZTswHoPiylAAAEjHsWbdam3TV6eN4E9YqPcjoOgDDDjDEAICAs3V6hBz7cqjmTMnXq8HSn4wAIQxRjAIDjapu8umH+KmX2jNOtZ410Og6AMMVSCgCA43732gYV7W3Q/CunKj6ajyYAzmDGGADgqPc2leiZpQW68uQhmjCol9NxAIQxijEAwDFVDR7d8uJaDUtP1PUzcpyOAyDM8f9VAADH/PGNjSqtadK/Lp7A1e0AOI53IQCAIz7aXKpnlhboipOGaExGD6fjAADFGADQ/eqavLr5hbXKSovXT2dkOx0HACSxlAIA4IA/v7VJxVUNeu7KqYpxu5yOAwCSmDEGAHSzJdsq9OhnO3TJ1EGchQJAQKEYAwC6TUOzTzc9v1qZvWJ106xhTscBgC9hKQUAoNvc806etpfX68nLJisuio8gAIGFGWMAQLdYV1Slhz7K14UTM3XC0FSn4wDAV1CMAQBdzue3+sVLa9UrPkq3zB7hdBwAOCCKMQCgyz3+2XatKazSr84eqeQ4t9NxAOCAKMYAgC61q6pBf1mYp2nZqTrn2H5OxwGAg6IYAwC61G8WbJDH59fvvnmMjDFOxwGAg6IYAwC6zKINJXpr/W79+LRsDUyJdzoOAHwtijEAoEvUNXn161fWKSc9QZdPy3I6DgAcEieRBAB0iXsW5am4qlHPz5mqqEjmYQAEPt6pAACdbn1xlR7+ZJvmTBrAZZ8BBA2KMQCgU/n9Vre9sl494qJ086zhTscBgA6jGAMAOtWLK4u0fMde3TxrOOcsBhBUKMYAgE5T3ejRnW9u1NjMHjpvfIbTcQDgsPDlOwBAp7lnUZ7K65r1yLxJiojgnMUAggszxgCATrFpd7Ue+2yH5kwaoNEZyU7HAYDDRjEGABw1a1u+cJcYE6n/OX2Y03EA4IhQjAEAR23B6mIt2Vahm84Yrp7xUU7HAYAjQjEGAByV2iavfv/6Ro3JSNZ3J2Y6HQcAjhhfvgMAHJX73t2sPTVNemDueLn4wh2AIMaMMQDgiG3ZU6OHP96m707I1LgBPZ2OAwBHhWIMADgi1lrd8dpGxUW5dNMsvnAHIPhRjAEAR+T93FJ9mFeqn8zIUUpCtNNxAOCoUYwBAIfN4/Prt69vUFZqvOZOGeh0HADoFBRjAMBhe/yzHcovrdOtZ49QVCQfJQBCA+9mAIDDUlHXrL+9k6dp2ak6ZVhvp+MAQKehGAMADsvf3slTXbNPvzp7pIzh9GwAQgfFGADQYXklNXryi5363uQByklPdDoOAHQqijEAoEOstfrtaxsUH+XST2fkOB0HADodxRgA0CHvbdqjjzaX6aczctQrPsrpOADQ6SjGAIBDavb69fvXNyorLV5zp3J6NgChiWIMADikJz7fofyyOt161gi5XXx0AAhNvLsBAL5WVYNH9723mdOzAQh5FGMAwNf65+Itqmrw6ObZwzk9G4CQRjEGABxU4d56PfLpdn1rXH+N6pfsdBwA6FIUYwDAQd29ME9G0s9OH+Z0FADochRjAMABrSuq0ksri3TpiYPVr0es03EAoMtRjAEAX2Gt1R/e2KiecW5dPX2I03EAoFtQjAEAX/F+bqk+3VquH5+WraQYt9NxAKBbUIwBAF/i81v98c2NGpQSp+9N5mIeAMIHxRgA8CXPLy9QXkmtbpo1XFGRfEwACB+84wEA9qtv9uruhXkaN6CHZh/Tx+k4ANCtKMYAgP0e+mib9tQ06ZdnjuBiHgDCDsUYACBJKq1p0gMfbNUZo9I1YVAvp+MAQLejGAMAJEn3vpunJq9fP5813OkoAOAIijEAQNvK6vT0kgLNmTRAWWkJTscBAEdQjAEA+uuiPEW5InTdaUOdjgIAjqEYA0CYW1dUpVdXF+uHJw5W78QYp+MAgGM6VIyNMbOMMbnGmC3GmJsPMuYCY8wGY8x6Y8xTnRsTANBV7no7V8mxbl1+UpbTUQDAUZGHGmCMcUm6X9JMSYWSlhpjFlhrN7QZky3pFkknWGv3GmN6d1VgAEDn+Ty/XB/kleqW2cOVHMulnwGEt47MGE+StMVam2+tbZb0jKRz2425XNL91tq9kmSt3dO5MQEAnc1aqz+/tUnpSdG65PhBTscBAMd1pBj3l1TQZruw9bm2ciTlGGM+McZ8boyZ1VkBAQBd452Ne7RiZ6V+OiNHMW6X03EAwHGHXEoh6UCXPrIHeJ1sSdMlZUj6yBhzjLW28ksvZMwVkq6QpAEDBhx2WABA5/D5re56e5OyUuN1/vgMp+MAQEDoyIxxoaTMNtsZkooPMOYVa63HWrtNUq5aivKXWGsftNZOsNZOSEtLO9LMAICj9PLKIuWV1OrG04cp0sUJigBA6lgxXiop2xgz2BgTJelCSQvajXlZ0imSZIxJVcvSivzODAoA6BxNXp/+uihPo/sna/YxfZyOAwAB45DF2FrrlXStpLclbZQ031q73hhzhzHmnNZhb0sqN8ZskLRY0v9Ya8u7KjQA4Mg9/cVOFVU26KZZwxQRcaDVcgAQnjqyxljW2jckvdHuudvaPLaSbmi9AQACVG2TV39/b4umZqXoxKGpTscBgIDCwjIACCMPf7xN5XXNumnWMBnDbDEAtEUxBoAwUVHXrAc/zNcZo9I1bkBPp+MAQMChGANAmPjn4i2qb/bqZ6cPczoKAAQkijEAhIHiygY99vkOfee4DGWnJzodBwACEsUYAMLA39/bLFnpJzO+cop5AEArijEAhLgd5XV6blmh5kzKVEbPOKfjAEDAohgDQIi7993NckUYXXPKUKejAEBAoxgDQAjbsqdWL68s0iXHD1LvpBin4wBAQKMYA0AI+9s7eYpxu3TlSVlORwGAgEcxBoAQtXFXtV5bs0uXnjBYKQnRTscBgIBHMQaAEPXXRXlKjInU5dOYLQaAjqAYA0AIWlNYqUUbSnT5tCwlx7mdjgMAQYFiDAAh6O6FeeoZ59YPThjkdBQACBoUYwAIMcu2V+iDvFJdefIQJcYwWwwAHUUxBoAQc/fCPKUmROviqQOdjgIAQYViDAAh5NMtZfosv1zXnDJEcVGRTscBgKBCMQaAEGGt1V8W5qpvcozmTBrgdBwACDoUYwAIEe/nlWrFzkpde+pQxbhdTscBgKBDMQaAEGCt1d0Lc5XZK1bnj890Og4ABCWKMQCEgLfXl2hdUbV+fGq2oiJ5aweAI8G7JwAEOb/f6p5FecpKjde3xvV3Og4ABC2KMQAEudfW7lJuSY1+OjNHkS7e1gHgSPEOCgBBzOvz62+L8jQsPVFnj+7rdBwACGoUYwAIYi+vKlZ+WZ2un5mjiAjjdBwACGoUYwAIUs1ev+59N0/H9E/SGaPSnY4DAEGPYgwAQeq55QUqqGjQjTOHyRhmiwHgaFGMASAINXp8+sd7W3TcgB6aPizN6TgAEBIoxgAQhJ5eslO7qhp14+nMFgNAZ6EYA0CQaWj26f7FWzUlq5eOH5LidBwACBkUYwAIMo99tl1ltU3MFgNAJ6MYA0AQqWn06P8+2KqTctI0cVAvp+MAQEihGANAEHnkk+3aW+/RjTNznI4CACGHYgwAQaKq3qN/fZSvmSPTdWxmD6fjAEDIoRgDQJA+E0lLAAAe/ElEQVT410f5qmn06gZmiwGgS1CMASAIlNc26eFPtumsMX01om+S03EAICRRjAEgCDzwYb4aPT5dPyPb6SgAELIoxgAQ4PZUN+rRT7frm2P7a2jvRKfjAEDIohgDQIC7f/EWef1WP2G2GAC6FMUYAAJYUWWDnl5SoPPHZ2hgSrzTcQAgpFGMASCA/eO9zZKk605jthgAuhrFGAAC1I7yOs1fVqg5kzLVv0es03EAIORRjAEgQN377mZFRhhdc8pQp6MAQFigGANAANqyp0YvryzSxVMHqndSjNNxACAsUIwBIADd885mxbhduurkIU5HAYCwQTEGgACzcVe1Xl+zSz84YZBSEqKdjgMAYYNiDAAB5q+L8pQYE6krpjFbDADdiWIMAAFkdUGlFm0o0eXTspQc53Y6DgCEFYoxAASQvy7KU484t35wwiCnowBA2KEYA0CAWLq9Qh/kleqqk4coMYbZYgDobhRjAAgQdy/MVWpCtC6eOtDpKAAQlijGABAAPt1Sps/zK/Sj6UMUFxXpdBwACEsUYwBwmLVWf1mYqz5JMbpo8gCn4wBA2KIYA4DD3s8t1YqdlbrutKGKcbucjgMAYYtiDAAOstbq7kW5yugZq/PHZzodBwDCGsUYABz09voSrSuq1k9Oy1ZUJG/JAOAk3oUBwCF+v9U9i/KUlRqvb43r73QcAAh7FGMAcMhra3cpt6RGP5mRrUgXb8cA4DTeiQHAAV6fX39blKdh6Yn6xph+TscBAIhiDACOeGllkfLL6nT9zBxFRBin4wAARDEGgG7X7PXrvvc265j+STpjVLrTcQAArSjGANDNnlteoIKKBt04c5iMYbYYAAIFxRgAulGjx6f73t2s4wb00PRhaU7HAQC0QTEGgG706KfbVVLdpJtmDWe2GAACDMUYALpJdaNH//vBVp2Uk6YpWSlOxwEAtEMxBoBu8tCH+aqs9+imM4Y5HQUAcAAUYwDoBmW1TXro4206a3RfHdM/2ek4AIADoBgDQDe4f/EWNXn9uuH0HKejAAAOgmIMAF2scG+9nvx8p847LkND0hKcjgMAOAiKMQB0sXvf2SwZ6Sczsp2OAgD4GhRjAOhCW/bU6IUVhZo7ZaD69Yh1Og4A4Gt0qBgbY2YZY3KNMVuMMTd/zbjzjDHWGDOh8yICQPC6e2GeYt0u/Wj6EKejAAAO4ZDF2BjjknS/pNmSRkqaY4wZeYBxiZJ+LOmLzg4JAMFoTWGl3ly3W5dNy1JKQrTTcQAAh9CRGeNJkrZYa/Ottc2SnpF07gHG/VbSnyU1dmI+AAhad72dq55xbl02bbDTUQAAHdCRYtxfUkGb7cLW5/YzxoyTlGmtfa0TswFA0Pp0a5k+2lyma04ZqsQYt9NxAAAd0JFibA7wnN2/05gISfdIuvGQL2TMFcaYZcaYZaWlpR1PCQBBxFqrP7+Vq77JMfr+lIFOxwEAdFBHinGhpMw22xmSittsJ0o6RtL7xpjtkqZIWnCgL+BZax+01k6w1k5IS0s78tQAEMAWbSjRqoJK/eS0bMW4XU7HAQB0UEeK8VJJ2caYwcaYKEkXSlqwb6e1tspam2qtHWStHSTpc0nnWGuXdUliAAhgPr/VXxbmanBqvM4bn+F0HADAYThkMbbWeiVdK+ltSRslzbfWrjfG3GGMOaerAwJAMHlpZZHySmp1w8wcRbo4VTwABJPIjgyy1r4h6Y12z912kLHTjz4WAASfRo9Pdy/M1bEZyTprdF+n4wAADhPTGQDQSf7z6XbtqmrUzbNHKCLiQN9bBgAEMooxAHSCvXXNun/xFp06vLemDklxOg4A4AhQjAGgE9y/eIvqmrz6+azhTkcBABwhijEAHKWCino99tkOnTc+Q8P6JDodBwBwhCjGAHCU7l6Yq4gI6fqZOU5HAQAcBYoxAByFdUVVenlVsS49YbD6Jsc6HQcAcBQoxgBwFO58c5N6xrl11fQhTkcBABwlijEAHKEP80r18ZYyXXdqtpJi3E7HAQAcJYoxABwBv9/qj29u0oBecfr+lIFOxwEAdAKKMQAcgZdXFWnjrmr97IxhiorkrRQAQgHv5gBwmFou/ZynMRnJOptLPwNAyKAYA8Bh+s+n21VU2aCbZw/n0s8AEEIoxgBwGMpqm3T/e1t02vDeOn5IqtNxAACdiGIMAIfhnkV5avD49IuzRjgdBQDQySjGANBBubtr9PSSnfr+lIEakpbgdBwAQCejGANAB1hr9bvXNygxxq2fnJbtdBwAQBegGANAB7yfW6qPNpfpx6dlq2d8lNNxAABdgGIMAIfg8fn1u9c3aHBqvOZyMQ8ACFkUYwA4hKeX7NTW0jrdMns4F/MAgBDGOzwAfI2qeo/uWZSnqVkpmjky3ek4AIAuRDEGgK/x9/c2q7LBo1vPHiFjuJgHAIQyijEAHMS2sjo9+tl2XTA+U6P6JTsdBwDQxSjGAHAQv399o9yuCN14eo7TUQAA3YBiDAAH8H7uHr2zsUTXnZqt3kkxTscBAHQDijEAtNPs9euOV1tOz3bpiYOcjgMA6CYUYwBo55FPtim/rE63fWOkoiNdTscBAHQTijEAtFFS3aj73t2s04b31inDejsdBwDQjSjGANDGnW9uksdn9auzRzodBQDQzSjGANBq2fYKvbSySJefNFiDUuOdjgMA6GYUYwCQ5PNb/XrBevVNjtE1pwx1Og4AwAEUYwCQ9MzSnVpfXK1fnDlCcVGRTscBADiAYgwg7FXWN+svb+dq8uBeOntMX6fjAAAcQjEGEPb+sjBXVQ0e3X7OKBljnI4DAHAIxRhAWFtVUKknv9ipi6cO0oi+SU7HAQA4iGIMIGx5fX798qW1SkuI1o2n5zgdBwDgMIoxgLD1+Oc7tL64Wrd9Y6QSY9xOxwEAOIxiDCAslVQ36u6FeTopJ01njeYLdwAAijGAMHXHaxvU7PPrDr5wBwBoRTEGEHY+yCvV62t26dpThnKFOwDAfhRjAGGl0ePTba+sU1ZqvK48OcvpOACAAMLlnQCElX8u3qId5fV66rLJio50OR0HABBAmDEGEDa27KnV/32Qr2+O7afjh6Y6HQcAEGAoxgDCgt9vdfMLaxQb5dIvzxrpdBwAQACiGAMIC098sUPLduzVr84eqbTEaKfjAAACEMUYQMgrqmzQn97cpGnZqfrOcf2djgMACFAUYwAhzVqrW19aK7+V/vCt0ZyzGABwUBRjACFtwepiLc4t1c/OGKbMXnFOxwEABDCKMYCQVVHXrN+8ukFjM3to3vGDnI4DAAhwFGMAIeuOV9erptGjP583Rq4IllAAAL4exRhASFq8aY9eXlWsH00fqpz0RKfjAACCAMUYQMipqvfo5hfXKLt3gn50yhCn4wAAggSXhAYQcn7z6nqV1TbroYsnctlnAECHMWMMIKS8tW63XlxZpGtPGarRGclOxwEABBGKMYCQUV7bpF++tFaj+iXp2lOHOh0HABBkWEoBICRYa/XLl9apptGrpy4fK7eLf/cDAA4PnxwAQsKC1cV6a/1u3XB6job14SwUAIDDRzEGEPRKqhv1q5fX6bgBPXT5tCyn4wAAghTFGEBQs9bq5y+sUbPPr7svGMuFPAAAR4xiDCCoPfH5Dr2fW6pbZo/Q4NR4p+MAAIIYxRhA0MorqdHvXt+o6cPSdPHUgU7HAQAEOYoxgKDU6PHpx0+vVGJMpO4671gZwxIKAMDR4XRtAILSn97apE27a/TIvIlKS4x2Og4AIAQwYwwg6CzO3aNHPtmueccP0inDezsdBwAQIijGAIJKaU2T/ue51RqWnqibZw93Og4AIISwlAJA0PD7rW56frWqG7168rIpinG7nI4EAAghzBgDCBoPfpSvxbmluvWsEVzdDgDQ6SjGAILC0u0VuuvtXJ01uq/mTuHUbACAzkcxBhDwymubdO1TK5TRM1Z//M5oTs0GAOgSrDEGEND8fqvr56/W3nqPXrx6opJi3E5HAgCEKGaMAQS0f76/RR/mlerX3xipY/onOx0HABDCKMYAAtZnW8v110V5OufYfrpo0gCn4wAAQlyHirExZpYxJtcYs8UYc/MB9t9gjNlgjFljjHnXGMM3YwAcld1Vjbru6ZUalBKvP3ybdcUAgK53yGJsjHFJul/SbEkjJc0xxoxsN2ylpAnW2jGSnpf0584OCiB8NHl9uuqJ5Wpo9uqBueOVEM3XIQAAXa8jM8aTJG2x1uZba5slPSPp3LYDrLWLrbX1rZufS8ro3JgAwoW1Vr9+Zb1WFVTq7guOVXY65ysGAHSPjhTj/pIK2mwXtj53MD+U9ObRhAIQvp5aslPPLC3QtacM1axj+jodBwAQRjry/5MHWthnDzjQmO9LmiDp5IPsv0LSFZI0YABfpAHwZct3VOj2Bes1fViarp+Z43QcAECY6ciMcaGkzDbbGZKK2w8yxsyQ9EtJ51hrmw70QtbaB621E6y1E9LS0o4kL4AQVVLdqKueWKF+PWJ173fHyRXBl+0AAN2rI8V4qaRsY8xgY0yUpAslLWg7wBgzTtIDainFezo/JoBQ1ujx6crHl6uuyasH505QchwX8QAAdL9DFmNrrVfStZLelrRR0nxr7XpjzB3GmHNah90lKUHSc8aYVcaYBQd5OQD4Er/f6sbnVmt1YaXu+e5YDevDl+0AAM7o0DmQrLVvSHqj3XO3tXk8o5NzAQgTf3snT6+v2aVbZg/XGaP6OB0HABDGuPIdAMe8tLJQ9723RRdMyNAVJ2U5HQcAEOYoxgAcsXR7hX7+/FpNzUrR777Jle0AAM6jGAPodjvL63Xl48uV0TNW//v94xQVyVsRAMB5fBoB6FbltU265JEl8vmt/j1vonrERTkdCQAASRRjAN2orsmrS/+zVMWVDXp43gQNTo13OhIAAPtRjAF0C4/Pr6ufXKG1RVX6x0XHafzAXk5HAgDgSzp0ujYAOBrWWv38+TX6MK9Ud357tGaOTHc6EgAAX8GMMYAud+dbm/TiyiLdODNHF04a4HQcAAAOiGIMoEs9+OFWPfBBvuZOGahrTx3qdBwAAA6KYgygyzz+2Xb94Y1NOmtMX91+zijOVQwACGgUYwBdYv6yAv3qlfWaOTJdf/vuWLkiKMUAgMBGMQbQ6V5ZVaSfv7BG07JT9Y+Lxsnt4q0GABD4+LQC0KneWrdbN8xfrYmDeunBuRMUHelyOhIAAB1CMQbQad7ZUKLrnl6h0f2T9fC8iYqNohQDAIIHxRhAp3hr3S5d9cRyjeibpEcvnaSEaE6TDgAILhRjAEdtwepiXfPUSo3JSNYTl01Wcqzb6UgAABw2pnQAHJUXlhfqf55frQmDeunheROZKQYABC0+wQAcsWeX7tTNL67V1KwUPXTJBMVF8ZYCAAhefIoBOCIPfZSv372+USfnpOmBueMV4+aLdgCA4EYxBnBYrLW6881NeuDDfJ05uo/u+e5YTskGAAgJFGMAHebx+XXzC2v1wopCzZ0yULefM4or2gEAQgbFGECH1Dd7dc2TK7Q4t1Q3zMzRdacOlTGUYgBA6KAYAzik8tomXfbYMq0uqNQfvjVaF00e4HQkAAA6HcUYwNfKK6nRDx9dqj3VTfrn947TrGP6Oh0JAIAuQTEGcFAf5JXq2idXKNrt0rNXTtXYzB5ORwIAoMtQjAEc0GOfbdftC9YrJz1R/543Uf17xDodCQCALkUxBvAlHp9fv3ttgx79bIdOG95b984Zx9XsAABhgU87APvtqWnUtU+u1JLtFbrsxMG65cwRnI4NABA2KMYAJEnLd1To6idWqLrRo3svHKtzx/Z3OhIAAN2KYgyEOWutHvtsh3772gb17xmrRy+dpBF9k5yOBQBAt6MYA2GstsmrX728Ti+tLNKpw3vrngvGKjnO7XQsAAAcQTEGwtTawipd9/QK7ayo1/UzWq5kF8F6YgBAGKMYA2HG77f698fb9Oe3Nyk1IVpPXT5FU7JSnI4FAIDjKMZAGCmrbdKN81frg7xSnT4yXX/6zhj1jI9yOhYAAAGBYgyEiTfX7tKtL69TTZNXvz13lL4/ZaCMYekEAAD7UIyBELe3rlm/XrBeC1YX65j+SXrq/LEa1ifR6VgAAAQcijEQwt7ZUKJbXlqrvXXNumFmjq6ePkRuV4TTsQAACEgUYyAEldU26XevbdDLq4o1vE+i/vODiRrVL9npWAAABDSKMRBC/H6r+csK9Mc3N6m+2asfn5ata08ZqqhIZokBADgUijEQIvJKavSLF9dq2Y69mjS4l/7wrdEa2jvB6VgAAAQNijEQ5KobPfrHe1v08MfblBgTqbvOG6PzxmdwxgkAAA4TxRgIUj6/1bNLC3T3wlxV1Dfr/PEZunn2CPXivMQAABwRijEQhD7dWqY7Xt2gTbtrNHFQT/3n7EkancGX6wAAOBoUYyCIbC6p0V1v52rhhhL17xGr+y86TmeO7sOyCQAAOgHFGAgCO8vr9bd38vTSqiLFR0XqZ6fn6LJpWYpxu5yOBgBAyKAYAwGspLpRf39vs55ZUiBXhNEV07J01clD1JN1xAAAdDqKMRCAiisb9OCH+Xp6yU75/FYXTsrUdadmKz0pxuloAACELIoxEEDyS2v1fx9s1Usri2St9M1x/fXjU7M1ICXO6WgAAIQ8ijEQANYVVel/P9iqN9buUpQrQhdNGqDLT8pSRk8KMQAA3YViDDjE6/Nr0YYSPfLJdi3ZXqGE6EhddfIQXXrCYKUlRjsdDwCAsEMxBrpZZX2znl1aoMc+26GiygZl9IzVL88coQsmZio51u10PAAAwhbFGOgG1lqt2FmpZ5fu1Kurd6nB49OUrF667RsjNWNEulwRnIcYAACnUYyBLlRR16wXVxTq2aUF2rynVnFRLp07tp8unjpII/slOR0PAAC0QTEGOpnH59fHm8v0wopCLVxfomafX+MG9NCfvjNaZ43pp4Ro/toBABCI+IQGOoHfb7V85169sqpIr6/Zpb31HvWIc+t7UwbowokDNKxPotMRAQDAIVCMgSNkrdX64mq9tmaXXl1drKLKBsW4IzRzZB+de2w/nZSTpqjICKdjAgCADqIYA4fB4/Nr6bYKLdxQooXrd6u4qlGuCKOTslP1P2cM08yR6YpnqQQAAEGJT3DgEGoaPfpkS5kWri/Ru5v2qKrBo+jICJ2Uk6brZ+botBHp6hUf5XRMAABwlCjGQDt+v9W64ip9mFeqD/PKtGLnXnn9Vj3i3JoxIl2nj0rXtOxUxUXx1wcAgFDCJzvCnrVWBRUN+nxbuT7eXKaPt5Spoq5ZknRM/yRdcVKWTspJ04SBPRXpYs0wAAChimKMsGOt1dbSWn2xrUJLWm+7qholSakJ0Zo+LE0nZafpxOxUpSZwaWYAAMIFxRghr67Jq3VFVVpdWKmVOyu1ZFuFyltnhNMSozV5cC9NHtxLkwanKLt3giK4Ch0AAGGJYoyQ4vX5lVdSq9WFlVpdUKlVBZXKK6mR37bsz+gZq5Nz0jQ5q6UID0qJkzEUYQAAQDFGEKtq8GjTrmpt3FWtTbtrtHFXtXJLatTo8UuSesS5dWxGD50+qo/GZiZrTEYPlkYAAICDohgj4NU3e5VfWqf8sjrltRbgTbtrVFTZsH9Mzzi3RvRN0kWTBurYzGQdm9FDA5kNBgAAh4FijIDg91vtrm7U1tJa5ZfW7b/PL61VcesX4yTJFWGUlRqv8QN76ntTBmhE3ySN6JOk9KRoSjAAADgqFGN0C2utqho8KqhoUMHeehXurd//uKCiXoV7G9Tk9e8fnxAdqSFp8ZqclaKs1HgN6Z2grLR4DUqJV4zb5eBvAgAAQhXFGEfNWqvqBq92Vzdqd3WjSqoaVbLvcXWjiiobVVhRr5om75d+LjnWrYyescrunahTh/fWwJR4DUlL0JC0eKUlMgMMAAC6F8UYB1Xf7FV5bbPKaptUUdes8tpmldc1q7y2SaW1TdrdpgDv+8JbW73io5SeFKO+yTGaNKinMnvFKaNnnDJ7xSqzV5ySYtwO/FYAAAAHRjEOAx6fX9UNHlW1u7V9bm+9R+WtBbistlnldU0HLLuSFOOOUGpCtPokxWhU/2SdNiJdfZJilJ4coz6tRTgtMZolDwAAIKh0qBgbY2ZJuleSS9JD1to72+2PlvSYpPGSyiV911q7vXOjhhdrrRo9ftU2eVXf7FVtk1d1TT7VNXlV1+xVXZNXte2265p8qm3y7i+8++7rmn1fe6wYd4R6xkUpJSFKveKjNSQtYf/jlIQopcRHKSUhuvU+SnFR/HsKAACEnkM2HGOMS9L9kmZKKpS01BizwFq7oc2wH0raa60daoy5UNKfJH23KwJ3N5/fyuPzq9nnV7PXL4/PL4/Xfnm7dX+T168mj08NHp8aPX41tt43eHxq8vi+tN3o8anRu2/Mf/c1tv58XZN3/0UpDiUqMkIJ0ZGKj3YpPipSSbFuZfaKU3Ks+yu3pNjINo9b7qMjmdkFAADoyNTfJElbrLX5kmSMeUbSuZLaFuNzJd3e+vh5Sf8wxhhrbQerXffYuKtaf3xzk5q9Pnl8rYXX+99i6/F+tQR3tJweSnRkhGLcLsW4W+5j3S5Fu12KiWyZrY1tsy/G7VJCdKTiolvu46MiFR8d+eXnoiOVENWy7XZFdE5IAACAMNaRYtxfUkGb7UJJkw82xlrrNcZUSUqRVNYZITuL31pVN3gU5YpQjDtCSTGRcrsi5I6MUJQrQm6XUVRkhNyufdsR+7e/si8yQlEu86UxUZERiol0KTaqteRGtpTc6MgIRURwhgUAAIBA1pFifKBG134etSNjZIy5QtIVkjRgwIAOHLpzjeqXrJevOaHbjwsAAIDA15H/gy+UlNlmO0NS8cHGGGMiJSVLqmj/QtbaB621E6y1E9LS0o4sMYD/b+/eQi6b4zCOf5+GoUgUUmYyo5xPESKRc06NG2oUTVwRE0WOceFKyKG4kRkXTEmOk8YxcmcYZ2PQpDAOGReixDT5uVjLnOyZcWHe/2J9P1d7r71rP2+/9ruetda/tSVJ0jbwT4rx28B+SWYnmQ7MBRZv8p7FwLz+8QXAa0NbXyxJkiRtyVaXUvRrhq8CXqK7XdvCqlqe5HZgWVUtBhYAjyZZSXemeO62DC1JkiT92/7RDWmragmwZJNtt23w+Dfgwn83miRJkjR1vM+XJEmShMVYkiRJAizGkiRJEmAxliRJkgCLsSRJkgRYjCVJkiTAYixJkiQBFmNJkiQJsBhLkiRJgMVYkiRJAizGkiRJEmAxliRJkgCLsSRJkgRYjCVJkiQAUlVtPjhZDXzZ5MNhd+DHRp+tyZzJ8DiTYXIuw+NMhsm5DE/LmexTVXts7U3NinFLSZZV1dGtc2g9ZzI8zmSYnMvwOJNhci7D81+YiUspJEmSJCzGkiRJEjDeYvxQ6wD6G2cyPM5kmJzL8DiTYXIuwzP4mYxyjbEkSZK0qbGeMZYkSZI2MtpinGR+ks+SLE9yZ+s8Wi/JdUkqye6ts4xdkruSfJrkwyTPJNm1daaxSnJW/z9rZZIbW+cRJJmZ5PUkK/p9ydWtM6mTZFqS95I83zqLOkl2TfJkv09ZkeT41pkmGWUxTnIKcD5weFUdAtzdOJJ6SWYCZwBftc4iAF4BDq2qw4HPgZsa5xmlJNOAB4GzgYOBi5Ic3DaVgLXAtVV1EHAccKVzGYyrgRWtQ2gj9wMvVtWBwBEMdD6jLMbAFcAdVfU7QFX90DiP1rsXuB5w8fsAVNXLVbW2f/omMKNlnhE7FlhZVV9U1RrgcbqDezVUVd9V1bv941/odvR7t02lJDOAc4GHW2dRJ8kuwEnAAoCqWlNVP7VNNdlYi/H+wIlJliZ5I8kxrQMJkswBvqmqD1pn0USXAS+0DjFSewNfb/B8FRawQUkyCzgSWNo2iYD76E6w/NE6iNbZF1gNPNIvcXk4yU6tQ02yXesA20qSV4G9Jrx0C93fvRvdpa9jgCeS7FveomOb28pcbgbOnNpE2tJMquq5/j230F02XjSV2bROJmzz/9VAJNkZeAq4pqp+bp1nzJKcB/xQVe8kObl1Hq2zHXAUML+qlia5H7gRuLVtrL/73xbjqjp9c68luQJ4ui/CbyX5g+73u1dPVb6x2txckhwGzAY+SALdJft3kxxbVd9PYcTR2dJ3BSDJPOA84DQPHptZBczc4PkM4NtGWbSBJNvTleJFVfV06zziBGBOknOAHYFdkjxWVRc3zjV2q4BVVfXXFZUn6Yrx4Ix1KcWzwKkASfYHpgM/Nk00clX1UVXtWVWzqmoW3ZfoKEtxW0nOAm4A5lTVr63zjNjbwH5JZieZDswFFjfONHrpjuIXACuq6p7WeQRVdVNVzej3I3OB1yzF7fX78q+THNBvOg34pGGkzfrfnjHeioXAwiQfA2uAeZ4JkyZ6ANgBeKU/k/9mVV3eNtL4VNXaJFcBLwHTgIVVtbxxLHVnJy8BPkryfr/t5qpa0jCTNFTzgUX9wf0XwKWN80zkL99JkiRJjHcphSRJkrQRi7EkSZKExViSJEkCLMaSJEkSYDGWJEmSAIuxJEmSBFiMJUmSJMBiLEmSJAHwJ45I34xw1CZaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.linspace(-6, 6, num = 1000)\n",
    "plt.figure(figsize = (12,8))\n",
    "plt.plot(x, 1 / (1 + np.exp(-x))); # Sigmoid Function\n",
    "plt.title(\"Sigmoid Function\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The logistic regression equation has a very simiar representation like linear regression. The difference is that the output value being modelled is binary in nature.\n",
    "\n",
    "$$\\hat{y}=\\frac{e^{\\beta_0+\\beta_1x_1}}{1+\\beta_0+\\beta_1x_1}$$\n",
    "\n",
    "or\n",
    "\n",
    "$$\\hat{y}=\\frac{1.0}{1.0+e^{-\\beta_0-\\beta_1x_1}}$$\n",
    "\n",
    "$\\beta_0$ is the intecept term\n",
    "\n",
    "$\\beta_1$ is the coefficient for $x_1$\n",
    "\n",
    "$\\hat{y}$ is the predicted output with real value between 0 and 1. To convert this to binary output of 0 or 1, this would either need to be rounded to an integer value or a cutoff point be provided to specify the class segregation point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "tmp = [0, 0.4, 0.6, 0.8, 1.0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 0.4, 0.6, 0.8, 1.0]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 1., 1., 1.])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(tmp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False, False,  True,  True])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(tmp) > 0.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making Predictions with Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\hat{y}=\\frac{1.0}{1.0+e^{-\\beta_0-\\beta_1x_i}}$$\n",
    "\n",
    "$\\beta_0$ is the intecept term\n",
    "\n",
    "$\\beta_1$ is the coefficient for $x_i$\n",
    "\n",
    "$\\hat{y}$ is the predicted output with real value between 0 and 1. To convert this to binary output of 0 or 1, this would either need to be rounded to an integer value or a cutoff point be provided to specify the class segregation point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = [[-2.0011, 0],\n",
    "           [-1.4654, 0],\n",
    "           [0.0965, 0],\n",
    "           [1.3881, 0],\n",
    "           [3.0641, 0],\n",
    "           [7.6275, 1],\n",
    "           [5.3324, 1],\n",
    "           [6.9225, 1],\n",
    "           [8.6754, 1],\n",
    "           [7.6737, 1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say you have been provided with the coefficient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "coef = [-0.806605464, 0.2573316]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "yhat 0.2106, yhat 0.0\n",
      "yhat 0.2344, yhat 0.0\n",
      "yhat 0.3139, yhat 0.0\n",
      "yhat 0.3895, yhat 0.0\n",
      "yhat 0.4955, yhat 0.0\n",
      "yhat 0.7606, yhat 1.0\n",
      "yhat 0.6377, yhat 1.0\n",
      "yhat 0.7261, yhat 1.0\n",
      "yhat 0.8063, yhat 1.0\n",
      "yhat 0.7628, yhat 1.0\n"
     ]
    }
   ],
   "source": [
    "for row in dataset:\n",
    "    yhat = 1.0 / (1.0 + np.exp(- coef[0] - coef[1] * row[0]))\n",
    "    print(\"yhat {0:.4f}, yhat {1}\".format(yhat, round(yhat)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning the Logistic Regression Model\n",
    "\n",
    "The coefficients (Beta values b) of the logistic regression algorithm must be estimated from your training data. \n",
    "\n",
    "* Generally done using [maximum-likelihood estimation](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation).\n",
    "\n",
    "* Maximum-likelihood estimation is a common learning algorithm\n",
    "\n",
    "* Note the underlying assumptions about the distribution of your data\n",
    "\n",
    "* The best coefficients would result in a model that would predict a value very close to 1 (e.g. male) for the default class and a value very close to 0 (e.g. female) for the other class. \n",
    "\n",
    "* The intuition for maximum-likelihood for logistic regression is that a search procedure seeks values for the coefficients (Beta values) that minimize the error in the probabilities predicted by the model to those in the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning with Stochastic Gradient Descent\n",
    "\n",
    "Logistic Regression uses gradient descent to update the coefficients.\n",
    "\n",
    "Each gradient descent iteration, the coefficients are updated using the equation:\n",
    "\n",
    "$$\\beta=\\beta+\\textrm{learning rate}\\times (y-\\hat{y}) \\times \\hat{y} \\times (1-\\hat{y}) \\times x $$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Scikit Learn to Estimate Coefficients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[-2.0011, 0],\n",
       " [-1.4654, 0],\n",
       " [0.0965, 0],\n",
       " [1.3881, 0],\n",
       " [3.0641, 0],\n",
       " [7.6275, 1],\n",
       " [5.3324, 1],\n",
       " [6.9225, 1],\n",
       " [8.6754, 1],\n",
       " [7.6737, 1]]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.array(dataset)[:, 0:1]\n",
    "y = np.array(dataset)[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-2.0011],\n",
       "       [-1.4654],\n",
       "       [ 0.0965],\n",
       "       [ 1.3881],\n",
       "       [ 3.0641],\n",
       "       [ 7.6275],\n",
       "       [ 5.3324],\n",
       "       [ 6.9225],\n",
       "       [ 8.6754],\n",
       "       [ 7.6737]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 1., 1., 1., 1., 1.])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf_LR = LogisticRegression(C=1.0, penalty='l2', tol=0.0001, solver=\"lbfgs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
       "                   multi_class='warn', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_LR.fit(X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 1., 1., 1., 1., 1.])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_LR.predict(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.99853453, 0.00146547],\n",
       "       [0.99740804, 0.00259196],\n",
       "       [0.98643844, 0.01356156],\n",
       "       [0.94830288, 0.05169712],\n",
       "       [0.75430207, 0.24569793],\n",
       "       [0.02307807, 0.97692193],\n",
       "       [0.21456501, 0.78543499],\n",
       "       [0.04771639, 0.95228361],\n",
       "       [0.00766657, 0.99233343],\n",
       "       [0.02199284, 0.97800716]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_LR.predict_proba(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Further Resources\n",
    "\n",
    "[A comparison of numerical optimizers for logistic regression](https://tminka.github.io/papers/logreg/)\n",
    "\n",
    "[PDF: A comparison of numerical optimizers for logistic regression](https://tminka.github.io/papers/logreg/minka-logreg.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification Exercise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset2 = [[ 0.2,  0. ],\n",
    "            [ 0.2,  0. ],\n",
    "            [ 0.2,  0. ],\n",
    "            [ 0.2,  0. ],\n",
    "            [ 0.2,  0. ],\n",
    "            [ 0.4,  0. ],\n",
    "            [ 0.3,  0. ],\n",
    "            [ 0.2,  0. ],\n",
    "            [ 0.2,  0. ],\n",
    "            [ 0.1,  0. ],\n",
    "            [ 1.4,  1. ],\n",
    "            [ 1.5,  1. ],\n",
    "            [ 1.5,  1. ],\n",
    "            [ 1.3,  1. ],\n",
    "            [ 1.5,  1. ],\n",
    "            [ 1.3,  1. ],\n",
    "            [ 1.6,  1. ],\n",
    "            [ 1. ,  1. ],\n",
    "            [ 1.3,  1. ],\n",
    "            [ 1.4,  1. ]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "X = np.array(dataset2)[:, 0:1]\n",
    "y = np.array(dataset2)[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true,
     "source_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
       "                   multi_class='warn', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_LR = LogisticRegression(C=1.0, penalty='l2', tol=0.0001, solver='lbfgs')\n",
    "\n",
    "clf_LR.fit(X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true,
     "source_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1.])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = clf_LR.predict(X)\n",
    "clf_LR.predict(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true,
     "source_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0.],\n",
       "       [0., 0.],\n",
       "       [0., 0.],\n",
       "       [0., 0.],\n",
       "       [0., 0.],\n",
       "       [0., 0.],\n",
       "       [0., 0.],\n",
       "       [0., 0.],\n",
       "       [0., 0.],\n",
       "       [0., 0.],\n",
       "       [1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.column_stack((y_pred, y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_LR.predict(np.array([0.9]).reshape(1,-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}