3_logistic_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/samdavanloo/ISE-ML/blob/main/3_logistic_regression.ipynb)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification: Logistic regression\n",
    "Logistic regression is a classification model that is very easy to implement but performs very well on linearly separable classes. It is one of the most widely used algorithms for classification in industry. Logistic regression is a linear model for binary classification that can be extended to multiclass classification.\n",
    "\n",
    "The \"odds ratio\" of an event (class label $y=1$) is defined as $p/(1-p)$ where $p$ is the probability of the event.\n",
    "\n",
    "The \"logit function\" is then the natural logarithm of the odds ration:\n",
    "$$\n",
    "logit(p) = \\log\\frac{p}{1-p}\n",
    "$$\n",
    "The logit function takes as input values in the range of 0 to 1 and transforms them to values over the entire real-number range, which can then be modeled by a linear function:\n",
    "$$\n",
    "logit(p(y=1|\\mathbf{x})) = w_0+w_1x_1+\\cdots+w_px_p=w_0+\\mathbf{w}^T\\mathbf{x}.\n",
    "$$\n",
    "Note taht we are actually interested in predicting the \"probability\" that a certain sample belongs to a particular class, which is the inverse form of the logit function. It is also called logistic \"sigmoid function\" and defined as\n",
    "$$\n",
    "\\phi(z) = \\frac{1}{1+e^{-z}},\n",
    "$$\n",
    "where \n",
    "$$\n",
    "z=w_0+\\mathbf{w}^T\\mathbf{x}.\n",
    "$$ \n",
    "\n",
    "Given some \"threshold\", the predicted probability can simply be converted into a binary outcome via the function:\n",
    "$$\n",
    "\\hat{y} =\n",
    "\\begin{cases}\n",
    "1 \\quad \\text{if}\\ \\phi(z)\\geq \\text{threshold} \\\\\n",
    "0 \\quad \\text{otherwise}\n",
    "\\end{cases}\n",
    "$$\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning (estimating) the parameters\n",
    "Parameters of the logistic regression model are generally estimated by maximizing the \"likelihood\" function. Assuming that the individual samples in our dataset are independent of one another, the likelihood function can be written as\n",
    "$$\n",
    "L(\\mathbf{w},w_0) = \\Pi_{i=1}^n p(y_i|\\mathbf{x}_i) = \\Pi_{i=1}^n \\phi(z(\\mathbf{x}_i;\\mathbf{w},w_0))^{y_i} (1-\\phi(z(\\mathbf{x}_i;\\mathbf{w},w_0)))^{1-y_i},\n",
    "$$\n",
    "or log-likelihood which has the form\n",
    "$$\n",
    "l(\\mathbf{w},w_0) = \\log L(\\mathbf{w},w_0) = \\sum_{i=1}^n \\Big(y_i\\log(\\phi(z(\\mathbf{x}_i;\\mathbf{w},w_0)))+(1-y_i)\\log(1-\\phi(z(\\mathbf{x}_i;\\mathbf{w},w_0)))\\Big).\n",
    "$$\n",
    "Instead of maximizing log-likelihood, it is a common practice to minimize negative log-likelihood as\n",
    "$$\n",
    "\\min_{\\mathbf{w}\\in \\mathbb{R}^p, w_0\\in \\mathbb{R}}\\ \\cal{L}(\\mathbf{w},w_0) \\triangleq \\sum_{i=1}^n \\Big(-y_i\\log(\\phi(z(\\mathbf{x}_i;\\mathbf{w},w_0)))-(1-y_i)\\log(1-\\phi(z(\\mathbf{x}_i;\\mathbf{w},w_0)))\\Big).\n",
    "$$\n",
    "\n",
    "Note that with $n=1$ the objective function is equivalent to\n",
    "$$\n",
    "\\cal{L}(\\mathbf{w},w_0) = \n",
    "\\begin{cases}\n",
    "-\\log(\\phi(z(\\mathbf{x};\\mathbf{w},w_0))) \\ \\ \\quad \\quad \\text{if} \\ \\  y=1 \\\\\n",
    "-\\log(1-\\phi(z(\\mathbf{x};\\mathbf{w},w_0))) \\quad \\text{if} \\ \\ y=0.\n",
    "\\end{cases}\n",
    "$$\n",
    "Let's quickly check the optimization objective for $n=1$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEKCAYAAADw2zkCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA1eElEQVR4nO3dd5iU1dn48e/ZOtt7Yyvs0pe+IIKAYC9RsRsktoQYk7yJJm+a6fn5vr5pJppEQ9RYsKCxxhCMhWYBpJelLSwLy+6yvfeZ8/vjDAsouG1mnpnZ+3Ndc83uzDPPufcRn3tOV1prhBBCDG0BVgcghBDCepIMhBBCSDIQQgghyUAIIQSSDIQQQgBBVgdwNvEJibolJI702DDiI0KsDkcIIbxTSyU0HIPUiWzeuq1aa500kNN4LBkopQ4DTYAd6NZaF3ze8dnZ2dRd/Et+fOU47jxvuCdCFEII37Pm17DqAfjJRlRQSMlAT+PpmsF8rXV1Xw4MUAqA1s5utwYkhBA+rbMZAkMhMHhQp/HaPgOlIDhQ0dJptzoUIYTwXgkjYeyVgz6NJ5OBBv6jlNqslFrSlw+EhwTR0iE1AyGEOKupi+H6Jwd9Gk82E83WWpcppZKBd5RSe7XWa089wJkklgBkZWWRFRZEU7skA3/U1dVFaWkp7e3tVocyZNhsNjIyMggOHlxzgvBPHksGWusy53OlUuo1YAaw9lPHLAWWAhQUFOhoWzCNbV2eClF4UGlpKVFRUeTk5KCc/UPCfbTW1NTUUFpayvDhMiDDryy7HoJtcNOyQZ3GI81ESqkIpVTUiZ+Bi4FdvX0uyhZEY7skA3/U3t5OQkKCJAIPUUqRkJAgNTF/1FQBjsH3rXqqzyAF+EAptR3YCPxLa72ytw9F24KlmciPSSLwLLnefqq9AWwxgz6NR5qJtNaHgEn9/Vx0mDQTCSHE52pvgNDoQZ/Ga4eWwolmIqkZCCHEGTkc0NHokpqBVyeDaFswzR3d2B2yAY8QQnyGo8sMLU2fOuhTeXcyCDND4JqldiA8rK2tjXnz5mG3961jrrOzk7lz59LdfeZ/q7NmzQLg4YcfZuzYsSxatMhlsd55550kJyeTn5/vsnMKHxEUClc9AqMvG/SpvDoZRNlMl4aMKBLutnz5cpYsWcKvf/1rAJ588kmuvfZaAgMD+/T5kJAQLrjgApYvX37G9z/66CMA/vKXv7BixQqee+451wQO3H777axc2et4DOGPHA5w0dbFXp0Mom2mZiDJQLjTww8/zL59+7jjjjt49dVXAXjuuee4+uqrASguLmbevHkAbNmyBaUUNTU12O128vPzaW1tBeCaa645600+MjKSu+++m0OHDnHVVVfx0EMPnfb+T37yE/74xz/2/H7//ffz8MMP9yn+uXPnEh8f378/WviHIx/DL+OheN2gT+W1S1gDRIc5awZt0kzkz37xz90UljW69JzjhkXzsy+M7/W41tZWfvnLX3LgwAG6u7t55pln6Ozs5NChQ+Tk5AAQGxtLU1MTAI888ggzZ86krq6ODz/8kIsuuojw8HAA8vPz+eSTT85a1mOPPcbKlStZtWoViYmJp7131113ce211/Ktb30Lh8PBiy++yMaNGwGYM2dOT/mn+u1vf8uFF17Yp+sh/FR7A2gHhEYO+lTenQykZiDcbNWqVWRnZxMXFwdAUlISZWVlxMbG9hwTExNDa2srNTU1lJeXM3v2bOrq6li6dCm///3ve44LDAwkJCSEpqYmoqKi+hVHTk4OCQkJbN26lePHjzNlyhQSEhIAWLdu8N/6hJ/qcH6JcsHQUq9OBjHODuSGVkkG/qwv3+DdZe3atVx88cWnvRYWFnbaTN2AANOa+re//Y277rqLwsJCduzYgd1uZ9SoUad9tqOjA5vNNqBYvvzlL/PUU09RUVHBnXfe2fO61AzEWbXWmuewuEGfyquTQUKk2eGspqXT4kiEv/roo4/4zne+Q1VVFUlJSTQ2NlJeXo7dbqe9vb3nxh4QEMCbb77J2rVrKS0t5be//S0PPvjgaeeqqakhKSlpwAvBLVy4kJ/+9Kd0dXXx/PPP97wuNQNxVq3VoALBFjvoU3l1B3J4SBC24ABqWzqsDkX4odWrVxMcHMwDDzxAbm4ul112GcuWLWPMmDFcfPHFfPDBBz3HhoSEcNlllxEUFER0dDQtLS1ceeXpa8ivWrWKyy+/fMDxhISEMH/+fG688cY+j2ICuOWWWzj33HPZt28fGRkZPPHEEwOOQfiYjBkw65sQ4IJbudbaKx/Tpk3TWms963/f0/cu36qFfyksLLQ6hM+1ZcsWfeutt/brMwsXLtR79+4dcJl2u11PmjRJ79+/f8Dn6I23X3cxOMAmPcB7rlfXDMA0FdU0SzOR8KwpU6Ywf/78fk06u+aaaxg9evSAyissLCQvL48LLriAkSNHDugcYghqrQW7a/pUvbrPACA+QpKBsMapnbi9CQkJ4Utf+tKAyxo3bhyHDh0a8OfFEPX4hZA2EW54atCn8vqaQXxECLXSgSyEEJ/VWg0RSS45ldcng4SIEGqkA1kIIU5n7zKTzsITez+2D7w+GcRHhNLe5aC1U2YhCyFEj9Ya8xyR4JLTeX0ySIhwzjWQfgMhhDippdo8D52agUkG0m8ghBCnCE+AC34KqRNccjqvH010YhayJAMhhDhFdBrM+Y7LTuf1NYOEiFAAqpulE1kIIXo0V0JD6dDYzwAgOdokg8omSQZCCNFj/V/gj5OHTjKwBQcSExZMRUN77wcL4SK+tO3lypUrGT16NHl5eZ9ZPE/4scZyiEp1zbpE+EAyAEiNtlHRKMlAuI+vbntpt9v5+te/zr///W8KCwt54YUXKCwsdMm5hZdrKoOoNJedzieSQUqMjeOSDISb+PK2lxs3biQvL48RI0YQEhLCzTffzBtvvNH/iyB8T2O56UR2Ea8fTQSQGh3KvgrXbosovMzfr/jsa+OvgRlfgc5WeO6Gz74/+YswZRG01MBLn1oX6I5/9alYX9/2sr6+nszMzJ7XMjIy2LBhQ5/+duHjmsohz3WbG/lIMrBR1dRBt91BUKBPVGaEj/D1bS9ffvnlz7ymlOpX2cIHORxwxe8gIc9lp/SJZJASY8Ohobq5k9SYgW0pKLzc532TDwn//PcjEvpcE/g0X9/2MiMjg6NHj/a8VlpayrBhwwZUvvAhAQEw6WaXntInkkFqtPmfq6KxXZKBcClf3/ayu7ubAwcOUFxcTHp6Oi+++OJpnxV+qrkSaovN8tXBYS45pU+0uaTFmD+2rL7N4kiEP/GHbS+DgoL405/+xCWXXMLYsWO58cYbGT9+/IBjED5i/9vw5MXQVOG6cw50izR3P05se6m11g1tnTr7+2/px1YXuWhzOGE1b99+Uba9FF7tvV9p/fNYrbs7T3sZf972EiDaFkxMWDBH61qtDkUMEbLtpfBqdSUQnQGBA2uSPBOf6DMAyIwP42itNBMJz5FtL4XXqi+BuGyXntInagYAmXHhlErNQAghoP4IxGa59JQ+VDMI5/29lWitZRy1EGJou+4JsEW79JQerRkopQKVUluVUm/197MZcWF0dDuoktVL/YZ20WqLom/kevuRnNku29TmBE83E30L2DOQD2bGmSn/0onsH2w2GzU1NXKD8hCtNTU1NQOeECe8SG0x7HoFOj47EXEwPNZMpJTKAK4AHgDu6+/nM+NNMjhS28q07HjXBic8LiMjg9LSUqqqqqwOZciw2WxkZGRYHYYYrKJ3YcV34d5CCO3fsiefx5N9Bn8AvgecNXql1BJgCUBW1umdI1nx4QQGKA5VtbgxROEpwcHBDB8+3OowhPA9NUUQHAHRrl12xCPNREqpK4FKrfXmzztOa71Ua12gtS5ISko67b2QoACy48M5WNXszlCFEMK7Ve+HxJHg4oE0nuozmA1cpZQ6DLwILFBKLevvSUYkRXCwUmoGQoghrPoAJI7q/bh+8kgy0Fr/UGudobXOAW4G3tda39rf8+QmRVJc3YLdIZ2OQoghqLMVGo66JRn4zDwDMMmg0+6gtK6V7IQIq8MRQgjPCrLBt3aYZxfz+AxkrfVqrfWVvR/5WbnJJgFIv4EQYkgKCDDLUESluP7ULj+jG41IjASQfgMhxNC079+w/lG3nNqnkkFcRAgJESEUVUrNQAgxBO1YDhsec8upfSoZAIxKiWJvRaPVYQghhOdV7IJk92xe5HPJYPywaPZWNNFtd1gdihBCeE5ni5lwljbRLaf3uWQwblg0Hd0ODlVLv4EQYgg5vhvQLl+g7gSfSwbjh8UAsLusweJIhBDCg+pKQAVAqtQMAMhNiiAkKIDCMuk3EEIMIRNvgB8egxj3LDboU5POAIICAxiTGsVuSQZCiKEmJNxtp/a5mgGYTuTdZY2yFr4QYmiwd8Oy62HfSrcV4ZPJYFxaNA1tXZQ1tFsdihBCuF/1fih6B9rd11fqk8lgQkYsANuP1lsahxBCeMQx5+r/wya7rQifTAbj0qIJDQpg0+E6q0MRQgj3O7oewuIgYaTbivDJZBASFMCkzFg2l9RaHYoQQrjfkQ2QeY5ZqM5NfDIZABRkx7G7rJG2TrvVoQghhPvYuyA2E3IXuLUY300GOXF0OzTbpN9ACOHPAoNh8WtwzlfdWozPJoOpWXEA0lQkhPBvDs+0fvhsMogND2FkciSbSqQTWQjhx57+Arx2t9uL8dlkAKapaEtJHQ7ZE1kI4Y+6O6B0E4QnuL0o304G2fE0tnezt6LJ6lCEEML1jm0GewdkzXR7UT6dDM4bmQjA2gNVFkcihBBuUPQeqEDImeP2onw6GaRE2xiTGsWafZIMhBB+qOhdyJgOYbFuL8rnVi39tHmjknjyw2JaOrqJCPX5P0cIIQytYdptZuaxB/h0zQBMMuiyaz4+WGN1KEII4TpKQcGdMH6hR4rz+WQwLSeOsOBA6TcQQviXwx9CY5nHivP5ZBAaFMis3ATW7JdkIITwEw4HvHwbvPNTjxXp88kAYN7oJEpqWjlc3WJ1KEIIMXgVO6ClCnIv8FiRfpEM5o5MAmD1vkqLIxFCCBcoesc8u3lxulP5RTLISYxgZHIkK3dXWB2KEEIM3p5/QnoBRKV4rEi/SAYAl+ansrG4lprmDqtDEUKIgWssh/LtMO4qjxbrV8nAoeGdwuNWhyKEEAMXnQb37oYpiz1arN8kg3Fp0WTFh7NilzQVCSF8XEwGhMd7tEi/SQZKKa6YmMaHRdVUS1OREMIX1R2GZdfD8UKPF+03yQDgmsnp2B2at7Z7bqKGEEK4zM6XzUii0EiPF+1XyWB0ahRjUqN4fZskAyGEj9EadrwMWedCbJbHi/dIMlBK2ZRSG5VS25VSu5VSv3BXWddMSWfb0XqKZQKaEMKXVOyA6n0w4QZLivdUzaADWKC1ngRMBi5VSrllt4ZrJqcToODlTUfdcXohhHCPLc9CYKjHFqb7NI8kA200O38Ndj7csldlaoyN+aOTeXlzKd12hzuKEEII10saDefe4/FRRCd4rM9AKRWolNoGVALvaK03nOGYJUqpTUqpTVVVA1947qbpmVQ1dbBKNr0RQviKGV+BC39uWfEeSwZaa7vWejKQAcxQSuWf4ZilWusCrXVBUlLSgMuaPyaZpKhQXtx4ZOABCyGEpxx4F7raLQ3B46OJtNb1wGrgUneVERwYwM3TM3l/XyVHalrdVYwQQgxexS547jrYuNTSMDw1mihJKRXr/DkMuBDY684yF52TTaBSPLv+sDuLEUKIwdm4FILCYMqtlobhqZpBGrBKKbUD+ATTZ/CWOwtMjbFxaX4qL35ylJaObncWJYQQA9NaCztegok3WtZxfIKnRhPt0FpP0VpP1Frna61/6Yly75idQ1N7N69uPeaJ4oQQon+2PA3dbXDOV62OxL9mIH/a1Kw4JmfG8re1h2SYqRDC+5R8DCPOh5TxVkfi38lAKcXXzs/lSG0r/9pZbnU4Qghxui8uhxuetjoKwM+TAcBFY1PIS47k0dUH0dot89yEEKJ/7N3QVgdKQVis1dEAA0gGSqkIpVSgO4Jxh4AAxd3zctlb0cR7e2SPZCGEF9j9GjyUD5V7rI6kR6/JQCkVoJT6olLqX0qpSsyQ0HLngnO/UUqNdH+Yg3P15GFkxofx0Lv7cTikdiCEsJC9G9Y8aFYmTRxtdTQ9+lIzWAXkAj8EUrXWmVrrZGAOsB54UCll7QDZXgQHBvDtC0axu6yRlbtlJzQhhIV2vAg1RTD/fgjwnpZ61Vs7ulIqWGvdNdhj+qugoEBv2rTJZeezOzQXP7QGpRRvf3sugQHKZecWQog+6e6ARwogIgG+ssr0GbiQUmqz1rpgIJ/tNS2duMkrpWy9HePNAgMU9100mqLKZl6TeQdCCCscXgcNR2DBj12eCAarP3WUT5RSv1NK5bktGje7LD+ViRkx/PbtfbR2yqxkIYSH5V0IX/sYci+wOpLP6E8ymIRZYO4hZ2fylUp5WWrrRUCA4idXjqOisZ2/rjlkdThCiKGk2TmaMWWc19UKoH/JIAbYDfwCeAX4NVDsjqDcaXpOPFdMTOOvaw9S3tBmdThCiKGgusgMJd3+otWRnFV/kkEN8CxwI5AOLAU8ssaQq/3g0jE4NDz4b7cunCqEEGaj+//cD4EhkLvA6mjOqj/JoADYD0wACoGHtdZPuiUqN8uMD+fuebm8sa2MDw5UWx2OEMKf7X0L9q+Eed+DyGSrozmrPicDrfUWrfUdwCIgD1irlPqR2yJzs3vOzyUnIZwfv76T9i671eEIIfxRRxOs+B6kTICZX7M6ms/V52SglFqtlNoEfADcDsQB17spLrezBQfywMIJHK5p5c+riqwORwjhj0o3QXs9fOEPEBhsdTSfK6gfx94O1AMN2k9WfJudl8jCKek8tuYgV0xMY0xqtNUhCSH8Se58uHe35RvX9EVf1iZSAFrrw1rr+jMlAl8bYnqqH18xlmhbMPct305nt+x5IIRwgc5W2OPczNEHEgH0cW0ipdQ3lVJZp76olApRSi1QSj0N3Oae8NwvITKU/712AoXljfzp/QNWhyOE8Afv/RKWLzKb3fuIviSDSwE78IJSqkwpVaiUOgQcAG4BHtJaP+XGGN3u4vGpXDc1gz+vPsj2o/VWhyOE8GXFa2HDozDjq5Cab3U0fdbrQnWnHaxUMJAItGE2ud/rrv4DVy9U15vG9i4ufWgttuBA3vzmeUSG9qc7RQghMBvc/3UuBIXCV9dBSLhHi3frQnXOAv5bKfURkK21Ltda1wOlgPW7OLtItC2Y3980mcM1Ldz/2k7ZFU0I0T9awxtfh6YKuPZvHk8Eg9XXoaV5wL2Y0UQAaK2bgC+4ISbLzByRwH0XjeKNbWUs/+So1eEIIXyJUjDxJrj8N5A+1epo+q2vbSHvARdhdjkDQCmVCMx2R1BWuuf8PDYU1/KzN3czKTOWsWky3FQI0YvuTggKgfHXWB3JgPWpZqC1fgloAIqUUp8opR4AZgH73BmcFQICFA/dNJmYsGDuXraZ+tZOq0MSQnizpuPw5+mw7XmrIxmU/ixH8QiQBfwMCAS+CzS5KS5LJUaG8uitUymvb+cbz2+l2y7zD4QQZ9DdActvNctTp/jOyKEz6dcGnFrrNq31Cq31D7TWc4FfuSkuy03Ljuf/XZPPB0XVPLBij9XhCCG8jdbw1n1QuhGu+QukTbQ6okEZ1PhJrfUaVwXijW6cnsneiiae/LCYMalR3DQ9q/cPCSGGhg2PwbZlMO/7MH6h1dEMWr9qBkPRjy4fw5yRidz/2i7W7K+yOhwhhLfobIExV8K8H1gdiUtIMuhFUGAAf140lbzkSL62bDM7SxusDkkIYSW7c//0ud+FG5+BAP+4jfrHX+Fm0bZgnr5zBnHhIdzx1EZKalqsDkkIYYXKPWbkUKlzdYSAQGvjcSFJBn2UEm3jmbtm0O3Q3PbkRqqaOqwOSQjhSQ2lsOw60zwUkWR1NC4nyaAfcpMieeK26Rxv7GDR4+upaZaEIMSQ0FgGT10JHc2w6B8Ql211RC4nyaCfpmXH8cTtBZTUtLLo8Q3UtcikNCH8Wks1PP0F87z4VZ8fQno2HkkGSqlMpdQqpdQepdRupdS3PFGuu8zKTeTx2wo4VN3CrU9soKG1y+qQhBDuEhoN6dNg0cuQMaAFQX2Cp2oG3cB3tNZjgZnA15VS4zxUtlvMGZnEXxdP48DxZhY/KTUEIfxO03EzszgoBK5dCtnnWh2RW3kkGTiXvd7i/LkJ2AOke6Jsd5o/OplHb53K3oomblr6Mccb260OSQjhCnUl8OQlsHyxmWk8BHi8z0AplQNMATZ4umx3uGBsCk/dPp3SujZueOxjjta2Wh2SEGIwqvbBk5dCWy1c/CuzNPUQ4NFkoJSKBF4Bvq21bjzD+0uUUpuUUpuqqnxntu+svESe/8pMGtu7uP6xjzhw3C/X7xPC/5Vtg79fBo5uuH0FZM6wOiKP8VgycG6Z+QrwnNb61TMdo7VeqrUu0FoXJCX51jjeyZmxLF9yLlrD9Y99zPpDNVaHJIToD61hxXchOALuXOlT+xe7gqdGEyngCWCP1vr3nijTCqNTo3jla7NIigpl8RMbeHVLqdUhCSH6wmE3zUHXPwl3vQ0JuVZH5HGeqhnMBhYDC5RS25yPyz1Utkdlxofzyt2zmJ4Tz30vbef37+yX/ZSF8FYOO7x9P/zjTnA4IDYLoodZHZUlBrWEdV9prT8AhkYvDBATHsxTd8zgR6/t5OH3DlBS08L/XTcRW7D/rGMihM/raILX7oa9b8GMJcDQ/tLmkWQwFIUEBfCb6ycyPDGC37y9j6LKZh67dRqZ8eFWhyaEqC6C5Yugej9c+iDM/JrVEVlOlqNwI6UUX5+fxxO3FXCktpWr/vQB6w74zigpIfySvRueu95MKFv8miQCJ0kGHnDB2BTe/MZ5JEWFctuTG3l09UHpRxDC0xwO8wgMgoV/ha+ugRHnWx2V15Bk4CHDEyN47Z7ZXDYhjf9buZclz26WJSyE8JSm4/DcdbDud+b3rHNMZ7HoIcnAgyJCg/jTLVP4yZXjWL2vksv+uE7mIwjhbkXvwmOzoeQjiPSt+UueJMnAw5RS3HXecF67ZzZhIYHc8rf1/P4/++i2O6wOTQj/0t0J//mx2ZAmIgmWrIZpt1sdldeSZGCR/PQY3vrmeVw3NYOH3y/ipqXrZV0jIVypag+sfxSmfxm+8j4kj7U6Iq+mvLUjs6CgQG/atMnqMDzijW3HuP+1XWit+eHlY1l0ThZqiCyOJYRLOexQvAZyF5jfaw4OqdnESqnNWusBbbogNQMvcPXkdN6+dy5TsuL48eu7WPzERkrrpJYgRL/UHIS/Xw7PLoRjW8xrQygRDJYkAy+RHhvGs3fN4IGF+Ww9Uself1jHCxuPyBBUIXrjcJjmoEdnm6ahax6DYVOsjsrnSDLwIkopFp2Tzcpvz2ViRgw/fHUnix7fwMGqZqtDE8I7aQ0v3AwrfwDD58A962HyLUNmDwJXkj4DL+VwaJ7feIT/W7mXji4Hd5+fyz3n58r6RkIAdLZCkA0CAmDb8+a1SZIEpM/ADwUEKG6dmc1735nHZRNSefi9A1z6h7WynIUQ+/4Nf54B254zv0/+onkM8UQwWJIMvFxylI0/3jyFZXedg1KKxU9s5JsvbKW8oc3q0ITwrLrD8OIi0ywUEgmJI62OyK9IM5EPae+y8+jqgzy65iABCu6el8tX5+YSFiJNR8LPbfirmUAWEARz/xvO/QYEhVgdldeRZqIhwhYcyL0XjeK9++ZxwdgU/vDuARb8bjVvbDsmo46E/3HYobvD/BybDfnXwTc3w5z7JBG4gdQMfNjG4lp++dZudh1rZEpWLD+5chxTs+KsDkuIwTv8Aaz8IYy6BBb82OpofIbUDIaoGcPjefPr5/Hr6ydSWtfGtX/5iCXPbGL/8SarQxNiYMp3wLLr4akroLUWUsZbHdGQITUDP9HS0c0THxSzdO0hWju7WTglg3svGklGnOysJnzER4+YfgFbLJx3r9mKMkT+/fbHYGoGkgz8TG1LJ4+uLuLpj0tAwxfPyeIbC/JIjAy1OjQhPqux3DxHp5laQeHrMOu/ICzWyqh8liQD8Rll9W08/N4BXtp0lNCgQBafm81X5owgKUqSgvACDcfgwz/C5qdMx/DCR62OyC9IMhBndbCqmUfeO8Cb28sICQrgizOy+eq8EaRE26wOTQxF9Ufhg4dg67OgHTDpZpjzXYgfbnVkfkGSgehVcXULf15VxGtbjxEYoLhleiZ3n59LWkyY1aGJoWTFf8Omv8OUW02/QFy21RH5FUkGos+O1LTyl9VF/GNzKUrBwinpLJk7grzkKKtDE/6oYqfpGJ52O2TPguZKsHdCTIbVkfklSQai30rrWvnrmkO8vPko7V0OLhiTzFfmjuCc4fGysY4YHK3h0Cr48GHzHBwBl/6PbDnpAZIMxIDVtnTyzMeHeebjEmpbOpmUEcOSublcMj6FoECZhiIGYNl1ZhP6yBQ4524ouAPCZDKkJ0gyEIPW1mnnlS2lPL7uEIdrWsmMD+NLM3O4sSCTmPBgq8MT3qylBnYsN/MCAoNgyzOAgok3QpCMXvMkSQbCZewOzTuFFTy+rphNJXWEBQdyzZR0bp+Vw+hU6VcQpyjfDhuWws6Xwd4Bt71lNpgRlpFkINxi17EGnvn4MG9sK6Oj28HMEfHcPiuHC8dKE9KQ1lwFy2+Fo+shONwMD52xBJLHWh3ZkCfJQLhVXUsnL35ylGXrSzhW38awGBs3Tc/ixukZMjR1qKgrger9MPIis+fw8zdA7gKYvEhmC3sRSQbCI+wOzbt7jrNsfQnrDlQToGDBmGRunp7F+aOTpLbgb+xdZlexzU/BwfchPAG+s8/0CwivNJhkIP9VRZ8FBiguGZ/KJeNTOVLTyvJNR3hpUynv7tlEarSNG6dnctP0TNJjpbbg83a/biaItVRCdDrM+76ZKCaJwG9JzUAMSpfdwft7K3lh4xHW7Df7M5+Xl8h1UzO4ZHyq7MLmKzqaoPBNSJ9q2v6PbjTLRky7HfIuhAD57+gLpJlIeIXSulZe3lTKK1tKKa1rIzI0iMsnpHLd1Aym58QTECCT2byKwwGH18K2F2DPm9DVaraUlM1kfJbXJwOl1JPAlUCl1jq/L5+RZOC7HA7NxsO1vLK5lBU7y2nptJMZH8a1UzK4dmo62QkRVocotIZHZ0PlbgiNhvELYfIXIfMckBnoPssXksFcoBl4RpLB0NLa2c3buyv4x+ZSPjpYg9YwOTOWqyYN48qJaSTL6qme0XAMdr9qmn9ufMbc8DcshfB4GHMFBEs/jz/w+mQAoJTKAd6SZDB0ldW38eb2Mt7YVsae8kYCFMwckcBVk4ZxWX6azHR2tdZa2P0a7HoFSj4CNAybCre+YpKA8Dt+kwyUUkuAJQBZWVnTSkpKPBKb8Lyiyibe3FbGm9vLOFzTSnCgYt6oJL4waRgLxiQTZZPEMCBt9YA2awHtfg1evh0SR8OE680mMgm5Fgco3MlvksGppGYwNGit2XmsgTe3lfHWjnIqGtsJCQxg7qhELstP48JxKcSESWL4XK21sPdfUPgGHFoN838Ic74DXW1QUwQp+dIPMETIPAPhs5RSTMyIZWJGLD+6fCxbjtSxYmcFK3eV8+6eSoIDFbPzErk8P42LxqUQFxFidcjew+GAF26CovdA2yE2G2Z+DUZeYt4PDoPUCdbGKHyG1AyEV9Jas720gX/vLGfFrnKO1rYRGKA4Z3g8F41L4cKxKWTGh1sdpmfVHIR9K6DuMFzxO/PaW/eBLQbGXQ1pk6QGMMR5fTORUuoF4HwgETgO/Exr/cTnfUaSgThBa83uskZW7CznP4XHKapsBmBMahQXj0vhonGp5KdH++emPJV7YMdLJglU7TWvpU6AL78ny0OLz/D6ZDAQkgzE2RRXt/Bu4XHeKTzOppJaHBrSYmxcODaFC8elcM7weGzBPjpjtrMVDq+DjOlmxM+GpbDyB2bLyDFXwOjLIC7H6iiFl5JkIIasmuYO3t9byTuFx1l3oJq2LjthwYHMzkvg/NHJzB+T7P1rJdUWw4F34MB/TCLoboer/2zWAmpvBEe3DAUVfSLJQAigvcvOx4dqWLW3kvf3VlJa1wbA6JQo5o9JZv7oJKZlx1m/ump3J7TXQ2QyNJbB7537AMTnwsiLYdTFkD1bmoFEv0kyEOJTtNYcrGph1d5KVu2rZGNxLd0OTZQtiNm5icwZlcjckUme64SuOWiWgT64CorXQu75cNMy897W5yBrpswBEIMmyUCIXjS1d/FhUTWr91Wxdn8VZQ3tAAxPjGDuyETmjEzi3NwEIkJdNNq6qx2CnUttvHCL6QAGM/wzdwGMuRJGXuiasoRwkmQgRD+cqDWs3V/FugNVrD9US1uXneBAxdSsOGbnJXJubgKTMmIJCepjk1JXO5R+Ytr8D60xo4D++4Bp6tn+olkiOncBxI+Q4Z/CbSQZCDEIHd12Nh+uY82BKtbtr6awvBGAsOBACnLiODc3gXNHJDAhPeZkf4O9yzwHBsO25+Gte03HrwqA1ImQdwHM/paZAyCEh0gyEMKF6lo62VBcw8cHa/j4UA37jzcTiJ3poUe5Nr6Yc9VuhjVugxueInD0JVC2DXYsh5w5Zgio7AksLCLJQAhXczigqwVCo6gp3U/0U/MJ7jaT3fY70vnYMY5/Bl1C/IgppuaQm8Co5CjZwEdYStYmEmKwHHY4vguOrDft/oc/gHHXwBf+QMKwPJi6CDJnQM4coh0xRB+qZoSz5vCfwuMAJESEMHNEAjNzE5ieEyfJQfgUqRmIoamjGepLIGW8+f0vs8yuXwAxWTB8Loy90sz47UVpXatpUnImh3LnSKUoWxBTs+KYnhPHtOx4JmfGyp7Qwq2kmUiI3jRVmA1ejm6AIx9DxS6ISILv7DWje7Y+B4EhkHUOxGYNuBitNUdr29hUUssnh+vYXFLL/uOmeSkoQDE+PYaC7JMJIilKJpYJ15FkIMSpHA6o2mNu/FMWmxE/K74HG/8KQWGQUWAmeWXONMM9A9w7I7m+tZMtR+rYdNg8tpfW09HtACA7IZyC7HgKcuIoyI4jNylSmpbEgEkyEKL2EOz8h9njt3QjtDeY15esgWGTofoAdDSaYZ+B1m6W09ntYFdZA5sO17LpcB2bS+qoaekEIDY8mGlZcUzNjmNyZiwTMmKIll3fRB9JMhBDR3eH6egt3QzHNsO02yH7XLPMw7MLIWn0yW/9WTPNCp9ePslLa01xdQubSupMgiip41BVC2BCz02KZFJGLJOzYpmcEcuYtCiCrV5fSXglSQbCP2kNXa0QEgFNx+HFL0LFDrCbb9FEpsAl/2P29+3uhO42v5nk1dDaxfbSerYdrWf7UfN8ovYQEhRA/rBoJmXGMtn5yIoP98/9HES/SDIQ/qG5ynzbP7YZjm2CY1tgwg1wxW/B3g3PXW82dskogPRpEJ3u9d/6XUVrTWld22nJYeexhp6+h7jw4J7kMCkzlkkZscTLFqFDjiQD4XuaKszM3a5WyL/WvPZQPjQcNUs6JI8zN/xRl8KYyy0N1Vt12R3sq2gyNYgj9WwvredAZTMn/pdOjw1j/LBoJqTHkJ8RQ/6wGBm95OckGQjfsPlp2PsvKN8GzWaiFrHZ8O0d5ue9/zLNPGmTITTSqih9WlN7FztLG9h5zDx2lzVSXN3S835qtI389Gjy02NMkkiPISXaZmHEwpVkBrLwDlpD/REo325u+OXboXo//Nc2CAiEip3m/dwFZvP2tMmm2eeEMVdYFLj/iLIFMysvkVl5iT2vNbZ3UVjWyK5jDexyJon39lb21CCSokLJd9YgxjuTRFqMTfoghhhJBmJgutrNWP6KnTDuavONft3v4P1fmfdVICSPhZy5ZvnmsFi4/DdDpo3fm0Tbgs0yGSMSel5r6eimsLyxJznsPtbImv1VOJwJIiYsmLFpUYxNi2ZcWjRj06IZmRJJaJDMoPZXkgxE7xwOMzGrYhd88JAZ2ll9ALTdvB+bDSPmwahLICzOfONPGX9yc5cTJBF4jYjQIKbnxDM95+Teym2ddvZUNLL7WAOF5U3sKW/khY1HaO8yndRBAYrcpEjGDYvuSRRj06JJjJR+CH8gyUCcZO+CykKzMcvx3eamX7ELLnkAJt5ohnQeWW+adsZ+wTyn5EPccPP51AmnN/sInxIWEsjUrDimZsX1vGZ3aA7XtLCnvJE95Y0UljXy8cEaXtt6rOeY5KjQnsQwJjWKUSlR5CZHSC3Cx0gH8lBk7zYzdk/c+FMnmEXZTt2cPTAEksaYGbtTbjUTu4Rwqm3pZG95I4XOx57yJooqm+iym/tJYIBieGIEo1NMchidGsmolCiyEyIIlOU23EY6kMWZORxmqGZXq2m/1xr+tsB84z8xcUsFwMx7TDKISoMbnjbHxo+wfNkG4b3iI0I+01Hd2e3gcE0L+yqa2H+8ib0VTewqa2DFrvKezurQoADykiMZnRplEoXzWTqsrSfJwN9sfQ6Orjff+Cv3QGezWY75tn+aNvthkyHnPDOOP3msWb4hOMx8VikYf42V0QsfFhIUwChnTeBUrZ3dFFU29ySJfceb+bComle3nGxqigoNYpSziWl0SmTPzwkRIZIkPESaiXxNQ6lpx6/eB1X7zbN2wFfeN+8/e60Z1pk87uQNP20SpE+1NGwhPq2+tZP9x5vZd7yJ/RVN7DvexL6KJhraunqOiQkLJi85krykSPKSI8lNjiAvKYr0uDBpbjoDaSbyN92dUHsQqvaZcfo1B+GaR82IntX/C1uXmeMiks03++RxpglIKbj5uZPf9IXwYrHhIcwYHs+M4SdHNGmtqWrqYG9FE0WVzRRVNVNU2cx7e4+zfNPRnuNCgwIYkRRJblKESRbOR05CBLZg6bgeCEkGVmqtNUM0a4rM6BxbNKx/FN6+/+SwTTCbrbTVQUSCad+fshgSR0F4/GfPKYlA+DClFMnRNpKjbcwdlXTae3UtnRx0JocTz9tL6/nXzpN9EgEKMuPDT9YkkiLJdSaKmDDpA/s8kgzcrbvDjNyJSjMTrw5/CO/+HGoOmBv8CQl5ZpetYVNgzn2QOBqSRkHCSAgJP3nciW0ahRhi4iJCKIiIpyDn9C9B7V12DlW19NQiDlY1c7CymXVF1XQ6F/IDM9M6NymC4YknHpEMTwwnMz5chsEiycA1tDZj9INCTJv+R4+Yb/vVB8xoHu2A6/9uFmQLskFQqJm1mzDSJIHEkWbiFpg1+LNmWvv3COFDbMGBjBsWzbhh0ae9bndojta29tQiiiqbOVTdwn92H+9ZDhxMbSIjLpycxAhGOBPFiZ+HxQ6dvgnpQO6vrjbY8ZL5tn/qY/79MOsbUHfYbK6ekHvyRp8wEnJmQ/Qwq6MXQmD2iyiuaaG4upni6laKq83Ph6tbae7o7jkuJDCArIRwhjuTQ44zWYxIjCApKtTrRjpJB7KrFa81nban3uxHXQIX/hxQ8M9vQUCQ2UUrfoQZujlssvlsbDb86JgsvSCEF4sJD2ZyuNn/4VRaa6qaOyiuauFwTQuHqlt6fl6zv+q0ZqeIkMCe5DA8MYKchAiyE8LJig/3ykTRm6GZDOqPmmacU2/2ccPh0v8x7//jLmiphMBQiB9ubvjxI8x7wTb49k7zLT/gDO2MPvYPQAhxklKK5CgbyVE2zjllYT8wzU5l9W0crmmhuLqFQ84ksfNYAyt2lvcs8gcQFhxIVnw4WQnhZMeHmySREEF2fDjpcWFeuW2pfyaD9gaoLYb6EtNsU1divslf/mvz/kuLoWyr+TnIZm70J9bXAVj0EoQnnv2GH5vp9j9BCOFdAgMUmfGmw3nOyNNHOnV2Oyita6WktpWjta2U1Jx4tLDuQFXPYn9g+iiGxYY5axGmNpHtTBxZ8eFE2awZ9eSxZKCUuhT4IxAIPK61fnDAJ7N3mY7ZuhM3+8PQWgNX/8m8//o9sPetk8efWEnzhAt/YZZhSMiFyFQzfv9Uw6YMODQhxNAT4pz3MCLps5syaa2pbOqgpKaVI7WtHKlpocSZMN7eXUHtKZ3ZYJb6yHLWJkySiDC1jPhwkqNCCXBTh7ZHOpCVUoHAfuAioBT4BLhFa114ts8UTJ2sN735BNQVn7zhX/E7MxLn3z+ADY+ePDgg2IzFv2e9GdFTvM4M24zLgbhsv9kkXQjhf5rauyipcdYonEniSG0LJTWtlNW3ndb8FBIUQEZsGBnx4WTGhZmaSlw4mfFhZMWHExcR6vUdyDOAIq31IQCl1IvA1cBZkwEVO+HxBSd/j0iGed83TTT515qVNuOyzQ0/Ku305pzhc9zyRwghhKtF2YLJd25B+mmd3Q7K6tt6mp+O1rZytK6Vo7Vt7Citp7616wxnHBhPJYN04Ogpv5cC53z6IKXUEmCJ89cO9YvGXSffbYTvZbkxRK+WCFRbHYQXkOtwklyLk+RanDR6oB/0VDI4UyPXZ9qntNZLgaUASqlNA63u+Bu5FoZch5PkWpwk1+IkpdSAJ2d5anxTKXDqEJwMoMxDZQshhOiFp5LBJ8BIpdRwpVQIcDPwpofKFkII0QuPNBNprbuVUt8A3sYMLX1Sa727l48tdX9kPkOuhSHX4SS5FifJtThpwNfCa9cmEkII4TneNydaCCGEx0kyEEIIYW0yUEpdqpTap5QqUkr94AzvK6XUw873dyil/HYj3z5ci0XOa7BDKfWRUmqSFXF6Qm/X4pTjpiul7Eqp6z0Znyf15Voopc5XSm1TSu1WSq3xdIye0of/R2KUUv9USm13Xos7rIjTE5RSTyqlKpVSu87yfv/vnVprSx6YjuSDwAggBNgOjPvUMZcD/8bMU5gJbLAqXi+4FrOAOOfPlw3la3HKce8DK4DrrY7bwn8XsZiZ/FnO35OtjtvCa/Ej4P+cPycBtUCI1bG76XrMBaYCu87yfr/vnVbWDHqWqNBadwInlqg41dXAM9pYD8QqpdI8HagH9HottNYfaa1P7JO5HjNXwx/15d8FwDeBV4BKTwbnYX25Fl8EXtVaHwHQWvvr9ejLtdBAlDIbCURikkE3fkhrvRbz951Nv++dViaDMy1RkT6AY/xBf//OuzBZ3x/1ei2UUunAQuAxD8Zlhb78uxgFxCmlViulNiulvuSx6DyrL9fiT8BYzITWncC3tNYOhqZ+3zut3M+gL0tU9GkZCz/Q579TKTUfkwzOc2tE1unLtfgD8H2ttd3XdpPqp75ciyBgGnABEAZ8rJRar7Xe7+7gPKwv1+ISYBuwAMgF3lFKrdNaN7o5Nm/U73unlcmgL0tUDJVlLPr0dyqlJgKPA5dprWs8FJun9eVaFAAvOhNBInC5Uqpba/26RyL0nL7+P1KttW4BWpRSa4FJmCXj/UlfrsUdwIPaNJoXKaWKgTHARs+E6FX6fe+0spmoL0tUvAl8ydkzPhNo0FqXezpQD+j1WiilsoBXgcV++K3vVL1eC631cK11jtY6B/gHcI8fJgLo2/8jbwBzlFJBSqlwzGrAezwcpyf05VocwdSQUEqlYFbwPOTRKL1Hv++dltUM9FmWqFBK3e18/zHMSJHLgSKgFZP5/U4fr8VPgQTgL85vxN3aD1dq7OO1GBL6ci201nuUUiuBHYADs4vgGYcb+rI+/rv4FfCUUmonppnk+1prv1zaWin1AnA+kKiUKgV+BgTDwO+dshyFEEIImYEshBBCkoEQQggkGQghhECSgRBCCCQZCCGEQJKBEEIIJBkIIYRAkoEQZ6WUulYp9fApv4cppdYopQLPcnyIUmqtUsrKZV6EGBBJBkKc3RRgyym/34lZLtp+poOdSyu/B9zkgdiEcClJBkJ8ilJqlFLqHeDbwM+VUt92vrUIsxYQSqm7nbuLbVNKFSulVjmPed15nBA+RZajEOIUSqlQYAOwGHPjn4XZSSwH2K+1Tv3U8cGYHdd+rbX+p7MJqUJrneTRwIUYJKkZCHG6izBbKpYBjVrrCqAdiALqz3D8H4H3tdb/BHA2IXUqpaI8E64QriEdXUKcbjJml6xJwA6lVDLQhFn50XbqgUqp24Fs4BufOkcoJoEI4TMkGQhxukZgImbv3B3AL4A/a63rlFKBSimb1rpdKTUN+C4w59StFZVSCUCV1rrLiuCFGChpJhLidMuAkZj9I76G2XT8Eed7/+HkdqPfAOKBVc5O5Medr8/HrCUvhE+RDmQhzkAptR1YcOr2okqpKcB9WuvFn/O5V4Efaq33eSBMIVxGagZCfIpzRFHUp/eZ1lpvxdQEzjrpDHhdEoHwRVIzEEIIITUDIYQQkgyEEEIgyUAIIQSSDIQQQiDJQAghBJIMhBBCIMlACCEE8P8BTQkK5KpHidMAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "def sigmoid(z):\n",
    "    return 1.0 / (1.0 + np.exp(-z))\n",
    "def cost_1(z):\n",
    "    return - np.log(sigmoid(z))\n",
    "def cost_0(z):\n",
    "    return - np.log(1 - sigmoid(z))\n",
    "z = np.arange(-10, 10, 0.1)\n",
    "phi_z = sigmoid(z)\n",
    "c1 = [cost_1(x) for x in z]\n",
    "plt.plot(phi_z, c1, label='$\\mathcal{L}(w)$ if y=1')\n",
    "c0 = [cost_0(x) for x in z]\n",
    "plt.plot(phi_z, c0, linestyle='--', label='$\\mathcal{L}(w)$ if y=0')\n",
    "plt.ylim(0.0, 5.1)\n",
    "plt.xlim([0, 1])\n",
    "plt.xlabel('$\\phi$(z)')\n",
    "plt.ylabel('$\\mathcal{L}(w)$')\n",
    "plt.legend(loc='best')\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that $\\phi(z)$ is the \"probability of class 1\". We can see that the cost approaches 0 if we correctly predict the class."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the chain rule, one can show that\n",
    "\\begin{align*}\n",
    "\\nabla_{\\mathbf{w}}\\mathcal{L}(\\mathbf{w},w_0) &= \\sum_{i=1}^n(\\phi(z(\\mathbf{x}_i;\\mathbf{w},w_0))-y_i)\\mathbf{x}_i. \\\\ \n",
    "\\nabla_{w_0}\\mathcal{L}(\\mathbf{w},w_0) &= \\sum_{i=1}^n(\\phi(z(\\mathbf{x}_i;\\mathbf{w},w_0))-y_i),\n",
    "\\end{align*}\n",
    "which are used in the Gradient Descent optimization algorithm below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LogisticRegressionGD(object):\n",
    "\n",
    "    \"\"\"\n",
    "    Logistic Regression Classifier using gradient descent.\n",
    "\n",
    "    Parameters\n",
    "    ------------\n",
    "    eta : float\n",
    "    Learning rate (between 0.0 and 1.0)\n",
    "    n_iter : int   Passes over the training dataset.\n",
    "    random_state : int    Random number generator seed for random weight initialization.\n",
    "\n",
    "    Attributes\n",
    "    -----------\n",
    "    w_ : 1d-array    Weights after fitting.\n",
    "    cost_ : list    Sum-of-squares cost function value in each epoch.\n",
    "\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, eta=0.05, n_iter=100, random_state=1):\n",
    "        self.eta = eta\n",
    "        self.n_iter = n_iter\n",
    "        self.random_state = random_state\n",
    "\n",
    "    def fit(self, X, y):\n",
    "        \"\"\" Fit training data.\n",
    "        Parameters\n",
    "        ----------\n",
    "        X : {array-like}, shape = [n_samples, n_features]\n",
    "        Training vectors, where n_samples is the number of\n",
    "        samples and\n",
    "        n_features is the number of features.\n",
    "        y : array-like, shape = [n_samples]\n",
    "        Target values.\n",
    "        Returns\n",
    "        -------\n",
    "        self : object\n",
    "        \"\"\"\n",
    "\n",
    "        rgen = np.random.RandomState(self.random_state)\n",
    "        self.w_ = rgen.normal(loc=0.0, scale=0.01,\n",
    "        size=1 + X.shape[1])\n",
    "        self.cost_ = []\n",
    "\n",
    "        for i in range(self.n_iter):\n",
    "            net_input = self.net_input(X)\n",
    "            output = self.activation(net_input)\n",
    "            errors = (y - output)\n",
    "            self.w_[1:] += self.eta * X.T.dot(errors)\n",
    "            self.w_[0] += self.eta * errors.sum()\n",
    "            cost = (-y.dot(np.log(output)) -\n",
    "            ((1 - y).dot(np.log(1 - output))))\n",
    "            self.cost_.append(cost)\n",
    "        \n",
    "        return self\n",
    "\n",
    "    def net_input(self, X):\n",
    "        \"\"\"Calculate net input\"\"\"\n",
    "        return np.dot(X, self.w_[1:]) + self.w_[0]\n",
    "\n",
    "    def activation(self, z):\n",
    "            \"\"\"Compute logistic sigmoid activation\"\"\"\n",
    "            return 1. / (1. + np.exp(-np.clip(z, -250, 250)))\n",
    "\n",
    "    def predict(self, X):\n",
    "            \"\"\"Return class label after unit step\"\"\"\n",
    "            return np.where(self.net_input(X) >= 0.0, 1, 0)\n",
    "            # equivalent to:\n",
    "            # return np.where(self.activation(self.net_input(X))\n",
    "            # >= 0.5, 1, 0)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will work on the famous Iris dataset. We will assign the petal length and petal width of the 150 flower samples to the\n",
    "feature matrix X and the corresponding class labels of the flower species to the vector y."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class labels: [0 1 2]\n"
     ]
    }
   ],
   "source": [
    "from sklearn import datasets\n",
    "import numpy as np\n",
    "iris = datasets.load_iris()\n",
    "X = iris.data[:, [2, 3]]\n",
    "y = iris.target\n",
    "print('Class labels:', np.unique(y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To evaluate how well a trained model performs on unseen data, we will further split the dataset into separate training and test datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1, stratify=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to keep in mind that it only works for binary classification tasks. So, let us consider only Iris-setosa and\n",
    "Iris-versicolor flowers (classes 0 and 1):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_01_subset = X_train[(y_train == 0) | (y_train == 1)]\n",
    "y_train_01_subset = y_train[(y_train == 0) | (y_train == 1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need a plotting function that is defined below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from matplotlib.colors import ListedColormap\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot_decision_regions(X, y, classifier, test_idx=None,resolution=0.02):\n",
    "    # setup marker generator and color map\n",
    "    markers = ('s', 'x', 'o', '^', 'v')\n",
    "    colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')\n",
    "    cmap = ListedColormap(colors[:len(np.unique(y))])\n",
    "    # plot the decision surface\n",
    "    x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
    "    x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
    "\n",
    "    xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),\n",
    "    np.arange(x2_min, x2_max, resolution))\n",
    "    Z = classifier.predict(np.array([xx1.ravel(), xx2.ravel()]).T)\n",
    "    Z = Z.reshape(xx1.shape)\n",
    "    plt.contourf(xx1, xx2, Z, alpha=0.3, cmap=cmap)\n",
    "    plt.xlim(xx1.min(), xx1.max())\n",
    "    plt.ylim(xx2.min(), xx2.max())\n",
    "    for idx, cl in enumerate(np.unique(y)):\n",
    "        plt.scatter(x=X[y == cl, 0], y=X[y == cl, 1],\n",
    "        alpha=0.8, c=colors[idx],\n",
    "        marker=markers[idx], label=cl,\n",
    "        edgecolor='black')\n",
    "        \n",
    "    # highlight test samples\n",
    "    if test_idx:\n",
    "        # plot all samples\n",
    "        X_test, y_test = X[test_idx, :], y[test_idx]\n",
    "        \n",
    "        plt.scatter(X_test[:, 0], X_test[:, 1],c=\"g\", edgecolor='black', \n",
    "        alpha=1.0,linewidth=1, marker='o',s=100, label='test set')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we will instanciate the class defined above and fit logistic regression using the fit methos:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/6c/mzld6tfn7g1cw7rzfxzv544h0000gp/T/ipykernel_11766/271874885.py:21: UserWarning: You passed a edgecolor/edgecolors ('black') for an unfilled marker ('x').  Matplotlib is ignoring the edgecolor in favor of the facecolor.  This behavior may change in the future.\n",
      "  plt.scatter(x=X[y == cl, 0], y=X[y == cl, 1],\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEGCAYAAAB7DNKzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAwlElEQVR4nO3de3hU9bXw8e8CacAKCMjFgApaUMY7pdNjFUrLqVXbgFY91nZOC/V9fASt9nYmPfalFnP6vs30YN9aS6xVwTY9GCsgRim9RCMjtkWg3AxKQbDGoFBCASuGS9b7x96TTJKZySRz2XNZn+eZZ2bv2dl7DT7Omt/vt/bvJ6qKMcYYE08frwMwxhiT2yxRGGOMScgShTHGmIQsURhjjEnIEoUxxpiETvI6gEwYNOg0HTlyrNdhGGNM3tixY/3fVXV4rPcKMlGMHDmW++5b53UYxhiTN2bMkDfivWddT8YYYxKyRGGMMSYhSxTGGGMSKsgxilj69DnGsGGN9Ov3vtehxHXsWH/27x9Da2s/r0Mxxpg2RZMohg1r5PTTBzJo0FhExOtwulBVDh3aDzSyb984r8Mxxpg2RdP11K/f+wwaNCwnkwSAiDBo0LCcbvEYY4pT0SQKIGeTRESux2eMKU5FlSiMMcb0nCWKLPrDH1bxkY+cy6RJH+JHP/qB1+EYY0xSLFFkyYkTJ/iP/7idX//6N/zpTw0sXbqEV19t8DosY4zpVtFUPfXEzE9+hMP79nbZP3D4CFY893Kvzrl+/VrOPvtDjB17NgCf+9znWblyBeed50spVmOMyTRLFDEc3reXtad1nRvLHyN5JGvPnrcYPfqMtu3S0jGsX//nXp/PGGOyxbOuJxE5Q0SeF5FtIvKKiNwV45hpInJQRDa6j+96EWs6xFqb3KqcjDH5wMsWxXHgm6q6QUQGAutF5Peq2rnjPqyqn/UgvrQqLR3DW2+92bbd1NTIqFGlHkZkjDHJ8axFoap7VHWD+/owsA0Y7VU8mTZp0kfYufOvvPHGLo4ePcqyZY9z9dUzvA7LGGO6lRNjFCIyFrgUiNVpf5mIbAKagG+p6itxznErcCvA8OFnZijS3jvppJMIhR7g+us/zYkTJ/jiF7/CxInnex2WMcZ0y/NEISKnAEuBr6nqoU5vbwDOUtV3ReQa4ClgfKzzqOpDwEMA48dP7jog0AMDh4+IOXA9cPiIVE7LlVdew5VXXpPSOYwxJts8TRQi0g8nSfxKVZd1fj86cajqShFZKCKnqerfE5337cbjrF0Lfn/v4uptCawxxhQiL6ueBHgE2Kaq98U5ZpR7HCLix4l3f3fnHnXS36mvamDt2nRGbIwxxcnLFsXlwL8DW0Rko7vvbuBMAFV9ELgBmCMix4EjwOc1Vp1pJ4NHDWBavwbqq2D7dh8TJsC112biIxhjTOHzLFGo6otAwhsJVPUB4IHenL+isoR55Q2wpYH6Oh/XXGP3LBhjTG94PpidSRWVJQDMK2/g/cMTPI7GGGPyU1FMClhRWUJfaeXt3Ud4+28t/POfXkdkjDH5oygSBcDwoSc4b2wLA/v8k8P7jngSwx13fIXx40dw2WUXeHJ9Y4zpjaJJFBGjx/ShX58TTuvi7fjHdR4y734IvXs33zyLJ59clfqJjDEmi4ouUQCcc+ZxRvX/B/2Ovsvbu7u2LhYvhp/+tD05qDrbixendt3LL5/KkCFDUzuJMcZkWVEmCoBTRw3gnDOPt7UuIuMWqvDuu7B0aXuy+OlPne13301Py8IYY/JJQVc9JeOcM4/zVuNR3t9/gsP7+jJq7ABuv915b+lS5wFw/fVw++1gM4MbY4pN0bYooo0e06e9dfG3FkRoSxYRliSMMcXKEkWUc848Dq2t7HmjhR/81/EO70WPWRhjTDGxRNHJuWe1sHLp+6xcJVz5yeM895zT7RQ9ZtFbt9xyM1deeRk7drzG+eeP4Ze/fCR9gRtjTIYU/RhFZyIwcGg/Zt18lGs+A++8eYLrr3fu8D7llNS6nx55ZEmaojTGmOyxRBHDrbOOouokhX+8fYS33z+V68v6cPpZJV6HZowxWWddT3FEWg6njhrAeWNbEG216T+MMUWpqBJFEjOUx3Xe2Bb6cYzD+5z5ojIhlfiMMSZTiiZR9D92jP2HDqX0ZXzOmcc5b2wLtLbyduPx7v+gB1SVQ4f2c+xY/7Se1xhjUlU0YxRj9u+nEdj394SrqCbn6FHePvxB9rzTh0Gn9k39fK5jx/qzf/+YtJ3PGGPSoWgSRb/WVsbt25e2801cu5IpVTfDiBEwbBjBYNpObYwxOaVoup7Szu8nPGcJwWEPw7YGQuXdLuVtjDF5yRJFKvx+yoI+wnOWwN53CIVg7VqvgzLGmPSyRJEOfj/h6fMp3b2G+iprXRhjCotniUJEzhCR50Vkm4i8IiJ3xThGROR+EdkhIptFZJIXsSYlEKBmYXOH1oUxxhQCL1sUx4FvqupE4F+A20XE1+mYq4Hx7uNWoCq7IfaC27pgWwOhubssYRhj8l7cqqckf70fU9Utvbmwqu4B9rivD4vINmA00BB12EzgF+rc/PAnETlVRE53/zZ3BQKEJyyB7duZUncPoZDPqqKMMXkrUXnsC8DLQKJp8MYBY1MNQkTGApcCf+701mjgzajtRndfl0QhIrfitDo4c/jwVENKnd/vtC6Y35YsAEsYxpi8kyhRvKyqn0z0xyLyXKoBiMgpwFLga6p6qPPbMf4k5q3VqvoQ8BDA5PHjc2cujECAMPOpbZpEaFuZtS6MMXkn7hhFd0ki2WMSEZF+OEniV6q6LMYhjcAZUdtjgKZUrumJQMApo3XHLqqrrYzWGJM/ej1GoaobUrmwiAjwCLBNVe+Lc9jTwB0i8jjwUeBgzo9PJBIIEGyqpfqlA9TXDaG+3loXxpjcl6jraYH73B+YDGzC6Qq6CGcs4YoUr3058O/AFhHZ6O67GzgTQFUfBFYC1wA7gPeA2Sle03NlQR9lNEP1/Uypu4fqah+BgNdRGWNMfHEThap+AsD9NX9rpLpJRC4AvpXqhVX1RRIPlONWO92e6rVyktu6CNVBaMtISi8cZgnDGJOTkrmP4rzoElhV3QpckrGIikhk+o/gsIdpqnPGLowxJtckM3vsNhF5GKjGqTgKANsyGlUx8fsp8wMhp3VRjVNGa60LY0yuSKZFMRt4BbgL+BrODXF5P1aQa8qCPoITa5mw5UlrXRiTAZ3XLLMFJZPXbYtCVd8XkQeBlar6WhZiKlrOQDfUuq2LtRN8+P1eR2VM/lu1Co4cgWuvBREnSTz1FAwYAFdd5XV0ua/bFoWIzAA2Aqvc7UtE5OkMx1XUyoI+po1ocGainW2tC2NSoeokiXDYSQ6RJBEOO/utZdG9ZMYo7gH8QD2Aqm50p9wwGVRRWQLspDbUYK0LY1Ig4rQkwEkO4bDzesqU9haGSSyZMYrjqnow45GYmDq0Lsr32x3dxvRCdLKIsCSRvGQSxVYR+QLQV0TGi8hPgJcyHJeJUlFZQnj6fKbxPPVVDZYsjOmhSHdTtEg3lOleMoniq8D5QAvwP8BBnOonk02BABWVJW2tC5svypjkRI9JTJkCCxY4z9FjFrH+JtF2sUlmjOLDwHdV9TuRHe48UCnN9WR6p6KyhHnlDbClgfo6H2BjF8YkIuJUN0WPSUS6oQYM6Nr9ZBVSXSWTKH4LvCwi/6aq77j7HgZyd1nSAucMdMO88gbqq8Dv77wwoDEm2lVXOV/4kaQQSRadk0R0hRQ4x0S3RqLPUUySSRSvAT8E6kXkFlV9iW7maDLZUVFZwk1zDxCa3QADBhBcOM7rkIzJWZ2/4GN94VuFVGzJjFGoqj4DzAAeEJE7iLN4kMm+moXNhBftpJQmJ2EYY1JiFVJdJZMoBEBV/wpMAabiTDVuckjNwmZKBziti1DI62iMyV9WIdVVt4lCVS+Nev1PVf034OyMRmV6pWZhM+Hp8yndvcZaF6agZKsKqScVUsWUOOImChEJus/3d36QhvUoTIYEAu2ti7m7vI7GmJStWtXxSzryZb5qVfqvFa9CasQIeC1qprtMxpCLErUoIlOJr4/zMDmsZmEzHDliycLkNS/mabrqqq5jEhMmwN69xTtXlGgBfsrJ48fruvviLcNdXG6aO5SmI0OsKsrkregv5ohsVyHlQgyZNmOGrFfVybHeS9T1VCsiT8d7ZC5ck06RqihrXZh8lQtVSLkQg5cSdT39N7AA2AUcAX7uPt4FtmY+NJNObcmifD+h8v1eh2NM0nKhCikXYvBS3EShqi+o6gvApap6k6rWuo8vAFdkL0STLuE5SwhfOBf2vmOtC5NVva1ainxB19d3rEKqr0/PPE2trYm3o2PoyVxRhSaZ+yiGi0hbOayIjAOGp+PiIvKoiOwVkZgtFBGZJiIHRWSj+/huOq5btPx+CAQ6tC5sUSSTaalULYnAs89CczOUlTnbZWXO9rPPxp6nKdlrLVgA3/9+e3JobXW2FyzoGkOsSqgpU2LPFVWIkkkUX8OZvqNeROqB53HWz06HxUB302yFVfUS93Fvmq5b9MJzljCN52mqa7CuKJMxqVYtnTgBJSVw6BDcdZezfdddznZJibPdm2u1tsJ778Hmze3J4vvfd7bfe69ry6JzJVQkWRTLJIEJq55EpA9wA7ACOM/d/aqqtqQtAGe1vGdU9YIY700DvqWqn+3JOa3qqQfWrmVK1c0wYiTBymFeR2MKUKoVQydOwJ13QmNj+74xY+D++6Fv395fKzo5RFx0EXznO9AnmZ/QBaZXVU8AqtoK3KGqLaq6yX2kLUkk6TIR2SQivxGR8+MdJCK3isg6EVm376AtyJc0v5/wnCVt4xbWujDplmrFUN++TlKIFitJ9PRaffo4SSFasSaJ7iTzT/J7EfmWiJwhIkMjj4xH5tgAnKWqFwM/AZ6Kd6CqPqSqk1V18vDBg7MUXoHw+wkv2kl41qNOwrBkYdIo1YqhSIsi2p13dux26s21Ii2KaNFjFqZdMoniK8DtwGra78pel8mgIlT1kKq+675eCfQTkdOyce2iFN26CGGTCxogtXmWUq0Yiu52GjMGli1znhsbnf3Hj3e81vLlsHp1x2utXt31WtHdThddBDU1znP0mIVpl8ykgONiPLIyKaCIjBJxGo0i4seJ137uZpLfT3j6fMKlN8E2G+gudqnOs5RqxVDfvtC/f8cxifvvd7ZbWqC2tmMCWL3aSR4zZzrnnjkThg6F11/veK0+feDkkzuOSXznO872ySdb91NnySxchIhcAPiA/pF9qvqLVC8uIkuAacBpItII3AP0c8//IM5A+hwROY5z09/ntRDnHMk1gQAA4QlLmFJ1M6HQMKZNw5ZcLTLpWu0t2dXl4lmwwGlZRMYk+vaFH//YSRKdY4sMTz71FFx3HaxY4ZTSXnBB13i/+U2n5RBJCpFkYUmiq27nehKRe3C+zH3ASuBq4EVVvSHj0fWSVT2lUXU1N710pzNf1EQfwaDXAZlsyuU5juLFpgovvthxXy7Em+t6XfXkugGYDrytqrOBi4GSNMZncpk7bXl4+nynK8rGLYpKLs9xFC+2667rui8X4s1nySSKI26Z7HERGQTsxRYuKj6BQHuymLvLEkaRyOU5juLFtnx51325EG8+S2aMYp2InIozIeB6nEkB12YyKJOjAgHCE5bA9u1MqbuHUMi6ovJB5775ZMcWOlcsRY9RgDNQHN2fr+o8ovdFxgBixQCx40rm2NZWZ/xh9WqYOrU9tqfdea3LypyWRXS81rLovW4TharOdV8+KCKrgEGqujnR35gC5vc7lVHMb0sWgCWMHLVqlTMgHfmSjHz5DxjQ/fQT8SqWwKkiWrGi43m//nVn0PnHP3aSQ6QE9R//gCuu6HhsZaVznvLyjnG9/jqcfXb3x65Y4YxDDB3ascLpueecwe7rrusYb7HMyZQpidajmNT5AQwFTnJfm2IWCBCcWEuQShu7yFHpWB0u1hxHM2c6X+bR5122DN5+27m/IXrupE2b4P33O97LsHy5s6zoa691jGv1ameepehjn3qq/djlyzsee+qpsH+/kzQiyeOkk5wWRkSxzcmUKXGrnkTkefdlf2AysAkQ4CLgz6qas1ONW9VTllVXM6XuHkqn+5gwwcpoc0mmqpZinffyy6GhoevcSXff7XQJRR97xRXO9TvHNXOm84WfTCVTvGOti6l3elX1pKqfUNVPAG8Ak9zpMT4MXArsyEyoJi+5rQteWkN9VYNNXZ5DMlW1FOu8n/tc7LmT+vbteux118WOq0+f5CuZ4h1rSSL9kql6Ok9Vt0Q2VHUrcEnGIjJ5qSzoo2ZhM8GJtTTVWbLIFZmqWop13mXLYs+ddOJE12OXL48dV2tr8pVM8Y61Cqf0S6bqaZuIPAxUAwoEgG0ZjcrkrbKgD0K1hOogtGUkpRcOi9zobbKsu6qlZH99d65CilVxtGwZPPEEHD0KkyY5LYnIGMWdd0K/fvDxjzvHLl/u3FUNMGNGe1yrV8PWrc6d1N1VMsU7Nhx24o0MZseK3/RcMi2K2cArOIsVfQ1ocPcZE1NZ0Ed4zhKCwx621oWH0rEyW6y5nmJVHF13nXPOwYM7zp108cXOnEzDhrUfe+21cMopzrHRcU2d6lRIda5kihwbXck0daozJ1MkSUT2jxgB27e3x9/TualMbMmUx74P/Mh9GJMcv58yP22ti2qcMlprXWRXKvMsxZvrafVq58u8ubm9RHbFChgyxElCkXP36dM+kL16dcdjTzrJOTYikhQg9rGxKpli3XNx7rnt1Vi9nZvKdJXMXE+XA98DziIqsWRrBtnesKqn3FIbamDt/nOo3+ujdLrPkkUeiVc11ZOKo55UXqVapZXLc1PlulTnenoEuA+4AvhI1MOYpJQFfVRUlrQNdK+1+/rzRryqqZ5UHPWk8irVKq1cnpsqnyWTKA6q6m9Uda+q7o88Mh6ZKThlQR/TRjRQX2XJIl/Eq5rqScVRTyqvUq3SyuW5qfJZMlVPz4vID4FlQNt62aq6IWNRmYJVUVnCvPIG6qugfvEAps0aZzfo5ZDofvzIXdThcMfKohdeiF9xBB1/wfek8irVKq10VXmZrpJJFB91n6P7rhT4ZPrDMcWgorIE1i5h3tKLqa86gt/v8zokQ9d5ocAZWD5xomMV0tatTnVS54oj6FpNlWi+qFSOjSXVvzfxdTuYnY9sMDt/zCtvoX6vDyb6bBU9D8X6NZ7ofoepU2PPHhvvy7gnM9j2drbbdP19sUo0mJ3sUqifAc6n41Ko96YnPFPMKipLqA3VsnZ/A/VVPmtdeCT613c43N5dU1bWPidTZF90S6LzORKdPxPHZuLvTVfdDmaLyIPATcBXcSYFvBGnVNaYtIhURZUOOEBotg10eyVWxVC8OZnsy7e4JFP19DFV/RJwQFXnA5cBZ2Q2LFOMahY2UzrgANsXryE0u8HrcIpOrIqheHMyFWCPtUkgqaVQ3ef3RKQUOAaMS8fFReRREdkrIlvjvC8icr+I7BCRzbYORuGrWdjcljAsWXT9Qk70Bd3amng70Xki5a6RMYoFC5ypwGtrnTurI/umTOm4DkUq8Zr8kUyieMZdCvWHwAZgN/B4mq6/GEi0pMjVwHj3cStQlabrmhzXlizm7iJUXpy37cSaZynevEULFrQvGATtCwctWJDceVescFaXi64Yuu46Z0qMc89Nbq6onsRr8ksyiSKkqv9Q1aU4YxPnAf+Vjour6mqgOcEhM4FfqONPwKkicno6rm1yX83CZsKzHqX08KuE5u7yOpys6snqdK2tzspwmzd3XF1u82Znf3TLItF5zz67vQwWnOfy8vYlSCP7Yq0Yl47V9EzuSmaupw2qOqm7fb0OQGQs8IyqXhDjvWeAH6jqi+52HVCuqusSndPKYwvPlNnnwIABlH5sXNHMFdWTeYuik0PERRe1z+Ta2/NmKl6Te3o115OIjBKRDwMDROTSqLWzpwEnZybUrmHE2Bczs4nIrSKyTkTW7Tt4MMNhmWwLL9pJcOwT8NKaomld9GTeosi03tFiJYmenrcnbJ6lwpWo6+nTwH8DY4AFUY9vAHdnPjQAGulYYTUGaIp1oKo+5C7XOnn44MFZCc5kV2QVPY4cKYpk0ZN5iyItimjRYxa9PW9P2DxLhSvRmtmPuWtmz1LVT0bW0FbVGaq6LEvxPQ18ya1++hecCQr3ZOnaJkeFF+2klCZCsxsKNmF0vlM6UcVRdLfTRRdBTY3zHD1m0ZvzZipek3+SGcweIyKD3C/rh0Vkg4hcmY6Li8gS4I/AuSLSKCK3iMhtInKbe8hK4HVgB/BzYG46rmvyX83CZsKLdhZs66Inq9P16eOs9hY9JvGd7zjbJ5/csfspHavepRqvyT/JDGZvUtWLReTTwO3APGBRugazM8EGs4vLlNnnwIiRAAQrh3kcTXr1ZN6i1taOSaHzdm/Pm6l4TW5JdeGiyH/ma3ASxCZiDzIb44nwnCWEL5wLe98puHsuejJvUeekEC9J9PS8PWHzLBWmZBLFehH5HU6i+K2IDAQS3PNpTJb5/RAIEJ6zpC1ZVFd7HZQxhSOZRHEL8G3gI6r6HvABYHZGozKmN/x+wnOWMI3naaprKLjWhTFeSXgfBYCqtqrqBlX9h7u9X1U3Rx9jTM7w+6moLOnQujDGpCZRi2JlEn+fzDHGZJ/bumDvO04ZrSUMY3otUaK4WEQOJXgcBkZmK1BjeszvJ7xop7UujElR3BXuVLVvNgMxJmP8fsIsYUrVzYRCTvlsMOhxTMbkkWQGs43Jf34/4enzCZfeBNsaCIW8DsiY/GGJwhSPQMApo50+vy1Z2LKrxnTPEoUpPm6yKN29hvoqa10Y052kEoWI9BWRUhE5M/LIdGDGZFQg4MwXFdW6MMbE1m2iEJGvAu8AvweedR/PZDguY7IjqivK7uY2Jra4VU9R7gLOVVWrLTSFKRAg2FRLqA5CW0bCsGFWFWVMlGS6nt4EbMk4U9DKgj6nKurCuda6MKaTuC0KEfmG+/J1oF5EngVaIu+rqs3jbQqLuxh3pHVRjS96tzFFK1HX00D3+W/u4wPuA+KsW21MISgL+iBUy9otDdTv9VGNz5KFKWqJ7syeDyAiN6rqr6PfE5EbMx2YMV4qC/ooA2pD7a0LSxamWCUzRvGfSe4zpuCUBX0EJ9Y605bP3WVjF6YoJRqjuBpnsaLRInJ/1FuDgOOZDsyYXOG0LnZSG2ogVFfG2gk+/H6vozImexKNUTQB64EZ7nPEYeDrmQzKmFxUFvSxtryB+iqor/dRWmoD3aY4JBqj2ARsEpFfqeqxLMZkTM6qqCyB6vnUNk2y1oUpGom6nrbgVjdJjBXSVfWiVC8uIlcBPwb6Ag+r6g86vT8NWAHscnctU9V7U72uSY9PfuMbHDrY9RabQYMH89x9BVw9HQhQBm2tC9wyWksYplAl6nr6rPt8u/v8S/f5i8B7qV5YRPoCPwU+BTQCL4vI06ra0OnQsKp+tssJjOcOHTzIusGDu+yfHCN5FKKKyhLmlTewffEBmo4MAax1YQpT3KonVX1DVd8ALlfVoKpucR/fBj6dhmv7gR2q+rqqHgUeB2am4bzGZE1FZQk1C5uZNqKB+qrOv3GMKQzJlMd+UESuiGyIyMeAD6bh2qNxpgeJaHT3dXaZiGwSkd+IyPnxTiYit4rIOhFZt69IftGa3FFRWcK0EQ3O+txzd9k6F6agJJMobgF+KiK7RWQ3sBD4Shqu3XXgo+sd3xuAs1T1YuAnwFPxTqaqD6nqZFWdPDxGd4gxmVZRWUJ4zhKmDVxvrQtTULpNFKq63v2ivgi4WFUvUdUNabh2I3BG1PYYnJLc6GsfUtV33dcrgX4icloarm1MZvj9VFSWUDrggNO6sHUuTAFIVPUUUNXqqMkBI/uBtEwK+DIwXkTGAW8Bnwe+0Olao4B3VFVFxI+T2Gy68wwafeONcKxrNfT7qow7rWOOfufAAUYdOMCYIUM67B8Uo0VXbBVSNQubqQ29yNr9DYRm+wgu8nkdkjG9lqjqKTIOMTDBMb2mqsdF5A7gtzjlsY+q6isicpv7/oPADcAcETkOHAE+r6o2IWEmHTvGW/36ddk9qqWla4XT4MFMPniQdY880u1pi7FCKjJf1E1zndbFtDlWFWXyU6Ib7n7mvqxU1fczcXG3O2llp30PRr1+AHggE9c2JltqFjZz01zYvngN9VVDrHVh8k4yg9lbRWSNiPxARK4RERspNqaHahY2U7OwuW3swph8ksxg9oeAm4EtODfhbRKRjRmOy5iCVLOwGYDQ3F2Eym24zeSHbhOFiIwBLgemAJcCrwA1GY7LmIIVXrST8KxHKT38KqG5u7r/A2M8lmgwO+JvOBVK/0dVb8twPCYD4lUyvdfaysl9Ov5WONHaypCWFko6HdtC14Hndw4c4Bgw+ZZbOuyPVck0yB347ixWhVRR8Pup8TczZfYQQnN3UfqxcTYTrclZySSKS4ErgC+IyLeBvwIvqGr3pS4mNySoZOq8/0hLCx8C3vrQhzrsj1XdNPmWW5KuZCrEEth0CC9qX+ci9NIAggvHeR2SMV0kM0axCXgMWAQ8B3wcmJfhuIwpGmVBH+FFO+HIEeuKMjkpmTGKdcAfgeuAV4Gpqjo2w3EZU3TCi3ZSSlPbfFHG5Ipkup6uVtV9GY/EGONWRTUzZfY5hMr3E6wc5nVIxiTV9WRJwpgsC89ZAnvfIVS+38pojeeSaVEYj/VknqRYFU7vtbYyqqWly9+3QJf9R4F+wKgdO7oeO2NGh30KDN23jwGdKqeIMXBebHM9pczvJ8wS2L6dKXX3ECrHWhfGM5Yo8kCP5kmKUeE0qqWFPXSd130U8HanfRcD/wN0XvhjFPB2pyVxW1UpBd46++xu4yrGuZ5S5vc7CWPCEqZU3WzJwngm0eyxn0v0h6q6LP3hGGO6cFsXU6puJhQaxrRptj63ya5ELYqyBO8pYInCmGzx+wlvn89NL91JfdUQ6peOtNaFyZpEs8fOzmYgxphuBALUBJph7aq21kUw6HVQphgkNUYhIp/B6bbuH9mnqvdmKihjTAJu68IZ5B4JwyxhmMzqNlGIyIPAycAngIdxFhOypeOzKNY8SY0HDtCPrvMsva/K6E5VTy3A6THO24IzSN153/R4x8ZYMyrWHFCx5m+yuZ7SLBAgPCGqKirks2RhMka6WzBORDar6kVRz6cAy1T1yuyE2HOTx4/XdQVecplonqVkVpzL9nlNBlVXM6XuHpjoo7QUm1zQ9MqMGbJeVSfHei+ZhYuOuM/viUgpcAywmcuMyRWBAOHp8wlSSVNdA6GQ1wGZQpNMonhGRE4FfghsAHYDj2cwJmNMTwUCzuSC0+fDtgaqq70OyBSSZAazQ6raAiwVkWdwBrQzsoa2MSZFgQDBplpCdRCqAyba2IVJXTItij9GXqhqi6oejN6XChG5SkReE5Ed7loXnd8XEbnffX+ziExKx3WNKWSRacutdWHSJdGd2aOA0cAAEbmU9hkgBuFUQaVERPoCPwU+BTQCL4vI06oavfL81cB49/FRoMp9LnqZqiKy6qQCEt262DKS0guH2UC36ZVEXU+fBmYBY4DoEqJDwN1puLYf2KGqrwOIyOPATCA6UcwEfqFOadafRORUETldVfek4fp5LVMT6dkEfYWlLOijrHo+tU2TCNWVUY3PkoXpsUR3Zj8GPCYi16vq0gxcezTwZtR2I11bC7GOGQ0UfaIwJmmBgDMfT8hpXVTji+w2JinJjFGsEZFHROQ3ACLiE5FbuvujJHSezBScOaR6eoxzoMitIrJORNbtsxlJjemiLOgjOLGWCVuepKnOxi5M8pJJFIuA3wKl7vZ24GtpuHYjcEbU9higqRfHAKCqD6nqZFWdPNz6042JqSzoo6KyhODEWprqGlhrcyyYJCSTKE5T1SeAVgBVPQ6cSMO1XwbGi8g4EfkA8Hng6U7HPA18ya1++hfgoI1PGJO6sqCPaSMaqK9qIDTbWhcmsWTuo/iniAzD7fKJfGGnemFVPS4id+C0VvoCj6rqKyJym/v+g8BK4BpgB/AeYDPaGpMmFZUlwE5qQw2E6mDtBJ+tc2FiSmaup0nAT4ALgK3AcOAGVd2c+fB6pxjmejImneaVt1C/1wcTfbYwUpFKNNdTty0KVd0gIh8HzsUZXH5NVY9182fGmDxSUVkC1fOZt+VG6qt8gLUuTLtkphnvD8wFrsDpfgqLyIOqatN4GFNIAgEqgHnlDdRXAW4ZrSUMk8wYxS+AwzjdTwA3A78EbsxUUMYY71RUljCvvIHtiw/QdGQI1rowySSKc1X14qjt50VkU6YCMsZ4zxnobmZe+R7qq8Dv93kdkvFQMuWxf3ErnQAQkY8CazIXkjEmV1RUllA64ACh2Q2E5u7yOhzjkWQSxUeBl0Rkt4jsxpk59uMiskVEcrbyyRiTHjULmwnPWcK0gesJzW7o/g9MwUmm6+mqjEdhjMltfj8VfrhprtO6sHUuiku3LQpVfSPRIxtBGmNyQ83CZoITayndvcZaF0Ukma4nY4xpUxb0UbOwuW3swhQ+SxTGmF6pWdgMYIPcRcAShTGm18KLdsKRI1YVVeAsURhjUhJetLM9YViyKEiWKIwxadGWLMr3Ewp5HY1JJ0sUxpi0CS/aSfjCubDNuqIKiSUKY0x6BQIdWhe2il7+s0RhjMmI8JwlTON5ZxW98v1eh2NSYInCGJMZfj8VlSWE5yyBve9YsshjliiMMZnl97cni7m7LGHkIUsUxpjMc5NFeNaj1rrIQ5YojDHZ4fd3bF2EsDLaPGGJwhiTXX4/4enzCVLplNFa6yLneZIoRGSoiPxeRP7qPg+Jc9xud92LjSKyLttxGmMyJBCgLOjr0LqwMtrc5VWL4ttAnaqOB+rc7Xg+oaqXqOrk7IRmjMkat3VRunuNU0ZrXVE5yatEMRN4zH39GHCtR3EYY7wWCDir6E2f73RFWbLIOV4lipGqugfAfR4R5zgFfici60Xk1kQnFJFbRWSdiKzbd/BgmsM1xmRcINCeLObusoSRQzKWKETkDyKyNcZjZg9Oc7mqTgKuBm4XkanxDlTVh1R1sqpOHj54cMrxG2M8EAg4ZbQf+7a1LnJIMmtm94qq/mu890TkHRE5XVX3iMjpwN4452hyn/eKyHLAD6zOSMDGmNwQKaNlPlPq7qG62gdAIOBxXEXMq66np4Evu6+/DKzofICIfFBEBkZeA1cCW7MWoTHGW4EAwYm1BJoqaaqz1oWXvEoUPwA+JSJ/BT7lbiMipSKy0j1mJPCiiGwC1gLPquoqT6I1xniiLOhzymjdsYvqaq8jKk4Z63pKRFX3A9Nj7G8CrnFfvw5cnOXQjDG5KBAg2FRLqA5CdVA63WddUVlkd2YbY/JCWdBHeNFOghNraaqz1kU2WaIwxuSVsqCvLVmEyvdbwsgCSxTGmLwTGbcIDnvYWhdZ4MkYhTHGpCwQoAwg5IxdVGNltJliLQpjTF4rC/qYNqKBCVuetNZFhliLwhiT9yoqSwCodVsXayf48Ps9DqqAWIvCGFMwIq2L+qoGQrMbbOryNLEWhTGmoDiti53MK2+hvgrAWhepshaFMaYgVVSWtLUuTGosURhjClYkWYRmO4Pc1hXVO5YojDEFraKyhODEWiZsedJaF71kicIYU/DKgj4qKksoHXCA0GxLFj1licIYUzRqFja3JQtLGMmzRGGMKSo1C5sJL9pprYsesERhjClKba2LubsIle/3OpycZonCGFO0ahY2E/7Ytyk9/Kq1LhKwRGGMKW6BQIfWhc0V1ZUlCmOMwWldTBu4Hl5aY11RnYiqeh1D2onIYeA1r+PIkNOAv3sdRIbYZ8tP9tnyU+fPdpaqDo91YKHO9fSaqk72OohMEJF19tnyj322/GSfzWFdT8YYYxKyRGGMMSahQk0UD3kdQAbZZ8tP9tnyk302CnQw2xhjTPoUaovCGGNMmliiMMYYk1BBJQoRuUpEXhORHSLyba/jSScReVRE9orIVq9jSScROUNEnheRbSLyiojc5XVM6SIi/UVkrYhscj/bfK9jSjcR6SsifxGRZ7yOJd1EZLeIbBGRjSKyzut40klEThWRJ0XkVff/vcsSHl8oYxQi0hfYDnwKaAReBm5W1YKYwEVEpgLvAr9Q1Qu8jiddROR04HRV3SAiA4H1wLWF8N9NRAT4oKq+KyL9gBeBu1T1Tx6HljYi8g1gMjBIVT/rdTzpJCK7gcmqWnA33InIY0BYVR8WkQ8AJ6vqP+IdX0gtCj+wQ1VfV9WjwOPATI9jShtVXQ00ex1HuqnqHlXd4L4+DGwDRnsbVXqo4113s5/7KIxfZoCIjAE+AzzsdSwmeSIyCJgKPAKgqkcTJQkorEQxGngzaruRAvnCKRYiMha4FPizx6Gkjds1sxHYC/xeVQvmswH/DwgCrR7HkSkK/E5E1ovIrV4Hk0ZnA/uARW634cMi8sFEf1BIiUJi7CuYX2+FTkROAZYCX1PVQ17Hky6qekJVLwHGAH4RKYhuQxH5LLBXVdd7HUsGXa6qk4Crgdvd7t9CcBIwCahS1UuBfwIJx3QLKVE0AmdEbY8BmjyKxfSA23+/FPiVqi7zOp5McJv29cBV3kaSNpcDM9x+/MeBT4pIQU3QrapN7vNeYDlO93YhaAQao1q3T+IkjrgKKVG8DIwXkXHu4Mzngac9jsl0wx3wfQTYpqr3eR1POonIcBE51X09APhX4FVPg0oTVf1PVR2jqmNx/l97TlUDHoeVNiLyQbe4Ardb5kqgICoOVfVt4E0ROdfdNR1IWDxSMLPHqupxEbkD+C3QF3hUVV/xOKy0EZElwDTgNBFpBO5R1Ue8jSotLgf+Hdji9uUD3K2qK70LKW1OBx5zK/L6AE+oasGVkRaokcBy53cMJwH/o6qrvA0prb4K/Mr9Uf06MDvRwQVTHmuMMSYzCqnryRhjTAZYojDGGJOQJQpjjDEJWaIwxhiTkCUKY4wxCVmiMDlHRGaJSGkSxy0WkRuS3Z+GuO6Oej02mZl83Vh2ichtCY65RESuSWOcs0TkgRTPsVtETnNfv5TOmETk6yLyt1RjNNljicLkollAt4nCA3d3f0hM/6GqDyZ4/xIgbYmip0Qk4f1UqvqxdF5PVX8EfDed5zSZZYnCZJT7y/tVEXlMRDa7c+Cf7L73YRF5wZ107bcicrrbEpiMczPQRhEZICLfFZGXRWSriDzk3s2d7PW7XMPdXy8ile56EdtFZIq7/2QRecKNtUZE/iwik0XkB8AAN6ZfuafvKyI/F2etid+5d193F8+N7ufYJCKr3Rue7gVucs99k4j4ReQld8K2lyJ30Lq/ypeJyCoR+auIhKLOO9v9HC/g3MQY2V/mfoa/iMgfRGSku/977r/l74BfiMgw9zP8RUR+RtTcaSLyrvt8rxvjRhF5S0QWufsD7r/jRhH5mXuDYdyYTB5SVXvYI2MPYCzO5IyXu9uPAt/CmXL7JWC4u/8mnLvpwZkTaXLUOYZGvf4lUOa+XgzcEOOai4EbkrjGAvf1NcAf3NffAn7mvr4AOB6JBXi30+c6Dlzibj8BBOLFErW9BRjtvj7VfZ4FPBB1zCDgJPf1vwJLo457HRgM9AfewJnf7HTgb8Bw4APAmsj5gCG031j7v6I+8/dw1v4Y4G7fD3zXff0Z97/ZaZ0/t7s9GNgMfBiYCNQC/dz3FgJfShRTrM9sj9x+FMwUHianvamqa9zX1cCdwCqcL+Lfuw2EvsCeOH//CREJAicDQ4FXcL6cunNuN9eITEC4HueLH+AK4McAqrpVRDYnOP8uVd0Y4xyJrAEWi8gTUdfvbDDO1B/jcb6w+0W9V6eqBwFEpAE4CzgNqFfVfe7+GmCCe/wYoMZtSX0A2BV1rqdV9Yj7eirwOQBVfVZEDsQKzG3N/Qr4kaquF2fanA8DL7v/xgNwplT/aIKYTJ6xRGGyofM8MYrTtfGKqiZeglGkP86v1Mmq+qaIfA/n13QyurtGi/t8gvb/F5Lu1or6+8g5uu16UtXbROSjOL/aN4rIJTEOqwCeV9XrxFmjoz7BNSNxx5uL5yfAfar6tIhMw2lJRPyzc3jdxe/+faOqLnK3BXhMVf8z+iARuTbJ85k8YGMUJhvOlPY1eW/GWRL0NWB4ZL+I9BOR891jDgMD3deRpPB3cdas6Ek1U6JrxPMi8G/u8T7gwqj3jokzJXqvicg5qvpnVf0u8HecrqPozwtOi+It9/WsJE77Z2CaO87QD7gxzrm+nOAcq4EvujFejdNl1Tn2z+IsNXxn1O464AYRGeEeM1REzuomJpNnLFGYbNgGfNntxhmKs2DKUZwv/UoR2QRsBCLVNYuBB8WZTbYF+DlO3/5TONPJJ6Wba8SzECe5bAbKcfriD7rvPQRsjhrM7o0fisgWcUprVwObgOcBX2QwGwgB/1dE1uB0lyWkqntwfun/EfgDsCHq7e8BvxaRME5iimc+MFVENuBMqf23GMd8E6caLTJwfa86a5v/b5yV4DYDv8dZAz1RTCbP2OyxJqPcrpNnVDUvVnZzK3b6qer7InIOzi/mCW7S6c35FuN8/ifTGGbeE5FZON2Jd3gdi+mejVEY09HJwPNud4kAc3qbJFwHgQoROU0T30tRNETk68BtOKsamjxgLQpjjDEJ2RiFMcaYhCxRGGOMScgShTHGmIQsURhjjEnIEoUxxpiE/j+5IRccmHQ2fwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lrgd = LogisticRegressionGD(eta=0.05,n_iter=1000,random_state=1)\n",
    "lrgd.fit(X_train_01_subset,y_train_01_subset)\n",
    "\n",
    "plot_decision_regions(X=X_train_01_subset,y=y_train_01_subset,classifier=lrgd)\n",
    "plt.xlabel('petal length [standardized]')\n",
    "plt.ylabel('petal width [standardized]')\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a logistic regression model with scikit-learn\n",
    "In the following code, we will use the sklearn package to fit logistic regression. First, we will standardize the features using the StandardScaler class from scikit-learn's preprocessing module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/6c/mzld6tfn7g1cw7rzfxzv544h0000gp/T/ipykernel_11766/271874885.py:21: UserWarning: You passed a edgecolor/edgecolors ('black') for an unfilled marker ('x').  Matplotlib is ignoring the edgecolor in favor of the facecolor.  This behavior may change in the future.\n",
      "  plt.scatter(x=X[y == cl, 0], y=X[y == cl, 1],\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAzSklEQVR4nO3de3xU1bnw8d9KMoSEQGIICnI1iIIECBLHktYDXqpc5IAVD+gbeqweL8eqreKh7dFXXyu+x6Joa2mLLVJf5YBYaqMIAiqCekINCeESIFKDBNBQLumEhIRkLvv9Y89OJpOZySSZzMyeeb6fTz5hZu+ZWfOxPKyu9TzrUZqmIYQQwrwSIj0AIYQQ3SOBXAghTE4CuRBCmJwEciGEMDkJ5EIIYXJJkfjQfv2ytIsuGhGJjxZR5Nw5SEivJSUFeqvekR6OEFGvfFf5aU3TBng/H5FAftFFI3jxxZJIfLSIMrtPH2bQtyu4bsToSA9FiKg3Mnlkla/nZWlFRFTz4WxqzsC60gpOOE5EejhCmJIEchFRViuMOT0Dpy0Dm9MW6eEIYUoSyEVUcJwYSHk5VDRVRHooQpiOBHIRFXKzsmmqGE15ObLEIkQnSSAXUSM3K1uWWIToAgnkIqoYSyxbj8gSixDBkkAuokpuVjZ1G2dQc0aWWIQIlgRyEXWsVnDaMvhsj002P4UIggRyEZVyHPnUbZwhmSxCBEECuYhaViuSySJEECSQi6hmpCUeOG6L9FCEiFoSyEXUM8r4JZNFCN8kkIuoZ7UimSxCBCCBXJiCsV7+2R6bBHMhvEggF6ZhVH7KerkQbUkgF6aS48jnVGWGpCQK4UECuTAdKeMXoi0J5MJ0jDJ+IYROArkwJasVTlVmSGchIZBALkwsx5Evm59CIIFcmFxDUb7kl4u4J4FcmJqclCiEBHIRA3Ic+S2Ha0kmi4hHEshFTJCGFCKeSSAXMcNzmUWCuYgn3Q7kSqmhSqmPlVIHlVL7lVI/CsXAhOgKyWQR8SgUM3IHsFDTtDHAt4AfKqWuCMH7CtElUsYv4k23A7mmadWapu1y/7kOOAgM7u77CtEdRhm/BHMRD0K6Rq6UGgFMBD73ce1epVSJUqqktvZUKD9WiHaMzkLfVEd6JEL0vJAFcqVUGvBn4Meapp31vq5p2u81TcvTNC0vPX1AqD5WCL9ys6SzkIgPIQnkSikLehD/b03T3g7FewoRCkZKoiyxiFgWiqwVBbwKHNQ07cXuD0mI0DHaxJWXS365iF2hmJF/G1gAXKeU2u3+kTNGRdSQ/HIR65K6+waapn0GqBCMRYgek+PIZ3fFYWw5FQxMGhjp4QgRUlLZKeJG8+FsOY9FxCQJ5CJuGOvlch5Lq6rKKp585EkmDJrApSmXMmHQBJ585EmqKqsiPTTRCRLIRVyxWqGpYjSf7bHF/cx826ZtzJw8k7X711JfUI/2uEZ9QT1r969l5uSZbNu0LdJDFEHq9hq5EGaTm5VN8cZsmLGRE0NOxOWaeVVlFQ9+/0Ea5zbCUI8LmeC41oHjUgcPfv9BNuzYwPCRwwHYvmU7q/+4muNVxxkyfAjjJoxj3559LY/v+MEdTLlxSmS+UJyTGbmIS/HekOLVZa9iz7W3DeKehoJ9gp2Vv1kJ6EF8+W+Xk/9QPo9tfozcH+TyduHbXHj1hTy2+THyH8pn+W+Xs33L9vB9CdFCArmIW0ZDings43/nzXdwTHAEvMeR66BwTSEAq/+4mmmLpnHJxEtITEokbXAacxbP4YsdX5CYlMglEy9h2qJprP7j6jCMXniTQC7iWvPh+CzjP2c7B+kd3JTuvg84XnWcYeOGtVxqbm5mRN4Izhw90/LcsHHDOF51vCeGKzoggVzENc9MlnhaYumT0QdqO7ip1n0fMGT4EI7uO9pyqVevXhwpOUL/Yf1bnju67yhDhg/pieGKDkggF3HPyGSJpzL+2fNnk7QncK5D0u4k5tw+B4A7fnAHm5Zs4quyr3A6nNR/XU/hE4VcPvlynA4nX5V9xaYlm7jjB3eEYfTCm2StCIGeyVJuO8Fne2x8ZwJRmcninTXS1SyRZc8t46P3P8LxtQMuxfeG5zGw7LFw1/K7AFo+Z/WvV/NW1VsMGT6E7835Hvs+38cLb73AkOFDuP+B+yVrJUIkkAvhluPIp9xWhM1pi7pAbmSNTFs0jWHjhnF031GWL1kO0Knguey5Zbxd+Dbzls2jsbaRlfeuxDHeAXnoa+a1+kzcssfCsteXtaQeGp8jgTo6ydKKEB6itbOQd9ZIV7NE/rT6T8xZPIdLJ1/KuGnj+M/t/8n4AeNRyxXqWUXaqjTmj5vPhh0bmDptas98GRFyMiMXwkNuVja7K6CcCjImRE+x0OFDh3H8t4Ml31vC+drz9E7vzbdu/RZHDx1td29VZRWvLnuVd958h3O2c/TJ6MPs+bO5+8G7qT9bz4i8EW3u75vVF03TUO6z7zRNC8dXEiEkgVwIL8Z6+YHjNgaOiHwg37ZpG0e+PMKRC47gWuCCdDhfe55Pyj6BL/Xrxux526ZtPPj9B7Hn2nEUOCAd6mvrWbtnLW9PfpvUtFSOlBzh0smXsv+D/a1LK/8OWrrW5t5lry+TWblJqEj86ztqVJ724oslYf9cIYJVXAx9Z2wksz9cN2J0xMZRVVnFzMkz25fSG46BWqUYmzsWl93Fwf0H0Qo0v/cmrUniwhEX8t1Hv8uaR9fQfFuz33tT1qXw1PNP8eGmD31usIZq8zVY4f68aDQyeWSppml53s/LjFwIH6xW4PQMym1FrDtTwXcmZERkmSWYUnp1tcKZ4qT6QDXKqtCG+pmcDQXyYEDzANb9ZB3NOX6CuPve5vHNvPhfL3L/yvvbbbACIdl8DVaoNntjlWx2ChGAUcZvc9oi8vnBlNK7cl18VfwVDbUNuCa6At7ryHVQeagSpSk9UyUA50QntTW1PjdYQ7X5Gqxwf57ZSCAXogNGQ4pIZLIEW0rfVNtEc31z0GX3wb5vc11zm6eMMnzvkn3Paz0h3J9nNhLIhehAJBs4B1tKn5yeTK+0XkGX3Qf7vr369mrzlFGG712y73mtJ4T788xGArkQQTDK+A8ct4X1c4MppU/YncAl1ktITU8loSzwX2mj7D6Y900sSyQ9M72lLN+zDN+7ZL+nS/TD/XlmI5udQgSp+XA2NaMrWHemgrmTwpPJcveDd/P25LdxXOrwm12ifa6RmJvIxQMupra4Fi4jYNl9TkEO7657F+dfnQFL9LWdGjPvm0nRr4tayvK9y/A9S/ZDUaLvLzPF1xEBciRAK0k/FKKTypOKmDIlfOextOSGT7DjyHX4LKVvl0fu5977fnwfRcVFTFs0jdq/1/Lbe36LfZy9TYl+QlkCSXuTuPVnt1K5vTJsAdNXZsqmJZskYHvwl34oSytCdJLjxMCwdhaaOm0qG3ZsYP64+aStSgtYSt/RveX7yluyP3Kn5bL4k8XkDcoj8Q+JsBiS/18yU6+YyjPbnuHG+28Ma2aIZKZ0nczIheiC3acPkzw6fEssoTIzfyaPbX6MxKTEluecDicv3PQCgN9rG4o2RHRs4fh8M5AZuRAhlJuVjdOWYYrOQlWVVTz5yJNMGDSBil0VPJD9AK8vep2TX50E/GeinPzqJL+793f8bf/fuDTlUiYMmsCTjzxJVWVVj4xTMlO6TmbkQnRRtJTxB9Lm7JUJjoDr4NBardmyfj7eDpNoXWvfk4Rlt6VHzmGRNfKO+ZuRSyAXohuMYB5tSyzbt2xnxa9XsGP7joBnr6g3FDfMuAEtQeN41XFSeqdgd9jZv3t/wNelrEthw44Nbc4r7+o4PbNUxk0Yx749++L6PJVAZGlFiB5gtYLTlsG60oqoaRNnzGyb0ppI+FZCh+e07NixgwuvvpDHNj/GrMWzOHnyJAnWwK+zT7Cz8jcrQzLO/IfyeWzzY+Q/lM///PV/uOMHd7ChaAOvrHlFgniQJJAL0U05jnyctoywFwv5Y2R/7P94P84JzoD3uia6aG5s5osdX7RkitjO2HBODPw6R66DwjWFIRmnZKl0nxQECRECDUX51GRs7NZJid4NIVL6pjB4xGC+rvqaxrONbRpEAH6bRxjnkpyvPR/UeSr2ejtnjp5peaozZ7Z0h7/zU96qeqtb7xuPJJALEQLGsbe7Tx/mQP+KTjekaNcQ4hQ0FDbwtz5/g+/TpkHEn6x/Ahe4rnL5bB5x+djLObrvKL3Te+vBPDPAB9eCJc1C/2H9W57qldaL5trmDl/XJ6NPp76jNyNL5ZKJl7Q8J1kqXSNLK0KEUPPhbGrO0Km0xKrKKh78/oM0zm3Eca37yNp3gDuAG9EDaqL+23Gtg+Z5zTRrzXoWite1xrmNHNh7gMKnChl77VgS9yT6+VRdQlkCvVJ6cfnky1vOMMnon0FiWeDXGWe2BGP7lu3cd/t9zMyfyX2338f2LdsB3+envLnoTf5R849294rAZEYuRAh5NqQ4MSS4np/tmkcUA1cScLORSe77prW/5rrSRVZGFufrz+P6qyvgeSrshNvuu41jnx/jhbdeYMjwITz6s0d5+j+epvEy/12JLHss3LX8rg6/WzANIYzzU1J6p5DcJ5mZT82U5hGdFJL0Q6XUSuBm4KSmaTkd3S/phyLWGZWfwayXTxg0gfqC+taljOeBuwm8tFEDvAr8h+9raavS2FO9p1PntHjq6uu83Xf7feQ/lN9m+eSrsq8o+nURr6x5pcv3xqueTj98jfZzA//OdW+TRIhol5uVHfSxt+2aPDQQ1GYjDf6vGRuRnTmnxVNXX+etMw0hpHlE14VkaUXTtE+UUiOCfkFdHSz5BSz6SSg+XoiolJuVTXnlCSoGVTA62X/BUJ+MPtTXeszIU9GbPnSw2Uiq/2tJvZOoqqxi+MjhDB85nKdffJqnX3y6U+Pv6us8dWZDUzY/uy5sm51KqXuVUiVKqZLmFHf7qCW/0EvjhIhRjhMDO2wT167JwzhgVwdvvMt9ny+l4OjvYObkmWzbtK1zAw6xzjSEkOYRXReyEn33jPy9YNbI80aN0kpefBGKi1m/LU1/UmbnIkYVF8PwBRv9nsdSVVnFzMkzaZzr3lysAVYAt+N/k3I1cA/tZ+3HgDXAvwHnQldK74u/JhBdva+z98ajHj9rpUuB3G39kgP6H6Ze6972FyK2lCcVkZhh87v52W5z8RRQCEykTdMHtUuRWJaIy+nCdaWrzTV2uX9uAUbp75u0NYn54+Z3a3nEFzngKjKiOpADbWfnEtBFDCpPKmLCd2x+18urKqtY+ZuVFK4p5JztHMqi6D+sP2dPnaWptole/Xox6dZJXDn7SvoP6c8z334GLUGDRvT18nGAlbazdI8MllCSDJPI8BfIQ7LZqZRaA0wFspRSx4GnNE17tVNvYrUyywqsWgWHdrF+25Vw8WAoKAjFEIWIuIaifMozNkKO/81P74nV2Cljmf7D6bw470VGXjOS0rdL2fHHHfTO6I1m1+B+YECAD+1EKb33EQGeZf/eSzOByus78z4iNCJyjK3PGbk3WT8XMchXfrmmwfbN/s8NTyhNwOl0oqwK10RX61JKCVAGfI+WpZR2gpyR+zu33N/54/5m5O8+/i5f7P8i6PcRnRNV55EHFcgNq1ax/huZnYvYsfv0YQZ9u4LrRoxmzYq+vPPmk5R8/graAh/nf9cAf0Av1/e38WlsbvpIVzTWyAdcOIA/rf4T9WfrSUpMIuuiLEBP+bth2g16Jedc/5WcKetSeOr5p/hw04ct55Y7LA5ueeqWljXywqcKqdhVQfO85h4/xzxemTeQg8zORUwxmlFoGmz8+WeUlv2axssbcF3van/zJvQF0BsCvOEHgJP2JXnuwDn/zvls/WQrcxbP4XzdebYt38bk70/msqsu4/zp8yy/azlnhpzBeb3/o2sTP0qk//H+3L/y/pbA/eaiN+nbpy/n6s4xZPgQnHYnn579tPW8GB96avM1Xpi7sYTVyqxFVzDr4l167vmqVZJ/LkzLaoUxp2fgqs2govI1ms836UsmvuxDP3clkEnAbvTZu1P/nbQ1iZR1KSx7fRkfbPqAOYvncOnkS/l8zefM/N8zGXPDGM6ePRv0+ePOiU5qa2rbnB0+f8l8Lsi8oKUJxM7/2akvpwQQinPMRXvmCOSGggJmTa1nFuth28dSUCRMLceRT1N9I/Z6u/+S/GDL9ZvwW0pff7aeEXkjAKg5VsPgcYPpldoLl1P/xyPY88eb65rbPOVdPt/uqAE/79Pdc8xFe+Y7/dBq1Wfo4F5y+VgP6rLkIqJEdXUlhe8tZdv2VZyvq6d33zSmTilgzs0LGTRoZJt7e/XqhyXtHPZau++S/CDL9dMu8L+hmdYvjSMlR7h08qVkDs3k631fc9Goi0hI1OdxwZ4/3qtvrzZPeZfPtztqwM/7dPccc9GeuWbk3txLLoDMzkVUKC19n4cXjmeLbQWNd9ahPaHReGcdW2wreHjheEpL32+5V9Ng6IB76JWcjCrz81cxiHL9js4Gv+2O2yh8opAvd3zJ1bdfzYZnNnDww4P069cv6PPHE8sSSc9MD1g+3+6ogS6MVXSNOTY7gyEboiLCqqsreXjheJrmNfjN2khem8rLS/e2zMw3bYLtnzzK/kO/hC5mrQSTCbLsuWUhzVrxVT7f7qiBLo5V+GfurJVOkHJ/ESm/+8MDbLGtwHm93e89iR9ZuCnjXu6/Z1nLc5oGa9a8z5833IIz14FrotMjV9xCYqki0QKOXA3XlfYunw0eSKjOHw/V+wjf4iaQA21n5yAzdBEW8wr60XhnXYcNIVJf68ebq2rbXaqurmTlxkXs+3QTjTa92fJlY+ZRZ3sUgDOnX6b+3GrsDfX0yejDnNvncNcP7wrZ7Nb7iICufkao3ke0F1+B3JMUFIkwmT07Ae0JTe+h6c8pYDmk9OnrcyPUu/JT02BO/uCWlxcWfY1SPf5VRJQydx55dxQU6Bui33ytb4gK0UN6903Tl0P8+RuwErgavxuhuVnZOG0ZfLbHRrX9BK/+sm0+36u/TMeYe0VgDiaiVOwHcrd2BUVChNjUKQUklll8X6wB/oK+aXkj+vJLov7beb2dpnkNPLd0LtXVleQ48jl/cDR//NUFrH8rjVn/Us+8u85y4UAH699K49VfpuNy6UF9zYq+If0O/jrei+gWN4EckNm56FFzbl5IUplFP//EWzH62eK+sjnQn3dMtPPOey8BoBT8o9bBmO8e5q4f1dJQn8Dfq5O4cKCDPmkuVv4qnfVvpXGuLiFkM3PjjPH8h/J5bPNj5D+Uz/LfLpdgbgKxv0buj7F2bpANURECpaXv89zSuTRPsKNNas0w4RXgPjq1EVpcDGnTN3JNbgYXJQ7k1V+m8+7atJY18ln/Us/dP64N2Zq5nDEe/eJ3jdwf9+xcCopEKE2aNJ2Xl+5leua9JCzvB4sT9N/NBFW+3lhf3/LQagVXrb5e/kVzRbugHcogDtLF3sziN5B7mLXoCmZNrW89v0WIbjh5soqiHdtwuc6C5sLlPKsfhrEUeBp4Hv1UwxqvF9ZCSlpam6dyHPnUbZzBvn3wX0ub2lzz3PgMBaOLvSfpYm8OEsgNUu4vguAdOL0fr1nzc/73z6/Hdtl+eAC9gXIv4Cr0M8OfAO5GD+wr0DNZ3BLLLEz5pwVt3s/lgquugo9/dTWb3riYq289TGHR18z6l/qWjc9QBXPpYm9e5js0q4fNWnRF28O4QNbPBaCX0zc2wpw5+makpkFhIaSkwLRpsHfvVtasewq+j76pWQO8Q/vy+kz088Uvp03He63YwvHLHsHlgoQEPYgvWrSJc47f4NKOoKlUijbUcfPbGoOHD2Fs7j306XttyJZXjHL71b9ezVtVbzFk+BBppmwSEsh9MfqHgr4puuQXUlAU5zRND+Kffqo/njNHD+KffgrXXKNff2Xlw/rM2wjaxehniQfIVGEi8GdFUk0Klwxbx6FDI3n2WXj8cT2I1yU+w4yfzeCyvGlUlO6m6LXPmPrD67jo4izeX7KUm4adA0IXaKfcOEUCtwnFb9ZKJ7Wc4SKz87hlzMCNYA56EDdm6P98q4J/pzUz5Xn0ZZQOMlUSXunFb391gIED9SC+d69+ydJ3FvOe/xaTrsvmxN+/JG1gH6oPnmDrsg/54ev/zvHdxyWjJM74y1qRGXmQZi26QmbncU4pPWh7BnIjiO/duxVcwKvozSBSCbophKu5mR89NoHmcw307puGXSsgiYXAEXL/aR4AdkcTvXpfwNDxQ6g5+g/qG5wMyhkkGSUCkM3OzpGCopjU0Qam5/OFhW2fKyzUNzifePp6uBp9Bm5saPYicMk+7uu9oMl5Dm2+XrLvyltBszYeR3Mquz+pAsCSlEzz+Wa+Lj9O1tABaM5EDu2sYsCwAZ37siImyYy8C9rMzg2y5GJKHW1gGjyXVYzllMJC2Lx5K9+c8tjgNGSir3+XAt8NMIBd7vvG0rrxeaMdxthpemMP7y52AN/jsrxMKrbupui1/+Haf53JyT0n2bx0C/Memt3p77x9y3ZW/3G133PFhflIIO+qggK93Rzu9fMlv5Az0E0mmA1MIyNEKT24e66Jz5kDf1nvtcHpyYqeYjjaz/Vj6IH839AD/5XoG6TT9PsTrnbRvP9iNj3zVz7ufYSUlL5g78vm5zdx4YARTJr0IL0HXcTWIxVcN2J0UN/ZKMOftmgaw8YN4+i+oyxfshxAgrmJyWZnqEiHIlPqaAPT1/2ez7fb4PT2N+BtIBc94Bsl+7vcP7cAo9z31qCvsf9H62N/Z5cbiouh74yNzJ0UXCCXMnxzkxL9niYFRaZkzKw9+Qvixv1t2Am8oTkKuAt9pv0qsNj924E+Ex/lcW86+gapx2PPkn1frFZw2jLYeqQi4H0GKcOPTbK0EmLtCopkdh7VPDcwGxq2csr2MLNv3a8HWgsMHTqW++56mfHjr6O6upLC95aybfsqGuvqSUxO0v8GVQHZAT4kEVB0nIpYi57t4vHYu2TflxxHPgfPbGQrHS+xGGX4njNyKcM3PwnkPcFdUNSydg6Sshhm3ksg3o+N54xllb59f86XVU/pyx95tCyBHCvZzxNPX8/Ubxewo+RtHBPtOO/UTzV01tqhBFgLzKXt7NpTCajETCitQ/uu/36e7ALGtT5MLLMwdcoCv7d7qts4A2Zs5MSQEwxMGuj3vjt+cAfLl7RdI9+0ZBP3P3B/UJ8jopOskYeJFBSFT7CZKMa9lZVb2bz1+vaZJ4ZjwOvAHPTsEl/XVwP30H7G7X5tZt83ONt4H447Gvx/hpG1kqk/Tl6bystL9zJo0Migvnd5UhGJGTZycmB0sv+ZuWStmJcUBEWYFBSFR2cyUUAP7D/8cYDME9zPX4UebH0F8qHom5l/Bm6ldUOzBNgJFw94mtOnC7hkeH+OrvVxVnkJUIb+DwWQ+JGFpDILP124LuggDvoSS/FGKGcj3/T3v8wiZfixR2bkESCz857V2UyUDjNPoH1Gia/rr6BPjRqAREhIzOGizF+RmnodmZlQUwMORyW2updoOP8GjvP1JPdJ5cILR3Dy70doamggJS2NqVMWMPvmRzoVxD0ZmSxGA2cRO2RGHkVkdt6zApXS+9RR5gm0zyjxdb0Z+BngBBYncMnwfS2f+fjj8NhjYLGMZEDmMpYuXRbSphCerFbYXTGaz6jgOxOQYB4HJP0wUrzL/SVlMWT8ldL7/T+fFoIrpU8N8notqKQ0bDb9MzUNnn229fON8Xk+DrXcrGyctgwOHLeF/s1F1AlJIFdKTVNKfaGU+lIp9dNQvGe8MNrNSYei0PAupV+6VP/96ae+g/mmTZCRPlZfpw6khDYZJe14ZJyoUgu9LAtobIQLL4QLLtBPNMzMhO9+V//9ySf6eFwu/femTV37voHkOPI5VZnBCceJ0L+5iCrdDuRKqUTgN8B04ArgdqXUFd1937gjBUUh4a+U/ppr9Oe9UxIbGyG118uwE30z05dj6Nf9bYYapfZW/c+JuyyMyn6k5fPy8mD8eP33+fP6WnlmJvTuDe+8o/8j09jYMzNzx4mBfLbHRkVTcAVDwpy6vdmplJoM/B9N025yP/4ZgKZp/+XvNfG+2dkhKffvtmDyyI3nCwuhsPDn/KO+fR65kXli5JG3yzgpRc84uQHUGQvstDCw/zpSU6dzzTVwyy365xpdf4zP++ST1vEE2ogNhd2nDzPo28GfxyKiV0+W6A+m7VzmuPs50VUyO+8276AYqOR+zhzo3/9JLh7wEZbdOfBb4BngtzDs7zksfuojHn30DV5eupfpmfeSsLwfLFbwu14kllqgUZG6rR/TM+9l2KC99OkzHaVagzjoQdzz8zzH05NBHPT18lOVwZfxC/MJRdaKr/8JtpvmK6XuBe4FGDZAzlAORrty/4sHw2WXxdQJi55l7+fr6undN42pUwqYc/PCLqffdYbnxmhKynX0a9oHKZCRoQfXa66Bce6174EDR3JR1jKyhy1reb0xmwb9fQ5ZWt+7sLB9kPa3EdvTwbyhKJ+ajI1UDKoIWCwkzCkUgfw4bVcPhwDfeN+kadrvgd+DvrQSgs+ND0b/0OJiOLSe9duuhEOHYiJlsbT0fZ5bOrdN2XtjbR1bylawdeH/46cL1zFp0vSQf66xzGIE1e3b4Z/+SX/u3Xf1e665Rn9spDDOnt26nu15HrlxXdPgs898X/OuMPX3Hj0ZzK1WKN44g3I2kjEhcBm/MJ9QBPKdwCil1CXA18B89L7hIpSsVj2oExvnn1dXV/Lc0rk0zWto15DBeb0d52V2nls6t1Ml6sHwLt8/fBjs9taN0FmzoLwcvvoKHn1Uf01Kir404msT1bgO/q91dKa59309xWqFclsGn+2xSX55jOl2INc0zaGUehDYjH7O20pN0/Z3e2TCr5aCom2YdnZe+N5SHBPtAcviHRPtvPPeS9x/zzI/N3WOd/n+7NnQ0AAnT0JJiV60U1gI//hH63KK5yx52rT2zSY8rwe6ZujoPXpajiOf3RWHseVUSCCPIVKib3It5f4mm53PK+hH4511HZbFd9RYobO8y/c1jZby+XBlkUSaUcLf0eFaIvpIY4kYNWvRFcy6eJe+Gbpqlf5jAufr6oMqi++osUJneTeSUEqfiYcziyTSrFb92NvycqRYKEbIWSuxwOgfauSfm2D9vHffNBprO5iRezVWCEWGi3fWiGf5vMNRyT/qljJ3/ioc58OfQRNOnuexZPa3SY65ycmMPJa488/NUO4/dUoBiWWWgPd4NlYoLX2fhxeOZ4ttBY131qE9odF4Zx1bbCt4eOF4Skvfx+Vq+3rPx8aZJ8ayyne+Ay+8oC+r7N0LCQnvc+LMeOrGrMB+j+/3N94nVuRmZTPm9AxqzsjM3OxkjTyGRfP6eXV1JQ8vHN8+a8Xg0VgB6PDehFWpjL1sL888M5KEBD2IP/sspKbC2LGtmSqbN+sbnEYGCcCnn1Zy8MvxAZs+JK9N5Vcv7OXzz0f6bFBhZsE2pBCRJ2vkcSiaZ+eDBo3kpwvXkbw2lcSPLPp53k6gRm+skLw2taWxQjAZLq5JdvZXvMSzz7YG8b174dw5PXAbh2bddFNrfnhjo/548PClaHkdZ9D88tcv9ei5KJGS48inqWI031RHeiSiq2RGHifazM4hambo1dWVvPPeS2zb/gaN9fU+GysEm+GS8Eo/LB7n0Y4f37qRGajRRNDvv7wf//q/amN2M/Rg1kYy+yPr5VFMGkvEOaPcH9brG6LbPo6KJZdBg0Zy/z3LAuaKB5vhotnr9bPF3R5/vPWMk0CNJoJ9f5e9PmaDOLQ2cJYyfvORpZV4YlSHeqcsRrnefdOCavygLGltnjKWWTpqNBHs+ydY0gI3qDA5z7REYS4SyOOVd4eiKD5hMZgMF0os4FrA+PGwdq2+rLJ3LyxeDH/5S+BGE8Fm0Fw2aoHfBhWxwmoFpy2DdaUVksliIhLI45wZZudzbl5IUpklYOOHhFILY0c/0rKc8vjjejDv00fPXAnUaCKY908qs/Djhx7x2aAi1uQ48nHaMrA5bZEeigiSbHaKFtG6IQpeJyVObG3skFhmIanMwk8XrmPixOkta+LQ2swBOm40Ecz7T5o03W+DilgjZfzRyd9mpwRy0ZZ7Vr7+myv1x1HUoSiYDJdofn+z2X36MMmjK/jOhAw5YCtKSCAXnbdqlR7QoyC7RUSGtImLLlIQJDqvoCBqC4pEeDQfzqbmDNImLspJIBeBeaYrRnl2iwg9qxXGnJ7BqcoMyWKJYhLIRXC8Z+fFxRLU44jjxEA+22Ojoklm5tFIKjtF8Iz+oatWwaFd+vq5STsUic7JzcqmeGM25WyEHKn8jDYyIxedV1BgqoIiERpWKzRVjKa8HJmZRxkJ5KJbzFBQJEInNytbTkqMQhLIRffJ7DyuGJksUsYfPSSQi5CR2Xl8MDJZnLaMSA9FuMlmpwgtd//Q9UsOtOaeXzxYNkRjkJ7JUkFOjk02PyNMZuSiR8xadEVrhyJjyUXEFGO9vLxcen5GmgRyE6usruaBP/yOfgXzSJg9m34F83jgD7+jsjqKdqK8C4pkySWm5GZl47RlcOC4LdJDiWsSyE3q/dJSxi98mBW2LdTd2Yj2hEbdnY2ssG1h/MKHeb+0NNJDbMsoKJLZecxpKMqXMv4Ik0BuQpXV1cxd+hwN85qwX+/Ue00mAplgv95Jw7wm5i59Lrpm5iCz8xhldBaqORPpkcQvCeQmtPS9QuwTHQG7vtsnOnjpvXfCOq6gec/OJWXR9KSzUGRJIDehVdu3YZ/oDHiPfaKTN7ZvC8+AusKYnUvKYswwOgvJenn4SSA3ofq680F1fa+vbwzLeLpNCopihrFeLjPz8JJAbkJpfXsH1fU9LS0lLOMJFZmdm59RLNRUMVpm5mEkgdyECqZMxVKWGPimEkh2urju0UfDM6hQ8Z6dy6aoKUlDivCSQG5CC2+eg6UsKWDX99QSxY6sizhb29HUPTpJQZG5SUOK8JJAbkIjBw1i3cKfkro2GctHiVADOIEasGyB1DcU6/oPZKTFEumhdp+kLJqa0ZBCgnnPkkBuUtMnTWLv0pe5N+Mm+r2WCouh3/IE7q1IZ++gYUzv0yfSQwwtKSgyJan8DI9uBXKl1G1Kqf1KKZdSql1nZ9F5gcruva9N/I8f40Jj1/MvkXNBJv8rLY3X688yqqqKfkcreaDmJE3OwGmKpiKzc1PKceRzqjJDmlH0IKVpWtdfrNQYwAW8AjymaVpJMK/LGzVKK3nxxS5/bqx6v7SUuUufwz7RoeeJpwO1YClLJGGnggSFa5Kr/bXSBJqb7CRdDfY8Wq+VgHMnvPefTzF90qTIfrlQKy5m/ba01sdTr9UXZkVU2n36MMmjK8jsD9eNkJMSu2pk8shSTdPaTZq7dYytpmkHAZRS3XkbQduy+zYVm5l6cQ8lwO20v3a9Ey5zwmp3EM/0uHYjMAZu/q+fc0XaBSQnts106Zeezlaz/oNq9A8Fd1D/WPqHRjGj52fmgo2RHkpMCtsauVLqXqVUiVKq5JRJMyl6UsCy+2JgEgFL8pnkvs/HNVeexhSclKSnt/kxa0ZLO+4lFykoim5WK5yqlDL+ntBhIFdKfaiUKvfxM7szH6Rp2u81TcvTNC1vQHpHZYnxJ2DZ/T7gyg7e4Er3fb7kwRv1dV0fnEm0pCtKQVHUMsr4bU5bpIcSUzoM5Jqm3aBpWo6Pnyg9kcmcApbdNxBUST4N/q/V211dHpupyOw86jUU5VNejmx+hpCkH0aJgGX3qQRVkk+q/2tplvj6T91mdm5UiIqoYLXS0llIgnlodDf98Bal1HFgMrBBKbU5NMOKPwHL7scBuzp4g13u+3wpgQVpfbs+OLPyPmFRUhajhtEmToRGtwK5pml/0TRtiKZpyZqmXaRp2k2hGli8CVh2bwVKCViST6n7Ph/XEkoU20kkr7a2zU+/eNqrkIKiqNN8OJvycjkpMRS6lUfeVZJH7tv7paXM/L9Pk5QH9qtoyQdnF7ATUEAe+samcW0nqBLARXzlkXfHqlWs/+ZKuHiwpCtGgfKkIhIzbHxnQgYDkwZGejhRzV8euQTyKHPR7NnclqT4jebSz09JRV8yacmZRs9OaXBfOw856Xry+BScvFFfR73dRZolgQVpfdlOIvteey38XyTaeRYULfpJZMciKE8qYsBImxQLdaBHCoJE6CUpxbIEC687m6h7CDgIfIpeEOSg7X8xJ5AA5bU14ITyJBjby8LLWQO5LlXf+cyLlVzxUDMKilatYr2x1CIz9IjJceRTXllExaAKRidLMO8sCeRRqiAhgd+95oJG4CogC/gQfVnFc2mlBCgD5gADYH+Jnet3fsPTaRfwZP/+kRm8mRQUMMv9x/VLDujr51LuHxGOEwMpL7dBjgTzzpKllTC47tFHfVZR9ktP54tjx8Bub3muweUiGTgLNFqA7wN9gBW0L9E3HAPWAP+GXqJ/DHgdMu3QOzmZy4cO9fv5pi3R7ymy5BJRxcUwfMFGWWLxQ5ZWIuhsbS0lPjJE8mprwW7na49zwwc2NXEC6J2EPhMfCmxCn4UHKtG/En39fJr78VVw8e5eJPfpF/jzRVvuJReZnUeG1QoHz8BWKiSYd0J8VYmYSBPoGSrQtRL9PChvbu6BkcWHdgVFImzGnJ5BzRkpFuoMmZFHKwetZflGiX4N7bNWjIwW7xL9dPd7iK7znp1fPFh/XjZEe1zdxhmUs5GMCSckJTEIMiOPVkm0luWnAnvQ18mTgLuBJ9y/k9zP76FtiX4t8s90iBiz81mXfSEFRWFitYLTliFt4oIkf9Uj4LpjxzjrdFLtcuF0uchragKgn8e57kmAowS4ERgFbAYKaHceOTcAlwOrgDEe10ogp1evnvwa8cW9Tt4mZVHSFXtUjiOf3RWHseVUyKy8AxLIw6BfenqbjcVqu50PEhJItFg439wM7syhmzSNJmAg4HChl90bwbmj88ivRE9VBD1rZSc0paZgHBnsa2Mzrkr0Q6mggFnFxazf5p6dS3ZLj9HL+Cv4pr9sfgYi6YcRkHf33X6zSEpefRUAdes/w3T03HEncB+t3X98qUFfYskFdsLTt93Bk/Pnh3jkoh0p9+9xxcXQd8ZGKeHHf/qhrJFHKzswAT03vJmgzyPP+fswPnpqsQTxcPE+jGvVKjkDPcSMY29lvdw/WVqJVhb0DctMWs8jDzQjr4V+/VLZ99KycIxOeDLK/YuLgS/0JZdtH8uSSwjlZmVTbjvBgeM2Bo6I71m5LzIjj1Jjhw7Vy+8hqPPILWWJLJgytYdHJQKyWls7FIF0KAqxHEd+S89PyTFvSwJ5BBibn4HOB3/5rvv0o2uPoeeJ7yLgeeSWsiQeublTbVRFD5KCop6R48inqWI031RHeiTRRTY7o9jP33yTp/60uu2hWRPRM1iMM8fLErGUJbFu4U/lzPEotX7JAf0PUu4fEsbmZ2Z/4i6TRc4jN6mte/fyoz/+nvKjR/UN0CRISFBoDuiblsKCKVN55ObZjBw0KNJDFYF4HsYlAb3bjGCek0NcnZQogVyIaFBcDIcOScpiCOw+fZjk0RVxlZYo6YdCRAOrVU9ZXHSFlPt3U25WtpTxu0kgFyJCZi26glkX72rNPxedluPIx2nLwOa0RXooESWBXIhIktl5t+mdheL72FsJ5EJEgXazc8k/D1puVrZ+7G05cbvEIoFciGjhnp3PYn1r/rkE9KAYx94eOG6L9FAiQgK5ENHGCOhSUNQpDUX51JyBrUfib4lFArkQ0UrK/TvFam1tExdvSywSyIWIcjI77xwjJTGeNj8lkAthBr5m5zJD9ynHkd+y+RkvyywSyIUwkZb+oYeWygw9AKtVb+AcL8ssEsiFMBuP6lApKPLPsyFFrJNALoSZSUFRQLlZ2QCsK43tJRYJ5ELEAJmd+zfm9AyctoyYXmKRQC5ErJDZuV+OEwNjOpOlW4FcKfW8UqpCKbVXKfUXpVRGiMYlhOiiNrNz4yfO5WZlx3Rnoe7OyD8AcjRNGw8cAn7W/SEJIbrN2AyVgqIWuVnZnKrMiMmUxG4Fck3Ttmia5nA//CswpPtDEkKEkhQUtYrVMv5QrpHfBbzv76JS6l6lVIlSquRUbW0IP1YI0SEp9wdiN7+8w1ZvSqkPAV99lB7XNO0d9z2PA3nA97QgesdJqzchIsizfyjAop9EbiwRUp5URGKGzXRt4vy1ekvq6IWapt0Q6LpS6l+Bm4HrgwniQogIs1qZZfR+XrWK9Ut+EXf9Q3Mc+eyuOIwtp8JUgdyf7matTAN+AvyzpmkNoRmSECJs4jhlsflwdsx0FuruGvkyoC/wgVJqt1JqeQjGJIQIs3gsKDJK+GMhmHc3a+VSTdOGapqW6/65P1QDE0KEWRzOzo38crOflNjhGrkQIr7MWnRF69q5IYY3RHOzsuF0NuW2Ik4MOWHKNXMp0RdCtBenBUVmLeOXQC6ECCheCopyHPmmLeOXQC6E6FicFBSZtYxfArkQImjxMDs3yvjNtMQigVwI0Tnes/MYS1k0yvjNtMTSYYl+T5ASfSFiy/olB/Q/TL1Wj4Qx4GDWRoCoKuP3V6IvM3IhRLe1FBRt+zhmZudGZyGb0xbpoXRIArkQIjS8C4piYEO0oSjfFJWfEsiFECEVS7NzzzL+aD72VgK5ECL0vGfnJp6hG2X8B47bIj0UvySQCyF6jFEdavaUxebD2VHdWUgCuRCi53mnLJqM1apvfp6qzIj0UHyKSPqhUuoUUBX2D/YtCzgd6UFEUDx/f/nu8cus33+4pmkDvJ+MSCCPJkqpEl95mfEinr+/fPf4/O4Qe99fllaEEMLkJJALIYTJSSCH30d6ABEWz99fvnv8iqnvH/dr5EIIYXYyIxdCCJOTQC6EECYngRxQSj2vlKpQSu1VSv1FKZUR6TGFi1LqNqXUfqWUSykVM+lYgSilpimlvlBKfamU+mmkxxNOSqmVSqmTSqnySI8l3JRSQ5VSHyulDrr/N/+jSI8pVCSQ6z4AcjRNGw8cAn4W4fGEUznwPeCTSA8kHJRSicBvgOnAFcDtSqkrIjuqsHoNmBbpQUSIA1ioadoY4FvAD2Plv70EckDTtC2apjncD/8KDInkeMJJ07SDmqZ9EelxhJEV+FLTtMOapjUDbwKzIzymsNE07ROgJtLjiARN06o1Tdvl/nMdcBAYHNlRhYYE8vbuAt6P9CBEjxkMHPN4fJwY+cssgqeUGgFMBD6P8FBCIinSAwgXpdSHgK9+TY9rmvaO+57H0f/v13+Hc2w9LZjvHkeUj+ckBzeOKKXSgD8DP9Y07WykxxMKcRPINU27IdB1pdS/AjcD12sxllzf0XePM8eBoR6PhwDfRGgsIsyUUhb0IP7fmqa9HenxhIosraBnMQA/Af5Z07SGSI9H9KidwCil1CVKqV7AfODdCI9JhIFSSgGvAgc1TYup7u8SyHXLgL7AB0qp3Uqp5ZEeULgopW5RSh0HJgMblFKbIz2mnuTe1H4Q2Iy+2fWWpmn7Izuq8FFKrQF2AJcrpY4rpe6O9JjC6NvAAuA699/z3UqpGZEeVChIib4QQpiczMiFEMLkJJALIYTJSSAXQgiTk0AuhBAmJ4FcCCFMTgK5EEKYnARyIYQwuf8PYBsSmyD1a1kAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "sc = StandardScaler()\n",
    "sc.fit(X_train)\n",
    "X_train_std = sc.transform(X_train)\n",
    "X_test_std = sc.transform(X_test)\n",
    "X_combined_std = np.vstack((X_train_std, X_test_std))\n",
    "y_combined = np.hstack((y_train, y_test))\n",
    "\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "lr = LogisticRegression(C=100.0, random_state=1)\n",
    "lr.fit(X_train_std, y_train)\n",
    "\n",
    "plot_decision_regions(X_combined_std,y_combined,classifier=lr,test_idx=range(105, 150))\n",
    "#plt.xlabel('petal length [standardized]')\n",
    "#plt.ylabel('petal width [standardized]')\n",
    "#plt.legend(loc='upper left')\n",
    "#plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The probability that training examples belong to a certain class can be computed using the \"predict_proba\" method. Each row contains the class membership of one flower for three classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2.26332469e-20, 6.62130359e-05, 9.99933787e-01],\n",
       "       [9.99870019e-01, 1.29981440e-04, 6.34998610e-21],\n",
       "       [9.99992811e-01, 7.18853591e-06, 1.23962827e-23]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.predict_proba(X_test_std[:3, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Avoid overfitting via regularization\n",
    "Overfitting is a common problem in machine learning, where a model performs well\n",
    "on training data but does not generalize well to unseen data (test data). If a model\n",
    "suffers from overfitting, we also say that the model has a high variance, which can\n",
    "be caused by having too many parameters that lead to a model that is too complex\n",
    "given the underlying data. Similarly, our model can also suffer from underfitting\n",
    "(high bias), which means that our model is not complex enough to capture the\n",
    "pattern in the training data well and therefore also suffers from low performance on\n",
    "unseen data.\n",
    "\n",
    "We will regularize the logistic loss function with $\\ell_2$-norm penalty as\n",
    "$$\n",
    "\\cal{L}_{\\lambda}(\\mathbf{w},w_0) \\triangleq \\sum_{i=1}^n \\Big(-y_i\\log(\\phi(z_i;\\mathbf{w},w_0))-(1-y_i)\\log(1-\\phi(z_i;\\mathbf{w},w_0))\\Big) + \\frac{\\lambda}{2}||\\mathbf{w}||_2^2,\n",
    "$$\n",
    "where $\\lambda$ is the regularization parameter. Increasing $\\lambda$ increases the regularization strength which pushes elements of $\\mathbf{w}$ to zero. This change to the objective function will slighly change the update in the GD algorithm which will not discuss.\n",
    "\n",
    "The parameter \"C\" that is implemented for the \"LogisticRegression\" class in scikit-learn is the inverse of the $\\lambda$ parameter above. Consequently, decreasing the value of C increases the regularization strength and vice versa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2P0lEQVR4nO3deXxU5fX48c+ZrCQkIYQAgQBhkz0ECCCiLO4LSqUKuCK2UkSstT9bba1+1Vpbq11ExV1ARcSiFou4b4AbJLLILmDAALIEIYTsyfn9cQcMkEwGksmdJOf9et3XzF3m3pNLmJN7n+eeR1QVY4wxpioetwMwxhgT3CxRGGOM8ckShTHGGJ8sURhjjPHJEoUxxhifLFEYY4zxKdTtAAKhRYsWmpKS4nYYxhhTb2RmZu5V1cTK1jXIRJGSkkJGRobbYRhjTL0hIlurWme3nowxxvhkicIYY4xPliiMMcb41CDbKCpTUlJCdnY2hYWFbofS6ERGRpKcnExYWJjboRhjTkKjSRTZ2dnExMSQkpKCiLgdTqOhquTk5JCdnU3Hjh3dDscYcxIaza2nwsJCEhISLEnUMREhISHBruSMqccaTaIALEm4xM67MXWgvDxgu25UiaK+mTlzJjt27Kh2u+uuu4558+b5vbymHnjggSPvs7Ky6N27d60fwxhzAsrLYcYFsOXTgOzeEkUQ8zdR1LWKicIYEwQ8HjjzTxAaGZjdB2Sv5jhZWVl0796dCRMmkJqaymWXXUZ+fj4AmZmZDB8+nAEDBnDeeeexc+dO5s2bR0ZGBldddRVpaWkUFBRw3333MXDgQHr37s2kSZM4kdEJKzsGwIgRI7j99tsZNGgQp5xyCosXLwYgPz+fsWPHkpqayrhx4xg8eDAZGRnccccdFBQUkJaWxlVXXQVAWVkZN9xwA7169eLcc8+loKCgls+eMaZS+ftgzRvO+45nQPvBATmMq72eROR5YBSwW1WPu38hzs3tR4ALgXzgOlX9uqbHvfd/a1i7I7emuzlKzzax/N/FvXxus2HDBp577jmGDh3K9ddfz/Tp07nlllu4+eabmT9/PomJicydO5c777yT559/nscee4yHH36Y9PR0AKZOncrdd98NwDXXXMOCBQu4+OKLq42tpKSkymMAlJaWsnTpUhYuXMi9997LBx98wPTp04mPj2fVqlWsXr2atLQ0AP72t7/x2GOPsWLFCsBJgN9++y1z5szhmWeeYezYsbz22mtcffXVJ3kmjTF+KTwAL14Ke9ZDu8EQ2yZgh3K7e+xM4DHghSrWXwB09U6DgSe8r/VSu3btGDp0KABXX30106ZN4/zzz2f16tWcc845gPPXeVJSUqWf//jjj/n73/9Ofn4++/bto1evXn4lig0bNvg8xpgxYwAYMGAAWVlZACxZsoRbbrkFgN69e5Oamlrl/jt27HgkkVTchzEmQIoOwkuXwa41MH52QJMEuJwoVHWRiKT42GQ08II691i+FJFmIpKkqjtrctzq/vIPlGN7/4gIqkqvXr344osvfH62sLCQKVOmkJGRQbt27bjnnnv87nJa3TEiIiIACAkJobS09Mhn/HX484f3YbeejAmg4nx4eRxsz4Sxs+CU8wJ+yGBvo2gLfF9hPtu7rF7atm3bkS/rOXPmcPrpp9OtWzf27NlzZHlJSQlr1qwBICYmhoMHDwIcSQotWrQgLy/vhHoz+TpGVU4//XReffVVANauXcs333xzZF1YWBglJSV+H98YU4s2vg3bvoCfPwM9qr+jUBuCPVFU1gG/0j91RWSSiGSISMaePXsCHNbJ6dGjB7NmzSI1NZV9+/Zx4403Eh4ezrx587j99tvp27cvaWlpfP7554DTvXXy5MmkpaURERHBDTfcQJ8+ffjZz37GwIED/T6ur2NUZcqUKezZs4fU1FQefPBBUlNTiYuLA2DSpEmkpqYeacw2xtSh3j+HKV85r3VETuQWQ0ACcG49LaiiMfsp4BNVneOd3wCMqO7WU3p6uh47HsW6devo0aNHrcV9orKyshg1ahSrV692LYYTUVZWRklJCZGRkWzevJmzzjqLjRs3Eh4eflL7c/v8G1OvlZXAm7+G9Ouhnf9/JJ4IEclU1fTK1rndmF2dN4GpIvIKTiP2gZq2Txj/5OfnM3LkSEpKSlBVnnjiiZNOEsaYGigrhddvcLrBJg8IWKLwxe3usXOAEUALEckG/g8IA1DVJ4GFOF1jN+F0j53oTqQ1l5KSUm+uJsBpH7FRAo1xWXkZzJ/iJIlz/gwDf+lKGG73erqimvUK3FRH4RhjTPAoL4cFv4FVc2Hkn2Dor10LJdgbs40xpnHSMijYD2fcBsN/52oowd5GYYwxjYsqFOdBRAxcPhPE/b/n3Y/AGGPMTz66H549Gwp+BE8IBEGZfksUQaymZcar8+STT/LCC8dXT6lYOnzFihUsXLjwyLp77rmHhx9++ISPZYzxw6cPweKHndpNEXFuR3OEJYogFugy45MnT+baa6/1uc2xicIYEyCfPQIf3w+p42HUv53S4UEieCJp4Oq6zPju3bsZMGAAACtXrkRE2LZtGwCdO3cmPz//qKuDzMxM+vbty5AhQ3j88ccBKC4u5u6772bu3LmkpaUxd+5cwCnpMWLECDp16sS0adMCds6MaTRWvAzv3w29xsDox4MqSUBjThQzLjp+WvqMs644v/L1y2c76w/lHL/ODxs2bGDSpEmsWrWK2NhYpk+ffqQE+Lx588jMzOT666/nzjvv5LLLLiM9PZ3Zs2ezYsUKmjRpwtSpU1m2bBmrV6+moKCABQsWVHmsli1bUlhYSG5uLosXLyY9PZ3FixezdetWWrZsSVRU1FHbT5w4kWnTph1VODA8PJz77ruPcePGsWLFCsaNGwfA+vXreffdd1m6dCn33nuv1X0ypqY6nwlDpsKYpyEk+PoYNd5E4YJjy4wvWbLkqBLgaWlp3H///WRnZ1f6+Y8//pjBgwfTp08fPvroo2oL+5122ml89tlnLFq0iD/+8Y8sWrSIxYsXc8YZZxy13YEDB9i/fz/Dhw8HnLEufLnooouIiIigRYsWtGzZkl27dvl7CowxFWUtcZ68jmkN5/0FQsLcjqhSwZe66srEt6peFx7le310gu/1VajrMuNnnHHGkauI0aNH8+CDDyIijBo16qjtVPW42Hw5tqz44dLkxpgT8M08pzTHyDth2G1uR+OTXVHUobouMz5s2DBeeuklunbtisfjoXnz5ixcuPDIVc1hzZo1Iy4ujiVLlgAwe/bsI+sqxmCMqSVr34TXJ0H7IXDqFLejqZYlijpU12XGU1JSACdhgDPGRLNmzYiPjz9u2xkzZnDTTTcxZMgQmjRpcmT5yJEjWbt27VGN2caYGtjwDsy7HtoOgCvnOncwgpzrZcYDwcqMBx+3z78xQaEoDx5JhWbt4dr5EBk8z0rU5zLjxhjTcEQ0havmQXxKUCWJ6titpzpS38qMG2Nq0bavfup+37Y/RDV3N54TZInCGGMCaXsmzL4MvnrSeUarHmpUiaIhtsfUB3beTaO1cxW8eCk0iYdr36wXDdeVaTSJIjIykpycHPvSqmOqSk5ODpGRkW6HYkzd2rUWXhgN4TEw4X8Q19btiE5ao2nMTk5OJjs7mz179rgdSqMTGRlJcnKy22EYU7eyl0FoBEx4E+I7uB1NjTSaRBEWFkbHjh3dDsMY09CVlznjSAyYAL0uhchYtyOqMVdvPYnI+SKyQUQ2icgdlawfISIHRGSFd7rbjTiNMcYv+7fBE0Mh6zNnvgEkCXDxikJEQoDHgXOAbGCZiLypqmuP2XSxqo46bgfGGBNMcnfArIsh/0cIj3Y7mlrl5hXFIGCTqm5R1WLgFWC0i/EYY8zJObjLSRKH9sI1r0ObNLcjqlVuJoq2wPcV5rO9y441RERWisjbItKrbkIzxhg/FeY6vZtyd8BV/4HkSqtg1GtuJorK6lof23f1a6CDqvYFHgX+W+XORCaJSIaIZFjPJmNMnQlv6lxBXPEKdDjN7WgCws1EkQ20qzCfDBw1QLSq5qpqnvf9QiBMRFpUtjNVfVpV01U1PTExMVAxG2OMI+szp/Ha44FLn4ROw92OKGDcTBTLgK4i0lFEwoHxwJsVNxCR1uIdUUdEBuHEm1PnkRpjTEUrX3FuN717p9uR1AnXej2paqmITAXeBUKA51V1jYhM9q5/ErgMuFFESoECYLzao9XGGLeowid/hU8fhJQz4JJpbkdUJxrNeBTGGFMjpUUwfyp88yqkXQWj/g2h4W5HVWtsPApjjKmpshLYsx7OvAvO+H9wAuPM13eWKIwxxpd930HTls6gQ7/8wKnf1Mg0muqxxhhzwrZ+Ac+cCW/d5sw3wiQBliiMMaZy38yDFy5xRqMbdpvb0bjKbj0ZY0xFqrDoYfj4fugwFMa9VO+GLq1tliiMMaaiQ3ucYUtTxzvdXxvp7aaKLFEYYwxA0UEIi3Yarid9AnHJjapnky/WRmGMMfu+cxqtP/mrM9+snSWJCuyKwhjTuH2/DOaMh/LSBl2vqSbsisIY03iteQNmjYKIGOcZiZTT3Y4oKNkVhTGmcTqwHV6fBG36w/iXITrB7YiCliUKY0zjouq0P8S1hatfh+SBEBbpdlRBzRKFMZXIySti3c6DrN15gHU7D7JuZy7FZeV0bx1D99axdGsdQ4/WsSTHN8HjsUbPeqNgP/znOuh/LfQeAx3PcDuiesEShWnUysqV7/YeYt3OXNbuzGWdd9qVW3Rkm9axkfRIiiE81MPaHbm8vfoHDhddjg4P4RRv8uiRFEO3Vs77uKgwl34iU6X922D25ZCzCVLHuh1NvWKJwjQaeUWlrPcmgrU7D7J2Zy4bfsilsKQcgFCP0KVlU4Z2bkHPNrH0SHKm5tFHl5LOLy5l46481u/MZf0PztXGwm92MmfptiPbtImLpFvrGLonxdK9dQw9kmLp2CKasBDrP+KK7Znw8ninVPjVr1vvphNkicI0OKrKjgOFrNvx01XC2p25bM3JP7JNXJMweiTFcOWgDvRIiqFnm1i6tGxKRGhItfuPCg8lrV0z0to1O+qYu3KLWPdDLht+OHgkiSzZtJeSMufyIzzEQ+eWTenROuZIEunROobEmAjE+uwHzo9ZMOMiaJoI1y2AxG5uR1TvVDtwkYh0VNXvqlsWTGzgosajqLSMb3flHXPr6CAHCkqObJOSEEWPpFh6eq8QeraJJSkusk6+nItLy9myN4/1Ow9WSCIH+SG38Mg28VFhdG8dS/ekmCNtIKe0iqFJePVJy/jpq6eh16VOsjCV8jVwkT+J4mtV7V/JDgfUYoy1yhJFw3S4gbliUti0O4/Scud3ODLM420rcJJBz6QYurWOpWlE8F04788vZn2FK4/1Pxxkww8HKSgpA5xOOSkJ0UcSx+Ek0i4+yhrP/VFWCu/fBX3HQ1Jft6OpF05qhDsR6Q70AuJEZEyFVbGA9SUztaa8XDlQUELOoSL25hWTk1dMzqGiI6/ZPxYc18DcKjaCHkmxnNm95ZHEkJIQTUg9+RJtFhXOqZ0SOLXTT333y8uVbfvyWf+DN3l4k+I7a35qPI8KD+GmkV24aWQXlyKvBwpzYd5E2PSBU7fJEkWN+fpTqxswCmgGXFxh+UHghto4uIicDzwChADPqurfjlkv3vUXAvnAdar6dW0c2wSOqpJfXEZOXjF7vV/4+ypJAnvzisg5VMy+Q8WUlR9/ZSsC8VHhtIyJ4LTOLY7cOuqRFENC04ZX0dPjEVJaRJPSIprzeycdWV6x8fydNT/w0Lsb6JkUy8juLV2MNkgdyIbZY50hSy+eBgMmuB1Rg+DPrachqvpFrR9YJATYCJwDZAPLgCtUdW2FbS4EbsZJFIOBR1R1cHX7tltPta+4tJx9h5wv932HKn7ZF5Pj/cLPySti78EiDubnQUkhEZTgQdmFU8u/p2TRJjyfxMhymkeUkxBeTkh0PLvbnElCdAQDd80lvmwv0Z4SIikhgmI8rXo44xMDrHgZQsIhtq3zsFRMEoQ0rm6ohSVlXDr9c344UMDCW84gKa6J2yEFj33fwfPnQ0k+jJ0Fnc90O6J65aRuPVWwSUT+CKRU3F5Vr69hXIOATaq6xRvkK8BoYG2FbUYDL6iTzb4UkWYikqSqO2t47EotW59Fk72rjlqmCvlxp1DSpAUhRT/SdN8aKqZWBfKa9aAkIp6wwhya/rj2qPUoHExIpSQ8lvCCXTTdv9F7G0GPrN/foj+lYU2JPLSD6NxvQRXVcsrLyykvL2NXi1Mp9kQRlbuZ2AMbnHVlZagqWl7Gty3OpsQTQfP9a2iRtw7KyynXctBytLycpS0upURDaJ+bQdtD65zlWg7lzuuC+GspU6XvwUWkFG048jm0jPwS5S8lV5JbWMqNIW8yxLOGCCmhI8X0oIQDNOWWyL+Q0DScv+T/mdSipXhC1LlGBPLiurLl8g9IaBpB0n8uxrN9GRThTAAR6XDeFOf9EwucPu6hERDWBEIjj35i9r0/QX5OhZMr0P8auORRZ/aDe6BJcyeJxHqnmNYNKplEhoXw+JX9uPjRJfx6znLm3HAqodbl1hGXDF3PgVOnQKuebkfToPiTKOYDi4EPgLJaPHZb4PsK89k4Vw3VbdMWOC5RiMgkYBJA+/btTyqgv89ewH9C7jxu+c3FU/lf+WkM8axhTvhfjls/sfh3fFzej3M8GTwT/s/j1o8tuoul2oOfeZbw7/Dpx62/qOgB1mgKV4V8wF/Cnj9u/fCif7JVW/OrkP/xh7A5x63/XWEie4nj1tA3uCL0jePW3/19f0pDmnBb+ftcqAuOWleOMK1sDB4RxhR8xdlFH1COBxVB8ZAfEsvafr8joWkEZ22Ppu1+D57wZoSGNyEsMorQ2NZ8OeosZ2fLr4Ufhx/1Rd80OpHU5GbeH/QhKCl0vvxDvVNEzE/BTF7su7Tzr5dD7g7I3e7U6cndAYmnOOtKi2DpM1Ccd/RnTp0C5/8VSgqcuj6Hr0YOJ5IWXevd6GWdEpvywJg+3PLKCv71wUZ+d153t0NyV+Ys6HaB0x4x+jG3o2mQ/Ln1tEJV02r9wCKXA+ep6i+989cAg1T15grbvAX8VVWXeOc/BH6vqpm+9n2yt54yv91Gk71rnGNXWF7YrDOlTVoQUpRL1P4NHLtBYbNTKItoRkjRfpoc2Hz855ufQnl4LKEFOUTkZnl/No58KRbG90DDoggt2EN4XrZ3vYeQ0BBEQihP6EpIeBNCi/YRWpBDiCcET0gIIR4PnpAQPM3aERIaRkjJQTwlhxAJAfF4J4Em8c5raRFoeYV1HidSTwP5i1QVCg94k8kOyM2GxB7QfjDk7nTGPz6wHUoO/fSZ8x6AITc5fe3n/eLoJBLbBtqf6rwGodvnreLVzO+ZNXEQw05phN0+C3Phoz/D0qfhjNvgrLvcjqheq+mtpwUicqGqLqzluLKBdhXmk4EdJ7FNrRnQtT109XU10hznDlxV4oGOPtY3Azr7WB8H+OrNEoVzCqoQGgdN4nysb3gNwEcRgSbNnOnYWw+xSTB1WYVk4r0qaeE936XFEB4Nu9bCt+8797kBLp/p9L/PWgJv3wF9x0GfsRDTqg5/sMrdc0kvVny/n1vnruDtW86gZWwj6Yy4/WvIeA5Wv+78Ow2ZCiP/6HZUDZo/VxQHgWig2DsJoKoaW6MDi4TiNGafBWzHacy+UlXXVNjmImAqPzVmT1PVQdXt2xqzTY2oQuF+J5HEtXWuyHauhAW3OqUgJAS6nAV9r4DuoyA0vNpdBsqm3Qe5+NHPSGvXjJd+ObjedA8+Yfn7ICIWQkLho7/AF49Dn59D/wmQXOkfweYE1eiBu0Dy9mr6N07T5/Oq+hcRmQygqk96u8c+BpyP0z12oqpWmwEsUZiA2bMBVs6BlXOdMZZv2wjhUZC3G6ITXRk+c15mNrf9ZyW3nNWVW885pc6PHzDl5ZC1CL5+Adb9Dy6fBd0vhIIfwRN6dPuWqbGaPpktwFVAR1X9s4i0A5JUdWnth1o7LFGYgCsvc3poJXZzrkCmD4HyEudJ4NTxzpjLdej/vbqS15dnM/sXgzmtS4s6PXatKymELx6D5S86bUeRzZzzOmgSJPi6dWtqwlei8KcVczowBLjSO58HPF5LsRlTP3lCfioup+VOg3jTVvDR/fDvPjDrYtj8cZ2Fc9/oXnRqEc0tc1ew52BR9R8INmWlztUaOM/KLH8R4trBmGfh/22ACx60JOEifxLFYFW9CSgEUNUfAfduyhoTbDwhzvMcExfCLSthxB3O2AeH9jrr83bDd4ucWykBEh0RyuNX9Se3oIRb566gvJIn3YPSvu/gwz/Dv3vDjAucnnkeD0xe4lR6Tb3cRp8LAv4kihLvU9QKICKJQOB+442pz+JTnETx6xXOCGrgtGnMuhge6etcceRsDsihu7eO5d5LerFk016mf7IpIMeoNd8vg1mXwLQ0WPJPaJ3qlNwQ75Oa1v4QVPzpHjsNeANoKSJ/AS4D/hTQqIyp70R++tIbeAPEtIGVL8Oih2HRQ9D+NJjwZq0/NT5uYDu+2JLDP9/fyMCU5gyuUHTQdbvXQVgUxHdw2nP2fQcj74S0K52nqk3Q8qvXk7eS7Fk4XWM/VNV1gQ6sJqwx2wSt3B2waq5TvO6ifzjLFj3sVDjtNNLp/llDeUWlXPzoEvKLS1n46zPcLaBYfAjWvOE8PZ291EmaFz3sdABQbTgPezYAJ9XrSURiVTVXRCqtb6Cq+2oxxlplicLUG0V5zi2p/L1OY3jqWOh7ZY1rFa3ZcYBLp3/OkE4JzLhuoDtjWLz3J8iYCcUHocUp0P9a59mT6HreK6uBOtleTy97XzOBjArT4XljTE1FNIXfroWxL0LbAfDlE/DEEFj+Uo1226tNHHeN6smnG/fw1KIttRRsNQp+hBVzODJ4RnkZ9BgFE9+Bm5bCaTdbkqinqrzOVdVR3ldfNSmMMTUVGgE9L3GmvD2weh50OdtZt/o1p1RF3yug67kn9BT41YPb8+XmHB5+bwODOsYzoEMAih+qwtbP4etZsHY+lBY6V0NJfZ1ijKZBqPaGqIhcCnykqge8882AEar638CGZkwj1DQRTr3xp/miPPh+Kaxf4JRQ73OZ05bR/UJn/aYPnXIj4nEaz8UDUQnQYQgiwt8HHSJy23pmvbSeU8akEdMk3NnP4VtbO1dBeanTxfdwocjIuJ8al/d957xWLCQZHu3U09qzEV65wnnwMCIW0q5ybi/ZiHINzklVjxWR5araL5CB1YS1UZgGpawUNn/k9Jpav9AprT55ibPu6RGwY/nR27c/Da5/23n/aDrkfHv0+q7nwlX/cd7/ozscPKZqf68xcPkM5/0DyU4bQ0X9r3XGACkphFevcbbvOdopZWLqrZpWj62sHSP4Rqs3pqEKCYVTznWmooPOdNjls5zbPeVlzhPiWu50QT3ssuehpICFq7KZ9dl3XHtqOy4a1OOn9Zc+6YzVcfizWu505T3s4n9DWcnR6xO8FXfDIn9KOKZB8+cLP0NE/olTtkNxhib1OR6EMSZAImKOfhgtvoPv7ZNSAbig3SDm/5jJLV/tpk2/dhy5HdBphO/P97nsZCM1DYg/nZhvxikvPhf4D04pj5sCGZQxpnaJCH//eV9ax0Uy9eXlHMgvcTskU49UmyhU9ZCq3qGq6ao6QFX/oKqHqvucMSa4xEWF8egV/diVW8jvX1uJm0MMmPqlykQhIv/2vv5PRN48dqqzCI0xtaZf+3huP787767ZxazPs9wOx9QTvtooXvC+PlwXgRhj6sYvz+jIl1tyeGDhegZ0aE6fZB/D5xqD71tPD3lfL1TVT4+d6iI4Y0ztExEevrwvLZqGc9PLX5NbaO0VxjdfiSJJRIYDl4hIPxHpX3GqqwCNMbUvPjqcR6/sx/b9BfzhtW+svcL45OvW093AHUAy8A+cyrGHKXBmAOMyxgTYgA7Nue3cbjz4znqGfJXA1adW09XWNFq+EsVOVb1ARO5W1ftq86DeirRzgRQgCxjrHTnv2O2ygINAGVBa1VODxpiT86thnfhySw73LVhLv/bN6NXG2ivM8Xzdeprmff1ZAI57B864Fl2BD73zVRmpqmmWJIypfR6P8M+xfYmPCmPqy8vJKyp1OyQThHwlihIRmQG0FZFpx041PO5oYJb3/SwCk4yMMX5IaBrBtPH92JpziDvfsPYKczxfiWIU8C7Ok9iZlUw10UpVdwJ4X1tWsZ0C74lIpohM8rVDEZkkIhkikrFnz54ahmdM4zK4UwK3nn0K81fsYO6y790OxwQZX+NR7AVeEZF1qrryRHcsIh8ArStZdecJ7Gaoqu4QkZbA+yKyXlUXVRHv08DT4FSPPdF4jWnspozswlff7eP/3lxDWvtmdG8d63ZIJkj4U+upQEQ+FJHVACKSKiJ/qu5Dqnq2qvauZJoP7BKRJO/+koDdVexjh/d1N/AGMMjPn8sYc4JCPMK/xqURExnGTbO/Jr/Y2iuMw59E8QzwB6AEQFVXAeNreNw3gQne9xOA+cduICLRIhJz+D1wLrC6hsc1xviQGBPBI+PT2LL3EHf9d43b4Zgg4U+iiFLVpccsq+mfGn8DzhGRb4FzvPOISBsRWejdphWwRERWAkuBt1T1nRoe1xhTjaFdWnDzmV157ets5mVmux2OCQL+jEexV0Q64zQsIyKXATt9f8Q3Vc0Bzqpk+Q7gQu/7LYCNqWiMC245qytLv8vhrv+upm9yHF1bxVT/IdNg+XNFcRPwFNBdRLYDvwEmBzIoY4y7QjzCI+P7ERUewk0vf01BcZnbIRkX+TMexRZVPRtIBLqr6umqujXwoRlj3NQqNpJ/jktj46487nnT2isas2oThYjEeYdC/RT4WET+ISL2nL8xjcDwUxKZMqIzczO+57/Lt7sdjnGJP7eenseptzTWO+UCMwIZlDEmePz2nFMYmBLPnW98w5Y9eW6HY1zgT6LorKr/570FtUVV7wU6BTowY0xwCA3xMO2KfoSHerjp5eUUllh7RWPj7wN3px+eEZGhQEHgQjLGBJukuCb8Y2xf1u3M5c8L1rodjqlj/nSPvRGYVaFd4kfguoBFZIwJSmd2b8WkYZ14etEWhnROYFRqG7dDMnWk2kShqiuAviIS653PDXRQxpjg9LvzurEsax93vPYNfdrG0SEh2u2QTB3wp9fTAyLSTFVzVTVXROJF5P66CM4YE1zCQjw8ekU/VJV/vLfR7XBMHfGnjeICVd1/eMY7Et2FAYvIGBPUkuOjuOrUDixYtYOtOYfcDsfUAX8SRYiIRByeEZEmQISP7Y0xDdwvTu9IqMfDU4u2uB2KqQP+JIqXgA9F5Bcicj3wPj+NTmeMaYRaxUby8wHJzMvIZnduodvhmADzp4TH34H7gR5AL+DP3mXGmEZs8vBOlJaX89yS79wOxQSYP91j8Zb3thLfxpgjOiREc1FqG176citTRnQhLirM7ZBMgPhz68kYYyp14/DOHCou44UvstwOxQSQJQpjzEnr2SaWkd0SmfF5lpUib8D8eY7iFn+WGWMapykju7DvUDFzl21zOxQTIP5cUUyoZNl1tRyHMaaeGpjSnPQO8Tyz+DtKysrdDscEQJWJQkSuEJH/AR1F5M0K08dATt2FaIwJdlNGdmb7/gLmr9jhdigmAHz1evocZ2zsFsA/Kiw/CKyqyUFF5HLgHpwut4NUNaOK7c4HHgFCgGdV9W81Oa4xJjBGdmtJ99YxPPnpZsb0a4vHI26HZGpRlVcUqrpVVT9R1SGq+mmF6WtVLa3hcVcDY4BFVW0gIiHA48AFQE/gChHpWcPjGmMCQES4cURnNu3O4721u9wOx9Qyfxqzx4jItyJyQERyReSgiNSogqyqrlPVDdVsNgjY5B0sqRh4BRhdk+MaYwLnoj5JtG8exROfbEJV3Q7H1CJ/GrP/DlyiqnGqGquqMaoaG+jAgLbA9xXms73LKiUik0QkQ0Qy9uzZE/DgjDFHCw3x8KvhnViZfYDPN1szZkPiT6LYparrTnTHIvKBiKyuZPL3qqCym5xV/pmiqk+rarqqpicmJp5ouMaYWvDz/skkxkTwxCeb3Q7F1KIqG7NFZIz3bYaIzAX+CxQdXq+qr/vasaqeXcPYsoF2FeaTAetSYUwQiwwL4Zend+Svb69nVfZ+UpObuR2SqQW+rigu9k6xQD5wboVlowIfGsuAriLSUUTCgfHAm3VwXGNMDVw5uD2xkaFM/9iuKhqKKq8oVHVioA4qIpcCjwKJwFsiskJVzxORNjjdYC9U1VIRmQq8i9M99nlVXROomIwxtSMmMoxrh6Tw+Ceb2LQ7jy4tm7odkqkhqa53gohMq2TxASBDVecHJKoaSk9P14yMSh/NMMbUgZy8IoY++BGjUtvw8OV93Q7H+EFEMlU1vbJ1/jRmRwJpwLfeKRVoDvxCRP5dSzEaYxqQhKYRjB/Ynv8u3872/QVuh2NqyJ9E0QU4U1UfVdVHgbNxnqi+FKfdwhhjjnPDsE4APGPDpdZ7/iSKtkB0hflooI2qllGhF5QxxlTUtlkTRqe15ZVl28jJs6+K+szfB+5WiMgMEZkJLAceFpFo4INABmeMqd9uHNGJotJyZn2e5XYopgb8GTP7OeA0nOco/gucrqrPquohVf1dYMMzxtRnXVrGcG7PVsz8PIu8opqWiDNu8VVmvLv3tT+QhFNOYxvQ2rvMGGOqdeOILuQWlvLyV1vdDsWcJF9lxn8LTOLoEuOHKXBmQCIyxjQoae2acVrnBJ5d/B0TTkshIjTE7ZDMCfJVZnyS93VkJZMlCWOM36aM6MLug0W8lrnd7VDMSfCnzHiUiPxJRJ72zncVkboo4WGMaSCGdkkgNTmOpxZtptSGS613/On1NAMoxmnQBqdY3/0Bi8gY0+CICFNGdGZrTj4LV//gdjjmBPmTKDqr6t+BEgBVLaDyEuDGGFOlc3u2pnNiNE98stkGNqpn/EkUxSLSBO9YECLSGXvQzhhzgjweYfLwzqzbmcsnG21wsfrEn0RxD/AO0E5EZgMfAr8PZFDGmIZpdFpb2sRF8oSVIK9X/Hng7j1gDHAdMAdIV9VPAhuWMaYhCg/1cMOwTizN2kdG1j63wzF+8qfX04s4iWKzqi5Q1b2BD8sY01CNG9iO+KgwpttwqfWGv72ekoBHRWSziLwmIrcEOC5jTAMVFR7KxKEd+Wj9btbtzHU7HOMHf249fQT8BbgLeBZIB24McFzGmAZswpAUosNDeMKuKuoFf249fQh8BowDNgADVbV7oAMzxjRccVFhXHVqBxas2sHWnENuh2Oq4c+tp1U4D9z1xhndrre3u6wxxpy0X5zekVCPh6dsYKOg58+tp1tVdRjOiHY5OG0W+2tyUBG5XETWiEi5iFQ6Rqt3uywR+UZEVoiIDYJtTAPSKjaSnw9IZl5GNrtzC90Ox/jgz62nqSIyF1gB/Ax4HrighsddjdOTapEf245U1bSqBv02xtRfk4d3orS8nOc++87tUIwPvsqMH9YE+CeQqaq1MvKIqq4Dp/6LMabx6pAQzUWpbZj95TamjOhCXJMwt0MylfDn1tNDqvpVbSWJE6TAeyKSKSKTfG0oIpNEJENEMvbssfIAxtQXk4d3Iq+olBe/yHI7FFMFfxqzT4qIfCAiqyuZRp/Aboaqan+cW103iciwqjZU1adVNV1V0xMTE2scvzGmbvRqE8eIbok8/1kWBcVlbodjKhGwRKGqZ6tq70qm+Sewjx3e193AG8CgQMVrjHHPlBFd2HeomLnLtrkdiqlEwBJFTYlItIjEHH4PnIvTCG6MaWAGdWxOeod4nln8HSU2sFHQcSVRiMilIpINDAHeEpF3vcvbiMhC72atgCUishJYCrylqu+4Ea8xJvCmjOzM9v0FzF+xw+1QzDH86fVU61T1DZxbSccu3wFc6H2/Behbx6EZY1wysltLureO4clPNzOmX1s8HusVGSyC9taTMaZxERFuHNGZTbvzeH/dLrfDMRVYojDGBI2L+iTRvnkU02241KBiicIYEzRCQzz8angnVn6/ny8257gdjvGyRGGMCSo/759MYkyEDWwURCxRGGOCSmRYCL84vSNLNu1lVfZ+t8MxWKIwxgShqwa3JzYylOkf21VFMLBEYYwJOjGRYVw7JIV31/7Apt15bofT6FmiMMYEpYlDU4gI9fDUp3ZV4a9A9RSzRGGMCUoJTSMYP7A9byzfzo79BW6HE/ReWbqNKbO/pqi09gsrWqIwxgStG4Z1AuCZxTZcqi/vrvmBP77xDfnFZXgCMM6PJQpjTNBq26wJo9Pa8srS79l3qNjtcILSV1tyuHnOclKTm/HE1f0JC6n9r3VLFMaYoDZ5eCcKSsqYacOlHmftjlx+OSuD9s2jmHHdQKLCA1O+zxKFMSaodW0Vw7k9WzHz8yzyitwYaDM4bcvJZ8KMpTSNDOWF6wcRHx0esGNZojDGBL0pI7uQW1jKy19tdTuUoLDnYBHXPP8VJWXlvHD9INo0axLQ41miMMYEvbR2zTitcwLPLv4uIL166pODhSVcN2Mpu3OLeP66gXRtFRPwY1qiMMbUC1NGdGH3wSJe/3q726G4prCkjEkvZLLhh4NMv7o//dvH18lxLVEYY+qFoV0SSE2O46lPN1NW3vhKkJeVK7fOXcEXW3J46PJURnZrWWfHtkRhjKkXRIQpIzqTlZPPwm92uh1OnVJV7pq/mrdX/8CfLurBpf2S6/T4liiMMfXGuT1b0ykxutENbPSvD77l5a+2ceOIzvzyjE51fnxXEoWIPCQi60VklYi8ISLNqtjufBHZICKbROSOOg7TGBNkPB5h8vDOrNuZyycb97gdTp148Ysspn34LWPTk/n9ed1cicGtK4r3gd6qmgpsBP5w7AYiEgI8DlwA9ASuEJGedRqlMSbo/CytLUlxkTzRCEqQL1i1g7vfXMPZPVrxwKV9kACU5/CHK4lCVd9T1cNPznwJVHbDbRCwSVW3qGox8Aowuq5iNMYEp/BQDzec0YmlWfvIyNrndjgBs+Tbvdw6dwXpHeJ57Mp+hAagNIe/gqGN4nrg7UqWtwW+rzCf7V1WKRGZJCIZIpKxZ0/juCQ1prEaP6gd8VFhDXa41FXZ+/nVixl0TmzKsxMGEhkW4mo8AUsUIvKBiKyuZBpdYZs7gVJgdmW7qGRZla1Xqvq0qqaranpiYmLNfwBjTNCKCg9l4tCOfLR+N68s3dagGra37MnjuhnLiI8OZ9b1g4hrEuZ2SASmghSgqmf7Wi8iE4BRwFla+b9yNtCuwnwysKP2IjTG1GcTh6bw2aa93PH6N7z1zU7+OqYPyfFRbodVI7tyC7nmuaUI8ML1g2gVG+l2SIB7vZ7OB24HLlHV/Co2WwZ0FZGOIhIOjAferKsYjTHBLSYyjDk3nMqfR/cic+uPnPevRbz45VbK6+nDeAfyS7j2uaXszy9m5sRBdEps6nZIR7jVRvEYEAO8LyIrRORJABFpIyILAbyN3VOBd4F1wKuqusaleI0xQcjjEa4ZksK7vxlGv/bx3PXf1Vz57JdszTnkdmgnpLCkjF++sIzv9h7i6WvT6ZMc53ZIR5GGdG/vsPT0dM3IyHA7DGNMHVJVXs34nvsXrKO0XPnded2YcFoKIR53upT6q7SsnMkvZfLh+t08dkV/LkpNciUOEclU1fTK1gVDrydjjKkxEWHcwPa899thnNqpOfctWMvYp75g0+48t0Orkqryh9e/4YN1u7nvkl6uJYnqWKIwxjQoSXFNeP66gfxzbF827c7jwmmLeeKTzZSWlbsd2nEefGcD/8nM5pazunLNkBS3w6mSJQpjTIMjIozpn8z7vx3GyG6JPPjOesY88TkbfjjodmhHPLt4C09+upmrBrfnN2d3dTscnyxRGGMarJYxkTx59QAeu7If238sYNSji5n24beUuHx18frX2dz/1jou7NOa+0b3dq00h78sURhjGjQRYVRqG967dRgX9E7in+9v5JLHPmP19gOuxPPx+t38ft4qTuucwL/GpQV9YztYojDGNBIJTSOYdkU/nr5mAHvzihj9+Gc8/O6GOh1aNXPrj9w4O5PuSTE8dc0AIkLdLc3hL0sUxphG5dxerfng1uH8LK0tj328iVHTlrB8248BP+7GXQe5fuYyWsVGMuO6QcREul+aw1+WKIwxjU5cVBj/GNuXGRMHkldUys+f+JwHFq6jsCQwVxfb9xdw7XNLCQ/18OL1g0mMiQjIcQLFEoUxptEa2a0l7906jPGD2vP0oi1c8MhiltVy6fJ9h4q59rmvOFRUyqyJg2ifUP/qUVmiMMY0ajGRYTxwaR9m/3IwJWXljH3qC+55cw2Hikqr/3A1DhWVcv3MZXz/YwHPTkinZ5vYWoi47lmiMMYYYGiXFrz7m2FMGJLCzM+zOP+RRXy+ae9J76+4tJwbZ3/Nquz9PHZFPwZ3SqjFaOuWJQpjjPGKjgjlnkt68eqvhhDq8XDls1/xh9e/4WBhyQntp7xc+d28lSzauIcHLu3Dub1aByjiumGJwhhjjjGoY3MW/voMJg3rxNxl2zj3X4v4ZMNuvz6rqvz5rbXMX7GD353XjfGD2gc42sCzRGGMMZVoEh7CHy/swWs3nkbTiFCum7GM2/6zkgP5vq8upn+ymRmfZTFxaApTRnSuo2gDyxKFMcb40K99PAt+fTpTR3bhjeXbOftfn/Lemh8q3faVpdt46N0NjE5rw10X9Qz60hz+skRhjDHViAgN4bbzujH/pqG0aBrBpBczuXnOcvYdKj6yzbtrfuCPb3zDsFMSeeiyvnjqQWkOfwVszGxjjGloereNY/5NQ3ny0808+tG3fL5pL/eO7kVi0whunrOcPsnNeOKq/oSHNqy/wW2EO2OMOQnrf8jl9/NWsSr7AKEeoUNCFP+ZfBrNo8PdDu2k+Brhzq4ojDHmJHRvHcvrN57Gs0u+46P1u/nXuLR6mySq40qiEJGHgIuBYmAzMFFV91eyXRZwECgDSqvKdsYY44bQEA+Th3dm8vCG0bupKm7dSHsf6K2qqcBG4A8+th2pqmmWJIwxxh2uJApVfU9VDxdS+RJIdiMOY4wx1QuGpvnrgberWKfAeyKSKSKTfO1ERCaJSIaIZOzZs6fWgzTGmMYqYG0UIvIBUFmBkztVdb53mzuBUmB2FbsZqqo7RKQl8L6IrFfVRZVtqKpPA0+D0+upxj+AMcYYIICJQlXP9rVeRCYAo4CztIo+uqq6w/u6W0TeAAYBlSYKY4wxgeHKrScROR+4HbhEVfOr2CZaRGIOvwfOBVbXXZTGGGPAvTaKx4AYnNtJK0TkSQARaSMiC73btAKWiMhKYCnwlqq+4064xhjTeLnyHIWqdqli+Q7gQu/7LUDfuozLGGPM8RpkCQ8R2QNsdTuOGmoBnPzwWg2LnYuj2fk4mp2Pn9TkXHRQ1cTKVjTIRNEQiEiGPWTosHNxNDsfR7Pz8ZNAnYtgeI7CGGNMELNEYYwxxidLFMHrabcDCCJ2Lo5m5+Nodj5+EpBzYW0UxhhjfLIrCmOMMT5ZojDGGOOTJQpjjDE+WaKoh7x1sDJFZJTbsbhNRH4mIs+IyHwROdfteOqa93dhlvccXOV2PG5r7L8PlamN7wtLFHVIRJ4Xkd0isvqY5eeLyAYR2SQid/ixq9uBVwMTZd2pjfOhqv9V1RuA64BxAQy3zpzgeRkDzPOeg0vqPNg6cCLnoyH+PhzrJP7f1Pj7whJF3ZoJnF9xgYiEAI8DFwA9gStEpKeI9BGRBcdMLUXkbGAtsKuugw+AmdTwfFT46J+8n2sIZuLnecEZHfJ772ZldRhjXZqJ/+fjsIb0+3Csmfj//6ZWvi9cKQrYWKnqIhFJOWbxIGCTtwgiIvIKMFpV/4ozXsdRRGQkEI3zy1AgIgtVtTywkQdGLZ0PAf4GvK2qXwc45DpxIucFyMZJFitooH/4ncj5EJF1NLDfh2Od4O9HU2rh+8IShfva8tNfhOD8xx9c1caqeieAiFwH7K2vScKHEzofwM3A2UCciHRR1ScDGZyLqjov04DHROQi4H9uBOaSqs5HY/l9OFal50NVp0LNvy8sUbhPKllW7VOQqjqz9kMJCid0PlR1Gs6XZUNX6XlR1UPAxLoOJghUdT4ay+/DsXz+v6np90WDvFStZ7KBdhXmk4EdLsUSDOx8VM7Oy9HsfBwtoOfDEoX7lgFdRaSjiIQD44E3XY7JTXY+Kmfn5Wh2Po4W0PNhiaIOicgc4Augm4hki8gvVLUUmAq8C6wDXlXVNW7GWVfsfFTOzsvR7HwczY3zYUUBjTHG+GRXFMYYY3yyRGGMMcYnSxTGGGN8skRhjDHGJ0sUxhhjfLJEYYwxxidLFMbUARFpLSKviMhmEVkrIgtF5BS34zLGH5YojAkwb4XbN4BPVLWzqvYE/gi0cjcyY/xjRQGNCbyRQEnFSqaqusK9cIw5MXZFYUzg9QYy3Q7CmJNlicIYY4xPliiMCbw1wAC3gzDmZFmiMCbwPgIiROSGwwtEZKCIDHcxJmP8ZtVjjakDItIG+DfOlUUhkAX8RlW/dTEsY/xiicIYY4xPduvJGGOMT5YojDHG+GSJwhhjjE+WKIwxxvhkicIYY4xPliiMMcb4ZInCGGOMT5YojDHG+PT/AWuZABST8YMtAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "weights, params = [], []\n",
    "for c in np.arange(-5, 5):\n",
    "    lr = LogisticRegression(C=10.**c, random_state=1)\n",
    "    lr.fit(X_train_std, y_train)\n",
    "    weights.append(lr.coef_[1])\n",
    "    params.append(10.**c)\n",
    "weights = np.array(weights)\n",
    "plt.plot(params, weights[:, 0],label='petal length')\n",
    "plt.plot(params, weights[:, 1], linestyle='--',label='petal width')\n",
    "plt.ylabel('weight coefficient')\n",
    "plt.xlabel('C')\n",
    "plt.legend(loc='upper left')\n",
    "plt.xscale('log')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see in the plot, the weight coefficients shrink as parameter C decreases, i.e., regularization parameter $\\lambda$ increases."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "acc99c6b55fd2fc6304828719d442d344302507c40af92bb67542866173c2113"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}