note/SVM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 背景"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们从一个二分类问题说起，即给定N个二维向量，它们一半被标记为正例，一半被标记为负例，我们希望有一个模型，或者分类器，或者超平面，什么都好，它的输入是一个二维向量，输出是一个一维，或者二维向量，代表它属于正例或者负例的概率。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在科技发达的现代，能解这个二分类问题的方法数不胜数，感知机、Logistic Regression、SVM、神经网络等等。在本节中我们主要介绍SVM，即支持向量机，Support Vector Machine。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 线性回归（Linear Regression）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们希望从Linear Regression引出支持向量机，先来看看Linear Regression能不能解这个问题，我们首先写几行代码生成实验数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD8CAYAAAB0IB+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJztnX2sHUd5xp/XDqS+lEJybcCQ3HMTlEYkfKTJVQRtiWiJ8qUqaVqogk5TVJBuSRWpIFVqoivRqOj+QShFqQpEtyWC2keEFkpJITQkFIqKxMdxGjuOghsn2MbETYzDZx0Fgt/+MXvw+nj37uzZmdmZ3ecnrc65e/buzuzMPPPOO+/OiqqCEEJI99nQdgIIIYSEgYJPCCE9gYJPCCE9gYJPCCE9gYJPCCE9gYJPCCE9gYJPCCE9gYJPCCE9gYJPCCE94ZS2E5Bn8+bNuri42HYyCCEkKXbs2PE9Vd1SdVxUgr+4uIjxeNx2MgghJClEZL/NcXTpEEJIT6DgE0JIT6DgE0JIT6DgE0JIT6DgE0JIT6DgE0JICEYjYHER2LDBfI5GwZMQVVgmIYR0ktEIWF4Gjh41f+/fb/4GgOEwWDJo4RNCyCzUsdhXVo6L/YSjR83+gNDCJ4SQutS12A8cKD5P2X5P0MInhJC61LXYFxbq7fcEBZ8QQupS12JfXQXm5k7cNzdn9geEgk8IIXWpa7EPh8DaGjAYACLmc20t6IQtQMEnhJD6zGKxD4fAvn3AsWPmM7DYAxR8QogPIog590okFntdGKVDCHFLJDHn3hkOk8sPLXxCSDUJxpyTk3Ei+CJyh4g8KSK7c/tuEZHvisgD2XaVi2sRQgIzsdj37wdUj1vsZaIfOua86+4jh7iy8D8K4IqC/R9Q1Quy7W5H1yKEhCTmmPO6nVHPcSL4qvoVAE+5OBchJDJijjmn+6gWvn34N4rIrszlc5rnaxFCfBBzzHkkSxakgk/B/zCAlwO4AMAhAO8vOkhElkVkLCLjw4cPe0wOIZGRiu855phzl+6jVMqjCarqZAOwCGB33d/y20UXXaSEdILt21UHA1UR87l9+8m/z82pGs+z2ebmTj4uFqry0xau7mNq5TEFgLHa6LTNQVYnmhJ1AFtz398F4M6qc1DwSSewEY/B4MTfJ9tg0Faq08VFZ5R4edgKvphjmyEiHwfwBgCbATwB4C+zvy8AoAD2AfgTVT203nmWlpZ0PB43Tg8hrbK4aKJFphkMjHsDMG6DorYnYtwgJCyJl4eI7FDVparjXEXpvEVVt6rqc1T1DFX9iKper6qvUtVXq+rVVWJPSGewmUi09T13za+cz8/mzWaLIW+RLF/sGz5pS4hrbMTDZiK0azHm0/k5csRsMeRtvfLoUqdr4/cJtdGHTyqJdfIwj+0EYFVeEvcrn0RZfmLJW1F5JDKZi9CTti42Cn4PqSPgiTQ+VXXTMYkUi6KI69SGoSw/MectkU7XVvDp0iHtUddl0eZTlXWH9S7i0LvmV7ZJd2x569iDXRR8Mhsu/Jp1BTxE4yvKV1u+9Ehei+eMovzkidFn3rVO12YYEGqjSycRXLlW6rosbIfXs7pTyvI1P9/esD6FOYs65PMzP2+2mH3msaWnBNCHT7zhyq9Z9zxFjW/SabgQDJtJRV/+5q4J+6zE6DNPoGwo+H3HZyV1NZk4izhP8pUXe1fWuM2kog8RSsSKDELXJqoDQcHvC22Ekrm0wmbtmHxY42XnnJ+vHlk0IUarVrUdyzbWexE5FPyUSM3nvF6HEkokfFjjNvkqG1k0yWeMVm1bow4bt10XcNxOKPip0AWf82TyzYcYllHHGq+ThjYehophxOQzTTa0XZ9C4qEzpeCnQpOGFcLnPMsyvyFEoq1Rhg9rPMYlfkOOOmKMjvKJh86Ugp8KTRqWLyt3QpNlfn2LxCR9XfExx7bEb0gLv82Raht46Ewp+KnQpGH5tnJt0mYzykjdIssTc0SNSyEJOYJqKzqqLWjh91jwmy601bb7oso6i0UMXRJrXLZrIQkVAdbWfExb0IffY8FXrRbziciGFlIbAelLVEUKhBh9+HD12I4m8k/mFj2la3vOkAQy1Cj4sRPrZGhVGmZZ5jcWUkmnDW2M+FT9TeY2bQ9tv0Ky5SgjCn7MxD4ZOp3WlEWy7VGSD9q0Xtt6MMqmPdjMLYWKMgpsqFHwY4aToWGIYZRUlKaYonHq0lZnU3ct/Z5FGVHwY4aToWEIPUqaxS0RYpVRl3mwPcY1dS38kB1TBFFGQQUfwB0AngSwO7fvdAD3Angk+zyt6jzOBT9Wd0QfJ0PbKIuQo6Qmbrq6afD5LEAME522aatKZ6g6V8fC74IPH8AlAC6cEvxbAdyUfb8JwHurzuNU8FOrvFVRCSmLfL6zClkWtqOktp9Z8LXKaN18tekqsiHWtlHVGQUw1IK7dAAsTgn+HgBbs+9bAeypOodTwU+p8vqMcW6LGPznNqOkkG4Wl3WySvxmyVeMi7ilQsudUQyC/4Op379fdQ6ngp9i5Y2lk/Jp8YYui6q8hHSz+OjQy845yzo0sdQ/UptkBB/AMoAxgPHCwoK7O5Bi5Y2hk/Jt8c5SFj59sbG7WapwGSHStOxjnTPrATEIfrsunRTdIzF0Ur4t3rpl4bscfblZYpqYrpOvWfMQe3vreGcUg+C/b2rS9taqc/QmSqeMGBqNT4t3lskr351gDPe8CWX3x9WKqU3TEcOIOvUytiB0lM7HARwC8DMABwG8HcA8gC9mYZlfBHB61Xl6E4e/Hm13UrFZvCHcXG3f8ybYRnz5zlcM7sgyYu6MHMEHr8hsxGYN+Yw5T1Xkp4khLzGLasydkSMo+CGIoaH5IKZ8hYxs6Ur5tUEsI40iYu6MHEHB9w1FIxyhIlt8CUDbgheKWJ8t6dpT6wVQ8KdJXTRsSEVY2k6n6zdDrZeXGASvCp/lEUvn2vZT356h4Ofx0ehi8wumICyqcaTTVoRciHmMhkEe3+XR9jLFqZXHjFDw8/go5NgqTmzpKSOGdNoIgyvxiM0wmMZ3eYQs7y6Ux4xQ8PP4KOQYLNU8qVTkWNLpYsmF0Ovn+MB3eYRsJyHKo213ZAkU/Dx9CO1LpSKnUhauxCM2w2CaEB1STHWryRIYEZclBT9PxAXljCZhcSHvT5MG1/ScdfAtHrHQpbYxS3nUWWl0lgXpAkHBnybmRueKWa2SWCIp8r/XESEf6e+CmNvShTxMcGEolAl72RaB29RW8MUcGwdLS0s6Ho/bTka3WFwE9u8/ef9gAOzbZ75v2GCq7jQiwLFjPlNXjE2a8/hK/2gErKwABw4ACwvA6iowHM5+PhIXZfWsLmX1MiAiskNVl6qO2xAiMaRFDhyo3r+wUHzM9P7RyDSSDRvM52jkIoXrp81mv2366zIcmoZ87Jj5pNh3i7L6VMb8PDA3d+K+uTljCCQCBb/r2Ijh6mp1RR6NgOVlYxGpms/lZT+iX1fAbdLfBUJ1uH2hrD6VCftttwFra8aiFzGfa2tpGQI2fp9QWxJLK6Tm73Tliw7p559lIjG1cqlLrJOrKd/3JoEOkQFO2nog1kZXhYvKG3qZ4pheUh0DMcbzp9oe8iQm7GVQ8H0QY6MLhe+8d0E8fBJy/R9b+tweIsNW8Lvlw/ft46w7mdglfPvJV1aAo0dP3Hf0qNlPmk9MT9qGCHD99W7mYvrcHlLFplcItTWy8ENYiH23aHwOf2NZciFWmtTvov91UYf73h4iAr1z6YR6RJxuBz9QPI5T1rHO2uGW3VsXriG2hyjon+D7tBC7NJkYa14oHgYf96GsbbjoWDsy6Zk6/RN8n4tydUWIqob2beeL4uGnHldZ+G2XO2mMreB7X1pBRPYB+DGAnwN4Vtd5/LfR0gqTB4PyE39zc80fjKj7mH/M2DxKnmK+uoSPZSKK2oaIuc5gwCUjOkBsSyv8lqpeYJOgmRkO/TwF16VIBJs0p5ivLuFjmYiitrFtmxF8LhnRK7oVlulj7ZNZGmBZeGjbj8bbiEbT9WdIM3yFv3JdIAL49+ED+DaA+wHsALC83rFRPnhV14dfdvwNN7Q/FxC7D58YOJdBaoJYJm0BvDT7fBGAnQAumfp9GcAYwHhhYcHrTZmZOg2wbIJs40b3k3GzEGuUDiFkZmwFP+h6+CJyC4CfqOpfF/3eynr4rtc8L5t0K6OtNecJIZ0hiklbEXmeiDx/8h3AZQB2+7xmLXws+VvmA9+4sd7xhBDiGN+Tti8G8F8ishPANwB8TlX/3fM17fGxfkvZpNvycj/WbCeERItXwVfVx1T1Ndl2vqrGpW4+Qi7LwkM/9KH0X55ACEmafr/TtksPVRFCeksUPvzo6cur8QghBH0XfF9P5/qm7Qe4CCFJckrbCWid4TB+gc8zvS7KJLIISCsfhJDg9NvCTxG+GYoQMiMUfFeEWj+nS4u5EUKCQpeOC8rcLF/9KvCxj7l1vywsFEcW8QEuQkgFtPDz5K3xzZvNZmOZl7lZ1tbCPdjFyCJCSAX9FPwiN8v0MgtHjpjNZsmFMnfKz39e73gbUo0sIoS0Tv8evCp7M9amTUbg16PsgayyB7g2biwWfT7YRQhxCB+8KqPM/VIl9kC5Zc71cwghCdA/wW/iTimbGOX6OYSQBOifS6fM/TI/Dzz99MnW/wQXL0QnhBAP0KVTRpn75bbbTrTG5+fNRsucENIR+heHPxHtsrdcUdQJIR2lf4IPpLd+DiGEOKB/Lh1CCOkpFHxCCOkJFHxCCOkJFHxCCOkJ3gVfRK4QkT0isldEbvJ9PUIIIcV4FXwR2QjggwCuBHAegLeIyHk+r0kIIaQY3xb+xQD2qupjqvpTAHcCuMbzNQkhhBTgW/BfBuA7ub8PZvt+gYgsi8hYRMaHDx/2nBxCCOkvvgVfCvadsHiPqq6p6pKqLm3ZssVzcgghpL/4FvyDAM7M/X0GgMc9X5MQQkgBvgX/mwDOEZGzROS5AK4DcJfnaxJCCCnA61o6qvqsiNwI4B4AGwHcoaoP+bwmIYSQYrwvnqaqdwO42/d1CCGErA+ftCWEkJ5AwSeEkJ5AwSeEkJ5AwSeEkJ5AwSeEkJ5AwSeEkJ5AwSeEkJ5AwSeEkJ5AwSeEkJ5AwSeEkJ5AwSeEkIaMRsDiIrBhg/kcjdpOUTHe19IhhJAuMxoBy8vA0aPm7/37zd8AMBy2l64iaOETQkgDVlaOi/2Eo0fN/tig4BNCSAMOHKi3v00o+IQQ0oCFhXr724SCTwghDVhdBebmTtw3N2f2xwYFnxDSCqlEtlQxHAJra8BgAIiYz7W1+CZsAUbpEEJaIKXIFhuGwzTSTQufEBKcWCJb8qOMzZvNlvqIYz1o4RNCghNDZMv0KOPIkeO/pT7iKMObhS8it4jId0XkgWy7yte1CCFp4Suypc68QNEoI0+ssfRN8O3S+YCqXpBtd3u+FiEkEXxEtkws9v37AdXjVnqZ6NuMJmKMpW8CffiEkOD4iGypOy9gM5qIMZa+Cb4F/0YR2SUid4jIaUUHiMiyiIxFZHz48GHPySGExMJwCOzbBxw7Zj6b+srrzgsUjTLyTEYcXQkfBRoKvojcJyK7C7ZrAHwYwMsBXADgEID3F51DVddUdUlVl7Zs2dIkOYQQT6QgenXnBaZHGfPzZsuPOIB6bqLYEVX1fxGRRQCfVdVXrnfc0tKSjsdj7+khhNgzHc0CGOs3toeLfKRzcdGI/DSDgRmVxIKI7FDVparjfEbpbM39eS2A3b6uRQjxRywx81X4mBeIIXzUJT59+LeKyIMisgvAbwF4l8drEUI8EVr0mriPXM8LpLQwmg3eBF9Vr1fVV6nqq1X1alU95OtahPSBtp4KDSl6dUMrfVM0sSti0pXik7kMyyQkAaaF8MgRs4UQxZCrQcbiPpp0rtdfD2zaZCZzASP2k2nPkGXgCgo+iZIyazYVq8p1VEubT4UWRbNs2mTE0HUZhHAfVZVNUef69NMm3+vFuMQ4r3ESqhrNdtFFFykh27erzs2pmua1/jY3Z45vM62DgaqI+dy+vTj9TdMpUn0vRFzlqhwfecszGBTnbTBwc36b9JelwWYLUQZFABirhca2LvL5jYJPVOs3OFdiUJcy8Zifd59Om3sS4j7EIMhF/zPd6TZJv03nGltdpOCTZKnb4Nqyqup2TE3SWTXqmdyzKsFrSlnZuCyDOgJet4OwSX9Zuc7Pr18GRdetk5cmUPBJsjS18EM1srodU1PrL5+v+fnjI4npdPh0c81i4fssj7rpsTl+vU6kqAzK8uXb/ZWHgk+SpYkPP2Qjq2MJhk6DL9eC7f2dCKPvDqnuiKNu+pt0UiHLhoJPkqbMkqqyqkI2MltLsI1Rhk83V1XebDpsV+UR24gjT8iyoeB3jDpDyT4TWgBDiUcZoS38JmnyUR4hR3R1oYVPwa/FesPiGCt4DMQogD6JUfBs5jZsy8OmQ2270y2DPnwKvjV1/NhdFrS6xCiAvolN8KosfNvy6EJZxhalE2R5ZFu4PPJxypZlLUPELBjVV0Yj85TjgQPA6aebfU89ZdZ7WV2NaxnfrlO0TPFkSYLBwL48UlmaOAZaXx6ZNKPuo+Sprt7ngrJH4bdtc7NiIimnaJmComWKt20zZVOnPLq2NHEMUPBLaPsNP3UEfLJ6X8xry/gklgW3+sZ6K1u6WKa4a0sTxwAFv4AYlmgtW5YVOP4qtsm+iVcu5hX7fHagMSy4lRKu8uK7ow25SmdvsHH0h9pimbQNEenhIvoglYiUPiy4lQou8xLbMgsp4TpfYJTO7PiuyK4aXSox56kLciodqw0u89Kl++KSWR5Ma1pfKfgN8F2RXZ0/lqdKq0jdEmzjadYJrvPlMi++V7ZMkSbLLzdptxT8Bvi2GF01uhjWjbGppKlbgm2l30f5us6Lz5UtU6TJ8stNDAgKfkNiWuGvrXTmaVJJU2/obaXfR0fTZlmk3vHb0GT55egtfABvBvAQgGMAlqZ+uxnAXgB7AFxuc76mgp/KcDFFAWxaSVMpmzLaSL8vV9J0Xm64IX6jIRWaLr88K6EE/xUAzgXw5bzgAzgPwE4ApwI4C8CjADZWna+J4KcmomUCEqswFt3fUC/d6CshOtlU3IKpYLuCqusFEIO6dAoE/2YAN+f+vgfA66rO00Twu1CZYu+01lvMLaZ0doUm9cH2f1OZ+E+Joo7Wd97bFvy/A/CHub8/AuBNJf+7DGAMYLywsDBzhrswXAzdacUaZkmO47uMUgntTR3fbcaZ4AO4D8Dugu2a3DHTgv/BAsH//apr9d3CDxm+2MRK70Ln2haxTbJ3od2kgO82Yyv4lUsrqOqlqvrKgu0z6/zbQQBn5v4+A8DjVddqQhcew266dkjVI/P5JSMAU+Xy2D4WzzVOZiPkkh22ZdSFdpMC0bQZm16hasPJFv75OHHS9jF4nrRVTX+46NtnW7VOua3F0RdfrGti9Zen0m5SSWcRnfDhA7gWxpp/BsATAO7J/bYCE52zB8CVNueLKQ6/LXz6bH29iajOO2f7DP3ls9MFI8NnedgKPl+A0hE2bDjZRQOc+GKUqpeqzM2ZdczrLGVb9LKLpufsKnyhx+zw3q0PX4DSM2x8hOstuTwYzCbMRUvk5vG5Ln1qSxbTXz47fBmKGyj4HaFMzPMvRnH1JqI8Ng3OR6OM4Z0FdSm6/xz92BHNpGfq2Ph9Qm2hfPg+n3hrkzYejLKZCF5vXmDWsmgznDC1p6RnJab8dMGH7xNw8bRiiipO1ypR29Eg0xOSk2tP39cmZdHWswBlwnPDDd0SpBgFNqYOKDYo+CU0tUhToM1okImVbjPKaFIWbVn4ZdfduLFbdSmVp76JgYJfgk1oYupPjM7SWF03OFdhomVlYWuBus6XTZq7UJdCGg0xjiZSg4JfQh8s/LoNyEeDa7IuuKtOyke+XFr4Mc8lhbTwubxDcyj4JfTBh69az7L10eBmXRc8hDg3yZcrH37s9TCk1Z36KzBjgIK/DjFbVm3go8HN4nJxXRahXiAyS5ROrCPNNtqGLwu/T8t5U/ATom3rw3eDSylfba9m2aaF25YvfZbrzuLOi6FD9QUFPxLa8DPPksa20+CD9fJVVC4h70NIC982XzE+21B2bKiFAlOBgh8BTSpmaOujqw8T1RH2SThpiLJo8vxCXWzrWCrvOQi9UGAKUPAjoEnFjGHCqquWv431F0LwZn1+oS6hXobStjusTgRYF+pxHgr+DISK2bapmKGH80XEMvpwTd1Y+pD5bStiSrVZXYnBHVYVAeZy1BQbFPwpYorZrqqYLhtKEwFJZYhfl7J7Mj/f/oimzYipybGzGD0hjYO2HrqLGQp+jrZ86TFUzCYC0lULfz3rr+2Xufhyq/gWP77cpV0o+Dna9KW3HfPfREC65sOv4zNvazG0VNwq03TVOEgFCn6OEL70WMMvm163K5ZU3cicNhdDS8GtMk3XjIPUoODncOlLnzV+u+3G2AXRbkLdyJyybZYRX0zRKz5hPWsPCn4OV770JvHbbTfGvlM3MseVhR9b9ArpJkEEH8CbATwE4BiApdz+RQBPA3gg2263OV+bUTo2NInfZmM8ThuWYN3IHFc+/BijV0j3CCX4rwBwLoAvFwj+7rrnazsOv4om8dtsjIYY5zJ8RrYweoWEIKhLpy+C3zR+O5XG6DOdfZvL4MiOhCAGwf8/AP8N4D8BvH6d/10GMAYwXlhY8H1fGjGLlZgavi3wvs1lcGRHQuBM8AHcB2B3wXZN7phpwT8VwHz2/SIA3wHwK1XXitXCbzuWPiS+LdI+WrxdMQZIvNgK/imoQFUvrTqm4H+eAfBM9n2HiDwK4FczSz4pRiNgeRk4etT8feQIMDcHbNsGDIftps0HBw7U21+X1dUT7ydg7ufqqpvzx8hw2M26QtJjg4+TisgWEdmYfT8bwDkAHvNxLd+srJwoToD5e2WlnfT4ZmGh3v66DIfA2howGAAi5nNtjYJISAgaCb6IXCsiBwG8DsDnROSe7KdLAOwSkZ0APgngHar6VLOktoNvizc2VleNxZ3HtQU+HAL79gHHjplPij0hYWgk+Kr6aVU9Q1VPVdUXq+rl2f5Pqer5qvoaVb1QVf/NTXLDM4vFOxoBi4vAhg3mczTykTI/0AInpLuI8ffHwdLSko7Hcbn5p334gLF419bM95UVY+0vLBy3gsuOp2gSQnwgIjtUdanyOAp+NaORvbBv2mQmdqcZDIz7ghBCXGMr+JVROqQ4ymJxsXgyd3rfhK76/Akh6eAlSicWfPrS6wq4qygXQgiZlc4K/sT3vn+/ebRn/37ztyvRLxPw+Xn/US6EEDILnRV83/HzZeGLt93GKBdCSJx01ofvO35+IuDTk7mT/RR4QkhsdFbwFxaMG6dovyv4yDwhJCU669IJ8cQoIYSkRGcFn0+MEkLIiXTWpQPQ5UIIIXk6a+ETQgg5EQo+IYT0BAo+IYT0BAo+IYT0BAo+IYT0BAo+IYT0hKjWwxeRwwAKno+tzWYA33NwHtfEmC6myZ4Y08U02RNjulylaaCqW6oOikrwXSEiY5uXAYQmxnQxTfbEmC6myZ4Y0xU6TXTpEEJIT6DgE0JIT+iq4K+1nYASYkwX02RPjOlimuyJMV1B09RJHz4hhJCT6aqFTwghZIpkBV9E3iwiD4nIMRFZmvrtZhHZKyJ7ROTykv8/S0S+LiKPiMgnROS5HtL4CRF5INv2icgDJcftE5EHs+PGrtMxda1bROS7uXRdVXLcFdn92ysiN3lO0/tE5FsisktEPi0iLyw5zvt9qsq3iJyaleverP4s+kjH1DXPFJEvicjDWZ3/s4Jj3iAiP8yV67sDpGvd8hDD32b3apeIXOg5Pefm8v+AiPxIRN45dUyQ+yQid4jIkyKyO7fvdBG5N9Oce0XktJL/fWt2zCMi8lanCVPVJDcArwBwLoAvA1jK7T8PwE4ApwI4C8CjADYW/P8/Abgu+347gBs8p/f9AN5d8ts+AJsD3bdbAPx5xTEbs/t2NoDnZvfzPI9pugzAKdn39wJ4bxv3ySbfAP4UwO3Z9+sAfCJAmW0FcGH2/fkA/qcgXW8A8NkQdci2PABcBeDzAATAawF8PWDaNgL4X5j49OD3CcAlAC4EsDu371YAN2Xfbyqq5wBOB/BY9nla9v00V+lK1sJX1YdVdU/BT9cAuFNVn1HVbwPYC+Di/AEiIgB+G8Ans10fA/C7vtKaXe8PAHzc1zUcczGAvar6mKr+FMCdMPfVC6r6BVV9NvvzawDO8HWtCmzyfQ1MfQFM/XljVr7eUNVDqnp/9v3HAB4G8DKf13TENQD+UQ1fA/BCEdka6NpvBPCoqrp4kLM2qvoVAE9N7c7XnTLNuRzAvar6lKp+H8C9AK5wla5kBX8dXgbgO7m/D+LkxjEP4Ac5kSk6xiWvB/CEqj5S8rsC+IKI7BCRZY/pmHBjNsS+o2RYaXMPffE2GKuwCN/3ySbfvzgmqz8/hKlPQchcSL8G4OsFP79ORHaKyOdF5PwAyakqjzbr0XUoN7BC36cJL1bVQ4DpxAG8qOAYr/cs6jdeich9AF5S8NOKqn6m7N8K9k2HItkcY4VlGt+C9a3731DVx0XkRQDuFZFvZRbCTKyXJgAfBvAemPy+B8bV9LbpUxT8b6NwLpv7JCIrAJ4FMCo5jdP7VJTMgn3e6k5dROSXAXwKwDtV9UdTP98P4774STYv868AzvGcpKryaOVeZfNxVwO4ueDnNu5THbzes6gFX1UvneHfDgI4M/f3GQAenzrmezDDy1MyK63oGCdpFJFTAPwegIvWOcfj2eeTIvJpGNfCzEJme99E5O8BfLbgJ5t76DRN2eTU7wB4o2bOzIJzOL1PBdjke3LMwaxsX4CTh+7OEZHnwIj9SFX/Zfr3fAegqneLyIdEZLOqels7xqI8nNcjS64EcL+qPjH9Qxv3KccTIrJVVQ9lrq0nC445CDPPMOEMmHlKJ3TRpXMXgOuyaIqzYHrvb+QPyATlSwDelO16K4CyEUNTLgXwLVU9WPSjiDxPRJ4/+Q4zgbm76FgXTPlQry251jcBnCMmkum5MMPjuzym6QoAfwHgalU9WnJMiPu+eLYwAAABaElEQVRkk++7YOoLYOrPf5R1UK7I5gg+AuBhVf2bkmNeMplLEJGLYdr2EY9psimPuwD8URat81oAP5y4NDxTOqIOfZ+myNedMs25B8BlInJa5m69LNvnBt+z1b42GLE6COAZAE8AuCf32wpMtMUeAFfm9t8N4KXZ97NhOoK9AP4ZwKme0vlRAO+Y2vdSAHfn0rEz2x6CcXH4vG/bADwIYBdMBdw6nabs76tgokEeDZCmvTB+ywey7fbpNIW6T0X5BvBXMJ0RAPxSVl/2ZvXn7AB1/TdhhvW7cvfoKgDvmNQtADdm92UnzMT3r3tOU2F5TKVJAHwwu5cPIhdN5zFdczAC/oLcvuD3CabDOQTgZ5lOvR1mrueLAB7JPk/Pjl0C8A+5/31bVr/2Avhjl+nik7aEENITuujSIYQQUgAFnxBCegIFnxBCegIFnxBCegIFnxBCegIFnxBCegIFnxBCegIFnxBCesL/A0w8AXKPQOhTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "data_count = 100\n",
    "\n",
    "x1_positive = np.linspace(-10, 10, data_count)\n",
    "x2_positive = 0.3 * x1_positive + 10 + np.random.randint(-5, 5, data_count)\n",
    "y_positive = np.array([1] * data_count)\n",
    "\n",
    "x1_negative = np.linspace(-10, 10, data_count)\n",
    "x2_negative = 0.3 * x1_negative - 10 + np.random.randint(-5, 5, data_count)\n",
    "y_negative = np.array([-1] * data_count)\n",
    "\n",
    "x1 = np.concatenate([x1_positive, x1_negative])\n",
    "x2 = np.concatenate([x2_positive, x2_negative])\n",
    "\n",
    "y_label = np.concatenate([y_positive, y_negative])\n",
    "\n",
    "plt.scatter(x1_positive, x2_positive, c='r')\n",
    "plt.scatter(x1_negative, x2_negative, c='b')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "那么Linear Regression希望做什么呢，对于一个Linear Regression模型（函数）："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "y = \\boldsymbol{w^Tx} + b\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "直观地，Linear Regression希望最最小化下式："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "loss(\\boldsymbol{w}, b) = \\frac{1}{N}\\sum^{N}_{i=0}(\\boldsymbol{w^Tx_i} + b - y_i)^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里使用平方损失函数其实问题很大，不过没关系，我们先忽略这点，稍后再来分析这里有什么问题。所以我们可以很自然的对$\\boldsymbol{w}, b$求梯度："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{aligned}\n",
    "        \\frac{\\partial loss(\\boldsymbol{w}, b)}{\\partial \\boldsymbol{w}} &= \\frac{2}{N}\\sum^{N}_{i=0}(\\boldsymbol{w^Tx_i} + b - y_i)\\cdot \\boldsymbol{x_i}, \\\\\n",
    "        \\frac{\\partial loss(\\boldsymbol{w}, b)}{\\partial b} &= \\frac{2}{N}\\sum^{N}_{i=0}(\\boldsymbol{w^Tx_i}+b-y_i)\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后通过梯度下降更新$\\boldsymbol{w}, b$："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{aligned}\n",
    "\\boldsymbol{w}_{t+1} &= \\boldsymbol{w}_t - \\eta \\frac{\\partial l(\\boldsymbol{w}_t, b_t)}{\\partial \\boldsymbol{w}_t} \\\\\n",
    "b_{t+1} &= b_t - \\eta \\frac{\\partial l(\\boldsymbol{w}_t, b_t)}{\\partial b_t}\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后通过几行代码简单实现一下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD8CAYAAAB0IB+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJztnX2MJdWZ3p+3e5hhGrphpmcYZrqnuweEEcbLYqaXGBN7WfwBRsiEZB2x6SUoXqljIqTsSpECammDsmop9sZZEcUf6qzROtBre7+8Zr3sYuysY2WlZd3jAB5iEwZ2ZtzzBcxgmKH5GvrNH1V3pvreqnurbp1z6pyq5yeV7r1169Y9p845T73nPe85JaoKQggh9Weg6gQQQghxAwWfEEIaAgWfEEIaAgWfEEIaAgWfEEIaAgWfEEIaAgWfEEIaAgWfEEIaAgWfEEIawrqqE5Bky5YtOjU1VXUyCCEkKPbs2fOyqm7tdZxXgj81NYWlpaWqk0EIIUEhIgfyHEeXDiGENAQKPiGENAQKPiGENAQKPiGENAQKPiGENAQKPiGEuGBxEZiaAgYGotfFRedJ8CoskxBCasniIjA7C6ysRJ8PHIg+A8DMjLNk0MInhJB+KGKxz82dFfsWKyvRfofQwieEkKIUtdgPHkw/T9Z+S9DCJ4SQohS12Ccmiu23BAWfEEKKUtRin58HhobW7hsaivY7hIJPCCFFKWqxz8wACwvA5CQgEr0uLDgdsAUo+IQQUpx+LPaZGWD/fmB1NXp1LPYABZ8QYgMPYs6t4onFXhRG6RBCzOJJzLl1ZmaCyw8tfEJIbwKMOSedGBF8EXlQRF4Ukb2JffeLyCEReTLebjHxX4QQx7Qs9gMHANWzFnuW6LuOOa+7+8ggpiz8PwBwc8r+31PVq+PtUUP/RQhxic8x50VvRg3HiOCr6g8AnDBxLkKIZ/gcc073USFs+/DvEZGnY5fPJsv/RQixgc8x554sWRAKNgX/SwAuBXA1gCMAPp92kIjMisiSiCy99NJLFpNDiGeE4nv2OebcpPsolPIog6oa2QBMAdhb9Lvktnv3biWkFjz8sOrkpKpI9Prww53fDw2pRp7naBsa6jzOF3rlpypMXcfQyqMNAEuaR6fzHJTrRG2iDmB74v1vAfh6r3NQ8EktyCMek5Nrv29tk5NVpTpcTNyMAi+PvIIv0bHlEJGvAbgBwBYAxwD8h/jz1QAUwH4A/1pVj3Q7z/T0tC4tLZVODyGVMjUVRYu0MzkZuTeAyG2Q1vZEIjcIcUvg5SEie1R1utdxpqJ0fk1Vt6vqOao6rqpfUdU7VfUXVPUqVf1kL7EnpDbkGUjM63uum185mZ8tW6LNh7x5snyxbTjTlhDT5BGPPAOhdYsxb8/P8ePR5kPeupVHnW66efw+rjb68ElPfB08TJJ3ALBXXgL3K3eQlR9f8pZWHoEM5sL1oK2JjYLfQIoIeCCNT1XN3JhE0kVRxHRq3ZCVH5/zFshNN6/g06VDqqOoy6LKWZVFu/Um4tDr5lfOk27f8laziV0UfNIfJvyaRQXcReNLy1dVvnRPHotnjLT8JPHRZ163m26eboCrjS6dQDDlWinqssjbve7XnZKVr9HR6rr1IYxZFCGZn9HRaPPZZ+5bejIAffjEGqb8mkXPk9b4WjcNE4KRZ1DRlr+5bsLeLz76zAMoGwp+07FZSU0NJvYjzq18JcXelDWeZ1DRhggFYkU6oW4D1Y6g4DeFKkLJTFph/d6YbFjjWeccHe3dsyiDj1atajWWra/XwnMo+CERms+52w3FlUjYsMbz5CurZ1Emnz5atVX1OvK47eqA4XZCwQ+FOvicW4NvNsQwiyLWeJE0VDEZyocek8005aHq+uQSCzdTCn4olGlYLnzO/Szz60Ikqupl2LDGfVzi12Wvw8foKJtYuJlS8EOhTMOyZeW2KLPMr22RaKWvLj5m35b4dWnhV9lTrQILN1MKfiiUaVi2rdw8acvTywjdIkvic0SNSSFx2YOqKjqqKmjhN1jwyy60VbX7opd15osYmsTXuGzTQuIqAqyq8ZiqoA+/wYKv2lvMWyLrWkjzCEhToipCwEXvw4arJ29vIjkzN22Wbt5zusSRoUbB9x1fB0N7paGfZX59IZR05qGKHp+qvcHcsu2h6kdIVhxlRMH3Gd8HQ9vTGrJIVt1LskGV1mtVE6PytIc8Y0uuoowcG2oUfJ/hYKgbfOglpaXJp2icolR1sym6ln7Doowo+D7DwVA3uO4l9eOWcLHKqMk85D3GNEUtfJc3Jg+ijJwKPoAHAbwIYG9i32YAjwN4Ln7d1Os8xgXfV3dEEwdDqygLl72kMm66ommwORfAh4HOvGnrlU5Xda6IhV8HHz6ADwO4pk3wPwfg3vj9vQA+2+s8RgU/tMrbKyohZJFP3qxclkXeXlLVcxZsrTJaNF9Vuory4Gvb6HUzcmCoOXfpAJhqE/xnAWyP328H8GyvcxgV/JAqr80Y56rwwX+ep5fk0s1isk72Er9+8uXjIm6hUPHNyAfB/3nb96/0OodRwQ+x8vpyk7Jp8boui155celmsXFDzzpnP+vQ+FL/SGGCEXwAswCWACxNTEyYuwIhVl4fblK2Ld5+ysKmL9Z3N0svTEaIlC17X8fMGoAPgl+tSydE94gPNynbFm/RsrBdjrbcLD4NTBfJV7958L291fxm5IPg/27boO3nep2jMVE6WfjQaGxavP0MXtm+CfpwzcuQdX1MrZhaNh0+9KhDL+McuI7S+RqAIwDeAbAM4DcAjAL4XhyW+T0Am3udpzFx+N2o+iblm8Xrws1V9TUvQ96IL9v58sEdmYXPNyNDcOIV6Q/frCGbMeehinw7PuTFZ1H1+WZkCAq+C3xoaDbwKV8uI1vqUn5V4EtPIw2fb0aGoODbhqLhDleRLbYEoGrBc4Wvc0vqNms9BQp+O6GLRh5CEZaq02n6yVDd8uKD4PXCZnn4cnOteta3ZSj4SWw0Ot/8giEIi6of6cwrQibE3EfDIInt8qh6meLQyqNPKPhJbBSybxXHt/Rk4UM68wiDKfHwzTBox3Z5uCzvOpRHn1Dwk9goZB8s1SShVGRf0mliyQXX6+fYwHZ5uGwnLsqjandkBhT8JE0I7QulIodSFqbEwzfDoB0XNySf6laZJTA8LksKfhKPC8oYZcLiXF6fMg2u7DmLYFs8fKFObaOf8iiy0mg/C9I5goLfjs+NzhT9WiW+RFIkvy8iQjbSXwcxz0sd8tDChKGQJexZmwdu07yCL9GxfjA9Pa1LS0tVJ6NeTE0BBw507p+cBPbvj94PDERVtx0RYHXVZurSyZPmJLbSv7gIzM0BBw8CExPA/DwwM9P/+YhfZNWzomTVS4eIyB5Vne513ICLxJAKOXiw9/6JifRj2vcvLkaNZGAgel1cNJHC7mnLsz9v+osyMxM15NXV6JViXy+y6lMWo6PA0NDafUNDkSEQCBT8upNHDOfne1fkxUVgdjayiFSj19lZO6JfVMDzpL8OuLrhNoWs+pQl7A88ACwsRBa9SPS6sBCWIZDH7+NqC2JphdD8naZ80S79/P0MJIZWLkXxdXA15OteJtDBM8BBWwv42uh6YaLyul6m2KeHVPuAj/H8obaHJIEJexYUfBv42OhcYTvvdRAPm7hc/ycvTW4PnpFX8Ovlw7ft4yw6mFgnbPvJ5+aAlZW1+1ZWov2k/MB0q22IAHfeaWYspsntIVTy3BVcbaUsfBcWYtMtGpvdX1+WXPCVMvU77bcm6nDT24NHoHEuHVdTxOl2sAPF4yxZN9Z+b7hZ19aEa4jtwQuaJ/g2LcQ6DSb6mheKR4SN65DVNkzcWGsy6Bk6zRN8m4ty1UWIenXtq84XxcNOPe5l4Vdd7qQ0eQXf+tIKIrIfwEkA7wI4rV2m/5ZaWqE1MSg58Dc0VH5iRNFp/j6TZyp5iPmqEzaWiUhrGyLR/0xOcsmIGuDb0gq/oqpX50lQ38zM2JkFV6dIhDxpDjFfdcLGMhFpbeOhhyLB55IRjaJeYZk21j7ppwFmhYdWPTU+j2iUXX+GlMNW+CvXBSKAfR8+gH8A8CMAewDMdjvWy4lXRX34WcfffXf1YwG++/BJBMcySEHgy6AtgB3x60UAngLw4bbvZwEsAViamJiwelH6pkgDzBogGxw0PxjXD75G6RBC+iav4DtdD19E7gdwSlX/c9r3layHb3rN86xBtyyqWnOeEFIbvBi0FZHzRGS49R7AxwHstfmfhbCx5G+WD3xwsNjxhBBiGNuDttsA/G8ReQrA3wP4S1X9a8v/mR8b67dkDbrNzjZjzXZCiLdYFXxVfUFVfzHerlRVv9TNRshlVnjoF78Y/sMTCCFB0+xn2tZpUhUhpLF44cP3nqY8Go8QQtB0wbc1O9c2VU/gIoQEybqqE1A5MzP+C3yS9nVRWpFFQFj5IIQ4p9kWfojwyVCEkD6h4JvC1fo5dVrMjRDiFLp0TJDlZvnbvwW++lWz7peJifTIIk7gIoT0gBZ+kqQ1vmVLtOWxzLPcLAsL7iZ2MbKIkCB48/SbeOGVF/CDAz/AE8tPOP3vZlr4aevnAGut9OPHzx7fyzLPcqe8+276/rITuwCz6/8QQozw2luvYfm1ZRx67VD0ejJ6Tb5/eeXlM8ffdOlN+Otfd7f4QPMmXmU9GWvjxrUin0bWhKysCVyDg+miz4ldhATFqq7i5ZWXuwr5odcO4eTbJzt+u3VoK8ZHxjE2Mobx4fGz70fGsevCXbh086Wl05d34lXzLPws90v7vjSyLPP5+fSbyF13rfXht/bT/UKIN5xePY0jJ490iPfyyeUz7w+dPIS33317ze8GZRA7hndgbGQM77vofbjp0pswPhIL+nAk6DuGd2DDug0V5ayT5gl+GXdK1sBoNzfL9dfT/UJIRay8s3JGsJOuluWTZ98fe/0YVnXtEuXnrjv3jHh/cOcHO4R8bGQM287bhsGBjFVwPaV5Lp0s98voKPDGG9mWvokHohNCjKCqePWtV8+6VTJcLSfeONHx2wvPvXCteMevSVfLpnM3QUQqyFl/0KWTRZb75YEHovcta3zz5ujziRO0zAlxyKqu4sXXX+w5+LnyzlrjTCDYdv42jA2P4ZJNl+BDEx/qEPKx4TGct/68inJWPc0T/F5RLhR1Qqzx9rtv4/DJw12F/PDJwzi9enrN79YNrDtjiV998dW49bJbz4h4S8i3D2/H+sH1FeUsDJrn0iGEWOHU26d6RrEce/1Yx++GzhnCzpGda6zwdp/51vO2YkA4bSgLunQIIUZQVZx440RnFEvb4Oerb73a8dvNGzefEe/d23enDn5esOGCoPzlIUPBJ6TBvLv6Lo6eOtpVyA+dPIQ3T7+55ncDMoCLz78Y4yPjeM/oe3Djrhs7Bj/Hhsew8ZyNFeWMpEHBJ6SmvHn6TRw+eXitkCet9JOHcOTkEbyraycHrh9cf0a8rx27NjWK5eLzL8a6AcpHaLDECAmQolP4WwyvHz4j3ldsvQLjw+Mdg59bhrbQxVJTrAu+iNwM4AEAgwB+X1X/k+3/JCRUTEzhHx8ZxwfGPtAh5GMjYxjZMFJBrogvWBV8ERkE8AUAHwOwDOCHIvKIqv5fm/9LiI+kTeFvF/KsKfzbh7djfGR8zRT+pKvFtyn8xE9sW/jXAtinqi8AgIh8HcBtACj4pFa0T+FPrsfSstaPnjoKxdow6NYU/rHhsTNT+Nt95iFO4Sd+YlvwxwD8LPF5GcA/Sh4gIrMAZgFggg/xIJ6hqvj5mz/vjGJps9JfefOVjt9eeO6FZ8T7qm1XrQlFbAl6aFP4SdjYFvy0mrzGxFHVBQALQDTxynJ6CDlD2hT+9iiWrCn8F513EcZHxtdM4W/3mTd5Cj/xE9uCvwxgZ+LzOIDDlv+TkI4p/GlC3m0K/9jIGKfwk9phW/B/COAyEdkF4BCAOwD8C8v/SWpOcgp/mpAvv7aMF19/seN3Q+cMnRHuG6ZuSF0tkVP4SZ2xKviqelpE7gHwGKKwzAdV9Rmb/0nCJW0Kf9rgZ7cp/GPDY2em8LcPfnIKP2k61uPwVfVRAI/a/h/iN+1T+NOEPG0Kv0CwfXg7xobHOqbwJxfb4hR+QnrDmbakNMkp/FmzP4+eOtp1Cv8vjf0Sbh++vWPwk1P4CTEHWxLpSvsU/rRJQ92m8I+NjOFjl35szRT+lshzCj8hbqHgN5T2KfxZg5+n3j7V8dutQ1sxNjKGnRfsxHXj13UIOafwE+InFPwa8s677+DoqaNdhfzwycNdp/BfufXKjin8YyNj2DG8A+euO7einBFCykDBD4zWFP6Oh1GcPOs/7zWF//qd13cI+fjIOKfwE1JzKPie0D6FP+thFD2n8F90Vcfa5ZzCTwgBKPhOSE7h77aGefsUfgDYdt42jI+MY9emXR1T+FszQs9ff34FuSKEhAYFvyQmpvC/f/v7cet7bu143ien8BNCTELB74KJKfy/PPnLqQ9uvui8iziFnxDilEYKfnIKf7fBzzxT+NOe98kp/IQQH6md4Hebwp98qlC3KfyXj16OG6du7BDyHcM7MHTOUEU5I4T4yuIiMDcHHDwITEwA8/PAzEzVqeqkdoL/nee/g1v+8JY1+9Km8LcveXvx+RfjnMFzKko1ISRUFheB2VlgJY65OHAg+gz4J/qi6s8zR6anp3VpaanUOY6cPIJvPfstTuEnhDhhaioS+XYmJ4H9+92kQUT2qOp0z+PqJviEEOKSgQEgTUZFgNVVN2nIK/gMEyGEkBJkPYrbx0d0U/AJIaQE8/PAUFssx9BQtN83KPiEkEpYXIz83wMD0eviYtUp6o+ZGWBhIfLZi0SvCwv+DdgCNYzSIYT4T0iRLXmYmQkj3bTwCSHOmZs7K/YtVlai/S5J9jK2bIm20Hsc3aCFTwhxzsGDxfbboL2Xcfz42e9C73FkYc3CF5H7ReSQiDwZb7f0/hUhpAnYimwpMi6Q1stIUkWPwza2XTq/p6pXx9ujlv+LEBIINiJbWhb7gQNRXHzLSs8S/Ty9CZc9DhfQh08IcY6NyJai4wJ5ehM+xtKXwbbg3yMiT4vIgyKyKe0AEZkVkSURWXrppZcsJ4cQ4gszM9HSA6ur0WtZX3nRcYG0XkaSVo+jLuGjQEnBF5HvisjelO02AF8CcCmAqwEcAfD5tHOo6oKqTqvq9NatW8skhxBiiRBEr+i4QHsvY3Q02pI9DqCYm8h3nKylIyJTAL6tqu/rdhzX0iHEP9qjWYDI+vVtcpGNdPqwMFoeKl9LR0S2Jz7eDmCvrf8ihNjDl5j5XtgYF/AhfNQkNn34nxORH4vI0wB+BcBvWfwvQoglXIteGfeR6XGBkBZGy4M1wVfVO1X1F1T1KlX9pKoesfVfhDSBqmaFuhS9oqGVtkkb2BWJ0hXizFyGZRISAO1CePx4tLkQRZerQfriPmrdXO+8E9i4MRrMBSKxbw17uiwDU1DwiZdkWbOhWFWmo1qqnBWaFs2ycWMkhqbLwIX7qFfZpN1c33gjyne3GBcfxzU6UFVvtt27dyshDz+sOjSkGjWv7tvQUHR8lWmdnFQViV4ffjg9/WXTKdL7WoiYylU2NvKWZHIyPW+Tk2bOnyf9WWnIs7kogzQALGkOja1c5JMbBZ+oFm9wpsSgKFniMTpqPp15romL6+CDIKf9pv2mWyb9eW6uvtVFCj4JlqINriqrquiNqUw6e/V6Wtesl+CVJatsTJZBEQEveoPIk/6sch0d7V4Gaf9bJC9loOCTYClr4btqZEVvTGWtv2S+RkfP9iTa02HTzdWPhW+zPIqmJ8/x3W4iaWWQlS/b7q8kFHwSLGV8+C4bWRFL0HUabLkW8l7fljDaviEV7XEUTX+Zm5TLsqHgk6DJsqR6WVUuG1leS7CKXoZNN1evvOW5YZsqD996HElclg0Fv2YU6Uo2GdcC6Eo8snBt4ZdJk43ycNmjKwotfAp+Ibp1i32s4D7gowDaxEfByzO2kbc88txQq77pZkEfPgU/N0X82HUWtKL4KIC28U3weln4ecujDmXpW5SOk+WR88Llkc+StSxrFiLRglFNZXExmuV48CCweXO078SJaL2X+Xm/lvGtO2nLFLeWJJiczF8eoSxN7AOVL49MylF0Knmoq/eZIGsq/EMPmVkxkWSTtkxB2jLFDz0UlU2R8qjb0sQ+QMHPoOon/BQR8NbqfT6vLWMTXxbcahrdVrY0sUxx3ZYm9gEKfgo+LNGatSwrcPZRbK19La+czyv22byB+rDgVkiYyovtG63LVTobQx5Hv6vNl0FbF5EeJqIPQolIacKCW6FgMi++LbMQEqbzBUbp9I/timyq0YUScx66IIdyY82DybzU6bqYpJ+JaWXrKwW/BLYrsqnz+zKrtBehW4JVzGZtYTpfJvNie2XLECmz/HKZdkvBL4Fti9FUo/Nh3Zg8lTR0S7Cq9NsoX9N5sbmyZYiUWX65jAFBwS+JTyv8VZXOJGUqaegNvar027jRVFkWod/481Bm+WXvLXwAnwLwDIBVANNt390HYB+AZwHclOd8ZQU/lO5iiAJYtpKGUjZZVJF+W66k9rzcfbf/RkMolF1+uV9cCf4VAC4H8P2k4AN4L4CnAGwAsAvA8wAGe52vjOCHJqJZAuKrMKZdX1cP3WgqLm6yobgFQyHvCqqmF0B06tJJEfz7ANyX+PwYgOt6naeM4NehMvl+0+q2mJtP6awLZepD3t+GMvAfEmk3Wtt5r1rw/xuAX098/gqAX8347SyAJQBLExMTfWe4Dt1F1zctX8MsyVlsl1Eoob2hY7vNGBN8AN8FsDdluy1xTLvgfyFF8P9Zr/9quoXvMnyxjJVeh5trVfg2yF6HdhMCtttMXsHvubSCqn5UVd+Xsn2ry8+WAexMfB4HcLjXf5WhDtOwy64d0mvKfHLJCCCqcknyTovnGif94XLJjrxlVId2EwLetJk8d4VeGzot/CuxdtD2BVgetFUNv7to22fba53yvBZHU3yxpvHVXx5KuwklnWnUwocP4HZE1vxbAI4BeCzx3Ryi6JxnAXwiz/l8isOvCps+W1tPIiryzNkmQ395/9TByLBZHnkFnw9AqQkDA50uGmDtg1F6PVRlaChax7zIUrZpD7soe866wgd69A+vXXf4AJSGkcdH2G3J5cnJ/oQ5bYncJDbXpQ9tyWL6y/uHD0MxAwW/JmSJefLBKKaeRJQkT4Oz0Sh9eGZBUdKuP3s/+fBm0DN08vh9XG2ufPg2Z7xVSRUTo/IMBHcbF+i3LKoMJwxtlnS/+JSfOvjwbQIunpZOWsWpWyWqOhqkfUCy9d/t17VMWVQ1FyBLeO6+u16C5KPA+nQD8g0KfgZlLdIQqDIapGWl5+lllCmLqiz8rP8dHKxXXQpl1jeJoOBnkCc0MfQZo/00VtMNzlSYaFZZ5LVATecrT5rrUJdcGg0+9iZCg4KfQRMs/KINyEaDK7MuuKmblI18mbTwfR5Lcmnhc3mH8lDwM2iCD1+1mGVro8H1uy64C3Euky9TPnzf66FLqzv0R2D6AAW/Cz5bVlVgo8H143IxXRauHiDST5SOrz3NKtqGLQu/Sct5U/ADomrrw3aDCylfVa9mWaWFW5UvvZ//7ced58MN1RYUfE+ows/cTxqrToMNuuUrrVxcXgeXFn7efPk4tyHrWFcLBYYCBd8DylRM19ZHXScTFRH2Vjipi7IoM3+hKHnrWCjPOXC9UGAIUPA9oEzF9GHAqq6Wfx7rz4Xg9Tt/oSiuHoZStTusSARYHepxEgp+H7iK2c5TMV1359PwpfdhmqKx9C7zW1XElGq5uuKDO6xXBJjJXpNvUPDb8Clmu1fFNNlQyghIKF38omRdk9HR6ns0VUZMtY7tx+hxaRxUNenOZyj4CarypftQMcsISF0t/G7WX9UPc7HlVrEtfny4S7VQ8BNU6UuvOua/jIDUzYdfxGde1WJoobhV2qmrcRAKFPwELnzpvoZflv3fulhSRSNzqlwMLQS3Sjt1Mw5Cg4KfwKQvvd/47aobYx1EuwxFI3Oytn56fD5Fr9iE9aw6KPgJTPnSy8RvV90Ym07RyBxTFr5v0SuknjgRfACfAvAMgFUA04n9UwDeAPBkvH05z/mqjNLJQ5n4bTbGs1RhCRaNzDHlw/cxeoXUD1eCfwWAywF8P0Xw9xY9X9Vx+L0oE7/Nxhjh41iGzcgWRq8QFzh16TRF8MvGb4fSGG2ms2ljGezZERf4IPivA/g/AP4XgA91+e0sgCUASxMTE7avSyn6sRJDw7YF3rSxDPbsiAuMCT6A7wLYm7LdljimXfA3ABiN3+8G8DMAI73+y1cLv+pYepfYtkibaPHWxRgg/pJX8NehB6r60V7HpPzmLQBvxe/3iMjzAN4TW/JBsbgIzM4CKyvR5+PHgaEh4KGHgJmZatNmg4MHi+0vyvz82usJRNdzft7M+X1kZqaedYWEx4CNk4rIVhEZjN9fAuAyAC/Y+C/bzM2tFScg+jw3V016bDMxUWx/UWZmgIUFYHISEIleFxYoiIS4oJTgi8jtIrIM4DoAfykij8VffRjA0yLyFIA/AfAZVT1RLqnVYNvi9Y35+cjiTmLaAp+ZAfbvB1ZXo1eKPSFuKCX4qvpNVR1X1Q2quk1Vb4r3/6mqXqmqv6iq16jqX5hJrnv6sXgXF4GpKWBgIHpdXLSRMjvQAiekvkjk7/eD6elpXVryy83f7sMHIot3YSF6PzcXWfsTE2et4KzjKZqEEBuIyB5Vne55HAW/N4uL+YV948ZoYLedycnIfUEIIabJK/g9o3RIepTF1FT6YG77vhZ19fkTQsLBSpSOL9j0pRcVcFNRLoQQ0i+1FfyW7/3AgWhqz4ED0WdTop8l4KOj9qNcCCGkH2or+Lbj57PCFx94gFEuhBA/qa0P33b8fEvA2wdzW/sp8IQQ36it4E9MRG6ctP2m4JR5QkhI1Nal42LGKCGEhERtBZ8zRgkhZC21dekAdLkQQkiS2lr4hBBC1kLBJ4SQhkDBJ4SQhkDBJ4SQhkDBJ4SQhkDBJ4SQhuDVevgi8hKAlPmxhdkC4GUD5zGNj+limvLjY7qYpvz4mC4dQ1CEAAAGAElEQVRTaZpU1a29DvJK8E0hIkt5HgbgGh/TxTTlx8d0MU358TFdrtNElw4hhDQECj4hhDSEugr+QtUJyMDHdDFN+fExXUxTfnxMl9M01dKHTwghpJO6WviEEELaCFbwReRTIvKMiKyKyHTbd/eJyD4ReVZEbsr4/S4ReUJEnhORb4jIegtp/IaIPBlv+0XkyYzj9ovIj+Pjlkyno+2/7heRQ4l03ZJx3M3x9dsnIvdaTtPvishPReRpEfmmiFyYcZz169Qr3yKyIS7XfXH9mbKRjrb/3CkifyMiP4nr/L9NOeYGEXk1Ua6/7SBdXctDIv5rfK2eFpFrLKfn8kT+nxSR10TkN9uOcXKdRORBEXlRRPYm9m0WkcdjzXlcRDZl/Pau+JjnROQuowlT1SA3AFcAuBzA9wFMJ/a/F8BTADYA2AXgeQCDKb//IwB3xO+/DOBuy+n9PIDfzvhuP4Atjq7b/QD+XY9jBuPrdgmA9fH1fK/FNH0cwLr4/WcBfLaK65Qn3wD+DYAvx+/vAPANB2W2HcA18fthAP8vJV03APi2izqUtzwA3ALgrwAIgA8AeMJh2gYBHEUUn+78OgH4MIBrAOxN7PscgHvj9/em1XMAmwG8EL9uit9vMpWuYC18Vf2Jqj6b8tVtAL6uqm+p6j8A2Afg2uQBIiIAbgTwJ/GurwL4J7bSGv/fPwfwNVv/YZhrAexT1RdU9W0AX0d0Xa2gqt9R1dPxx78DMG7rv3qQJ9+3IaovQFR/PhKXrzVU9Yiq/ih+fxLATwCM2fxPQ9wG4H9oxN8BuFBEtjv6748AeF5VTUzkLIyq/gDAibbdybqTpTk3AXhcVU+o6isAHgdws6l0BSv4XRgD8LPE52V0No5RAD9PiEzaMSb5EIBjqvpcxvcK4DsiskdEZi2mo8U9cRf7wYxuZZ5raItPI7IK07B9nfLk+8wxcf15FVF9ckLsQno/gCdSvr5ORJ4Skb8SkSsdJKdXeVRZj+5AtoHl+jq12KaqR4DoJg7gopRjrF4zr594JSLfBXBxyldzqvqtrJ+l7GsPRcpzTC5ypvHX0N26v15VD4vIRQAeF5GfxhZCX3RLE4AvAfgdRPn9HUSupk+3nyLlt6XCufJcJxGZA3AawGLGaYxep7RkpuyzVneKIiLnA/hTAL+pqq+1ff0jRO6LU/G4zJ8DuMxyknqVRyXXKh6P+ySA+1K+ruI6FcHqNfNa8FX1o338bBnAzsTncQCH2455GVH3cl1spaUdYySNIrIOwD8FsLvLOQ7Hry+KyDcRuRb6FrK8101E/juAb6d8lecaGk1TPDh1K4CPaOzMTDmH0euUQp58t45Zjsv2AnR23Y0jIucgEvtFVf2z9u+TNwBVfVREvigiW1TV2toxOcrDeD3KyScA/EhVj7V/UcV1SnBMRLar6pHYtfViyjHLiMYZWowjGqc0Qh1dOo8AuCOOptiF6O7998kDYkH5GwC/Gu+6C0BWj6EsHwXwU1VdTvtSRM4TkeHWe0QDmHvTjjVBmw/19oz/+iGAyySKZFqPqHv8iMU03Qzg3wP4pKquZBzj4jrlyfcjiOoLENWf/5l1gzJFPEbwFQA/UdX/knHMxa2xBBG5FlHbPm4xTXnK4xEA/zKO1vkAgFdbLg3LZPaoXV+nNpJ1J0tzHgPwcRHZFLtbPx7vM4Pt0WpbGyKxWgbwFoBjAB5LfDeHKNriWQCfSOx/FMCO+P0liG4E+wD8MYANltL5BwA+07ZvB4BHE+l4Kt6eQeTisHndHgLwYwBPI6qA29vTFH++BVE0yPMO0rQPkd/yyXj7cnuaXF2ntHwD+I+IbkYAcG5cX/bF9ecSB3X9HyPq1j+duEa3APhMq24BuCe+Lk8hGvj+oOU0pZZHW5oEwBfia/ljJKLpLKZrCJGAX5DY5/w6IbrhHAHwTqxTv4ForOd7AJ6LXzfHx04D+P3Ebz8d1699AP6VyXRxpi0hhDSEOrp0CCGEpEDBJ4SQhkDBJ4SQhkDBJ4SQhkDBJ4SQhkDBJ4SQhkDBJ4SQhkDBJ4SQhvD/AdleV9Ysh8HCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "w1 = np.random.normal(0, 0.002)\n",
    "w2 = np.random.normal(0, 0.002)\n",
    "b = np.random.normal(0, 0.002)\n",
    "\n",
    "training_steps = 1000\n",
    "\n",
    "eta = 0.001\n",
    "\n",
    "for step in range(training_steps):\n",
    "    grad_w1 = np.mean((w1 * x1 + w2 * x2 + b - y_label) * x1)\n",
    "    grad_w2 = np.mean((w1 * x1 + w2 * x2 + b - y_label) * x2)\n",
    "    grad_b = np.mean(w1 * x1 + w2 * x2 + b)\n",
    "\n",
    "    w1 -= eta * grad_w1\n",
    "    w2 -= eta * grad_w2\n",
    "    b -= eta * grad_b\n",
    "\n",
    "plt.scatter(x1_positive, x2_positive, c='r')\n",
    "plt.scatter(x1_negative, x2_negative, c='b')\n",
    "plt.plot(x1, -(w1 * x1 + b) / w2, c='g')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 梯度下降（Gradient Decent）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为什么能用梯度下降呢？这里快速预习一下泰勒公式："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "f(x) = \\sum^{N}_{n=0}\\frac{f^{(n)}(x_0)}{n!}(x - x_0)^n + R_n(x)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "泰勒公式的核心精神在于，某一个函数可以用该函数在某一点的函数值及其n阶导数值来近似表示，在我们的问题里，对于$\\boldsymbol{w}$就有："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "loss(\\boldsymbol{w}_{t+1}) = loss(\\boldsymbol{w}_t + \\Delta{\\theta}) \\approx loss(\\boldsymbol{w}_t) + loss'(\\boldsymbol{w}_{t})\\Delta{\\theta}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果希望梯度下降在本次迭代中使得损失函数值变小："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "loss(\\boldsymbol{w}_{t+1}) -  loss(\\boldsymbol{w}_{t}) = \\nabla loss(\\boldsymbol{w}_{t})\\Delta{\\theta} < 0\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过观察可以发现，$\\nabla loss(\\boldsymbol{w}_{t})$与$\\Delta{\\theta}\\ $都是二维向量，当$\\Delta{\\theta} = -\\eta \\nabla loss(\\boldsymbol{w}_{t})$时，即两向量反向时，其内积有最小值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 平方损失函数的问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "理想情况下，初始化$\\boldsymbol{w}, b$后，在误差反向传播计算梯度时，那些被错误分类的点，会带来很大误差，同时贡献可观的梯度用来更新权重。但是平方损失却不能很好的达到我们的目的。\n",
    "\n",
    "最小化平方损失函数的目的在于对样本所有的数据点，通过调整$\\ \\boldsymbol{w}, b$，使他们的输出更靠近他们的标签，即-1或者1，而不是完美地解决我们的二分类问题，或者说，解决了二分类问题是最小化平方损失函数的一个顺带的结果。\n",
    "\n",
    "同时，直觉地，一些被分类正确的点，反而会通过平方损失函数计算出极大的误差，贡献出可观的权重来修正自己，希望自己靠近标签，这是不合理的。\n",
    "\n",
    "既然这个问题是平方损失函数带来的，那么理论上我们就应该通过重新设计损失函数解决这个问题。事实上，我们最原始的目标函数，或者损失函数，应该是："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "loss(\\boldsymbol{w}, b) = \\sum^{N}_{i=0}\\delta \\left( \\boldsymbol{w^Tx_i} + b \\ne y_i \\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但是这个损失函数是不可微的，也就没有办法用梯度下降的方式求解了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 合页损失函数（Hinge Loss）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "事实上，带Sigmoid函数的交叉熵损失函数，或者带Sigmoid激活函数的平方损失函数也能解决这个问题，而前者一般叫Logistic Regression，而后者，好像还没有名字。在这里我们选用合页损失函数（Hingle Loss）："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "loss(\\boldsymbol{w}, b) = \\sum^{N}_{i=0} \\max(0, 1 - y_i \\cdot (\\boldsymbol{w^Tx_i} + b))\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于一维铰链损失函数，它的图像将会是：\n",
    "\n",
    "直观地，我们可以看出，对于一个样本点，如果分类正确，那么$y_i$与$\\boldsymbol{w^Tx_i} + b$应该同号，且如果他们的乘积大于1，那么损失函数的值为0，如果他们的乘积大于0（分类正确），小于1，那么损失函数的值就等于$1 - y_i \\cdot (\\boldsymbol{w^Tx_i} + b)$，而对于分类错误的情况，可以发现$y_i$与$\\boldsymbol{w^Tx_i} + b$应该异号，则损失函数的值为$1 - y_i \\cdot (\\boldsymbol{w^Tx_i} + b)$。\n",
    "\n",
    "不同于平方损失，我们可以发现，Hinge Loss在我们的二分类问题中更为合理，相对于平方损失函数“认为”模型对所有样本的预测结果都应该靠近标签，即-1或1而言（即与标签越相似越好），Hingle Loss则“认为”一些已经“显然”被分类正确的样本（$1 - y_i \\cdot (\\boldsymbol{w^Tx_i} + b) < 0$）显然地不应该传播误差，贡献梯度更新权重，即显然被分类正确的样本点不应该参与更新$\\ \\boldsymbol{w}, b$，因为他们的梯度为0。\n",
    "\n",
    "而$0 < 1 - y_i \\cdot (\\boldsymbol{w^Tx_i} + b) < 1$，这些“险些”被分类错误的点，但是仍然被分类正确的点，被Hingle Loss“认为”分类还不够完美，仍然需要反向传播误差，贡献梯度更新权重。\n",
    "\n",
    "而对于分类错误的点，它们的处理方法则与平方损失函数相同，直接反向传播误差贡献梯度更新权重即可。\n",
    "\n",
    "从上我们还可以看出Hingle Loss的另外一个好处，即一些“显然”被分类正确点的不参与梯度计算，从而降低了一些计算的复杂度。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 合页函数的梯度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "而给定Hinge Loss的函数，则$\\boldsymbol{w}$的梯度则有如下形式："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}  \n",
    "\\frac{\\partial loss(\\boldsymbol{w}, b)}{\\partial \\boldsymbol{w}} =\n",
    "\\left\\{  \n",
    "             \\begin{array}\n",
    "              \\\\-y_i \\cdot \\boldsymbol{x_i}, &  y_i \\cdot (\\boldsymbol{w^Tx_i} + b ) < 1\\\\  \n",
    "              0, & y_i \\cdot (\\boldsymbol{w^Tx_i} + b ) > 1   \n",
    "             \\end{array}  \n",
    "\\right.  \n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "而对于$b$的梯度，则有如下形式："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}  \n",
    "\\frac{\\partial loss(\\boldsymbol{w}, b)}{\\partial b} =\n",
    "\\left\\{  \n",
    "             \\begin{array}\n",
    "             \\\\-y_i, &  y_i \\cdot (\\boldsymbol{w^Tx_i} + b ) < 1\\\\  \n",
    "             0, & y_i \\cdot (\\boldsymbol{w^Tx_i} + b ) > 1   \n",
    "             \\end{array}  \n",
    "\\right.  \n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后通过梯度下降更新$\\boldsymbol{w}, b$："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{aligned}\n",
    "\\boldsymbol{w}_{t+1} &= \\boldsymbol{w}_t - \\eta \\frac{\\partial l(\\boldsymbol{w}_t, b_t)}{\\partial \\boldsymbol{w}_t} \\\\\n",
    "b_{t+1} &= b_t - \\eta \\frac{\\partial l(\\boldsymbol{w}_t, b_t)}{\\partial b_t}\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过几行代码简单实现一下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD8CAYAAAB0IB+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJztnX+MHdd1379nlz/EFckVuaSoJblvVxYUVbJjqeLCNutKsCpXv9BYVhMVSreqUAfYSoWAJkCBSlggFRosUDt1AxX1D2xjIa60kO0mdaM6ihXZjRs0QBwvXUmmaquiFHJJiqJo0iIpkaJE7ukfM285b3bmvZk3c+/cO/P9AIP33rx5b+6de+93zpx75oyoKgghhNSfgaoLQAghxA4UfEIIaQgUfEIIaQgUfEIIaQgUfEIIaQgUfEIIaQgUfEIIaQgUfEIIaQgUfEIIaQirqi5AlC1btujExETVxSCEEK/Ys2fPz1V1a6/tnBL8iYkJLCwsVF0MQgjxChE5kGU7unQIIaQhUPAJIaQhUPAJIaQhUPAJIaQhUPAJIaQhUPAJIcQG8/PAxAQwMBC8zs9bL4JTYZmEEFJL5ueB6WngzJng84EDwWcAmJqyVgxa+IQQ0g95LPaZmYti3+bMmWC9RWjhE0JIXvJa7IuLyf+Ttt4QtPAJISQveS32VivfekNQ8AkhJC95LfbZWWBoqHPd0FCw3iIUfEIIyUtei31qCpibA8bHAZHgdW7O6oQtQMEnhJD89GOxT00B+/cDS0vBq2WxByj4hBATOBBzbhRHLPa8MEqHEFIujsScG2dqyrv60MInhPTGw5hzspJSBF9EnhCRt0Rkb2TdYyJyWEReCJe7ytgXIcQybYv9wAFA9aLFnib6tmPO6+4+KpGyLPw/AHBHwvrfU9UbwuXZkvZFCLGJyzHneU9GDacUwVfVvwBwooz/IoQ4hssx53Qf5cK0D/9hEXkpdPlsMrwvQogJXI45dyRlgS+YFPyvALgKwA0AjgD4YtJGIjItIgsisnDs2DGDxSHEMXzxPbscc16m+8iX9iiCqpayAJgAsDfvd9Fl165dSkgteOop1fFxVZHg9amnVn4/NKQaeJ6DZWho5Xau0Ks+VVHWcfStPWIAWNAsOp1lo0x/FBN1AKOR978F4Bu9/oOCT2pBFvEYH+/8vr2Mj1dVan8p42TkeXtkFXwJti2GiDwN4FMAtgA4CuDfhJ9vAKAA9gP456p6pNv/TE5O6sLCQuHyEFIpExNBtEic8fHAvQEEboOksScSuEGIXTxvDxHZo6qTvbYrK0rn11V1VFVXq+pOVf2aqt6vqr+sqh9V1c/0EntCakOWicSsvue6+ZWj9dmyJVhcqJsj6YtNwzttCSmbLOKRZSK0bjHm8focPx4sLtStW3vU6aSbxe9ja6EPn/TE1cnDKFknAHvVxXO/8grS6uNK3ZLaw5PJXNietC1joeA3kDwC7sngU9VyTkwiyaIoUnZp7ZBWH5fr5slJN6vg06VDqiOvy6LKuyrzXtaXEYdeN79ylnK7Vrea3dhFwSf9UYZfM6+A2xh8SfWqypfuyGPxSiOpPlFc9JnX7aSb5TLA1kKXjieU5VrJ67LIenndrzslrV4jI9Vd1vswZ5GHaH1GRoLFZZ+5a+VJAfThE2OU5dfM+z9Jg6990ihDMLJMKpryN9dN2PvFRZ+5B21DwW86JjtpWZOJ/Yhzu15RsS/LGs8yqWhChDyxIq1Qt4lqS1Dwm0IVoWRlWmH9nphMWONp/zky0vvKogguWrWq1Vi2rh4Lx6Hg+4RvPuduJxRbImHCGs9Sr7QriyL1dNGqreqqI4vbrg6UPE4o+L5QB59ze/LNhBimkccaz1OGKm6GcuGKyWSZslB1f7KJgZMpBd8XigwsGz7nftL82hCJqq4yTFjjLqb4tXnV4WJ0lEkMnEwp+L5QZGCZsnLbFEnza1ok2uWri4/ZtRS/Ni38Kq9Uq8DAyZSC7wtFBpZpKzdL2bJcZfhukUVxOaKmTCGxeQVVVXRUVdDCb7DgF020VbX7opd15ooYlomrcdllC4mtCLCq5mOqgj78Bgu+am8xb4usbSHNIiBNiarwARtXHyZcPVmvJqJ35ibdpZv1P21iyVCj4LuOq5OhvcrQT5pfV/ClnFmo4opP1dxkbtHxUPUjJCuOMqLgu4zrk6HxsvosklVfJZmgSuu1qhujsoyHLHNLtqKMLBtqFHyX4WSoHVy4Skoqk0vROHmp6mSTN5d+w6KMKPguw8lQO9i+SurHLWEjy2iZdci6TdnktfBtnpgciDKyKvgAngDwFoC9kXWbATwP4NXwdVOv/yld8F11RzRxMrSKtrB5lVTETZe3DCbvBXBhojNr2XqV01afy2Ph18GHD+BmADfGBP8LAB4J3z8C4PO9/qdUwfet8/aKSvBZ5KMnK5ttkfUqqep7FkxlGc1brypdRVlwdWz0OhlZMNSsu3QATMQE/xUAo+H7UQCv9PqPUgXfp85rMsa5Klzwn2e5SrLpZimzT/YSv37q5WISN1+o+GTkguC/Hfv+F73+o1TB97HzunKSMmnx2m6LXnWx6WYxcUJP+89+8tC40v9IbrwRfADTABYALLRarfKOgI+d14WTlGmLt5+2MOmLdd3N0osyI0SKtr2rc2YNwAXBr9al46N7xIWTlGmLN29bmG5HU24Wlyam89Sr3zq4Pt5qfjJyQfB/NzZp+4Ve/9GYKJ00XBg0Ji3efiavTJ8EXTjmRUg7PmVlTC1aDheuqH1v4wzYjtJ5GsARAB8AOATgNwCMAPh+GJb5fQCbe/1PY+Lwu1H1Sco1i9eGm6vqY16ErBFfpuvlgjsyDZdPRiXBG69If7hmDZmMOfdV5OO4UBeXRdXlk1FJUPBt4MJAM4FL9bIZ2VKX9qsCV640knD5ZFQSFHzTUDTsYSuyxZQAVC14tnD13pK63bWeAAU/ju+ikQVfhKXqcpb9ZKhudXFB8Hphsj1cOblWfde3YSj4UUwMOtf8gj4Ii6ob5cwqQmWIuYuGQRTT7VF1mmLf2qNPKPhRTDSyax3HtfKk4UI5swhDWeLhmmEQx3R72GzvOrRHn1Dwo5hoZBcs1Si+dGRXyllGygXb+XNMYLo9bI4TG+1RtTsyBQp+lCaE9vnSkX1pi7LEwzXDII6NE5JLfatICgyH25KCH8XhhiqNImFxNo9PkQFX9D/zYFo8XKFOY6Of9siTabSfhHSWoODHcXnQlUW/VokrkRTR7/OIkIny10HMs1KHOrQpw1BIE/a0xQG3aVbBl2BbN5icnNSFhYWqi1EvJiaAAwdWrh8fB/bvD94PDARdN44IsLRksnTJZClzFFPln58HZmaAxUWg1QJmZ4Gpqf7/j7hFWj/LS1q/tIiI7FHVyV7bDdgoDKmQxcXe61ut5G3i6+fng0EyMBC8zs+XUcLuZcuyPmv58zI1FQzkpaXglWJfL9L6UxojI8DQUOe6oaHAEPAECn7dySKGs7O9O/L8PDA9HVhEqsHr9LQZ0c8r4FnKXwdsnXCbQlp/ShP2xx8H5uYCi14keJ2b88sQyOL3sbV4kVrBN39nWb5om37+fiYSfWuXvLg6uerzcS8S6OAY4KStAVwddL0oo/PaTlPs0kOqXcDFeH5fx0MUz4Q9DQq+CVwcdLYwXfc6iIdJbOb/yUqTx4NjZBX8evnwTfs4804m1gnTfvKZGeDMmc51Z84E60nxien22BAB7r+/nLmYJo8HX8lyVrC1FLLwbViITbdoTF7+upJywVWK9O+k35bRh5s+HhwCjXPp2LpFnG4HM1A8LpJ2Yu33hJt2bMtwDXE8OEHzBN+khVinyURX60LxCDBxHNLGRhkn1ppMevpO8wTfZFKuughRr0v7qutF8TDTj3tZ+FW3OylMVsE3nlpBRPYDOA3gAoDz2uX230KpFdo3BkUn/oaGit8Ykfc2f5fJciu5j/WqEybSRCSNDZFgP+PjTBlRA1xLrXCLqt6QpUB9MzVl5i64OkUiZCmzj/WqEybSRCSNjSefDASfKSMaRb3CMk3kPulnAKaFh1Z9a3wW0Siaf4YUw1T4K/MCEcC8Dx/A3wD4MYA9AKa7bevkjVd5ffhp2z/0UPVzAa778EkA5zJITuDKpC2A7eHr5QBeBHBz7PtpAAsAFlqtltGD0jd5BmDaBNngYPmTcf3gapQOIaRvsgq+1Xz4IvIYgHdU9d8nfV9JPvyyc56nTbqlUVXOeUJIbXBi0lZELhWRDe33AG4DsNfkPnNhIuVvmg98cDDf9oQQUjKmJ223AfjfIvIigL8G8Ceq+l3D+8yOifwtaZNu09PNyNlOCHEWo4Kvqq+r6vXh8mFVdUvdTIRcpoWHfvnL/j88gRDiNc1+pm2dbqoihDQWJ3z4ztOUR+MRQgiaLvim7s41TdU3cBFCvGRV1QWonKkp9wU+SjwvSjuyCPCrHoQQ6zTbwvcRPhmKENInFPyysJU/p07J3AhpOLaDZujSKYM0N8tf/iXw9a+X635ptZIji3gDFyFOoap4+723sXhysXM5tYiDJw9i8eQiPrHzE/jWvd+yVqZmh2XGiaZZ2Lw5WHfiRO+UC2nhnYODwIULK9cXCfs0lfefEJKLc+fP4dCpQ8tCfvDUwRXi/u4H73b8ZvXAaowNj6E13EJruIXdO3fjwckHC5cla1hmMy38pPw5QKeQHj9+cftelnmaOyVJ7Lttn4X2/svM/0MI6UBVcezMsZXWeUTY33znzRW/u/zSy9EabuHardfitqtuWxb29nL5pZdjQKrzpDfPwk+zkNet6xT5JNIsc5sWPiGkMO++/26HRX7w5EEsnlrs+HzuwrmO36xbtW6FgLeGWxjbGFjsOzfuxLrV6yqpDy38NNKiXOLrkkizzGdnk08iDzzQ6cNvr+eNXYQY48LSBbz5zptdfefHz3YadwLB9g3b0RpuYdfoLnz2ms+uEPbN6zZDRCqqVTk0T/CLuFPSJka7uVk++Um6XwgpkZPvnUx0sbSXw6cP4/zS+Y7fDK8dXvadf3zHxy9a5+G6HRt2YPXg6opqZI/muXTS3C8jI8DZs+mWPidGCTHOBxc+wOHTh7v6zk+dO9Xxm1UDq7Bz484VLpbo5+FLhiuqkR3o0kkjzf3y+OPB+36idAghPVFVHD97vNNvHrpa2uuOnD4CRacRumVoC8Y2juGqTVfhlolbVoj5FeuvwOBAyvMmSAfNE/xeUS4UdUL64uwHZzvCFJN852fPn+34zdrBtcvifftVt6+w0MeGxzC0eihljyQvzXPpEEJys6RLOPrO0a4x58fOHFvxu9H1oxfjzjfGoluGx7B1aKv3E6EuQJcOISQzp8+dThTxtrgfPHkQHyx90PGb9WvWL4v3rtFdHZOg7YnQtavWVlQjkgQFn5Cac37pPN44/UaniMfizt9+7+2O3wzKIHZs3LEc1XLvdfeuCFMcXjtM69wzKPiEeIyq4hfv/eLiBGjEb94W9sOnD2NJlzp+t+mSTWgNtzA+PI6bWjet8J2PbhjFqgHKQ91gixLiMPF8LUn+83i+ljWDa5bF+5Yrb0n0na9fs76iGpEqMS74InIHgMcBDAL4fVX9d6b3SYgPqCreevetrr7zXvla2pEtUf951flaiLsYFXwRGQTwJQB/H8AhAD8SkWdU9f+a3C8hLhDP1xK3zpPytQytHloW8I9u++gKv/nOjTtxyapLKqoR8R3TFv7HAOxT1dcBQES+AeBuABR84jUXli7gyDtHuvrO4/laBmQAo+tHl6Na7vlb96y4O7QO+VqIu5gW/B0ADkY+HwLw8egGIjINYBoAWnyIB3GEeL6WuHWelq8lmuc8fov/9g3bG5GvhbiLacFPMlU67vRS1TkAc0Bw45Xh8hCC9y+8j8OnDnf1nSflaxnbOIax4THcNH5Tx0To2PBYI/K1EP8xLfiHAIxFPu8E8IbhfZIGE8/XkhRznpavpTXcwtUjV+PWK2/tuIGoNdzCtku3MV8L8R7Tgv8jAFeLyJUADgO4D8A/NrxPUmPOfnB2+c7PNN95PF/LJasuWfaTR6NaohOhzNdCmoBRwVfV8yLyMIDnEIRlPqGqL5vcJ/GXeL6WpJjztHwtreEWrt92PX7ll35lhe98y9AWToQSAgtx+Kr6LIBnTe+HuM/pc6dTE28tnlzEoVOHEvO1jA+PY2x4bDlfS3TZsXEH1gyuqahGhPgF77QlpZCUryUu7t3ytewe2708ERr1nzNfCyHlQcEnPWnna+l4aEXswRVvnH5jRb6Wzes2ozXcwsRlE8v5WqLLFeuvYL4WQizC0UYS87XErfNu+VpuvfLWFTcQMV8LIe5Bwa857Xwt3XznR989uuJ32y7dhtZwC9dtvS4xsmXrpVuZr4UQz6Dge05SvpaouPfK13L9tutXJN9ivhZC6gkF32Ha+Vq6+c5PnD3R8ZsBGcD2DdvRGm5hcvtkR76W9rLpkk2cCCWkgVDwKyQpX0v0gc+HTh3CBb3Q8ZvLLrls2U++e+fuFb5z5mshhKRBwTdEO19LN9/56fdPd/ymna+lNdzCzeM3r3gK0djwGDau3VhRjQghvkPB74OkfC1x33lSvpatQ1s78rXEn0LEfC2EEJNQ8BNo52tJ8513y9fSGm7hjqvuWHEDEfO1EEKqpnGCn5SvJe47j+drEQhGN3Tma+mwzjeOMV8LIQ1mfh6YmQEWF4FWC5idBaamqi7VSmon+O+dfw+vnXgt1XeelK9lw5oNy+I9uX1yhe+c+VoIIWnMzwPT08CZM8HnAweCz4B7oi+q7jxzZHJyUhcWFgr9x3f3fRd3zt+5/HlQBrFz486LLpaNrRVhinxwBSGkXyYmApGPMz4O7N9vpwwiskdVJ3ttVzsLf9foLjz9q08vi/no+lFOhBJCjLG4mG99ldRO8LdeuhX3feS+qotBCGkIrVayhe/iI7qZDIUQQgowOwsMxQLwhoaC9a5BwSeEVML8fOD/HhgIXufnqy5Rf0xNAXNzgc9eJHidm3NvwhaooUuHEOI+PkW2ZGFqyo9y08InhFhnZuai2Lc5cyZYb5PoVcaWLcHi+xVHN2jhE0Ks40JkS/wq4/jxi9/5fsWRhjELX0QeE5HDIvJCuNxlal+EEL9Ii2ApGtmSZ14g6SojShVXHKYx7dL5PVW9IVyeNbwvQognmIhsaVvsBw4Aqhet9DTRz3I14WIsfRHowyeEWMdEZEveeYEsVxMuxtIXwbTgPywiL4nIEyKyKWkDEZkWkQURWTh27FjSJoSQGjI1FaQeWFoKXov6yvPOCyRdZURpX3HUJXwUKCj4IvI9EdmbsNwN4CsArgJwA4AjAL6Y9B+qOqeqk6o6uXXr1iLFIYQYwgfRyzsvEL/KGBkJlugVB5DPTeQ6VpKnicgEgO+o6ke6bVdG8jRCSLnEo1mAwPp17eYiE+V0ITFaFrImTzMZpTMa+XgPgL2m9kUIMYcrMfO9MDEv4EL4aJmY9OF/QUR+IiIvAbgFwG8Z3BchxBC2Ra+I+6jseQFT4aNVYUzwVfV+Vf1lVf2oqn5GVY+Y2hchTaCqu0Jtil7e0ErTJE3sigTl8vHOXIZlEuIBcSE8fjxYbIiizWyQrriP2ifX++8H1q0LJnOBQOzb054226AsKPjESdKsWV+sqrKjWqq8KzQpmmXdukAMy24DG+6jXm2TdHI9ezaod7cYFxfnNVagqs4su3btUkKeekp1aEg1GF7dl6GhYPsqyzo+rioSvD71VHL5i5ZTpPexECmrVumYqFuU8fHkuo2Pl/P/WcqfVoYsi402SALAgmbQ2MpFPrpQ8Ilq/gFXlhjkJU08RkbKL2eWY2LjOLggyEm/iZ90i5Q/y8nVtb5IwSfeknfAVWVV5T0xFSlnr6ue9jHrJXhFSWubMtsgj4DnPUFkKX9au46MdG+DpP3mqUsRKPjEW4pa+LYGWd4TU1HrL1qvkZGLVxLxcph0c/Vj4Ztsj7zlybJ9t5NIUhuk1cu0+ysKBZ94SxEfvs1BlscStF0GU66FrMe3LYymT0h5rzjylr/IScpm21DwidekWVK9rCqbgyyrJVjFVYZJN1evumU5YZfVHq5dcUSx2TYU/JqR51KyydgWQFvikYZtC79ImUy0h80rurzQwqfg56LbZbGLHdwFXBRAk7goeFnmNrK2R5YTatUn3TTow6fgZyaPH7vOgpYXFwXQNK4JXi8LP2t71KEtXYvSsZIeOStMj3yRtLSsaYgECaOayvx8cJfj4iKweXOw7sSJIN/L7KxbaXzrTlKa4nZKgvHx7O3hS2piF6g8PTIpRt5byX3N3lcGabfCP/lkORkTSTpJaQqS0hQ/+WTQNnnao26piV2Agp9C1U/4ySPg7ex9LueWMYkrCbeaRrfMlmWkKa5bamIXoOAn4EKK1rS0rMDFR7G117W9ci5n7DN5AnUh4ZZPlFUX0ydam1k6G0MWR7+txZVJWxuRHmVEH/gSkdKEhFu+UGZdXEuz4BNl1wuM0ukf0x25rEHnS8y574Lsy4k1C2XWpU7HpUz6uTGtaH+l4BfAdEcu6/9duau0F75bglXczdqm7HqVWRfTmS19pEj65SLjloJfANMWY1mDzoW8MVk6qe+WYFXlN9G+ZdfFZGZLHymSfrmIAUHBL4hLGf6qKmeUIp3U94FeVflNnGiqbAvfT/xZKJJ+2XkLH8C9AF4GsARgMvbdowD2AXgFwO1Z/q+o4PtyueijABbtpL60TRpVlN+UKylel4cect9o8IWi6Zf7xZbgXwvgGgA/iAo+gOsAvAhgLYArAbwGYLDX/xURfN9ENE1AXBXGpONr66EbTcXGSdYXt6AvZM2gWnYCRKsunQTBfxTAo5HPzwHY3et/igh+HTqT6yetbsncXCpnXSjSH7L+1peJf59IOtGarnvVgv+fAPyTyOevAfi1lN9OA1gAsNBqtfqucB0uF22ftFwNsyQXMd1GvoT2+o7pMVOa4AP4HoC9CcvdkW3igv+lBMH/1V77arqFbzN8sYiVXoeTa1W4Nsleh3HjA6bHTFbB75laQVU/raofSVj+uMvPDgEYi3zeCeCNXvsqQh1uwy6aO6TXLfPRlBFA0OWiZL0tnjlO+sNmyo6sbVSHceMDzoyZLGeFXgtWWvgfRuek7eswPGmr6v/lommfba885Vktjqb4YsvGVX+5L+PGl3ImUQsfPoB7EFjz5wAcBfBc5LsZBNE5rwC4M8v/uRSHXxUmfbamnkSU55mzTYb+8v6pg5Fhsj2yCj4fgFITBgZWumiAzgej9HqoytBQkMc8TyrbpIddFP3PusIHevQPj113+ACUhpHFR9gt5fL4eH/CnJQiN4rJvPS+pSymv7x/+DCUcqDg14Q0MY8+GKWsJxFFyTLgTAxKF55ZkJek48+rn2w4M+npO1n8PrYWWz58k3e8VUkVN0ZlmQjuNi/Qb1tUGU7o213S/eJSfergwzcJmDwtmaSOU7dOVHU0SHxCsr3v+HEt0hZV3QuQJjwPPVQvQXJRYF06AbkGBT+FohapD1QZDdK20rNcZRRpi6os/LT9Dg7Wqy/5ctc3CaDgp5AlNNH3O0b7GaxlD7iywkTT2iKrBVp2vbKUuQ59yabR4OLVhG9Q8FNogoWfdwCZGHBF8oKXdZIyUa8yLXyX55JsWvhM71AcCn4KTfDhq+azbE0MuH7zgtsQ5yL1KsuH73o/tGl1+/4ITBeg4HfBZcuqCkwMuH5cLmW3ha0HiPQTpePqlWYVY8OUhd+kdN4UfI+o2vowPeB8qlfV2SyrtHCr8qX3s99+3HkunFBNQcF3hCr8zP2UseoymKBbvZLaxeZxsGnhZ62Xi/c2pG1rK1GgL1DwHaBIx7RtfdT1ZqI8wt4OJ7XRFkXuX8hL1j7my3MObCcK9AEKvgMU6ZguTFjV1fLPYv3ZELx+71/Ii62HoVTtDssTAVaHfhyFgt8HtmK2s3RM25fzSbhy9VE2eWPpbda3qogp1WJ9xQV3WK8IsDKvmlyDgh/DpZjtXh2zzIFSREB8ucTPS9oxGRmp/oqmyoip9rb9GD02jYOqbrpzGQp+hKp86S50zCICUlcLv5v1V/XDXEy5VUyLHx/uUi0U/AhV+tKrjvkvIiB18+Hn8ZlXlQzNF7dKnLoaB75AwY9gw5fuavhl0f3WxZLKG5lTZTI0H9wqcepmHPgGBT9Cmb70fuO3qx6MdRDtIuSNzElb+rnicyl6xSTsZ9VBwY9Qli+9SPx21YOx6eSNzCnLwncteoXUEyuCD+BeAC8DWAIwGVk/AeAsgBfC5atZ/q/KKJ0sFInf5mC8SBWWYN7InLJ8+C5Gr5D6YUvwrwVwDYAfJAj+3rz/V3Ucfi+KxG9zMAa4OJdhMrKF0SvEBlZdOk0R/KLx274MRpPlbNpcBq/siA1cEPx3AfwfAP8LwE1dfjsNYAHAQqvVMn1cCtGPlegbpi3wps1l8MqO2KA0wQfwPQB7E5a7I9vEBX8tgJHw/S4ABwFs7LUvVy38qmPpbWLaIm2ixVsXY4C4S1bBX4UeqOqne22T8JtzAM6F7/eIyGsAfim05L1ifh6YngbOnAk+Hz8ODA0BTz4JTE1VWzYTLC7mW5+X2dnO4wkEx3N2tpz/d5GpqXr2FeIfAyb+VES2ishg+P5DAK4G8LqJfZlmZqZTnIDg88xMNeUxTauVb31epqaAuTlgfBwQCV7n5iiIhNigkOCLyD0icgjAbgB/IiLPhV/dDOAlEXkRwB8CeFBVTxQrajWYtnhdY3Y2sLijlG2BT00B+/cDS0vBK8WeEDsUEnxV/baq7lTVtaq6TVVvD9f/kap+WFWvV9UbVfV/lFNc+/Rj8c7PAxMTwMBA8Do/b6JkZqAFTkh9kcDf7waTk5O6sOCWmz/uwwcCi3duLng/MxNY+63WRSs4bXuKJiHEBCKyR1Une25Hwe/N/Hx2YV+3LpjYjTM+HrgvCCGkbLIKfs8oHZIcZTExkTyZG1/Xpq4+f0KIPxiJ0nEFk770vAJeVpQLIYT0S20Fv+17P3AguLXnwIHgc1minybgIyPmo1wIIaQfaiv4puPn08IXH3+cUS6EEDeprQ/fdPx8W8Djk7nt9RRuZJEiAAAGfUlEQVR4Qohr1FbwW63AjZO0vix4yzwhxCdq69KxcccoIYT4RG0Fn3eMEkJIJ7V16QB0uRBCSJTaWviEEEI6oeATQkhDoOATQkhDoOATQkhDoOATQkhDoOATQkhDcCofvogcA5Bwf2xutgD4eQn/UzYulotlyo6L5WKZsuNiucoq07iqbu21kVOCXxYispDlYQC2cbFcLFN2XCwXy5QdF8tlu0x06RBCSEOg4BNCSEOoq+DPVV2AFFwsF8uUHRfLxTJlx8VyWS1TLX34hBBCVlJXC58QQkgMbwVfRO4VkZdFZElEJmPfPSoi+0TkFRG5PeX3V4rID0XkVRH5poisMVDGb4rIC+GyX0ReSNluv4j8JNxuoexyxPb1mIgcjpTrrpTt7giP3z4RecRwmX5XRH4mIi+JyLdF5LKU7Ywfp171FpG1YbvuC/vPhIlyxPY5JiJ/LiI/Dfv8v0zY5lMicjLSrr9toVxd20MC/mN4rF4SkRsNl+eaSP1fEJFTIvKbsW2sHCcReUJE3hKRvZF1m0Xk+VBznheRTSm/fSDc5lUReaDUgqmqlwuAawFcA+AHACYj668D8CKAtQCuBPAagMGE338LwH3h+68CeMhweb8I4LdTvtsPYIul4/YYgH/VY5vB8Lh9CMCa8HheZ7BMtwFYFb7/PIDPV3GcstQbwL8A8NXw/X0AvmmhzUYB3Bi+3wDg/yWU61MAvmOjD2VtDwB3AfhTAALgEwB+aLFsgwDeRBCfbv04AbgZwI0A9kbWfQHAI+H7R5L6OYDNAF4PXzeF7zeVVS5vLXxV/amqvpLw1d0AvqGq51T1bwDsA/Cx6AYiIgD+HoA/DFd9HcBnTZU13N8/AvC0qX2UzMcA7FPV11X1fQDfQHBcjaCqf6aq58OPfwVgp6l99SBLve9G0F+AoP/cGravMVT1iKr+OHx/GsBPAewwuc+SuBvAf9GAvwJwmYiMWtr3rQBeU9UybuTMjar+BYATsdXRvpOmObcDeF5VT6jqLwA8D+COssrlreB3YQeAg5HPh7BycIwAeDsiMknblMlNAI6q6qsp3yuAPxORPSIybbAcbR4OL7GfSLmszHIMTfE5BFZhEqaPU5Z6L28T9p+TCPqTFUIX0t8G8MOEr3eLyIsi8qci8mELxenVHlX2o/uQbmDZPk5ttqnqESA4iQO4PGEbo8fM6Sdeicj3AFyR8NWMqv5x2s8S1sVDkbJsk4mMZfx1dLfuP6mqb4jI5QCeF5GfhRZCX3QrE4CvAPgdBPX9HQSups/F/yLht4XCubIcJxGZAXAewHzK35R6nJKKmbDOWN/Ji4isB/BHAH5TVU/Fvv4xAvfFO+G8zH8HcLXhIvVqj0qOVTgf9xkAjyZ8XcVxyoPRY+a04Kvqp/v42SEAY5HPOwG8Edvm5wguL1eFVlrSNqWUUURWAfiHAHZ1+Y83wte3ROTbCFwLfQtZ1uMmIv8ZwHcSvspyDEstUzg59Q8A3KqhMzPhP0o9TglkqXd7m0Nh2w5j5aV76YjIagRiP6+q/y3+ffQEoKrPisiXRWSLqhrLHZOhPUrvRxm5E8CPVfVo/IsqjlOEoyIyqqpHQtfWWwnbHEIwz9BmJ4J5ylKoo0vnGQD3hdEUVyI4e/91dINQUP4cwK+Fqx4AkHbFUJRPA/iZqh5K+lJELhWRDe33CCYw9yZtWwYxH+o9Kfv6EYCrJYhkWoPg8vgZg2W6A8C/BvAZVT2Tso2N45Sl3s8g6C9A0H/+Z9oJqizCOYKvAfipqv6HlG2uaM8liMjHEIzt4wbLlKU9ngHwT8NonU8AONl2aRgm9Yra9nGKEe07aZrzHIDbRGRT6G69LVxXDqZnq00tCMTqEIBzAI4CeC7y3QyCaItXANwZWf8sgO3h+w8hOBHsA/BfAaw1VM4/APBgbN12AM9GyvFiuLyMwMVh8rg9CeAnAF5C0AFH42UKP9+FIBrkNQtl2ofAb/lCuHw1XiZbxymp3gD+LYKTEQBcEvaXfWH/+ZCFvv53EVzWvxQ5RncBeLDdtwA8HB6XFxFMfP8dw2VKbI9YmQTAl8Jj+RNEoukMlmsIgYAPR9ZZP04ITjhHAHwQ6tRvIJjr+T6AV8PXzeG2kwB+P/Lbz4X9ax+Af1ZmuXinLSGENIQ6unQIIYQkQMEnhJCGQMEnhJCGQMEnhJCGQMEnhJCGQMEnhJCGQMEnhJCGQMEnhJCG8P8BvGdqrIVoJ9cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "for step in range(training_steps):\n",
    "    hinge_judge_term = y_label * (w1 * x1 + w2 * x2 + b)\n",
    "\n",
    "    mask_no_grad = hinge_judge_term > 1\n",
    "\n",
    "    grad_before_mean_w1 = -y_label * x1\n",
    "    grad_before_mean_w1[mask_no_grad] = 0\n",
    "    grad_w1 = np.mean(grad_before_mean_w1)\n",
    "\n",
    "    grad_before_mean_w2 = -y_label * x2\n",
    "    grad_before_mean_w2[mask_no_grad] = 0\n",
    "    grad_w2 = np.mean(grad_before_mean_w2)\n",
    "\n",
    "    grad_before_mean_b = -y_label * 1\n",
    "    grad_before_mean_b[mask_no_grad] = 0\n",
    "    grad_b = np.mean(grad_before_mean_b)\n",
    "\n",
    "    w1 -= eta * grad_w1\n",
    "    w2 -= eta * grad_w2\n",
    "    b -= eta * grad_b\n",
    "\n",
    "plt.scatter(x1_positive, x2_positive, c='r')\n",
    "plt.scatter(x1_negative, x2_negative, c='b')\n",
    "plt.plot(x1, -(w1 * x1 + b) / w2, c='g')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 支持向量机（Support Vector Machine）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "较为直观地，我们可以发现，随着梯度下降迭代的进行，渐渐地诸多样本点将不再反向传播误差，贡献梯度用来更新权重，因为他们已经“显然”被分类正确了，即$y_i \\cdot (\\boldsymbol{w^Tx_i} + b ) > 1 $，而那些直至迭代结束还在贡献梯度的样本点，将会被称之为支持向量，这也是支持向量机的来历。\n",
    "\n",
    "其实到这里支持向量机就介绍完了。\n",
    "\n",
    "有些观众朋友们可能就会问了，好像学过的支持向量机不是这么推导的呀，似乎应该是先写出最大化间隔，然后放缩，然后引出拉格朗日乘子法解等式约束的最值问题，然后引出KKT条件和不等式约束，然后构建广义拉格朗日方程使之与原始问题同解，然后领各种偏微分为0，解其对偶问题呢？\n",
    "\n",
    "是这样的没错，上面一系列复杂的推导似乎是凸优化（虽然我没系统的学过）的一部分，这些内容将会在下一节预习，所以在这里先简要的说一下结论：\n",
    "\n",
    "即如果我们为上面的Hinge Loss加上一个L2正则化项："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "loss(\\boldsymbol{w}, b) = \\sum^{N}_{i=0} \\max(0, 1 - y_i \\cdot (\\boldsymbol{w^Tx_i} + b)) + \\lambda \\Vert \\boldsymbol{w} \\Vert ^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "那么，通过这两种方法得到的结果本质上是相同的，为什么是相同的这里打算在下一节一起预习，所以这里先暂且结束了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这次主要从梯度下降的线性回归入手，从梯度下降和最小化Hinge Loss的角度预习了支持向量机，下一节将从传统凸优化的角度预习支持向量机。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}