Logistic-Regression-Classifier.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scipy.io\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import random\n",
    "import os\n",
    "\n",
    "from classifier import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following document will explain the model used for logistic regression, how the code was implemented, and the results on all the given datasets.\n",
    "\n",
    "By running \"logisticregression.py\" with the next argument as the desired dataset, this code can be run in the terminal for any specified dataset. The specific command would be:\n",
    "python logisticregression.py 'classify\\_d5\\_k3\\_saved2.mat'\n",
    "\n",
    "The example dataset shown can be replaced with any of the given datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Description of the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Logisitic regression is a discriminative graphical model, meaning that it directly maximizes $p(y|x)$. Logisitic regression is a modification of linear regression, by subsitution the hypothesis function of a linear mapping, to the sigmoid function. The use of the sigmoid function for hypothesis results in outputting values between 0 and 1, which are then translated to either 0 or 1 based on a threshold (of 0.5) for the binary classification task. The hypothesis with the sigmoid function is:\n",
    "$$p(y=1|x) = h_{\\theta} = \\frac{1}{1+\\exp{(-\\theta^Tx)}}$$\n",
    "$$p(y=0|x) = 1- h_{\\theta}$$\n",
    "Note in our case $y=1$ corresponds to class 2, and $y=0$ corresponds to class 1.\n",
    "\n",
    "To find the parameters $\\theta$, the maximum likelihod estimation is used. Training a model to find the parameters  can be done by maximizing the conditional log likelihood of the training data.\n",
    "\n",
    "The logistic regression cost function is:\n",
    "$$Cost(h_{\\theta}(x), y) = y*(-\\log(h_{\\theta}(x))) + (1-y)*(-\\log(1-h_{\\theta}(x)))$$\n",
    "\n",
    "For a training set with $m$ points, the likelihood of the parameters is:\n",
    "$$L(\\theta)= p(Y|X; \\theta)$$ \n",
    "Where, the $L$ represents the likelihood, the $Y$ and $X$ represent the class and data for all the points in the training set. Using the chain rules for probabilities and the previously given equation for $p(y^i|x^i;\\theta)$, this equation can be written as:\n",
    "$$L(\\theta)= \\prod_{i=1}^m (h_{\\theta}(x^i)^{y^i})(1-h_{\\theta}(x^i))^{1-y^i}$$ \n",
    "\n",
    "This can be transformed into the log likelihood:\n",
    "$$l(\\theta) = \\log L(\\theta)= \\sum_{i=1}^m y^i \\log (h_{\\theta}(x^i)) + (1-y^i) \\log(1-h_{\\theta}(x^i))$$\n",
    "\n",
    "The objective will be to maximize the log likelihood, and this can be done using gradient descent.\n",
    "\n",
    "In order to optimize the parameters, this can be done iteratively using gradient descent, to update the parameters at each step up the gradient towards the maximum. The update equation is:\n",
    "\n",
    "$$\\theta := \\theta + \\alpha \\nabla_{\\theta} l(\\theta)$$\n",
    "\n",
    "By taking gradient of the log likelihood function, this equation can be simplified to:\n",
    "$$\\theta := \\theta + \\alpha (y^i-h_{\\theta}(x^i))x_j^i$$\n",
    "\n",
    "This is the equation that was implemented in the code for gradient descent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training Algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section will show step by step the code that was used to execute the training of the model. This code was also consolidated into one function \"logistic_regression\" in the submitted python file. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First the files are loaded from the matlab file an formatted into the format required for use in python. The output of the format data function will be the training features and the class IDs in a list. In this code, $0$ was used to represent class 1, and $1$ was used to represent class 2. The features list, is a list of the feature vectors. Note the format data function also adds an extra column as required to enable setting parameters for the intercept, so that the parameters found are not restricted to passing through the origin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# load file\n",
    "mat = scipy.io.loadmat('classify_d4_k3_saved1.mat')\n",
    "features, classIDs = format_data(mat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Logistic Regression Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To begin logisitic regression, first set the required parameters, of the learning rate and the max number of steps for gradient descent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# logistic regression parameters\n",
    "\n",
    "num_steps = 500000\n",
    "learning_rate = 0.0001"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient descent for the parameters, $\\theta$, optimization will be used to update the parameters with the following equation: \n",
    "\n",
    "$$\\theta := \\theta + \\alpha (y^i - h_{\\theta}(x^i))x_j^i$$\n",
    "\n",
    "Where $\\alpha$ is the learning rate set in the previous section, the $x$ are the points in the training set, the $y$ is the class, and $h$ is the prediction of the class. The predictions $h$ are calculated using the sigmoid function.\n",
    "\n",
    "This equation will be executed in the following code. The loglikelihoods will be printed out as it proceeds to track the progress."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-1371.4056583\n",
      "-815.68939063\n",
      "-815.662721761\n",
      "-815.662696567\n",
      "Gradient is close to 0, reached convergence.\n",
      "Final Loglikelihood:\n",
      "-815.662696545\n"
     ]
    }
   ],
   "source": [
    "# initialize parameters:\n",
    "params = np.ones(features.shape[1])\n",
    "\n",
    "# gradient descent\n",
    "for step in range(0, num_steps):\n",
    "    scores = np.dot(features, params)\n",
    "    predictions = sigmoid(scores)\n",
    "\n",
    "    # Update weights with log likelihood gradient\n",
    "    errors = classIDs - predictions\n",
    "        \n",
    "    gradient = np.dot(features.T, errors)\n",
    "    params += learning_rate * gradient\n",
    "    \n",
    "    if np.linalg.norm(gradient) < 0.0001:\n",
    "        print(\"Gradient is close to 0, reached convergence.\")\n",
    "        print(\"Final Loglikelihood:\")\n",
    "        print(log_likelihood(features, classIDs, params))\n",
    "        break\n",
    "    \n",
    "    # Print log-likelihoods\n",
    "    if step % 10000 == 0:\n",
    "        print(log_likelihood(features, classIDs, params))\n",
    "        \n",
    "    \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following section will show step by step how the code generates the results from the parameters calculated for the model. The accuracy will be calculated and compared to the accruacy from a pre-built function in python sklearn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Format Test Data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First the data is formatted for the testing. This is done the same way as previously, however train is set to false, which means it will take data points not yet used in the training set. The train/test split is 80/20. The test data is taken as the last 20% of the data, and since this data was randomly generated, this method is acceptable (ie. it does not need to be shuffled before hand because it was already generated randomly)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_features, test_classIDs = format_data(mat, train = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Results on Test Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results from one dataset is shown below. The final class predictions is obtained from the sigmoid. The cut-off threshold to associate class 0 or 1 with a given point is set at $0.5$, and this is accomplished by the round function. \n",
    "\n",
    "The data is also plotted to visualize the success of the classification. The left half of the graph should be class 0 and the right half of the graph should be class 1. The accuracy is calculated as the number of correctly classified points after the rounding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnX+MZtdZ379n3p0JHePG5N0NimzvjA2uitNWcTJFRKlQ\ni6PWWZDdSm5lOqRbRDtiUlAQVK1Xg1AaaaWWqpC24KAlONnyvpBQKCWKHLk0IUJCSsK4iRM7qc3G\n2dkYR3idCho0InF2D3/ce7N37txzznPOPT/vfT7Sq5n3fe9777nnx/c85znPOVdIKcEwDMOMi5XU\nCWAYhmH8w+LOMAwzQljcGYZhRgiLO8MwzAhhcWcYhhkhLO4MwzAjhMWdYRhmhLC4MwzDjBAWd4Zh\nmBFyItWFT548KTc3N1NdnmEYpkieeOKJl6SUp0zHJRP3zc1N7O/vp7o8wzBMkQghDijHsVuGYRhm\nhLC4MwzDjBAWd4ZhmBHC4s4wDDNCWNwZhmFGCIs7w+TGcglsbgIrK9Xf5TJ1ipgCYXFnmJxYLoGd\nHeDgAJCy+ruz41fgufPwQ+b5yOLOMDr6GnDIRr23BxweHv3s8LD63AcxOo+c8VV21HxsX+/kyeoV\nqzOQUiZ5veENb5AMkzWLhZTr61JWzbd6ra5KubZ29DNAyvm8Or7vHBsbUgpR/d3dPfq++xshjp8b\nqD7vnqvveiY2NvrPv7Fhf66h+Lgf2+t1y3N9PVw+9l3Pw7UB7EuCxrK4M4wKVQNWvYSoxLthd1ct\n1qoGrrrmfO5HmHSdRx8uAmzq0HZ3q/vxJHZkfHZspnxcLKSczcx1xuHaLO4M40JbmGyEvWthLxb0\nc5isvfX1fjEEKgGxEV4bgXOxdE3WagCxM9KUqa7MbNHlo00eOFybxZ1hbBkqTO0Gbmv1d9PRtZYp\nHQXF8u0bTayvH7Wu53N1Z2ISYNv79iG0Oihl6tKh6Do+mzxgy51hIkBplCqf+1BBM4myTmxV1mOX\nPkESQsp777Xr1HQC7DricRE7isvIVKZDXEEq9xP1XtnnzjCR0AlTW0AWC73YugicTtQWi6pTsTlf\nn3CohIfiG6amdYjlvrZGFzuTy4hiQfucxLUd9akm4AmwuDOMLbb+6JUVO/Eyib5tumxFeKhVreo0\nuvni6tqaz2+cw9Uip/q8ffv2TWXU5L2HDoXFnWFssZlAdBFcnTWpc80MmdylpJlquVOEqTuquemm\n6tUWcF16qWWgi1aJKLSkEcJs5jUKiMWdYVyghv7ZCm5jKeomR1WRL74s993d/uNMPneqb5hiMZsi\nf6j34ponfZPJQ3zulFGK50liFneGCYkptE4nIBQR6vqPbV0dNj73pjPpRsvYLi6iCm5fzD5VIHd3\n7ecITJ2H6ySuzYjHIyzuDBMSk+Dqhv5UEezGv+t+N5+bRxy2C5hsoY5muqttqYt9VCMPamdn6jhM\n+OpkB8LizjChMQmuymIbMpwfsoTe5HNv/vre2sCUL9RVvK4WO6AvK6plTb0/24VllrC4M0wsXCxi\naqie7re2AmJjebpYnFSfe/e8pgVTzfGuwt6e7xiyhQNlZNI3Me55Dx0Wd4aJxRCL0CV6Zgg2Kyhd\nLNCu774bKdMnfH0x/H0x7yrLXQh1p9IVbxuh7R5LXUjWPYevzcpqWNwZJhZDGrDOGoyZXt+WPDVP\nVB3NyspxAVb53Hd3j3ZaFPeSSeT70k9ZnUyN6hkwycrizjAxcR16B2j8xjTaujUoaaJauS4Lq9qd\nQjtaZjY7ugsnNW27u+aOR5VPzcR1M2IwdV4BJrFZ3BmmBAIM28nXsX3ptgV23QUx9CSlaj8dU8dD\nGVENXUnrCIs7w5SC5wm3Xmwsdtt48CG7IPpyEany0CZt7Y5H9zvKaEF3f+xzZ5iR4VPEbc9l89AQ\nW0EaIsrde3FZGKRLr81K4u45TU/FosLRMgwzYnxacC7nMsXjDwnf00Wx2IqayxoAnetDF41kyj+b\nEUhEWNwZJiWuE4wUXPy4IX37OhF0gWLJU/zjuo3ITM+yldK8xURfekO51VqwuDNMKoZMMFJwjcAI\nJUIhI34onZLp+kMWfZkmX207TQ9lwOLOMKkYOsGYIAJjEKEjflxi0n09Yam9CKv9ahZk2ZSFp3Sy\nuDNMKqiTeH2rJymNXxXe143iiOkuCH0tisD7uL5q8VKfyNtuRuapU2ZxZ5hUmBbAqATIpvGrHnTt\nGvHSENl/TCLWWgAp1WWg8v3bhI16WtDE4s4wqXAVI5vGb+oIfE66UiYfQ+Jq8bp0VC5PvRr69C62\n3BmmIFyExabxmzoCFytxSNhgSFx33XTpYG0t96ZsKWXNPneGmSg2jT+E5W5jtXp+LqgWl3sZYu2r\nRi8+XEMcLcMwE8WXFeh7oZPKcrZZiu+Ky70M8W+ryiCT+QgWd4YZO74jSGw22Wp/H0PkbO8lt3BR\nj1DFXVTH6hFC3AfgPwOYAXiPlPLfd74/DeAigFvqYx6WUj6mO+fW1pbc3983XpthmIgsl8DeHnDl\nCnD6NHDmDHDxInB4qP7NxgZw+XK0JJJYLoGdnaPpXl8HLlwAtrfTpcsDQognpJRbpuNWCCeaAfhF\nAG8BcDeAHxRC3N057KcB/IaU8h4ADwF4xD7JDMMkZ3u7Eurr16u/jzxSCeJspv7NlSuxUkdne7tK\n98YGIET1dwTCbsMJwjHfDeCSlPI5ABBCvB/AAwA+1zpGAvir9f+vBPCCz0QyDJOQRhDf+tbKudHl\n9Om46aGyvT0pMe9itNwB3ArgS633z9eftXkHgB8SQjwP4DEAP953IiHEjhBiXwixf/XqVYfkMsyI\nWS6BzU1gZaX6u1ymTtENtreBH/3R45+vrgLnz/u7Ts55UBgUcRc9n3W77x8E8D4p5W0AzgD4VSHE\nsXNLKS9IKbeklFunTp2yTy3D5IgPQWp8xAcHlXV8cFC9z0nc3vQmYG3t6GeiTx4cKSEPSsI04wrg\njQAeb70/B+Bc55inAdzeev8cgFfrzsvRMkzWRF6YUkR0R+g0lpAHGQBf0TJCiBMAngVwL4A/BvCH\nAP6plPLp1jEfBvABKeX7hBDfBeAjAG6VmpNztAyTLTaRFpublYXZxTaCZGWl358tRDW5mQOh01hC\nHmSAt2gZKeU3APwYgMcBfB5VVMzTQoh3CiHurw/7KQD/UgjxJIBfB/DPdcLOMFmzt3c89O/wsPq8\niypSxDaCRDUpmdNkZeg0+jx/SN99KfMCFPM+xIvdMowVMVcH+tzAi0rqPdFzSaOvJf6h0hlzh0oF\n4BWqzGjw/fxRnw/DiJ02F0pIo8/z25ZfYStfWdxzIJO9KIrHtkHp9gZxfRhG4EenBSWUIOV639SR\nV+w9azzB4p6aDIZvo8GmQenyPaRFFwufe5QPEaQc67eujPvKOeZukx4Zv7jn2vgaMqgE2UMtQ5u8\n1B2bgdU1CN97lA+pi7nV7768MeVTzH3iPTJucc8gg43EEJLcOzhd+mzK0OZYXb7nJki26B4koasD\nIdqLbrfIFPXRZLHbzq2Y6m7CdjducS+hkbqk0abS5N7BmdLny4/e/d7UwHPOMxOUh2noOkifgkTZ\n+z1m3vq0wn09mCMQ4xZ3n1ZxLhEKtsfn3sGZ0ue7DKlD8txHOzqoD9OIUQcWC1pnE6s+uraHvvqQ\nedsat7iHjC1uKqyPhm8jJLb3lLv/2JQ+nw3IZUheIqZOLHYdoHQ0sdLic1SWedsat7jb+mtVAmuy\nhLrnDGn12VaozK0LY/om1Bi90q6Duoc2x4AykohZH/vap0ubzbxtjVvcpaQVmklAbIaVof21FDFs\n32+fXzDWMy0pUPKrPQRuhMql08y8MQYj9RyCS4RKTFyNwPlcyrW1vO6lxfjFnYKp0VMsjxBuhD50\nFVH13b33Hu+gMqqEXjpg6nUyeTJ9dFKnuSuK83k++Udts331Z3U1r3tpweIupXm4TvFhhpgAVKFq\nqLoQuNItVp/zJ0NELrUVzPiH2mZtR82J6wSLu5Q04Wi7BnRWcMqhP8V9FKrDUUENTTQ1iFid5nx+\n49zz+fH0TNW1M2aoZaqrgxl2+tMV9yG+M1+LbnyTm+VuygubvFLd23zux1paLKohdvf8a2tHzzml\nSdkumVmm3qDWQ10nkGGnP01xD+07S9UIclts4TqX0dcgVGXma0LLFCZJvaexkqFl6pWh8z4ZdvrT\nFPcxN1BVJU3R4ZgqvG2D6N5D24UytBxNy+TbaRizyKkYc5uxwXa+iy33yOKeYS9rTQlDZJ+Wex8+\ny5FquUtZRt5TsLmPMbSZkGTY6U9T3GP0siEFYGhFoqYtdGTJ0PvwWY4Un/tYRF1K+7zP0DLNjszq\nxzTFPXQvm3ohk4+0xYoJH9IgfOezLlomQ8tsELZ1aGz3TyEzsbZlmuIupZ+CS+V/GzJEpqatFEst\nVgMsJT+ouO6OWLDYWTGCzmzc4p7KNRLaPzlEaKhp000wTpGx+Zx1YbNTEG8TI+jMqeK+gtJYLoGd\nHeDgoCqWg4Pq/XI57Jybm8DKCnD2LHB4ePT7w0Ngbw84fbr/9ysr1Wtzc1g6zp8H1tePfra+Xn1u\nQpW27ueq44QYlvZSoeZbKfTVIQC4ds1feymZK1fsPi8ZSg8Q4uVsufvueSlbEDSvvrjy7mvoEM91\nVGLjc1dZqwVZL94YwTD9GO065GOh25jcNhOy3MsTd9/DaF2oXJ9w923W5VJRfESsdH9PPacq3aW6\nIoYyJvHqMrS9UDq/kvJvBJ35eMXdd89ru2/Lyor5GFPD8RHymEuoIZM3Q8va9PsSxbKkzqiH8Yq7\n78pkY7lTX6aGE7rBmSixQTJuDC3rmE/UYkiMV9yl9Nvzqiq/agm86UVpOEOHyj5cU77yMLfzpL5G\njgy5b5N4jy3aqADGLe6+UfmvbYV9NqM1nNSWuy98LogKPZLIcbSSa2fTTpdpZ9Vc6mIsMigzFncf\n2FrvPiep2sf2dTy+V3C6VFhfDTuGQNhcI9YoIrfORpUu3c6qud5HCDK5VxZ3H9i6bHyHl+kqk093\niGuF9TUkjzG0N+0O6aPjtCmTXC1el3QNCd/NceSiIpMyY3G3RVXRYljOKnxUpm76d3ePvh/SUY3B\ncvfVcdvWCdcOLbQgxvKh+2xDsTqJTOYXWNxtcInl7YpkiAoVIkaZ+qJco3Sfu8/8sO2gXC3k0PkU\noqPtE19f14npKmHLvUBxzzWWN9TEK+VlE1ZJ7eR0x8aOlvGdHy4PKLGtUzHEJcR8Tt/5hhgVbWIK\nLvvcCxT3XGN5Q8Uom14hKmwmDeObqMp0PndLZwxfdUyXia+OVpUvvp7/G9tVksE8gVdxB3AfgGcA\nXALwsOKYfwLgcwCeBvBrpnNGFXdTgeQcy7tYqPciN0G13H09jNolLakmEH1PVpfqMgmNzsDwkV8l\n5slAvIk7gBmALwC4E8AagCcB3N055i4AnwLwbfX7V5vOG03cqf503TGhKtDQiBmXe49hpfelQ3X9\nlItdfFthoa263EY/FHRtx0d+lZgnA/Ep7m8E8Hjr/TkA5zrH/CyAf0G5YPOKIu6LBX34Z/IH+65A\n1HMO7VhSTAR3r6/rYEZsYQUhA7eAFa5tpz3p2rRhnQFUUp4MxKe4PwjgPa33bwXwC51j/mct8H8A\n4OMA7jOdN7i4m0TF1mL0XYGoop1J+JUzOtfQmCysiQmMFbZ5o2u7Y6ozjvgU93/cI+7/tXPMhwD8\nNoBVAHcAeB7ALT3n2gGwD2D/9OnTYXPA5G9OaTHauClK9ynqfK5jaaQTdA0EJee2mwFUcac8iel5\nALe33t8G4IWeY35HSvmylPKL9eTrXd0TSSkvSCm3pJRbp06dIlzakeWyeuKMCurTjULQPElKhZTA\nyZM3npQz5OlMOaB6otHGBrC9HTctodjbUz+9i7HH9FSkMT41KQQm9QdwAsBzqCzyZkL1tZ1j7gNw\nsf7/JIAvAZjrzhv0Adk6dwx1cy/Xa5uGn9QIlrU1/SrZUpiCVVu66yw32HLXAs+hkGcAPIsqamav\n/uydAO6v/xcAfg5VKORnATxkOmcwcU/l46WKmE3s+VgqccmdE4XSXWe50J5EVbUTn2240HrpVdxD\nvIKJeyofL7WB26waZcuvDKYwOukSIoy0m4dNWzZFy/i6XiFlNl1xT2VFUYfmNnucsOVXDoVagU6E\nEMbY7bbg0RZV3CkTqmWRagJSNXHY/Xx7G7hwoZpQFAKYz4HZ7Pjv1tb8pHm5BDY3gZWVaqL25Mnq\n/83NG5O2IWlfP9Y1U7C9DVy+DFy/Xv0dy2RxHyEmkFWTpKEmT2NfLwWUHiDEK2iceworymTNmBZJ\n9W0x4HofFN9ljGFowUPfSeBav0JMILPlTgaTdcukRNVYXERuyMo+m61tQ1bmghvQ6BnS8YYo19iG\nQMGGB4t7Trg0BtcGZDNhO9TaMqGz8GLswzIVH7gLQwQ6lDD6KjPqeQqtgyzuOeEyjHUd+tpu85vC\ncnfdVpdKwVZZNHw8CCbHzjOXsg+YDhZ3nwytyLla7ql87kMeZUcpB3YHmRlrHuVyXwHTweLui6E9\ncHeylHoOnz73xkqbz9VPsQ9FnyC7WI02+cErRs3kYuH6JpeyD5gOFndfUHpgm4nURmQpjWhotEyq\nbX5NhB7J5GK95U6urpUh5FL2bLkXgKkH1llAOVS0HC00lzTZWEI53jMTh1zKnn3uBWASaN33usnN\nWNbS0KiIUJad7blt72OMVilDI5ey52iZzOhzaeh6YJ1FSZncDG1VuPr+crGAck0PM05y6Rg0sLi7\noBIQnc9aZ1FSFxTluJAoB5dSlwIaHlMwhRgQLO4uuAiazbYDOhdNKFwrbC5RBwwTixwNmh6o4j6+\njcOG4LKZUHcjsI2N6n2zcVR7Q6mNjf5zqDYd84EpfSqoG6ExzFgY2WZiLO5tXAWNuiNgqh0rXXYs\nLP3xfsz48b3j6MgMmnGJ+9DCDi1orlZ0CkpKawlMZevjWDTPIj44qJwnBwfV+yH5OjaDhuK7CfHy\n7nP3NRnCk3Z6OH/sKWSirihC+ccLqN+Y3IRq6smQGJUidcVjkXIjdd0cI7oAhZEzPXFPGd0RQ/Ry\nEFYWKTc48sg/qrrYbCc9lNSGlIbpiXtK4Ylx7RyEdaoilWJXUEaPbgM6H66Z1IaUhumJe8oCiSF6\nttcIYXlMUaR81CvKOTK2FLNF5ZYZ2u5U9Xw2y6J8pifuUqZrILlZ7qE6uswtmiD4Kltd3fSdr1Pp\nKEK1O8oDbxJ2ztMU91Tk5nNXVfr53E86piAcDTFGZT5FakodcKh7pewJ1S6fyHnO4h6bnKJldJbH\nGBt5SIYKL6XMfHYgU3OdhWh31D2hmvKJnOcs7lNGZ3mMtZGHoq+hC1FtJufy2z6LzqePd6qT3r5p\ndxqzmb4tRc5zqriPa4WqT0peUahbUVfoPhlHiFk229vA2bPVKt0GKYGLF/uv207b2bPA4eHR7w8P\ngb29o5/1rYwEgGvXqmvZrL7MYQl9yW2nob1lx8WL+pWrOeR5H5QeIMQra8s9J7+l67DT9SHUuZOi\nbKjDbtvhfPe3VEtRR+q6m/r6oYg5IW4A7JYZQC5+yyGVZqyNLEXZUFdD2k7E2V6POsxPOekdsnxy\nnsznaJlCxD2HmHIp40zmlUYKnzJ1NaRLCF0fqlHXbJZ/GYYqn7EaKw6wuA8hh5hyKeMLWQmdQQrL\nnboa0sfE6GIh5eqqe+fQLUPdU8RCEKJ8FothrqpYRGo/LO5D8BFTPrQyx4542d09LmA5WkapLDhV\nWbQ7Wh9po7h2VEaGyuKPmVchFmTp5jFyiQKKWC9Z3IcyNKbctdKZKnOIChNyn44QpBhh2EyqDkkb\nxbXTrVvUidxYZeqzfEydXS71M+KIksU9Fr4L1WSxhxAy3TVzsYxSE8syc7HcqRO5pZXpYmG+l/k8\nj9FlRBcqVdzLi3PPLYbW99NbVHHoQtAfkefrmkD6WN1ciPVkKlXMe0Nf3bJdu1BCmTZPWjLxla8M\nfwKTD1R5KmU6naL0AADuA/AMgEsAHtYc9yAACWDLdE4nyz3XGfMYw9CQw09qNAgTh3Z9ms+rl65u\n2VjuObQXCrajkdTumYjuVPhyywCYAfgCgDsBrAF4EsDdPcfdDOD3AXw8mLjnEn9ug63wh+zAVGnp\nm0ylLrFn0tNXfjm7MChQ7ic3V1OkQAif4v5GAI+33p8DcK7nuHcB+AEAHwsm7qXtm+Eq1LE2Q1pf\nr4TBde8UJj02k6klGUG6WP/c7y2wTvkU9wcBvKf1/q0AfqFzzD0Afqv+P5y4l2a555ReXQx2Lmmc\nAr47bhv3RUlG0OqqlGtrNGMkN1dT4HZPFXfKhKro+Ux+80shVgD8PICfMp5IiB0hxL4QYv/q1auE\nS3fwPXkZgvaE78FB/zEpNu9SXfPaNbvjGXeaScKDg6q522wIpsKmnHKdSN3bO77B2ssvAzfffHwC\n+5FH4kxsDyEXnTKpPwxuGQCvBPASgMv16y8AvACD9e4cCpnzKkrqEJkt92kSwqLTTYbnbN22Kc3d\nSiGgTsGjW+YEgOcA3IEbE6qv1Rz/MZOwyyHinjOUIXLK3SWpPndVGmN3rKk68lDXDSFiunLN1Qjq\nkpP7sgC8iXt1LpwB8CyqqJm9+rN3Ari/59i8xT2kYOhm+HNoZKp7p+RJ7DDUVGGvIa8bSsRyHs1S\nyDXEOVO8inuIVxJxD12JSrJAbAUh9r2lysuQ1x2biPnsVErvoCLC4t5HaMEopfG6pDO2XzSVHzb0\ndcciYqXU9REyLXFPtcnXkLSkxKWTy8Vyn8/DXM903RxHXynhfEoGVdzL21umi014WYxnHbafvRhq\nL5ihqMLndGF1Z84cfY4oEDa86/x5YHX1+Odf/WrYfTpyCWMD8ttHqY1LHWLiQukBQry8We42FgQP\nJStsra6+fIuxijXVc2BzGH2lqKs29+3Lcs8hrwsDk3HL5PJIvJKwFY5UQ/Axxj9TiZ3ntnXCR+eT\nIgJrBG1/OuJegu8vx0qlS1P3u7787RPZWEvrcyrbUMTu2Fzy2qW827+JuYAu9ajdY9uYjrinLjQT\nuaevi8oFY2qEIe7TxR0UuiON1VHH7thiBRtQVnCH6MBSGgqe28Z0xF3KPC3jhtKsT9fl7KHu0+bZ\nrqE70pgddWyjIEY9pW5yFqJtpHTxec7baYm7ihxEvzS/sW6VrS4vQ92nThC66QgtUL7Pb6qfMetv\njM6Eskc79ZohFuEVsu0Ei3su7pCxWO6m9Ia6T5MgtMs0dEfq8/y51M9umkJ2Jqo6MpvZ++1t8840\nAixo2wkW91xENcdGrMM1vaHukzKUb8q0JMs9l/pJxYfw27jYdPgO5V0s/E7udvPK8x70LO45uUNy\ncA/Z4JreEPdJmYRryjTUpG5zT/N5/wMkXM6fU/00ESrs0XWthG3e6ToDU/2yLQ9VXnncpZPFvTTL\niFHTCKzJcm8f62szq74nBJkeWE1BdT+ht1dwwUdbSjnq0XUGppGhbfoi6A6Le2nuEMZMLhEkbavP\nlcWi6ii6511by6eOmjpVG6s25XyFriPVCbtL3YowImNxl7I8dwhjJmaZ2kzmuhBje4UhLjaTOyyV\n5d6kj3pfqhFY18XWfs1mbmXLlvtIn8TEjAubyVwXYmwv7DrSMd177lsN9F2/3RnorHZqAEFf5xLh\nPlncGT9MefRjM5nrQs7RPab1Drb1YLE4Kqjzedq6pLu/oZFhgdvMdMV9ymLkm9TWVg7YTOa6nDtk\n/ppGBrq24rPjybEeDbm/xMEa0xT3HCtRm9I6Ho44ukGouhWyTtiGAIZa1JNjPRpyf4nDWKcp7jlW\noobcO54+SorFjkGq+H+fk6JNnYu5HD/XeuR6f2y5JxD3XCuRlMkrhBMlptknPsRtaKfu4/d99xCz\nrYytHiU21KYp7jlXopw7HhWLxfFwMR9x2CW4p3w14KF1MlS4ZOi2Emplby4krMPTFPecXR85dzwq\n+hbarK76X/WZSxm1GVJe7Ybfdw5qp75YDPu96dyhyiHkyl5mouIuZb5WYSmi1iZEh1RKJ+c60qKE\nT1LvN1SUTjutIdpKKWVcKNMV95zJteNREcKVVIp7ylWgKAufhkZlAP2/z6V+6co4lzQWDIs7M5wp\nW+6uIy2TK8ZHVEbf5mI5jQx16c4ljQXD4s4MJ4Rg5CRCJlyszFSLf3LqNFXpjrGXzgRgcS+F3Iep\nIdKX+z0PwXfnRc2r3NxdfenOLY2FwuJeAiVZsbmRcweRIm05We4qSkhjAbC4lwBXdje4UzxOzDwJ\nsWKWIcPiriMXq4+HqW5wp9hPjHodasUsQ4Yq7qI6Nj5bW1tyf38//oWXS2BnBzg8vPHZ+jpw4QKw\nvR03LZubwMHB8c83NoDLl+OmxZblEtjbA65cAU6fBs6fj5d/KyuVrHQRArh+PU4apkrJdXYkCCGe\nkFJumY5biZGYrNjbOyrsQPV+by9+Ws6frzqWNuvr1ec6lsuqka2sVH+Xy1ApVF9/Z6dq5FJWf3d2\n7NIx5B5On7b7nPHHlSt2nzPpoJj3IV7J3DK5uUJsh6k5+C2HukVSb6bFuMMuseSAfe4KSq+cOaTf\nZaFOuxObzYbfQ2rfberrp4I71uR4FXcA9wF4BsAlAA/3fP+TAD4H4DMAPgJgw3TOZOJeeuX0PfLw\nuVBHlafU/VZ8Pjs0pPCWXoeGMsaOraB78ibuAGYAvgDgTgBrAJ4EcHfnmL8HYL3+fxfAB0zn5WgZ\nR1KtgDT9TpcmSmfga/QRQ3hzGD0x/iiss/Yp7m8E8Hjr/TkA5zTH3wPgD0zn5Th3R3xWxFjb2uqO\n8d2YYghvbvM2zDAK66yp4k6JlrkVwJda75+vP1PxIwA+TDgv48L2dhW2ubFRhf5tbLiHcQ6JfNje\nrkLfrl+v0tBHE72iimKZzYbfQ5cY0RwcrTMuRhoBRBF30fOZ7D1QiB8CsAXgPyq+3xFC7Ash9q9e\nvUpPJXOUtrBevuwuir5E6syZ/s8PDqowxzNn+kM+L14cfg9dYgivawgrkycj7awp4v48gNtb728D\n8EL3ICHHo91xAAAMoUlEQVTEmwHsAbhfSvm1vhNJKS9IKbeklFunTp1ySS/jE6pImWLSH3tMfY2D\ng0rEz571M9owEUN4fY6emPSMtbM2+W0AnADwHIA7cGNC9bWdY+5BNel6F8UXJNnnng+myWWKj5/i\nU/ftv1wsjm4hO58fjc4pdcKcSUNBdQY+tx8QQpwB8C5UkTOPSinPCyHeWV/kg0KI/w3gbwL4cv2T\nK1LK+3XnTLb9AGMHZbm56pg2PrcGWC6BH/5h4OWXj36+tgY8+ihb0Myo8br9gJTyMSnlX5NSfoeU\n8nz92c9IKT9Y//9mKeW3SylfV7+0ws44kmLbAcpkU9+wtovKf2m6p77v9/aOCzsAfP3rabaRYJgc\noZj3IV7e3TIFDaucSBWLSw0Ta/K/CQmkpNN0T6rvYyyEYphMwaS2HyhsEYITqWJxXfKW2tGa7kn1\nvWr7goxjkxnGF9MS98IWITiRcuFMqFGR6Z50E7Wrq8c/W1sL/zg7hkkMVdzHseXvSBchHEEXi/u2\ntwEnTlSTlidOVO994iuuvospvlj1/cYG8N73AvP5jc9WViqf+9mzVT7YzEn42MKYYTJjHOJOXYSQ\neh/0Iahicb/zO4F3vxu4dq367Nq16r1vgfdBN/9Vi5ua+GJd/PH2NvDSS8BiUX3WROI0+WAj0Dnt\n8c8wvqCY9yFe0X3uIfzyQ4byLr/t+43K/zybudxROFT5v7trjrPXfW/alGw+N+cz7xXDFAQm5XOX\n0l0EXP3yQzoLHx1NOzpF9cqJUPMilAVUpnyewpwNMxqo4j6dZ6j6fu7mkGdJDn0OZd9zYLvMZsA3\nvmE+VyxCPfeUsoCqSzefc3quLsMY4GeodvG9OdCQSdyhE8B9PuIuOzu0c8WCkv8ucyKUBVRdrlw5\neq29vXh73zBMLCjmfYhX9L1l+lwhq6uVT9bFZz5kKD/UDaBzRcxmlR87BG1XUOPrt5kvcFmwZDMX\n0U6XLhb+ppvGvy6CGS2YnM+dQtsvP59XcdGuDTylzz2Fj1j39CVbEe7rTF3vSXVOXQe4shI//xjG\nEyzuJnwIpEpYKJEwQyNtfI5CKJgmb0NNjOoiVnSdJPXRftRrMUwmsLj3QXk0XCNUriIZaysE21HI\n0BWYpjyjCqMqHSoxns/V59J10IuFOs0qlw1b7kwBjF/cdft5N9+3RWR3l/ZQZ+qmVypCjghcr+ka\nY065hs396Tq+xcJ+SwGTtb+721+efXWBfe5MIYxb3FVCAFQNt09EKPHQqmNshHnoghgXy990TZUw\n23RkPnzupk6o3VlT8p/SkQ5xnTFMhoxb3HVWZOOmMAl596X7jY0vdqjl7vJ73e6JOveE7XVU0TLd\nEYBqROC6UZgq/6ewGyjDdBi3uNuuSqS81tftLcc+TK6HEEvhTVa1TWdn05F1XWO6vNVNcpq2+KV0\nOEO3cWCYQhinuLtGQbSFy/aYbiw2RRT6jqOKvmqyTzex2FxT91uqm4rakek6FNV5h8a6+xBltvaZ\nwhmfuNuISZ+YtScQqQLf9dGGiE3vS2vfi7JXuc7qp0ww+7gfXZ42+eiyUZiv/XhMD/pgkWcyZ3zi\nThUTIW5MqtoumOmzYE2jBR+rSm2sX5c8Uv1uiCXscj9D3CaUiCBTp0HpRNmKZzJnfOKuExPb8MXF\n4nhcuMra9fXMziHupG6aVCLmy7rVhZgOvR+XstGVlaqcutexSS/HuzMZMz5x10WE2DZQXShl+/c2\nFr4JlQDZTHaq3E2+FivZxJrb+typeabKD92WAZQRi81Ig1eqMhkzPnFXiSO1gVImLbuCaRIE1z3Y\nTROtq6v9K059RPPo0HVmfRO63fuhdlQ68TTlt005ta9jY7k3IaQMkyHjE3cp3Xyxze+oViZlSXz3\nOFMaXe6r77PQTwwydWame6Hms64zMl2/L2991wGXjpthIjFOce/Dp7+1Kzq2MesqAVH5rW2hTpi6\numZ8bTHQXNtl502V9a8LBaXONdiM3qj3yzCRmY64S2kWM4q/VSU6NjHrOreED0uQImJDJlVNcxEu\nIwTbtQF9111dpY0abDo06poHhsmMaYm7Cd1krM8HdcSwBE0i5rLKs3v+2Pud93VIfWsNfOJzspxh\nIsLi3sb3qkTXmPUYlqAPv3zsVZxDOyQXTD549rkzmUIV92k8Q3V7u3ompq9nZKqeBzqf65/n6fq8\nVht8PCvWd36ZGPpMWRe69zifVy9+hiozEkTVEcRna2tL7u/vJ7n2YJbL6gHU7YdUr69XggAAb387\n8JWvHP1N831owdClLVex2twEDg6Of76xAVy+HDs1DJM1QognpJRbpuOmYbn7RmfZbm8DL70ELBbx\nLF9q2nLl/PnjI5719epzhmGcYMudyYPlEtjbq1wxp09Xwp5zh8QwiaBa7idiJIZhjDSjHoZhvMBu\nGYZhmBHC4s4wDDNCWNwZhmFGCIs7wzDMCGFxZxiGGSHJQiGFEFcB9KxcIXESwEsek+OLXNMF5Js2\nTpcdnC47xpiuDSnlKdNBycR9CEKIfUqcZ2xyTReQb9o4XXZwuuyYcrrYLcMwDDNCWNwZhmFGSKni\nfiF1AhTkmi4g37RxuuzgdNkx2XQV6XNnGIZh9JRquTMMwzAaihN3IcR9QohnhBCXhBAPJ07LZSHE\nZ4UQnxZC7NefvUoI8btCiD+q/35bhHQ8KoR4UQjxVOuz3nSIiv9S599nhBCvj5yudwgh/rjOs08L\nIc60vjtXp+sZIcQ/CJiu24UQvyeE+LwQ4mkhxNvrz5PmmSZdSfNMCPEtQohPCiGerNP17+rP7xBC\nfKLOrw8IIdbqz19Rv79Uf78ZIl2GtL1PCPHFVp69rv48Zv2fCSE+JYT4UP0+bn5RHteUywvADMAX\nANwJYA3AkwDuTpieywBOdj77WQAP1/8/DOA/REjH9wJ4PYCnTOkAcAbAhwEIAN8D4BOR0/UOAP+6\n59i76/J8BYA76nKeBUrXawC8vv7/ZgDP1tdPmmeadCXNs/q+v7X+fxXAJ+p8+A0AD9Wf/xKA3fr/\ntwH4pfr/hwB8IGAdU6XtfQAe7Dk+Zv3/SQC/BuBD9fuo+VWa5f7dAC5JKZ+TUn4dwPsBPJA4TV0e\nAHCx/v8igH8Y+oJSyt8H8P+I6XgAwH+TFR8HcIsQ4jUR06XiAQDvl1J+TUr5RQCXUJV3iHR9WUr5\nf+r/vwrg8wBuReI806RLRZQ8q+/7z+u3q/VLAvg+AL9Zf97NryYffxPAvUII4TtdhrSpiFKWQojb\nAHw/gPfU7wUi51dp4n4rgC+13j8PfeUPjQTwv4QQTwghdurPvl1K+WWgaqwAXp0obap05JCHP1YP\niR9tua2SpKseAt+DyuLLJs866QIS51ntYvg0gBcB/C6qUcKfSim/0XPtb6ar/v7PAMxDpKsvbVLK\nJs/O13n280KIV3TT1pNun7wLwL8BcL1+P0fk/CpN3Pt6s5ThPm+SUr4ewFsA/CshxPcmTAuV1Hn4\nbgDfAeB1AL4M4D/Vn0dPlxDiWwH8FoCfkFL+f92hPZ8FS1tPupLnmZTympTydQBuQzU6+C7NtaPm\nVzdtQoi/AeAcgL8O4G8DeBWAfxsrbUKIHwDwopTyifbHmusGSVNp4v48gNtb728D8EKitEBK+UL9\n90UAv42q0v9JM8yr/76YKHmqdCTNQynln9SN8TqAX8YNN0LUdAkhVlEJ6FJK+T/qj5PnWV+6csmz\nOi1/CuBjqPzVtwghmqe5ta/9zXTV378SdPecj7TdV7u4pJTyawDei7h59iYA9wshLqNyHX8fKks+\nan6VJu5/COCuetZ5DdXkwwdTJEQIcZMQ4ubmfwB/H8BTdXrO1oedBfA7KdKnSccHAfyzOmrgewD8\nWeOKiEHHv/mPUOVZk66H6siBOwDcBeCTgdIgAPwKgM9LKX+u9VXSPFOlK3WeCSFOCSFuqf//KwDe\njGo+4PcAPFgf1s2vJh8fBPBRWc8WRkrb/2110gKVb7udZ0HLUkp5Tkp5m5RyE5VGfVRKuY3Y+eVr\nZjjWC9Vs97OofH57CdNxJ6pIhScBPN2kBZWv7CMA/qj++6oIafl1VMP1l1FZAT+iSgeqIeAv1vn3\nWQBbkdP1q/V1P1NX6te0jt+r0/UMgLcETNffQTXs/QyAT9evM6nzTJOupHkG4G8B+FR9/acA/Eyr\nDXwS1UTufwfwivrzb6nfX6q/vzNgWarS9tE6z54CsMCNiJpo9b++3t/FjWiZqPnFK1QZhmFGSGlu\nGYZhGIYAizvDMMwIYXFnGIYZISzuDMMwI4TFnWEYZoSwuDMMw4wQFneGYZgRwuLOMAwzQv4Sub7d\n+Ys6SjEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a13eb7eb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:\n",
      "0.64\n"
     ]
    }
   ],
   "source": [
    "test_scores = np.dot(test_features, params)\n",
    "preds = sigmoid(test_scores)\n",
    "acc = sum(np.round(preds) == test_classIDs) / len(preds)\n",
    "\n",
    "plt.plot(preds, 'ro')\n",
    "plt.show()\n",
    "\n",
    "print(\"Accuracy:\")\n",
    "print(acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Comparison to Sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We compared to the accuracy of sklearn to validate the success of our implementation of this algorithm. It is clear that the accuracies from our implementation is similar to the accuracy from sklearn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy from sk-learn: \n",
      "0.64\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "clf = LogisticRegression(fit_intercept=True, C = 1e15)\n",
    "clf.fit(features[:,1:], classIDs)\n",
    "\n",
    "acc_sklearn = clf.score(test_features[:,1:], test_classIDs)\n",
    "\n",
    "print('Accuracy from sk-learn: ')\n",
    "print(acc_sklearn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Results on All Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous steps have been summarized into a training function and testing function. The training function called \"logistic_regression\", will load the data and run the gradient descent as previously shown, and the test function will repeat the steps shown in the testing section. This will facilitate running these operations on all the given datasets. \n",
    "In this section we will run these functions on all the data sets and print an output of the results for comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results:\n",
      "File | Accuracy | sklearn Accuracy\n",
      "classify_d3_k2_saved1.mat| 0.655| 0.655\n",
      "classify_d3_k2_saved2.mat| 0.64| 0.64\n",
      "classify_d3_k2_saved3.mat| 0.6825| 0.6825\n",
      "classify_d4_k3_saved1.mat| 0.7325| 0.7325\n",
      "classify_d4_k3_saved2.mat| 0.7275| 0.7275\n",
      "classify_d5_k3_saved1.mat| 0.7175| 0.7175\n",
      "classify_d5_k3_saved2.mat| 0.7175| 0.72\n",
      "classify_d99_k50_saved1.mat| 1.0| 1.0\n",
      "classify_d99_k50_saved2.mat| 1.0| 1.0\n",
      "classify_d99_k60_saved1.mat| 1.0| 1.0\n",
      "classify_d99_k60_saved2.mat| 1.0| 1.0\n"
     ]
    }
   ],
   "source": [
    "results = []\n",
    "print(\"Results:\")\n",
    "print(\"File | Accuracy | sklearn Accuracy\")\n",
    "\n",
    "for file in os.listdir(\".\"):\n",
    "    if file.endswith(\".mat\"):\n",
    "        params = logistic_regression(data=file)\n",
    "        acc = test(params=params, data = file)\n",
    "        sk_acc = test_sklearn(data = file)\n",
    "        print(str(file) + \"| \"+ str(acc) + \"| \"+ str(sk_acc))\n",
    "        results.append([file, acc, sk_acc])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is clear this classifier work very well for the larger datasets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}