Classification kNN & SVM, Soft Margin Support Vector Machine.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Will implement K nearest neighbors algorithm. Breast cancer dataset which is from california, Irvine(UCI)\n",
    "\n",
    "https://archive.ics.uci.edu/ml/datasets.html\n",
    "\n",
    "https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn import preprocessing, neighbors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the data and add the names\n",
    "\n",
    "Attribute Information:\n",
    " \n",
    "   #  Attribute                     Domain\n",
    "   -- -----------------------------------------\n",
    "   1. Sample code number            id number\n",
    "   2. Clump Thickness               1 - 10\n",
    "   3. Uniformity of Cell Size       1 - 10\n",
    "   4. Uniformity of Cell Shape      1 - 10\n",
    "   5. Marginal Adhesion             1 - 10\n",
    "   6. Single Epithelial Cell Size   1 - 10\n",
    "   7. Bare Nuclei                   1 - 10\n",
    "   8. Bland Chromatin               1 - 10\n",
    "   9. Normal Nucleoli               1 - 10\n",
    "  10. Mitoses                       1 - 10\n",
    "  11. Class:                        (2 for benign, 4 for malignant)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('breast-cancer-wisconsin.data.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "cleaning data and dropping unnecessary data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.replace('?',-99999, inplace=True)\n",
    "df.drop(['id'], 1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   clump_thickness  uniform_cell_size  uniform_cell_shape  marginal_adhesion  \\\n",
      "0                5                  1                   1                  1   \n",
      "1                5                  4                   4                  5   \n",
      "2                3                  1                   1                  1   \n",
      "3                6                  8                   8                  1   \n",
      "4                4                  1                   1                  3   \n",
      "\n",
      "   single_epi_cell_size bare_nuclei  bland_chromation  normal_nucleoli  \\\n",
      "0                     2           1                 3                1   \n",
      "1                     7          10                 3                2   \n",
      "2                     2           2                 3                1   \n",
      "3                     3           4                 3                7   \n",
      "4                     2           1                 3                1   \n",
      "\n",
      "   mitoses  class  \n",
      "0        1      2  \n",
      "1        1      2  \n",
      "2        1      2  \n",
      "3        1      2  \n",
      "4        1      2  \n"
     ]
    }
   ],
   "source": [
    "print(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time to create X and y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.array(df.drop(['class'], 1))\n",
    "y = np.array(df['class'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = neighbors.KNeighborsClassifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "           metric_params=None, n_jobs=1, n_neighbors=5, p=2,\n",
       "           weights='uniform')"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9571428571428572\n"
     ]
    }
   ],
   "source": [
    "accuracy = clf.score(X_test, y_test)\n",
    "print(accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time to predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_measures = np.array([4,2,1,1,1,2,3,2,1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_measures = example_measures.reshape(1, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2]\n",
      "[[1. 0.]]\n"
     ]
    }
   ],
   "source": [
    "prediction = clf.predict(example_measures)\n",
    "print(prediction)\n",
    "print(clf.predict_proba(example_measures))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2 2]\n",
      "[[1. 0.]\n",
      " [1. 0.]]\n"
     ]
    }
   ],
   "source": [
    "example_measures = np.array([[4,2,1,1,1,2,3,2,1],[4,2,1,1,1,2,3,2,1]])\n",
    "example_measures = example_measures.reshape(len(example_measures), -1)\n",
    "prediction = clf.predict(example_measures)\n",
    "print(prediction)\n",
    "print(clf.predict_proba(example_measures))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### svm\n",
    "The objective of the Support Vector Machine is to find the best splitting boundary between data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import svm\n",
    "\n",
    "clf = svm.SVC()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9642857142857143\n"
     ]
    }
   ],
   "source": [
    "clf.fit(X_train, y_train)\n",
    "confidence = clf.score(X_test, y_test)\n",
    "print(confidence)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2]\n"
     ]
    }
   ],
   "source": [
    "example_measures = np.array([[4,2,1,1,1,2,3,2,1]])\n",
    "example_measures = example_measures.reshape(len(example_measures), -1)\n",
    "prediction = clf.predict(example_measures)\n",
    "print(prediction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "in this tutorial SVC is faster that kNN and more accurate than kNN. \n",
    "SVC handle pointless data better than kNN\n",
    "\n",
    "\n",
    "Radial Basis Function (RBF) kernel, purely since it's typically the default kernel used, and can take us to a proposed \"infinite\" number of dimensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The question is then how we might classify a total of 3 or more groups. Typically, the method is to do what is referred to as \"One Verse Rest\" or (OVR). The idea here is you separate each group from the rest. For example, to classify three separate groups (1, 2, and 3), you would start by separating 1 from 2 and 3. Then you would separate 2 from 1 and 3. Then finally separate 3 from 1 and 2. \n",
    "\n",
    "Another method is One-vs-One (or OVO). In this case, consider you have three total groups. The way this works is you have a specific boundary that separates 1 from 3 and 1 from 2, and this process repeats for the rest of the classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}