Iris Data set classifier (4).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Iris Data Set Classifier "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main aim of this tutorial is to develop a classification model for the Iris dataset which involves three types of plants that will help us to create an SVM margine classifier "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us start by loading the dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classes [0 1 2]\n",
      "Charactaristics  4.0\n"
     ]
    }
   ],
   "source": [
    "#Using the scikit-learn library to import the data set \n",
    "from sklearn.datasets import load_iris\n",
    "import numpy as np\n",
    "IrisDataSet = load_iris()\n",
    "IrisDataSetClasses = IrisDataSet.target\n",
    "print(\"Classes\", np.unique(IrisDataSetClasses))\n",
    "IrisDataSetCharacteristics = IrisDataSet.data\n",
    "print(\"Charactaristics \" ,IrisDataSetCharacteristics.size/ IrisDataSet.target.size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the data printed above it is clear that our data consists of 3 classes indicated by the numbers 0 ,1 & 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also the number of characteristics printed above indicates that each instance of the dataset (flower) can be described using four features :\n",
    "* Sepal Length\n",
    "* Sepal Width\n",
    "* Petal Length \n",
    "* Petal Width"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Normalizing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the next step after successfully loading and examining the data we will then need to normalize the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Using the preprocessing scikit-learn library which mainly helps in preparing datasets before \n",
    "  applying any process on it  we will import the Standard Scaler\n",
    "* Standard Scaler : \n",
    "                     - Standardize features by removing the mean and scaling to unit variance \n",
    "                     - Centering and scaling happen independently on each feature by computing the \n",
    "                       relevant statistics on the samples in the training set.\n",
    "                     - Mean and standard deviation are then stored to be used on later data using the transform method.\n",
    "                     - Standardization of a dataset is a common requirement for many machine learning estimators: \n",
    "                     they might behave badly if the individual feature do not more or less look like standard\n",
    "                     normally distributed data (e.g. Gaussian with 0 mean and unit variance).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data was successfully Normalized \n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "# Alternative to using the standard scaler\n",
    "#from sklearn.preprocessing import MinMaxScaler\n",
    "# Scaler = MinMaxScaler(feature_range=(0, 1), copy=True)\n",
    "StandardScaler = StandardScaler()\n",
    "NormalizedDataSet = StandardScaler.fit_transform(IrisDataSetCharacteristics, IrisDataSetClasses)\n",
    "print(\"Data was successfully Normalized \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to divide our dataset into 2 subsets and shuffle the data to avoid any preference towards one class more than the others\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training Set : Which is the largest chunk of data 80% which our model will be trained on to develop it "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test Set : Which is the smallest part of the data 20% to examine our model , portion of the data that the model won't be able to \n",
    "           see till it is fully developed to make sure our model will work successfully and with high amount of accuracy & react \n",
    "           as required to any foreign data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data was successfully split & shuffled \n"
     ]
    }
   ],
   "source": [
    "# import the train_test_split function from the model_selection library \n",
    "from sklearn.model_selection import train_test_split\n",
    "# The main objective of this function (train_test_split) : Split arrays or matrices into random train and test subsets\n",
    "# first argument : array to be shuffled and split\n",
    "# second argument : unique classes of data\n",
    "# third argument : portion of dataset assigned for testing \n",
    "# forth argument : enable shuffling the data\n",
    "# fifth argument : random state is the seed used for random number generator \n",
    "# 0.2 => 20% test set \n",
    "X_train, X_test, y_train, y_test = train_test_split(NormalizedDataSet, IrisDataSetClasses, test_size=0.2,shuffle = True, random_state=0)\n",
    "print(\"Data was successfully split & shuffled \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to choose the most suitable machine learning algorithm ? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following the scikit-learn algorithm cheat sheet we need to know if the number of data is enough \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of instances 150\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of instances\", IrisDataSet.target.size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Confirmed more than 50 samples , since our aim is to develop an SVM margine classifier and the data is labeled and we can predict it's categories we can safely ignore regression , clustering , dimensionally reduction and move on to classification \n",
    "\n",
    "* Going deep in classification we find our data points are less that 100,000 samples so our start point is linear SVC\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear SVC classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stands for :Linear Support Vector Classification\n",
    "\n",
    "\n",
    "Parameters : \n",
    "* loss => Specifies the loss function. ‘hinge’ is the standard SVM loss \n",
    "* multi_class => Determines the multi-class strategy ,if y contains more than two classes then assign its value to \"ovr\" trains n_classes one-vs-rest classifiers\n",
    "* random_state => The seed of the pseudo random number generator to use when shuffling the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linear SVC classifier was successfully developed\n"
     ]
    }
   ],
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "LinearSvcClassifier = LinearSVC(loss='hinge', multi_class='ovr', random_state=0)\n",
    "print(\"Linear SVC classifier was successfully developed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimizing our Hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are hyperparameters ??\n",
    "- A kind of parameters that cannot be directly learned from the regular training process ,because they tend\n",
    "  to express “higher-level” properties of the model such as its complexity or how fast it should learn.\n",
    "  Hyperparameters are usually fixed before the actual training process even begins.\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regularization Parameter 'C'\n",
    "\n",
    "* When we have a high degree linear polynomial that is used to fit a set of points in a linear regression setup, to prevent       overfitting, we use regularization, and we include a lambda parameter in the cost function. This lambda is then used to update   the theta parameters in the gradient descent algorithm.\n",
    "* In other words one might consider the C as how do we actually care about violating the developed margine \n",
    "* In our case the hyperparameter we want to optimize as much as possible is the regularization paramater 'C'\n",
    "* In order to obtain the most suitable 'C' we will use the GridSearchCV a hyperparameter optimization algortihm that will take \n",
    "  as input our model and a range of the regularization parameter to search and obtain the most suitable C\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to calculate C ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Unfortunetly there is no clear method was yet develop to ensure a successful and precise calculation for this parameter.\n",
    "* We will apply GridSearchCV : \n",
    "   Exhaustive search over specified parameter values for an estimator which is way to select the best of a family of models, parametrized by a grid of parameters.Alongside with cross validation.\n",
    "* Cross Validation: which is a method to robustly estimate test-set performance (generalization) of a model. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Range of C was successfully calculated\n"
     ]
    }
   ],
   "source": [
    "#Using the numpy logspace function : Return numbers spaced evenly on a log scale.\n",
    "# Argument #1: start of the range \n",
    "# Argument #2: end of the range \n",
    "# Argument #3: step size between elements \n",
    "RegularizationParameter = {'C': np.logspace(10^3, 10^10, base=3)}\n",
    "#print (\"Calculated C :\", RegularizationParameter)\n",
    "print (\"Range of C was successfully calculated\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search was accomplished successfully\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "# Argument #1: our previously developed linear SVC classifier\n",
    "# Argument #2: range of parameters \n",
    "# Argument #3: cross-validation generator or an iterable determines the splitting startegy \n",
    "#              assigning it to integer, to specify the number of folds in a (Stratified)KFold\n",
    "GridSearch = GridSearchCV(LinearSvcClassifier, RegularizationParameter, cv=5)\n",
    "print (\"Search was accomplished successfully\" )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally apply the grid search on our training set the previously split 80% in order to choose the best estimator "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classifier was successfully chosen\n",
      "The most suitable value for C is  347.85717768381505\n"
     ]
    }
   ],
   "source": [
    "GridSearch.fit(X_train , y_train)\n",
    "#Obtaining the best classifier\n",
    "ChosenClassifier = GridSearch.best_estimator_\n",
    "print(\"Classifier was successfully chosen\")\n",
    "\n",
    "print (\"The most suitable value for C is \" ,ChosenClassifier.C)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Concluding the results "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model with reacting to the training set (80%) with accuracy :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of model with the training set is : 0.9666666666666667\n"
     ]
    }
   ],
   "source": [
    "print('Accuracy of model with the training set is :', ChosenClassifier.score(X_train, y_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The developed model results in 96.67% accuracy relatively high "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model with reacting to the test set (20%) with accuracy :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of model with the test set is : 1.0\n",
      "When applying the model to foreign data it reacts with  100.0 % accuracy remarkably high accuracy when the model is generalized\n"
     ]
    }
   ],
   "source": [
    "print('Accuracy of model with the test set is :', ChosenClassifier.score(X_test, y_test))\n",
    "print (\"When applying the model to foreign data it reacts with \", ChosenClassifier.score(X_test, y_test) *100,\"% accuracy remarkably high accuracy when the model is generalized\"  )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Now let's try it with kernel =linear "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In other words training words we are training SVM with a linear function kenel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The chosen value of C is  10\n",
      "Training accuracy : 0.9666666666666667\n",
      "Testing accuracy : 1.0\n"
     ]
    }
   ],
   "source": [
    "from sklearn import svm\n",
    "from sklearn.svm import SVC\n",
    "LinearKernel = {'kernel': ['linear'], \n",
    "                     'C': [1, 10, 50,100, 1000]\n",
    "                    }\n",
    "gridSearch2 = GridSearchCV(SVC(), LinearKernel, cv=5)\n",
    "gridSearch2.fit(X_train ,y_train)\n",
    "ChosenClassifier2 = gridSearch2.best_estimator_\n",
    "\n",
    "print (\"The chosen value of C is \" ,ChosenClassifier2.C)\n",
    "print (\"Training accuracy :\",ChosenClassifier2.score(X_train, y_train) )\n",
    "\n",
    "print (\"Testing accuracy :\",ChosenClassifier2.score(X_test, y_test) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# kernel =rbf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When training an SVM with the Radial Basis Function (RBF) kernel:\n",
    "two parameters must be considered: \n",
    "#1 :\n",
    "The parameter C, common to all SVM kernels, trades off misclassification of training examples against simplicity of the decision surface. A low C makes the decision surface smooth, while a high C aims at classifying all training examples correctly.\n",
    "#2 :\n",
    "Gamma defines how much influence a single training example has. The larger gamma is, the closer other examples must be to be affected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The chosen value of C is  1\n",
      "Training accuracy : 0.9666666666666667\n",
      "Testing accuracy : 1.0\n"
     ]
    }
   ],
   "source": [
    "rbfKernel = {'kernel': ['rbf'], 'gamma': [1e-1, 1e-5],\n",
    "                     'C': [1, 10, 50,100, 1000]\n",
    "                    }\n",
    "gridSearch3 = GridSearchCV(SVC(), rbfKernel, cv=5)\n",
    "gridSearch3.fit(X_train ,y_train)\n",
    "ChosenClassifier3 = gridSearch3.best_estimator_\n",
    "\n",
    "print (\"The chosen value of C is \" ,ChosenClassifier3.C)\n",
    "print (\"Training accuracy :\",ChosenClassifier3.score(X_train, y_train) )\n",
    "\n",
    "print (\"Testing accuracy :\",ChosenClassifier3.score(X_test, y_test) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# kernel = poly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training the SVM with polynomial function kernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The chosen value of C is  10\n",
      "Training accuracy : 0.9666666666666667\n",
      "Testing accuracy : 1.0\n"
     ]
    }
   ],
   "source": [
    "polynomialKernel = {'kernel': ['poly'], 'degree' : [3],\n",
    "                     'C': [1, 10, 50,100, 1000]\n",
    "                    }\n",
    "gridSearch4 = GridSearchCV(SVC(), polynomialKernel, cv=5)\n",
    "gridSearch4.fit(X_train ,y_train)\n",
    "ChosenClassifier4 = gridSearch4.best_estimator_\n",
    "\n",
    "print (\"The chosen value of C is \" ,ChosenClassifier4.C)\n",
    "print (\"Training accuracy :\",ChosenClassifier4.score(X_train, y_train) )\n",
    "\n",
    "print (\"Testing accuracy :\",ChosenClassifier.score(X_test, y_test) )\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}