K_Fold_Method.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### The Problem Statement\n",
    "\n",
    "In Holdout method , we have seen a problem arising , with repeated train and test data splitiing , overlapping of test and training and testing data might and will occur. The K fold cross validation method comes to rescue here. We can provide any number here to replace K here. Suppose we are using a 3 fold cross validation where k is being replaced by 3. The workflow will be as below -\n",
    "\n",
    "    -   The entire dataset will be divided into 3 equal parts\n",
    "    -   Suppose the parts are names as p1,p2,p3\n",
    "    -   For training/testing the data , there will be 3 iteration here\n",
    "    -   1st iteration - p1 is the testing data / p2 and p3 is the training data - error estimate - e1\n",
    "    -   2nd iteration - p2 is the testing data / p1 and p3 is the training data - error estimate - e2\n",
    "    -   3rd iteration - p3 is the testing data / p1 and p2 is the training data - error estimate - e3\n",
    "    -   Final error estimate - (e1+e2+e3)/k - k is 3 here\n",
    "    -   The advantage here is that every set of data is being used for both training and testing.\n",
    "    -   Primarily used for classification models but also used for regression problems for finding the most robust cost function"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a basic example of 3 fold cross validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KFold(n_splits=3, random_state=None, shuffle=False)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# importing the libraries\n",
    "from sklearn.model_selection import KFold\n",
    "''' We are defining here that number of splits of k will be 3'''\n",
    "kf = KFold(n_splits=3)\n",
    "kf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3 4 5 6 7 8] [0 1 2]\n",
      "[0 1 2 6 7 8] [3 4 5]\n",
      "[0 1 2 3 4 5] [6 7 8]\n"
     ]
    }
   ],
   "source": [
    "''' The kf.split section here is taking a random numpy array data into consideration and splitting it into 3 subsets as per k value being 3'''\n",
    "''' The for loop is helping us to print train and test split data in each iteraion as output'''\n",
    "for train_index, test_index in kf.split([1,2,3,4,5,6,7,8,9]):\n",
    "    print(train_index, test_index)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Digit Classifier Data\n",
    "\n",
    "Now we will work on a data available in sklearn library , which is basically a image classifier , different images of blurred data is present in it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing libraries\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "import numpy as np\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits = load_digits()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.  0.  5. ...  0.  0.  0.]\n",
      " [ 0.  0.  0. ... 10.  0.  0.]\n",
      " [ 0.  0.  0. ... 16.  9.  0.]\n",
      " ...\n",
      " [ 0.  0.  1. ...  6.  0.  0.]\n",
      " [ 0.  0.  2. ... 12.  0.  0.]\n",
      " [ 0.  0. 10. ... 12.  1.  0.]]\n"
     ]
    }
   ],
   "source": [
    "print(digits.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 2 ... 8 9 8]\n"
     ]
    }
   ],
   "source": [
    "print(digits.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train test splitting without cross validation\n",
    "xtrain, xtest, ytrain, ytest = train_test_split(digits.data,digits.target,test_size=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now we will run a loop here to fit our training data into all imported models , Logistic regression , random forest and support vector\n",
    "lr = LogisticRegression(max_iter = 20)\n",
    "rf = RandomForestClassifier(n_estimators= 40)\n",
    "svm = SVC()\n",
    "models = [lr,rf,svm]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the score for model LogisticRegression(max_iter=20) is 0.9555555555555556\n",
      "the score for model RandomForestClassifier(n_estimators=40) is 0.9722222222222222\n",
      "the score for model SVC() is 0.9907407407407407\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\Anaconda\\envs\\github1\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:458: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n"
     ]
    }
   ],
   "source": [
    "for i in models:\n",
    "    i.fit(xtrain,ytrain)\n",
    "    print(f'the score for model {i} is {i.score(xtest,ytest)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\Anaconda\\envs\\github1\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:458: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n",
      "d:\\Anaconda\\envs\\github1\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:458: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n",
      "d:\\Anaconda\\envs\\github1\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:458: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n"
     ]
    }
   ],
   "source": [
    "# Now let's have a look how we can apply kfold cross validation here with a for loop\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "''' Function to get the score for each model'''\n",
    "def get_score(model, X_train, X_test, y_train, y_test):\n",
    "    model.fit(X_train, y_train)\n",
    "    return model.score(X_test, y_test)\n",
    "''' Defining the splits'''\n",
    "folds = StratifiedKFold(n_splits=3)\n",
    "''' Blank lists where scores of each model will be stored'''\n",
    "scores_logistic = []\n",
    "scores_svm = []\n",
    "scores_rf = []\n",
    "''' for loop to iterate through our digit data and finding out the scores based on 3 splits of all training and testing sets'''\n",
    "for train_index, test_index in folds.split(digits.data,digits.target):\n",
    "    X_train, X_test, y_train, y_test = digits.data[train_index], digits.data[test_index], digits.target[train_index], digits.target[test_index]\n",
    "    scores_logistic.append(get_score(LogisticRegression(), X_train, X_test, y_train, y_test))  \n",
    "    scores_svm.append(get_score(SVC(), X_train, X_test, y_train, y_test))\n",
    "    scores_rf.append(get_score(RandomForestClassifier(), X_train, X_test, y_train, y_test))   \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the logistic regression score set is [0.9215358931552587, 0.9415692821368948, 0.9165275459098498]\n"
     ]
    }
   ],
   "source": [
    "print(f'the logistic regression score set is {scores_logistic}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the support vector score set is [0.9649415692821369, 0.9799666110183639, 0.9649415692821369]\n"
     ]
    }
   ],
   "source": [
    "print(f'the support vector score set is {scores_svm}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the random forest score set is [0.9382303839732888, 0.9565943238731218, 0.9198664440734557]\n"
     ]
    }
   ],
   "source": [
    "print(f'the random forest score set is {scores_rf}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\Anaconda\\envs\\github1\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:458: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n",
      "d:\\Anaconda\\envs\\github1\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:458: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n",
      "d:\\Anaconda\\envs\\github1\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:458: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n"
     ]
    }
   ],
   "source": [
    "# using the sklearn inbuilt module finding the cross val score as above\n",
    "from sklearn.model_selection import cross_val_score\n",
    "logreg_cross_val = cross_val_score(LogisticRegression(), digits.data, digits.target,cv=3)\n",
    "svm_cross_val = cross_val_score(SVC(), digits.data, digits.target,cv=3)\n",
    "rf_cross_val = cross_val_score(RandomForestClassifier(),digits.data, digits.target,cv=3)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logistic regression cross validation score set is [0.92153589 0.94156928 0.91652755]\n",
      "support vector cross validation score set is [0.96494157 0.97996661 0.96494157]\n",
      "random forest cross validation score set is [0.93989983 0.95993322 0.93656093]\n"
     ]
    }
   ],
   "source": [
    "print(f'logistic regression cross validation score set is {logreg_cross_val}')\n",
    "print(f'support vector cross validation score set is {svm_cross_val}')\n",
    "print(f'random forest cross validation score set is {rf_cross_val}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}