MultiClass_LogisticRegression_fromScratch_Modified.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyMkaCWZN9vCWekNk8bBPYxf",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Ogunfool/Multi-class-logistic-regression-with-Python./blob/main/MultiClass_LogisticRegression_fromScratch_Modified.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Import Necessary Libraries\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "from scipy.special import softmax\n",
        "from sklearn.preprocessing import OneHotEncoder, StandardScaler\n",
        "from sklearn.datasets import load_iris\n",
        "from sklearn.model_selection import train_test_split, cross_val_score, KFold, GridSearchCV\n",
        "from scipy.special import expit, logit\n",
        "from matplotlib.animation import FuncAnimation\n",
        "from keras.datasets import mnist"
      ],
      "metadata": {
        "id": "uApi16ehhXuL"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Before we begin manipulating our data, let's create a checkpoint.\n",
        "\n",
        "1.   Checkpoints are places throughout our code where we store a copy of our dataset to avoid loosing a lot of progress incase we overwrite some variables mistakenly.\n",
        "2.   It is an extremely reliable practice when we need to clean, preprocess and process many parts of a dataset.\n",
        "\n"
      ],
      "metadata": {
        "id": "6Q4h4ndZLVan"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# npz is a combination of two or more npy files\n",
        "def checkpoints_z(filename, data_0, data_1):\n",
        "  np.savez(filename, data_0, data_1)\n",
        "  checkpoint_variable = np.load(filename + '.npz') #Load so that we always have an on-hand version of the checkpoint\n",
        "  return(checkpoint_variable)"
      ],
      "metadata": {
        "id": "XW7fOd-wpNST"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# I didn't process my data with the header - so a simple npy file works for me\n",
        "def checkpoints(filename, checkpoint_data):\n",
        "  np.save(filename, checkpoint_data)\n",
        "  checkpoint_variable = np.load(filename + '.npy') #Load so that we always have an on-hand version of the checkpoint\n",
        "  return(checkpoint_variable)"
      ],
      "metadata": {
        "id": "lJspzsYpOzEi"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Load datasets (Keras and Sklearn datasets) and preprocess.\n",
        "\n",
        "\n",
        "1.  Add bias feature set\n",
        "2.  Train, Val , Test split (Set shuffle to true)\n",
        "1.  Feature scaling using data standardization (Substract mean, then divide by standard deviation), features will have a mean of 0 and standard deviation of 1. Normalization is dividing a vector by its lenght and it transforms the data to a range between 0 and 1.\n",
        "2.  One-hot encoding for categorical data y_train.\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "0TaBFGAFue4W"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Keras MNIST Dataset Preprocessing."
      ],
      "metadata": {
        "id": "olrQtDJpoTzl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#loading\n",
        "(train_X, train_y), (test_X, test_y) = mnist.load_data()"
      ],
      "metadata": {
        "id": "n-0dcZAmucTA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "944a09e4-0311-42b4-e399-11a4ee2ba065"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11490434/11490434 [==============================] - 0s 0us/step\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Log test and train set\n",
        "checkpoint_train_set = checkpoints_z('train_set', train_X, train_y)\n",
        "checkpoint_test_set = checkpoints_z('test_set', test_X, test_y)"
      ],
      "metadata": {
        "id": "hXBRHdplpyv6"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def reshape_m(X):\n",
        "  return X.reshape(X.shape[0], np.prod(X.shape[1:]))"
      ],
      "metadata": {
        "id": "kOYPheClusM7"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X_train = reshape_m(train_X)\n",
        "X_test = reshape_m(test_X)"
      ],
      "metadata": {
        "id": "VsDp25bVyAlP"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X_train.shape, X_test.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IIrQDDtVyWn2",
        "outputId": "3017a65e-d4e9-47ee-ea22-9852668b7092"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((60000, 784), (10000, 784))"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Add Bias feature column to dataset\n",
        "X_train_bias = np.ones((60000,1))\n",
        "X_test_bias = np.ones((10000,1))"
      ],
      "metadata": {
        "id": "pd07XXP6F8Pw"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X_train_bias.shape, X_test_bias.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C6nH6oRVGcUP",
        "outputId": "b5651252-fbee-40f3-b7ac-c523479a77e6"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((60000, 1), (10000, 1))"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "X_train_bias.shape"
      ],
      "metadata": {
        "id": "Bo9mWVCmHILs",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e3508031-d0d6-4bec-d340-983cf2db869d"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(60000, 1)"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Concatenate with original dataset\n",
        "X_train = np.concatenate((X_train_bias, X_train), axis =1)\n",
        "X_test = np.concatenate((X_test_bias, X_test), axis =1)"
      ],
      "metadata": {
        "id": "X-egNO94GnbM"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X_train.shape, X_test.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qRLjMBnBHlIV",
        "outputId": "33402b15-17d2-438f-e2e9-cc2e90136943"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((60000, 785), (10000, 785))"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Since X,y data are already seperated - Let's split Train-Val-Test\n",
        "# Set shuffle to true to shuffle before split\n",
        "X_train, X_val, y_train, y_val = train_test_split(X_train, train_y, test_size = 0.1, random_state = 42, shuffle=True, stratify=train_y)"
      ],
      "metadata": {
        "id": "TtcD55plyl4M"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "y_test = test_y"
      ],
      "metadata": {
        "id": "BLgjhyKtzjs0"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X_train.shape, X_test.shape, X_val.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AOahpFDyzUeI",
        "outputId": "541004f3-9cc0-4472-b2ee-7d98a5835d5d"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((54000, 785), (10000, 785), (6000, 785))"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "y_train.shape, y_test.shape, y_val.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lcMKbNSqzZsB",
        "outputId": "bd503ec1-7e22-44be-a8a1-dc2e886cadeb"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((54000,), (10000,), (6000,))"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's normalize data - StandardScaler\n",
        "# Fit t0 training set and use mean and std to transform X_val and X_test\n",
        "scaler = StandardScaler().fit(X_train)\n",
        "X_train_scaled = scaler.transform(X_train)\n",
        "X_val_scaled = scaler.transform(X_val)\n",
        "X_test_scaled = scaler.transform(X_test)"
      ],
      "metadata": {
        "id": "M9bmzQZa0B53"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Now let's use one-hot encoding for categarical data - y\n",
        "y_df = pd.DataFrame(y_train)\n",
        "y_df.head()\n",
        "y_train_df = pd.get_dummies(y_df[0])\n",
        "y_train = y_train_df.to_numpy()"
      ],
      "metadata": {
        "id": "-xwSGhUg0FLF"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "y_train.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ifEHqOzn0Veo",
        "outputId": "2e8a76a3-a086-4232-de01-e05cc7aea681"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(54000, 10)"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Scikit learn IRIS dataset preprocessing.\n",
        "Note: The sklearn dataset is already in an array format in a dictionary, so you can easily trun it into a pandas data or numpy array by calling the np.array() or pd.Dataframe() functions on it."
      ],
      "metadata": {
        "id": "X8VhyNGDIpzv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Sklearn dataset\n",
        "iris_data = load_iris()\n",
        "iris_data.data.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hpGAkWoSIVFE",
        "outputId": "00c45f1a-57c6-4530-f130-7ba0ece9744d"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(150, 4)"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Loading data in different ways\n",
        "# Pandas\n",
        "df = pd.DataFrame(data=iris_data.data, \n",
        "                  columns=iris_data.feature_names)\n",
        "df.head()\n",
        "# You can convert the pandas dataframe to numpy array\n",
        "arr = df.to_numpy()"
      ],
      "metadata": {
        "id": "xJVnoCC_HlwM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Loading directly\n",
        "data_load = load_iris()\n",
        "X = data_load.data\n",
        "y = data_load.target\n",
        "list(data_load.target_names)"
      ],
      "metadata": {
        "id": "z9Ym6gOkk8CJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Since X,y data are already seperated - Let's split Train-Val-Test\n",
        "# Set shuffle to true to shuffle before split\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 2, shuffle=True, stratify=y)"
      ],
      "metadata": {
        "id": "j_mVFPjQMgYO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's normalize data - StandardScaler\n",
        "# Fit t0 training set and use mean and std to transform X_train and X_test\n",
        "scaler = StandardScaler().fit(X_train)\n",
        "X_train_scaled = scaler.transform(X_train)\n",
        "X_test_scaled = scaler.transform(X_test)"
      ],
      "metadata": {
        "id": "I9heq0a0PMF2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Now let's use one-hot encoding for categarical data - y\n",
        "y_df = pd.DataFrame(y_train)\n",
        "y_df.head()\n",
        "y_train_df = pd.get_dummies(y_df[0])\n",
        "y_train = y_train_df.to_numpy()"
      ],
      "metadata": {
        "id": "6ykpYvdUQ_Nu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "MultiVariantLogisticRegression Class"
      ],
      "metadata": {
        "id": "gSxtfQkTtpfH"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "AmOZ50P0ZjfB"
      },
      "outputs": [],
      "source": [
        "class MultivaraiantLogisticRegression:\n",
        "  def __init__ (self, X, y, ALPHA, LAMBDA, n_iter):\n",
        "    self.X = X\n",
        "    self.y = y\n",
        "    self.ALPHA = ALPHA\n",
        "    self.LAMBDA = LAMBDA\n",
        "    self.n_iter = n_iter\n",
        "    # self.m = X.shape[0]\n",
        "\n",
        "    # Initialize w\n",
        "    self.w = np.zeros(X.shape[1])\n",
        "\n",
        "  def predict(self, X):\n",
        "    return X @ self.w\n",
        "\n",
        "  def sigmoid(self,z):\n",
        "    return expit(z)\n",
        "\n",
        "  def hypothesis(self,X):\n",
        "    return self.sigmoid(self.predict(X))\n",
        "\n",
        "  def cost_function(self,X,y):\n",
        "    h = self.hypothesis(X)\n",
        "    unreg_term = (1/X.shape[0]) * np.sum(-y * np.log(h, where=h>0) - (1-y) * np.log(1-h, where=(1-h) > 0))\n",
        "    reg_term = (self.LAMBDA/(2*X.shape[0])) * (sum(self.w*self.w))\n",
        "    return unreg_term + reg_term\n",
        "\n",
        "  def grad(self,X,y):\n",
        "    unreg_term = (1/X.shape[0]) * (X.transpose() @ (self.hypothesis(X) - y))\n",
        "    reg_term = self.LAMBDA/X.shape[0] * self.w[1:]\n",
        "    reg_term_array = np.concatenate((np.zeros(1),reg_term))\n",
        "    return unreg_term + reg_term_array\n",
        "\n",
        "  def update(self,X,y):\n",
        "    self.w = self.w - (self.ALPHA * self.grad(X,y))\n",
        "\n",
        "  def accuracy(self, pred_y, y):\n",
        "    # prediction_prob = self.hypothesis()\n",
        "    # decision = prediction_prob.round()\n",
        "    return (sum(pred_y == y))/y.shape[0], (sum(pred_y != y))/y.shape[0]\n",
        "\n",
        "  def prediction(self, W, X):\n",
        "    return self.sigmoid(np.matmul(W,X.transpose()))\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Define some helper functions \n",
        "# Convergence Check as a function\n",
        "def convergence_check(cost_train_per_iteration):\n",
        "  changes_list = []\n",
        "  k = len(cost_train_per_iteration)\n",
        "  old_v = 0\n",
        "  for f in range(k-1,k-6,-1):\n",
        "    # print(f)\n",
        "    cur_v = cost_train_per_iteration[f]\n",
        "    new_change = abs(cur_v - old_v)\n",
        "    old_v = cur_v\n",
        "    changes_list.append(new_change)\n",
        "  if max(changes_list[1:]) <= SMALL_ENOUGH:\n",
        "    answer = True\n",
        "  else:\n",
        "    answer = False\n",
        "  return answer\n",
        "\n",
        "def early_stop(cost_train_per_iteration, cost_val_per_iteration):\n",
        "  # if last 5 training_cost is \n",
        "  train_val_cost_diff = np.array(cost_train_per_iteration[-3:]) - np.array(cost_val_per_iteration[-3:])\n",
        "  if (train_val_cost_diff[0] < train_val_cost_diff[1]) & (train_val_cost_diff[1] < train_val_cost_diff[2]):\n",
        "    print('Early stopping at iteration:', i)\n",
        "    answer = True\n",
        "  else:\n",
        "    answer = False\n",
        "  return answer\n",
        "\n",
        "# Let's create a plot function\n",
        "def animate(cost_train_per_iteration, cost_val_per_iteration):\n",
        "  plt.cla()\n",
        "  plt.plot(cost_train_per_iteration, label='Training cost')\n",
        "  plt.plot(cost_val_per_iteration, label='Validation cost')\n",
        "  plt.legend(loc='upper left')\n",
        "  plt.tight_layout()"
      ],
      "metadata": {
        "id": "oPfAqLPkzP6k"
      },
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "TRAINING ALL CLASSIFIERS. Note a few things\n",
        "\n",
        "1.   Use cost function as descent function (used to track convergence and for model diagnosis by plotting descent function to know if model is learning) but be careful of the math log error.\n",
        "2.  First ensure the model is learning (i.e first tune learning rate (alpha) hyperparameter) - Once we are sure that algorithm(s) are correct, and learning is happening, then other issues like inderfitting then overfitting can be treated.\n",
        "3. Is it learning? Let's plot descent function every 50 iterations\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "DgquByMnDtPK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Main Loop - But training with different y(s) and classifier objects\n",
        "# Instantiate the MultivaraiantLogisticRegression class object\n",
        "# Create classifier class objects (Track current classifier)\n",
        "SMALL_ENOUGH = 1e-3\n",
        "prediction_prob_per_classifier = []\n",
        "\n",
        "classifiers = [MultivaraiantLogisticRegression(X_train_scaled, y_train[:,ys], 0.1, 0, 5000) for ys in range(y_train.shape[1])]\n",
        "for index, classifier in enumerate(classifiers):\n",
        "  # Create necessary lists \n",
        "  cost_train_per_iteration = []\n",
        "  cost_val_per_iteration = []\n",
        "  # Track no of iterations\n",
        "  i = 0\n",
        "\n",
        "  # Let's define stopping criteria(s)\n",
        "  not_converged = True\n",
        "  while not_converged:\n",
        "    # Track cost on train and validation set per epoch\n",
        "    # Plot learning curves\n",
        "    CF_train = classifier.cost_function(classifier.X, classifier.y)\n",
        "    CF_val = classifier.cost_function(X_val_scaled, y_val)\n",
        "\n",
        "    # Append lists\n",
        "    cost_train_per_iteration.append(CF_train)\n",
        "    cost_val_per_iteration.append(CF_val)\n",
        "\n",
        "    # Update weight(s)\n",
        "    classifier.update(classifier.X, classifier.y)\n",
        "\n",
        "    # print(i)\n",
        "    # Check for convergence or perform early stopping\n",
        "    # Convergence check\n",
        "    if (i > 10):\n",
        "      if (convergence_check(cost_train_per_iteration)) == True:\n",
        "        print('Classifier', index,'converged at iteration:', i)\n",
        "        not_converged = False\n",
        "        break\n",
        "      \n",
        "\n",
        "    # # Early Stopping\n",
        "    # if (i > 10):\n",
        "    #   if (early_stop(cost_train_per_iteration, cost_val_per_iteration)) == True:\n",
        "    #     print('Early stop at:', i)\n",
        "    #     not_converged = False\n",
        "    #     break\n",
        "      \n",
        "\n",
        "    # Increment i\n",
        "    i+=1\n",
        "\n",
        "  # Let's track misclassification error and/ accuracy as well : EVALUATION METRIC\n",
        "  prediction_prob_per_classifier.append(classifier.hypothesis(X_val_scaled))\n",
        "\n",
        "\n",
        "# Choose the most confident class\n",
        "classifier_array = np.array(prediction_prob_per_classifier)\n",
        "classifier_array = classifier_array.transpose()\n",
        "target_class = np.argmax(classifier_array, axis=1)\n",
        "\n",
        "# Check accuracy and misclassification error\n",
        "classifier.accuracy(target_class, y_val)"
      ],
      "metadata": {
        "id": "RM-MK0nfxsA9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ed440afa-9089-403f-d108-2d430a38fc3c"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Classifier 0 converged at iteration: 77\n",
            "Classifier 1 converged at iteration: 94\n",
            "Classifier 2 converged at iteration: 1030\n",
            "Classifier 3 converged at iteration: 71\n",
            "Classifier 4 converged at iteration: 146\n",
            "Classifier 5 converged at iteration: 75\n",
            "Classifier 6 converged at iteration: 46\n",
            "Classifier 7 converged at iteration: 1152\n",
            "Classifier 8 converged at iteration: 117\n",
            "Classifier 9 converged at iteration: 125\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.8131666666666667, 0.18683333333333332)"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Tuning alpha"
      ],
      "metadata": {
        "id": "IoeLVIOvRG5Y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Main Loop - But training with different y(s) and classifier objects\n",
        "# Instantiate the MultivaraiantLogisticRegression class object\n",
        "# Create classifier class objects (Track current classifier)\n",
        "SMALL_ENOUGH = 1e-3\n",
        "prediction_prob_per_classifier = []\n",
        "accuracies_list = []\n",
        "alpha_list = [0.001, 0.003, 0.01, 0.03, 0.1, 0.3]\n",
        "\n",
        "for alpha in alpha_list:\n",
        "  classifiers = [MultivaraiantLogisticRegression(X_train_scaled, y_train[:,ys], alpha, 0, 5000) for ys in range(y_train.shape[1])]\n",
        "  for index, classifier in enumerate(classifiers):\n",
        "    # Create necessary lists \n",
        "    cost_train_per_iteration = []\n",
        "    cost_val_per_iteration = []\n",
        "    # Track no of iterations\n",
        "    i = 0\n",
        "\n",
        "    # Let's define stopping criteria(s)\n",
        "    not_converged = True\n",
        "    while not_converged:\n",
        "      # Track cost on train and validation set per epoch\n",
        "      # Plot learning curves\n",
        "      CF_train = classifier.cost_function(classifier.X, classifier.y)\n",
        "      CF_val = classifier.cost_function(X_val_scaled, y_val)\n",
        "\n",
        "      # Append lists\n",
        "      cost_train_per_iteration.append(CF_train)\n",
        "      cost_val_per_iteration.append(CF_val)\n",
        "\n",
        "      # if (i+1)%10 == 0:\n",
        "      #   # animate(cost_train_per_iteration, cost_val_per_iteration)\n",
        "      #   ani = FuncAnimation(plt.gcf(), animate(cost_train_per_iteration,cost_val_per_iteration), interval=1000)\n",
        "      #   plt.tight_layout()\n",
        "      #   plt.show()\n",
        "\n",
        "      # if (i+1)%50 == 0:\n",
        "      #   plt.close()\n",
        "      #   plt.plot(cost_train_per_iteration, label='Training cost')\n",
        "      #   plt.legend(loc='upper left')\n",
        "      #   plt.show()\n",
        "      #   print('classfier:',index)\n",
        "\n",
        "      # Update weight(s)\n",
        "      classifier.update(classifier.X, classifier.y)\n",
        "\n",
        "      # print(i)\n",
        "      # Check for convergence or perform early stopping\n",
        "      # Convergence check\n",
        "      if (i > 10):\n",
        "        if (convergence_check(cost_train_per_iteration)) == True:\n",
        "          print('Classifier', index,'converged at iteration:', i)\n",
        "          not_converged = False\n",
        "          break\n",
        "\n",
        "      # Increment i\n",
        "      i+=1\n",
        "\n",
        "    # Let's track misclassification error and/ accuracy as well : EVALUATION METRIC\n",
        "    prediction_prob_per_classifier.append(classifier.hypothesis(X_val_scaled))\n",
        "\n",
        "\n",
        "  # Choose the most confident class\n",
        "  classifier_array = np.array(prediction_prob_per_classifier)\n",
        "  classifier_array = classifier_array.transpose()\n",
        "  target_class = np.argmax(classifier_array, axis=1)\n",
        "\n",
        "  # Check accuracy and misclassification error\n",
        "  cla = classifier.accuracy(target_class, y_val)\n",
        "\n",
        "  accuracies_list.append(cla)"
      ],
      "metadata": {
        "id": "CcJRFHnmu5MH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's make predictions another way: Less computationally expensive and faster. Gather the weights from each classifier after training and use matrix multiplication."
      ],
      "metadata": {
        "id": "D94nwB4pHQuM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "weights = np.zeros((y_train.shape[1],X_train.shape[1]))\n",
        "for index, classifier in enumerate (classifiers):\n",
        "  weights[index] = classifier.w\n",
        "\n",
        "pred_train = classifier.prediction(weights, X_train_scaled)\n",
        "# # Make predictions (Train and Val set)\n",
        "trainset_predictions = np.argmax(classifier.prediction(weights, X_train_scaled), axis = 0)\n",
        "valset_predictions = np.argmax(classifier.prediction(weights, X_val_scaled), axis = 0)\n",
        "\n",
        "# Check accuracy\n",
        "print(classifier.accuracy(trainset_predictions, (y_df.to_numpy()).squeeze()))\n",
        "print(classifier.accuracy(valset_predictions, y_val))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w6e1I7EfHNXC",
        "outputId": "9142a0f8-bf60-4396-909d-43cb9fc3d655"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(0.8204074074074074, 0.17959259259259258)\n",
            "(0.8131666666666667, 0.18683333333333332)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Hyperparameter tuning - Grid search vs Interval/random search\n",
        "\n",
        "\n",
        "1.   Grid search\n",
        "\n"
      ],
      "metadata": {
        "id": "jnH_u2-A1RnL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from math import inf\n",
        "# GRID SEARCH\n",
        "# Create a dictionary of hyperparameters, loop through dico, train and choose best\n",
        "params_dict = {}\n",
        "alpha_list = [0.001, 0.003, 0.01, 0.03, 0.1, 0.3]\n",
        "gamma_list = [0.001, 0.003, 0.01, 0.03, 0.1, 0.3]\n",
        "best_so_far_accuracy = -inf\n",
        "\n",
        "params = []\n",
        "# List of tuple\n",
        "for i in alpha_list:\n",
        "  for j in gamma_list:\n",
        "    params.append((i,j))\n",
        "\n",
        "\n",
        "# Dictionary \n",
        "for key, val in enumerate(params):\n",
        "  params_dict[key] = val\n",
        "\n",
        "for ind in range(len(params_dict)):\n",
        "  SMALL_ENOUGH = 1e-3\n",
        "  weights = np.zeros((y_train.shape[1],X_train.shape[1]))\n",
        "\n",
        "  classifiers = [MultivaraiantLogisticRegression(X_train_scaled, y_train[:,ys], params_dict[ind][0], params_dict[ind][1], 5000) for ys in range(y_train.shape[1])]\n",
        "  for index, classifier in enumerate(classifiers):\n",
        "    # Create necessary lists \n",
        "    cost_train_per_iteration = []\n",
        "    cost_val_per_iteration = []\n",
        "    # Track no of iterations\n",
        "    i = 0\n",
        "\n",
        "    # Let's define stopping criteria(s)\n",
        "    not_converged = True\n",
        "    while not_converged:\n",
        "      # Track cost on train and validation set per epoch\n",
        "      # Plot learning curves\n",
        "      CF_train = classifier.cost_function(classifier.X, classifier.y)\n",
        "      CF_val = classifier.cost_function(X_val_scaled, y_val)\n",
        "\n",
        "      # Append lists\n",
        "      cost_train_per_iteration.append(CF_train)\n",
        "      cost_val_per_iteration.append(CF_val)\n",
        "\n",
        "      # Update weight(s)\n",
        "      classifier.update(classifier.X, classifier.y)\n",
        "\n",
        "      # Convergence check\n",
        "      if (i > 10):\n",
        "        if (convergence_check(cost_train_per_iteration)) == True:\n",
        "          print('Classifier', index,'converged at iteration:', i)\n",
        "          not_converged = False\n",
        "          break\n",
        "\n",
        "      # Increment i\n",
        "      i+=1\n",
        "      # if i > 1:\n",
        "      #   not_converged = False\n",
        "      #   break\n",
        "\n",
        "  for inx, classifier in enumerate (classifiers):\n",
        "    weights[inx] = classifier.w\n",
        "\n",
        "  # Make predictions on val set\n",
        "  valset_predictions = np.argmax(classifier.prediction(weights, X_val_scaled), axis = 0)\n",
        "  # Accuracy of current model\n",
        "  accuracy = classifier.accuracy(valset_predictions, y_val)\n",
        "\n",
        "  if accuracy[0] > best_so_far_accuracy:\n",
        "    best_so_far_accuracy = accuracy[0]\n",
        "    best_hyperparameter = params_dict[ind]"
      ],
      "metadata": {
        "id": "cAuGV8LkyVr9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "best_hyperparameter"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ac006bb4-9ce2-4f92-a714-d71166a43b6e",
        "id": "wg3J1CieyVr9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.003, 0.001)"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "best_so_far_accuracy"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9dd940e9-a472-40e2-dc71-4f88852b9076",
        "id": "9-uKOHFJyVr9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8311666666666667"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Conclusion: With this OOP-python-based multi-variate mlogistic regression model, we got an accuracy of 83% on the MNIST validation dataset."
      ],
      "metadata": {
        "id": "Ju66m6vsylEl"
      }
    }
  ]
}