Fine_Tuning_your_model.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyO4VPBwV93ZHI4P50Qqr5Ob",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dr-mushtaq/Machine-Learning/blob/master/Fine_Tuning_your_model.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**Table of Content**</p>"
      ],
      "metadata": {
        "id": "6gnFpKv1Q7cS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "*   Grid Search⭐️\n",
        "*   Random Search⭐️\n",
        "*   Bayesian Optimization⭐️\n",
        "*   Particle Swarm Optimization⭐️\n",
        "*   Hyperopt⭐️"
      ],
      "metadata": {
        "id": "K--v162wRBw4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**1-Grid Search**</p>"
      ],
      "metadata": {
        "id": "pA_LGwxlTOY4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Import Necessary Libraries**"
      ],
      "metadata": {
        "id": "vtCWSkXhThd0"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rnZdTEE9QdA6"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "from sklearn import datasets\n",
        "from sklearn.model_selection import train_test_split, GridSearchCV\n",
        "from sklearn.svm import SVC\n",
        "from sklearn.metrics import classification_report"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Load and Split the Dataset**"
      ],
      "metadata": {
        "id": "vyPh6trnVJ0a"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Load the iris dataset\n",
        "iris = datasets.load_iris()\n",
        "X = iris.data\n",
        "y = iris.target"
      ],
      "metadata": {
        "id": "Bq05RoevVOOi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Split the dataset into training and testing sets\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)"
      ],
      "metadata": {
        "id": "skeg87pWVR9D"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**1-Descion Tree**</p>"
      ],
      "metadata": {
        "id": "JaVEKZXKVZy3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import GridSearchCV\n",
        "from sklearn.tree import DecisionTreeClassifier"
      ],
      "metadata": {
        "id": "MvD_-F-eWQ83"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "n_samples = 24187\n",
        "n_features = 9"
      ],
      "metadata": {
        "id": "6nab1tGmWc6g"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "grid = GridSearchCV(DecisionTreeClassifier(random_state=1), cv=3, n_jobs=-1, verbose=5,\n",
        "                    param_grid ={\n",
        "                    'criterion': ['gini', 'entropy'],\n",
        "                    'max_depth': [None,1,2,3,4,5,6,7],\n",
        "                    'max_features': [None, 'sqrt', 'auto', 'log2', 0.3,0.5,0.7, n_features//2, n_features//3, ],\n",
        "                    'min_samples_split': [2,0.3,0.5, n_samples//2, n_samples//3, n_samples//5],\n",
        "                    'min_samples_leaf':[1, 0.3,0.5, n_samples//2, n_samples//3, n_samples//5]},\n",
        "                    )"
      ],
      "metadata": {
        "id": "kr-fp-RwWVlY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "grid.fit(X_train, y_train)"
      ],
      "metadata": {
        "id": "lTg4QTdhWmeN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print('Train Accuracy : %.3f'%grid.best_estimator_.score(X_train, y_train))\n",
        "print('Test Accuracy : %.3f'%grid.best_estimator_.score(X_test, y_test))\n",
        "print('Best Score Through Grid Search : %.3f'%grid.best_score_)\n",
        "print('Best Parameters : ',grid.best_params_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dEJTzntbW2lo",
        "outputId": "be2dece9-e195-4c54-9e4e-45d6f6f977df"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Train Accuracy : 1.000\n",
            "Test Accuracy : 1.000\n",
            "Best Score Through Grid Search : 0.933\n",
            "Best Parameters :  {'criterion': 'gini', 'max_depth': None, 'max_features': 'sqrt', 'min_samples_leaf': 1, 'min_samples_split': 2}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**2-SVM**</p>"
      ],
      "metadata": {
        "id": "oy9GixunemJz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the model\n",
        "model_svm = SVC()"
      ],
      "metadata": {
        "id": "-A_s50GCeyti"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the parameter grid\n",
        "param_grid_svm = {\n",
        " 'C': [0.1, 1, 10, 100],\n",
        " 'kernel': ['linear', 'poly', 'rbf', 'sigmoid'],\n",
        " 'degree': [2, 3, 4], # Only relevant for 'poly' kernel\n",
        " 'gamma': ['scale', 'auto']\n",
        "}"
      ],
      "metadata": {
        "id": "gd90gz98e2JL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Apply Grid Search**"
      ],
      "metadata": {
        "id": "S5GvTulje7pZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "grid_search_svm = GridSearchCV(estimator=model_svm, param_grid=param_grid_svm, cv=5,scoring='f1_micro', verbose=1, n_jobs=-1)\n"
      ],
      "metadata": {
        "id": "u2eHkBTqe_vh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Fit the model\n",
        "grid_search_svm.fit(X_train, y_train)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 183
        },
        "id": "eaVhQ_jsfr7n",
        "outputId": "9facc9aa-4384-4cac-d3f3-99b40b8f3826"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fitting 5 folds for each of 96 candidates, totalling 480 fits\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(cv=5, estimator=SVC(), n_jobs=-1,\n",
              "             param_grid={'C': [0.1, 1, 10, 100], 'degree': [2, 3, 4],\n",
              "                         'gamma': ['scale', 'auto'],\n",
              "                         'kernel': ['linear', 'poly', 'rbf', 'sigmoid']},\n",
              "             scoring='f1_micro', verbose=1)"
            ],
            "text/html": [
              "<style>#sk-container-id-2 {\n",
              "  /* Definition of color scheme common for light and dark mode */\n",
              "  --sklearn-color-text: #000;\n",
              "  --sklearn-color-text-muted: #666;\n",
              "  --sklearn-color-line: gray;\n",
              "  /* Definition of color scheme for unfitted estimators */\n",
              "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
              "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
              "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
              "  --sklearn-color-unfitted-level-3: chocolate;\n",
              "  /* Definition of color scheme for fitted estimators */\n",
              "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
              "  --sklearn-color-fitted-level-1: #d4ebff;\n",
              "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
              "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
              "\n",
              "  /* Specific color for light theme */\n",
              "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
              "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
              "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
              "  --sklearn-color-icon: #696969;\n",
              "\n",
              "  @media (prefers-color-scheme: dark) {\n",
              "    /* Redefinition of color scheme for dark theme */\n",
              "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
              "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
              "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
              "    --sklearn-color-icon: #878787;\n",
              "  }\n",
              "}\n",
              "\n",
              "#sk-container-id-2 {\n",
              "  color: var(--sklearn-color-text);\n",
              "}\n",
              "\n",
              "#sk-container-id-2 pre {\n",
              "  padding: 0;\n",
              "}\n",
              "\n",
              "#sk-container-id-2 input.sk-hidden--visually {\n",
              "  border: 0;\n",
              "  clip: rect(1px 1px 1px 1px);\n",
              "  clip: rect(1px, 1px, 1px, 1px);\n",
              "  height: 1px;\n",
              "  margin: -1px;\n",
              "  overflow: hidden;\n",
              "  padding: 0;\n",
              "  position: absolute;\n",
              "  width: 1px;\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-dashed-wrapped {\n",
              "  border: 1px dashed var(--sklearn-color-line);\n",
              "  margin: 0 0.4em 0.5em 0.4em;\n",
              "  box-sizing: border-box;\n",
              "  padding-bottom: 0.4em;\n",
              "  background-color: var(--sklearn-color-background);\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-container {\n",
              "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
              "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
              "     so we also need the `!important` here to be able to override the\n",
              "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
              "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
              "  display: inline-block !important;\n",
              "  position: relative;\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-text-repr-fallback {\n",
              "  display: none;\n",
              "}\n",
              "\n",
              "div.sk-parallel-item,\n",
              "div.sk-serial,\n",
              "div.sk-item {\n",
              "  /* draw centered vertical line to link estimators */\n",
              "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
              "  background-size: 2px 100%;\n",
              "  background-repeat: no-repeat;\n",
              "  background-position: center center;\n",
              "}\n",
              "\n",
              "/* Parallel-specific style estimator block */\n",
              "\n",
              "#sk-container-id-2 div.sk-parallel-item::after {\n",
              "  content: \"\";\n",
              "  width: 100%;\n",
              "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
              "  flex-grow: 1;\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-parallel {\n",
              "  display: flex;\n",
              "  align-items: stretch;\n",
              "  justify-content: center;\n",
              "  background-color: var(--sklearn-color-background);\n",
              "  position: relative;\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-parallel-item {\n",
              "  display: flex;\n",
              "  flex-direction: column;\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-parallel-item:first-child::after {\n",
              "  align-self: flex-end;\n",
              "  width: 50%;\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-parallel-item:last-child::after {\n",
              "  align-self: flex-start;\n",
              "  width: 50%;\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-parallel-item:only-child::after {\n",
              "  width: 0;\n",
              "}\n",
              "\n",
              "/* Serial-specific style estimator block */\n",
              "\n",
              "#sk-container-id-2 div.sk-serial {\n",
              "  display: flex;\n",
              "  flex-direction: column;\n",
              "  align-items: center;\n",
              "  background-color: var(--sklearn-color-background);\n",
              "  padding-right: 1em;\n",
              "  padding-left: 1em;\n",
              "}\n",
              "\n",
              "\n",
              "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
              "clickable and can be expanded/collapsed.\n",
              "- Pipeline and ColumnTransformer use this feature and define the default style\n",
              "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
              "*/\n",
              "\n",
              "/* Pipeline and ColumnTransformer style (default) */\n",
              "\n",
              "#sk-container-id-2 div.sk-toggleable {\n",
              "  /* Default theme specific background. It is overwritten whether we have a\n",
              "  specific estimator or a Pipeline/ColumnTransformer */\n",
              "  background-color: var(--sklearn-color-background);\n",
              "}\n",
              "\n",
              "/* Toggleable label */\n",
              "#sk-container-id-2 label.sk-toggleable__label {\n",
              "  cursor: pointer;\n",
              "  display: flex;\n",
              "  width: 100%;\n",
              "  margin-bottom: 0;\n",
              "  padding: 0.5em;\n",
              "  box-sizing: border-box;\n",
              "  text-align: center;\n",
              "  align-items: start;\n",
              "  justify-content: space-between;\n",
              "  gap: 0.5em;\n",
              "}\n",
              "\n",
              "#sk-container-id-2 label.sk-toggleable__label .caption {\n",
              "  font-size: 0.6rem;\n",
              "  font-weight: lighter;\n",
              "  color: var(--sklearn-color-text-muted);\n",
              "}\n",
              "\n",
              "#sk-container-id-2 label.sk-toggleable__label-arrow:before {\n",
              "  /* Arrow on the left of the label */\n",
              "  content: \"▸\";\n",
              "  float: left;\n",
              "  margin-right: 0.25em;\n",
              "  color: var(--sklearn-color-icon);\n",
              "}\n",
              "\n",
              "#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {\n",
              "  color: var(--sklearn-color-text);\n",
              "}\n",
              "\n",
              "/* Toggleable content - dropdown */\n",
              "\n",
              "#sk-container-id-2 div.sk-toggleable__content {\n",
              "  max-height: 0;\n",
              "  max-width: 0;\n",
              "  overflow: hidden;\n",
              "  text-align: left;\n",
              "  /* unfitted */\n",
              "  background-color: var(--sklearn-color-unfitted-level-0);\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-toggleable__content.fitted {\n",
              "  /* fitted */\n",
              "  background-color: var(--sklearn-color-fitted-level-0);\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-toggleable__content pre {\n",
              "  margin: 0.2em;\n",
              "  border-radius: 0.25em;\n",
              "  color: var(--sklearn-color-text);\n",
              "  /* unfitted */\n",
              "  background-color: var(--sklearn-color-unfitted-level-0);\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-toggleable__content.fitted pre {\n",
              "  /* unfitted */\n",
              "  background-color: var(--sklearn-color-fitted-level-0);\n",
              "}\n",
              "\n",
              "#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
              "  /* Expand drop-down */\n",
              "  max-height: 200px;\n",
              "  max-width: 100%;\n",
              "  overflow: auto;\n",
              "}\n",
              "\n",
              "#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
              "  content: \"▾\";\n",
              "}\n",
              "\n",
              "/* Pipeline/ColumnTransformer-specific style */\n",
              "\n",
              "#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
              "  color: var(--sklearn-color-text);\n",
              "  background-color: var(--sklearn-color-unfitted-level-2);\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
              "  background-color: var(--sklearn-color-fitted-level-2);\n",
              "}\n",
              "\n",
              "/* Estimator-specific style */\n",
              "\n",
              "/* Colorize estimator box */\n",
              "#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
              "  /* unfitted */\n",
              "  background-color: var(--sklearn-color-unfitted-level-2);\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
              "  /* fitted */\n",
              "  background-color: var(--sklearn-color-fitted-level-2);\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-label label.sk-toggleable__label,\n",
              "#sk-container-id-2 div.sk-label label {\n",
              "  /* The background is the default theme color */\n",
              "  color: var(--sklearn-color-text-on-default-background);\n",
              "}\n",
              "\n",
              "/* On hover, darken the color of the background */\n",
              "#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {\n",
              "  color: var(--sklearn-color-text);\n",
              "  background-color: var(--sklearn-color-unfitted-level-2);\n",
              "}\n",
              "\n",
              "/* Label box, darken color on hover, fitted */\n",
              "#sk-container-id-2 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
              "  color: var(--sklearn-color-text);\n",
              "  background-color: var(--sklearn-color-fitted-level-2);\n",
              "}\n",
              "\n",
              "/* Estimator label */\n",
              "\n",
              "#sk-container-id-2 div.sk-label label {\n",
              "  font-family: monospace;\n",
              "  font-weight: bold;\n",
              "  display: inline-block;\n",
              "  line-height: 1.2em;\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-label-container {\n",
              "  text-align: center;\n",
              "}\n",
              "\n",
              "/* Estimator-specific */\n",
              "#sk-container-id-2 div.sk-estimator {\n",
              "  font-family: monospace;\n",
              "  border: 1px dotted var(--sklearn-color-border-box);\n",
              "  border-radius: 0.25em;\n",
              "  box-sizing: border-box;\n",
              "  margin-bottom: 0.5em;\n",
              "  /* unfitted */\n",
              "  background-color: var(--sklearn-color-unfitted-level-0);\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-estimator.fitted {\n",
              "  /* fitted */\n",
              "  background-color: var(--sklearn-color-fitted-level-0);\n",
              "}\n",
              "\n",
              "/* on hover */\n",
              "#sk-container-id-2 div.sk-estimator:hover {\n",
              "  /* unfitted */\n",
              "  background-color: var(--sklearn-color-unfitted-level-2);\n",
              "}\n",
              "\n",
              "#sk-container-id-2 div.sk-estimator.fitted:hover {\n",
              "  /* fitted */\n",
              "  background-color: var(--sklearn-color-fitted-level-2);\n",
              "}\n",
              "\n",
              "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
              "\n",
              "/* Common style for \"i\" and \"?\" */\n",
              "\n",
              ".sk-estimator-doc-link,\n",
              "a:link.sk-estimator-doc-link,\n",
              "a:visited.sk-estimator-doc-link {\n",
              "  float: right;\n",
              "  font-size: smaller;\n",
              "  line-height: 1em;\n",
              "  font-family: monospace;\n",
              "  background-color: var(--sklearn-color-background);\n",
              "  border-radius: 1em;\n",
              "  height: 1em;\n",
              "  width: 1em;\n",
              "  text-decoration: none !important;\n",
              "  margin-left: 0.5em;\n",
              "  text-align: center;\n",
              "  /* unfitted */\n",
              "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
              "  color: var(--sklearn-color-unfitted-level-1);\n",
              "}\n",
              "\n",
              ".sk-estimator-doc-link.fitted,\n",
              "a:link.sk-estimator-doc-link.fitted,\n",
              "a:visited.sk-estimator-doc-link.fitted {\n",
              "  /* fitted */\n",
              "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
              "  color: var(--sklearn-color-fitted-level-1);\n",
              "}\n",
              "\n",
              "/* On hover */\n",
              "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
              ".sk-estimator-doc-link:hover,\n",
              "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
              ".sk-estimator-doc-link:hover {\n",
              "  /* unfitted */\n",
              "  background-color: var(--sklearn-color-unfitted-level-3);\n",
              "  color: var(--sklearn-color-background);\n",
              "  text-decoration: none;\n",
              "}\n",
              "\n",
              "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
              ".sk-estimator-doc-link.fitted:hover,\n",
              "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
              ".sk-estimator-doc-link.fitted:hover {\n",
              "  /* fitted */\n",
              "  background-color: var(--sklearn-color-fitted-level-3);\n",
              "  color: var(--sklearn-color-background);\n",
              "  text-decoration: none;\n",
              "}\n",
              "\n",
              "/* Span, style for the box shown on hovering the info icon */\n",
              ".sk-estimator-doc-link span {\n",
              "  display: none;\n",
              "  z-index: 9999;\n",
              "  position: relative;\n",
              "  font-weight: normal;\n",
              "  right: .2ex;\n",
              "  padding: .5ex;\n",
              "  margin: .5ex;\n",
              "  width: min-content;\n",
              "  min-width: 20ex;\n",
              "  max-width: 50ex;\n",
              "  color: var(--sklearn-color-text);\n",
              "  box-shadow: 2pt 2pt 4pt #999;\n",
              "  /* unfitted */\n",
              "  background: var(--sklearn-color-unfitted-level-0);\n",
              "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
              "}\n",
              "\n",
              ".sk-estimator-doc-link.fitted span {\n",
              "  /* fitted */\n",
              "  background: var(--sklearn-color-fitted-level-0);\n",
              "  border: var(--sklearn-color-fitted-level-3);\n",
              "}\n",
              "\n",
              ".sk-estimator-doc-link:hover span {\n",
              "  display: block;\n",
              "}\n",
              "\n",
              "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
              "\n",
              "#sk-container-id-2 a.estimator_doc_link {\n",
              "  float: right;\n",
              "  font-size: 1rem;\n",
              "  line-height: 1em;\n",
              "  font-family: monospace;\n",
              "  background-color: var(--sklearn-color-background);\n",
              "  border-radius: 1rem;\n",
              "  height: 1rem;\n",
              "  width: 1rem;\n",
              "  text-decoration: none;\n",
              "  /* unfitted */\n",
              "  color: var(--sklearn-color-unfitted-level-1);\n",
              "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
              "}\n",
              "\n",
              "#sk-container-id-2 a.estimator_doc_link.fitted {\n",
              "  /* fitted */\n",
              "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
              "  color: var(--sklearn-color-fitted-level-1);\n",
              "}\n",
              "\n",
              "/* On hover */\n",
              "#sk-container-id-2 a.estimator_doc_link:hover {\n",
              "  /* unfitted */\n",
              "  background-color: var(--sklearn-color-unfitted-level-3);\n",
              "  color: var(--sklearn-color-background);\n",
              "  text-decoration: none;\n",
              "}\n",
              "\n",
              "#sk-container-id-2 a.estimator_doc_link.fitted:hover {\n",
              "  /* fitted */\n",
              "  background-color: var(--sklearn-color-fitted-level-3);\n",
              "}\n",
              "</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=5, estimator=SVC(), n_jobs=-1,\n",
              "             param_grid={&#x27;C&#x27;: [0.1, 1, 10, 100], &#x27;degree&#x27;: [2, 3, 4],\n",
              "                         &#x27;gamma&#x27;: [&#x27;scale&#x27;, &#x27;auto&#x27;],\n",
              "                         &#x27;kernel&#x27;: [&#x27;linear&#x27;, &#x27;poly&#x27;, &#x27;rbf&#x27;, &#x27;sigmoid&#x27;]},\n",
              "             scoring=&#x27;f1_micro&#x27;, verbose=1)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow\"><div><div>GridSearchCV</div></div><div><a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.6/modules/generated/sklearn.model_selection.GridSearchCV.html\">?<span>Documentation for GridSearchCV</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></div></label><div class=\"sk-toggleable__content fitted\"><pre>GridSearchCV(cv=5, estimator=SVC(), n_jobs=-1,\n",
              "             param_grid={&#x27;C&#x27;: [0.1, 1, 10, 100], &#x27;degree&#x27;: [2, 3, 4],\n",
              "                         &#x27;gamma&#x27;: [&#x27;scale&#x27;, &#x27;auto&#x27;],\n",
              "                         &#x27;kernel&#x27;: [&#x27;linear&#x27;, &#x27;poly&#x27;, &#x27;rbf&#x27;, &#x27;sigmoid&#x27;]},\n",
              "             scoring=&#x27;f1_micro&#x27;, verbose=1)</pre></div> </div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow\"><div><div>best_estimator_: SVC</div></div></label><div class=\"sk-toggleable__content fitted\"><pre>SVC(C=1, kernel=&#x27;poly&#x27;)</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow\"><div><div>SVC</div></div><div><a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.6/modules/generated/sklearn.svm.SVC.html\">?<span>Documentation for SVC</span></a></div></label><div class=\"sk-toggleable__content fitted\"><pre>SVC(C=1, kernel=&#x27;poly&#x27;)</pre></div> </div></div></div></div></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print('Train Accuracy : %.3f'%grid_search_svm.best_estimator_.score(X_train, y_train))\n",
        "print('Test Accuracy : %.3f'%grid_search_svm.best_estimator_.score(X_test, y_test))\n",
        "print('Best Score Through Grid Search : %.3f'%grid_search_svm.best_score_)\n",
        "print('Best Parameters : ',grid_search_svm.best_params_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RP-WEc8_iN4r",
        "outputId": "505b2c2e-209a-43e4-be68-ef17ec9777c8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Train Accuracy : 0.981\n",
            "Test Accuracy : 0.978\n",
            "Best Score Through Grid Search : 0.971\n",
            "Best Parameters :  {'C': 1, 'degree': 3, 'gamma': 'scale', 'kernel': 'poly'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**2-Random Search**</p>"
      ],
      "metadata": {
        "id": "URtzYi394alD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn.model_selection import RandomizedSearchCV\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.datasets import load_iris\n",
        "from sklearn.metrics import accuracy_score"
      ],
      "metadata": {
        "id": "U1xM8dJ440uM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Load the Iris dataset\n",
        "data = load_iris()\n",
        "X = data.data\n",
        "y = data.target\n",
        "# Display basic information\n",
        "print(f\"Feature names: {data.feature_names}\")\n",
        "print(f\"Target classes: {data.target_names}\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P4yxZw4J40wH",
        "outputId": "34c55ff3-3c05-4f3a-deab-284d3b502fc6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Feature names: ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']\n",
            "Target classes: ['setosa' 'versicolor' 'virginica']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**RF**</p>"
      ],
      "metadata": {
        "id": "YW9IFz5m5AXb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Initialize the model\n",
        "model = RandomForestClassifier(random_state=42)"
      ],
      "metadata": {
        "id": "VfCBUepL5PW7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Parameter grid for Random Search\n",
        "param_dist = {\n",
        " 'n_estimators': [10, 50, 100, 200],\n",
        " 'max_depth': [None, 10, 20, 30, 40],\n",
        " 'min_samples_split': [2, 5, 10],\n",
        " 'min_samples_leaf': [1, 2, 4],\n",
        " 'bootstrap': [True, False]\n",
        "}"
      ],
      "metadata": {
        "id": "BAwRo6XK5Tsv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "# Split data into training and testing sets\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
      ],
      "metadata": {
        "id": "Ktmlbr6d5bzn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# RandomizedSearchCV setup\n",
        "random_search = RandomizedSearchCV(\n",
        " estimator=model,\n",
        " param_distributions=param_dist,\n",
        " n_iter=50, # Number of parameter combinations to try\n",
        " cv=3, # Number of cross-validation folds\n",
        " verbose=2,\n",
        " random_state=42,\n",
        " n_jobs=-1 # Use all processors\n",
        ")\n",
        "# Fit the model\n",
        "random_search.fit(X_train, y_train)\n",
        "# Display the best parameters\n",
        "print(\"Best Parameters:\", random_search.best_params_)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MpDES7985iX0",
        "outputId": "0d37507d-1142-4c7f-ef0b-b87a3948b148"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fitting 3 folds for each of 50 candidates, totalling 150 fits\n",
            "Best Parameters: {'n_estimators': 50, 'min_samples_split': 10, 'min_samples_leaf': 4, 'max_depth': 20, 'bootstrap': False}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**3-Bayesian Optimization**</p>"
      ],
      "metadata": {
        "id": "Wj3CTdx-6bIi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install scikit-learn scikit-optimize\n"
      ],
      "metadata": {
        "id": "dqPMXJFd6sgG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.datasets import load_iris\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.metrics import accuracy_score\n",
        "# Load Iris dataset\n",
        "iris = load_iris()\n",
        "X, y = iris.data, iris.target\n",
        "# Split into train and test sets\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
      ],
      "metadata": {
        "id": "HtpaToTb60Ap"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**RF**</p>"
      ],
      "metadata": {
        "id": "VHvmTDGy7x5T"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import cross_val_score\n",
        "from skopt import BayesSearchCV\n",
        "# Define the model\n",
        "model = RandomForestClassifier(random_state=42)\n",
        "# Define the parameter search space\n",
        "param_space = {\n",
        " 'n_estimators': (10, 200), # Number of trees\n",
        " 'max_depth': (1, 20), # Depth of trees\n",
        " 'min_samples_split': (2, 10), # Minimum samples to split\n",
        " 'min_samples_leaf': (1, 10) # Minimum samples per leaf\n",
        "}\n"
      ],
      "metadata": {
        "id": "LoPe15XX60Ci"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Define Bayesian optimization\n",
        "opt = BayesSearchCV(\n",
        " estimator=model,\n",
        " search_spaces=param_space,\n",
        " n_iter=32, # Number of iterations\n",
        " cv=3, # Cross-validation folds\n",
        " scoring='accuracy', # Scoring metric\n",
        " random_state=42\n",
        ")\n"
      ],
      "metadata": {
        "id": "md7vLvvu7LWf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Fit the optimizer\n",
        "opt.fit(X_train, y_train)\n",
        "# Print best hyperparameters and corresponding accuracy\n",
        "print(\"Best Hyperparameters:\", opt.best_params_)\n",
        "print(\"Best Cross-Validation Score:\", opt.best_score_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2kBdyn4W7RZd",
        "outputId": "fefa7aff-f9e9-44c6-97e4-aa4e8b70931e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Best Hyperparameters: OrderedDict([('max_depth', 19), ('min_samples_leaf', 3), ('min_samples_split', 8), ('n_estimators', 11)])\n",
            "Best Cross-Validation Score: 0.9666666666666667\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Train the model with best parameters\n",
        "best_model = opt.best_estimator_\n",
        "# Make predictions on the test set\n",
        "y_pred = best_model.predict(X_test)\n",
        "# Evaluate performance\n",
        "test_accuracy = accuracy_score(y_test, y_pred)\n",
        "print(\"Test Accuracy:\", test_accuracy)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lirdEptk7LYl",
        "outputId": "f84534b8-7a06-456f-f82c-e56f77545ee6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Test Accuracy: 1.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**4-Particle Swarm Optimization**"
      ],
      "metadata": {
        "id": "byzdO5Kq8P6r"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install pyswarm"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YyOErsIH8-2n",
        "outputId": "fc2ab96a-3acb-4681-8d7b-3954fcd23264"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting pyswarm\n",
            "  Downloading pyswarm-0.6.tar.gz (4.3 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.11/dist-packages (from pyswarm) (1.26.4)\n",
            "Building wheels for collected packages: pyswarm\n",
            "  Building wheel for pyswarm (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyswarm: filename=pyswarm-0.6-py3-none-any.whl size=4464 sha256=d395aee3a2ec4674f306294e6f30e20b58f3fd9e2e998d6654d174e9a740dd91\n",
            "  Stored in directory: /root/.cache/pip/wheels/bb/4f/ec/8970b83323e16aa95034da175454843947376614d6d5e9627f\n",
            "Successfully built pyswarm\n",
            "Installing collected packages: pyswarm\n",
            "Successfully installed pyswarm-0.6\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn.datasets import load_iris\n",
        "from sklearn.model_selection import train_test_split, cross_val_score\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from pyswarm import pso # PSO library"
      ],
      "metadata": {
        "id": "26LuIPKo8mqA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Load the Iris dataset\n",
        "data = load_iris()\n",
        "X = data.data\n",
        "y = data.target\n",
        "# Split into training and test sets\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)"
      ],
      "metadata": {
        "id": "U4hjHUq39Pn9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**RF**</p>"
      ],
      "metadata": {
        "id": "AssweVkT_MBE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def objective_function(params):\n",
        " n_estimators, max_depth = int(params[0]), int(params[1])\n",
        " # Define the RandomForestClassifier\n",
        " model = RandomForestClassifier(n_estimators=n_estimators, max_depth=max_depth, random_state=42)\n",
        " # Perform cross-validation and calculate accuracy\n",
        " scores = cross_val_score(model, X_train, y_train, cv=5)\n",
        " accuracy = scores.mean()\n",
        " # Since PSO minimizes the objective, return -accuracy\n",
        " return -accuracy"
      ],
      "metadata": {
        "id": "Dh54quda9XOe"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Define bounds for hyperparameters\n",
        "lower_bounds = [10, 1] # Minimum values for n_estimators and max_depth\n",
        "upper_bounds = [200, 20] # Maximum values for n_estimators and max_depth\n",
        "# Run Particle Swarm Optimization\n",
        "best_params, best_score = pso(objective_function, lower_bounds, upper_bounds, swarmsize=20, maxiter=30)\n",
        "# Print the results\n",
        "print(f\"Best Hyperparameters: n_estimators={int(best_params[0])}, max_depth={int(best_params[1])}\")\n",
        "print(f\"Best Cross-Validation Accuracy: {-best_score}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "awGMnSz89eAV",
        "outputId": "02de89b9-9827-43f5-d065-747f9891f338"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Stopping search: maximum iterations reached --> 30\n",
            "Best Hyperparameters: n_estimators=18, max_depth=4\n",
            "Best Cross-Validation Accuracy: 0.9523809523809523\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Train the RandomForestClassifier with optimal hyperparameters\n",
        "optimal_n_estimators = int(best_params[0])\n",
        "optimal_max_depth = int(best_params[1])\n",
        "final_model = RandomForestClassifier(n_estimators=optimal_n_estimators, max_depth=optimal_max_depth, random_state=42)\n",
        "final_model.fit(X_train, y_train)\n",
        "# Evaluate the model on the test set\n",
        "test_accuracy = final_model.score(X_test, y_test)\n",
        "print(f\"Test Set Accuracy: {test_accuracy}\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vPlI_s9e9wyI",
        "outputId": "38079a5e-d6b6-4c85-8548-0f8bd67c228c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Test Set Accuracy: 1.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**5-Hyperopt**"
      ],
      "metadata": {
        "id": "jEYLIr6z6pd_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from hyperopt import fmin, tpe, hp, STATUS_OK, Trials\n",
        "from sklearn.datasets import load_iris\n",
        "from sklearn.model_selection import cross_val_score\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "import numpy as np\n",
        "\n",
        "data = load_iris()\n",
        "X, y = data.data, data.target\n",
        "\n",
        "def objective(params):\n",
        "    # Ensure that 'max_depth', 'min_samples_split', and 'min_samples_leaf' are integers\n",
        "    params['max_depth'] = int(params['max_depth'])\n",
        "    params['min_samples_split'] = int(params['min_samples_split'])\n",
        "    params['min_samples_leaf'] = int(params['min_samples_leaf'])\n",
        "\n",
        "    clf = DecisionTreeClassifier(**params)\n",
        "    accuracy = cross_val_score(clf, X, y, cv=5, scoring='accuracy').mean()\n",
        "    return {'loss': -accuracy, 'status': STATUS_OK}\n",
        "\n",
        "space = {\n",
        "    'criterion': hp.choice('criterion', ['gini', 'entropy']),\n",
        "    # Use hp.randint to ensure integer values for max_depth\n",
        "    'max_depth': hp.randint('max_depth', 1, 11), # from 1 up to, but not including, 11\n",
        "    # Use hp.randint to ensure integer values for min_samples_split\n",
        "    'min_samples_split': hp.randint('min_samples_split', 2, 11), # from 2 up to, but not including, 11\n",
        "    # Use hp.randint to ensure integer values for min_samples_leaf\n",
        "    'min_samples_leaf': hp.randint('min_samples_leaf', 1, 6) # from 1 up to, but not including, 6\n",
        "}\n",
        "\n",
        "# Run the optimization\n",
        "trials = Trials()\n",
        "best = fmin(\n",
        "    fn=objective,\n",
        "    space=space,\n",
        "    algo=tpe.suggest,\n",
        "    max_evals=50,\n",
        "    trials=trials\n",
        ")\n",
        "\n",
        "print(best)"
      ],
      "metadata": {
        "id": "dSjb3quE59IL",
        "outputId": "bea63c9a-753e-4510-88c2-3ef2924bdd53",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "100%|██████████| 50/50 [00:01<00:00, 29.92trial/s, best loss: -0.9733333333333334]\n",
            "{'criterion': 0, 'max_depth': 3, 'min_samples_leaf': 1, 'min_samples_split': 2}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Convert discrete values to integers\n",
        "best['max_depth'] = int(best['max_depth'])\n",
        "best['min_samples_split'] = int(best['min_samples_split'])\n",
        "best['min_samples_leaf'] = int(best['min_samples_leaf'])"
      ],
      "metadata": {
        "id": "tLBQWypg6Fw8"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"Best Hyperparameters:\", best)"
      ],
      "metadata": {
        "id": "l2x4CLll6Lhe",
        "outputId": "01e10019-cf09-403d-9e29-3086419c2ed5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Best Hyperparameters: {'criterion': 0, 'max_depth': 3, 'min_samples_leaf': 1, 'min_samples_split': 2}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**References**</p>"
      ],
      "metadata": {
        "id": "PrwQyG2pQp3F"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "1.   [Hyperparameter Optimization of Machine Learning Algorithms\n",
        "](https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms)\n",
        "2.   List item\n",
        "\n"
      ],
      "metadata": {
        "id": "gpesJxDbADsl"
      }
    }
  ]
}