t81_558_class_08_4_bayesian_hyperparameter_opt.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CcROQcVVoUiH"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/app_deep_learning/blob/main/t81_558_class_08_4_bayesian_hyperparameter_opt.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Di3an3U1oUiJ"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "\n",
    "**Module 8: Kaggle Data Sets**\n",
    "\n",
    "- Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "- For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "V1W44MImoUiJ"
   },
   "source": [
    "# Module 8 Material\n",
    "\n",
    "* Part 8.1: Introduction to Kaggle [[Video]](https://www.youtube.com/watch?v=7Mk46fb0Ayg&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_1_kaggle_intro.ipynb)\n",
    "* Part 8.2: Building Ensembles with Scikit-Learn and PyTorch [[Video]](https://www.youtube.com/watch?v=przbLRCRL24&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_2_pytorch_ensembles.ipynb)\n",
    "* Part 8.3: How Should you Architect Your PyTorch Neural Network: Hyperparameters [[Video]](https://www.youtube.com/watch?v=YTL2BR4U2Ng&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_3_pytorch_hyperparameters.ipynb)\n",
    "* **Part 8.4: Bayesian Hyperparameter Optimization for PyTorch** [[Video]](https://www.youtube.com/watch?v=1f4psgAcefU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_4_bayesian_hyperparameter_opt.ipynb)\n",
    "* Part 8.5: Current Semester's Kaggle [[Video]] [[Notebook]](t81_558_class_08_5_kaggle_project.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ISBSbu8XoUiK"
   },
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SzfTEj7foUiK",
    "outputId": "daf55884-f88a-4d7b-f96e-d1285ad2de79"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: using Google CoLab\n",
      "Using device: mps\n"
     ]
    }
   ],
   "source": [
    "# Startup Google CoLab\n",
    "try:\n",
    "    import google.colab\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False\n",
    "\n",
    "# Nicely formatted time string\n",
    "\n",
    "\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return \"{}:{:>02}:{:>05.2f}\".format(h, m, s)\n",
    "\n",
    "\n",
    "# Make use of a GPU or MPS (Apple) if one is available.  (see module 3.2)\n",
    "import torch\n",
    "has_mps = torch.backends.mps.is_built()\n",
    "device = \"mps\" if has_mps else \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "print(f\"Using device: {device}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WydBc__qoUiL"
   },
   "source": [
    "# Part 8.4: Bayesian Hyperparameter Optimization for PyTorch\n",
    "\n",
    "Bayesian Hyperparameter Optimization is a method of finding hyperparameters more efficiently than a grid search. Because each candidate set of hyperparameters requires a retraining of the neural network, it is best to keep the number of candidate sets to a minimum. Bayesian Hyperparameter Optimization achieves this by training a model to predict good candidate sets of hyperparameters. [[Cite:snoek2012practical]](https://arxiv.org/pdf/1206.2944.pdf)\n",
    "\n",
    "- [bayesian-optimization](https://github.com/fmfn/BayesianOptimization)\n",
    "- [hyperopt](https://github.com/hyperopt/hyperopt)\n",
    "- [spearmint](https://github.com/JasperSnoek/spearmint)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "NNKv5zWgoUiL"
   },
   "outputs": [],
   "source": [
    "from scipy.stats import zscore\n",
    "import pandas as pd\n",
    "import logging\n",
    "import os\n",
    "logging.disable(logging.WARNING)\n",
    "\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat(\n",
    "    [df, pd.get_dummies(df['job'], prefix=\"job\", dtype=int)], axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat(\n",
    "    [df, pd.get_dummies(df['area'], prefix=\"area\", dtype=int)], axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['age'] = zscore(df['age'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('product').drop('id')\n",
    "x = df[x_columns].values\n",
    "dummies = pd.get_dummies(df['product'])  # Classification\n",
    "products = dummies.columns\n",
    "y = dummies.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9qetzyCcoUiM"
   },
   "source": [
    "Now that we've preprocessed the data, we can begin the hyperparameter optimization. We start by creating a function that generates the model based on just three parameters. Bayesian optimization works on a vector of numbers, not on a problematic notion like how many layers and neurons are on each layer. To represent this complex neuron structure as a vector, we use several numbers to describe this structure.\n",
    "\n",
    "- **dropout** - The dropout percent for each layer.\n",
    "- **neuronPct** - What percent of our fixed 5,000 maximum number of neurons do we wish to use? This parameter specifies the total count of neurons in the entire network.\n",
    "- **neuronShrink** - Neural networks usually start with more neurons on the first hidden layer and then decrease this count for additional layers. This percent specifies how much to shrink subsequent layers based on the previous layer. We stop adding more layers once we run out of neurons (the count specified by neuronPct).\n",
    "\n",
    "These three numbers define the structure of the neural network. The commends in the below code show exactly how the program constructs the network.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "d6j5HvCqoUiM"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import time\n",
    "import statistics\n",
    "from scipy.stats import zscore\n",
    "from sklearn import metrics\n",
    "from sklearn.model_selection import StratifiedShuffleSplit\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.utils.data import DataLoader, TensorDataset\n",
    "\n",
    "# Convert data to PyTorch tensors\n",
    "x_tensor = torch.FloatTensor(x).to(device)\n",
    "y_tensor = torch.LongTensor(np.argmax(y, axis=1)).to(\n",
    "    device)  # Convert one-hot to index\n",
    "\n",
    "\n",
    "class NeuralNetwork(nn.Module):\n",
    "    def __init__(self, input_dim, dropout, neuronPct, neuronShrink):\n",
    "        super(NeuralNetwork, self).__init__()\n",
    "\n",
    "        layers = []\n",
    "        neuronCount = int(neuronPct * 5000)\n",
    "        layer = 0\n",
    "\n",
    "        prev_count = input_dim\n",
    "        while neuronCount > 25 and layer < 10:\n",
    "            layers.append(nn.Linear(prev_count, neuronCount))\n",
    "            prev_count = neuronCount\n",
    "            layers.append(nn.PReLU())\n",
    "            layers.append(nn.Dropout(dropout))\n",
    "            neuronCount = int(neuronCount * neuronShrink)\n",
    "            layer += 1\n",
    "\n",
    "        layers.append(nn.Linear(prev_count, y.shape[1]))\n",
    "        layers.append(nn.Softmax(dim=1))\n",
    "        self.model = nn.Sequential(*layers)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.model(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LGfXROENoUiN"
   },
   "source": [
    "We can test this code to see how it creates a neural network based on three such parameters.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nCQC86b_oUiN",
    "outputId": "992541aa-360a-4b19-80d9-0cf01a515256"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NeuralNetwork(\n",
      "  (model): Sequential(\n",
      "    (0): Linear(in_features=47, out_features=500, bias=True)\n",
      "    (1): PReLU(num_parameters=1)\n",
      "    (2): Dropout(p=0.2, inplace=False)\n",
      "    (3): Linear(in_features=500, out_features=125, bias=True)\n",
      "    (4): PReLU(num_parameters=1)\n",
      "    (5): Dropout(p=0.2, inplace=False)\n",
      "    (6): Linear(in_features=125, out_features=31, bias=True)\n",
      "    (7): PReLU(num_parameters=1)\n",
      "    (8): Dropout(p=0.2, inplace=False)\n",
      "    (9): Linear(in_features=31, out_features=7, bias=True)\n",
      "    (10): Softmax(dim=1)\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Create and print the model\n",
    "model = NeuralNetwork(x.shape[1], 0.2, 0.1, 0.25).to(device)\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vgY6GprqoUiN"
   },
   "source": [
    "We will now create a function to evaluate the neural network using three such parameters. We use bootstrapping because one training run might have \"bad luck\" with the assigned random weights. We use this function to train and then evaluate the neural network.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "kbX4k-ynoUiN"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-4.381666020591725\n"
     ]
    }
   ],
   "source": [
    "# Evaluation function\n",
    "SPLITS = 2\n",
    "EPOCHS = 500\n",
    "PATIENCE = 10\n",
    "\n",
    "def evaluate_network(learning_rate=1e-3,dropout=0.2,\n",
    "                        neuronPct=0.1, neuronShrink=0.25):\n",
    "    \n",
    "    boot = StratifiedShuffleSplit(n_splits=SPLITS, test_size=0.1)\n",
    "    mean_benchmark = []\n",
    "    epochs_needed = []\n",
    "\n",
    "    for train, test in boot.split(x, np.argmax(y, axis=1)):\n",
    "        x_train = x_tensor[train]\n",
    "        y_train = y_tensor[train]\n",
    "        x_test = x_tensor[test]\n",
    "        y_test = y_tensor[test]\n",
    "\n",
    "        model = NeuralNetwork(x.shape[1],  \n",
    "                        dropout=dropout,\n",
    "                        neuronPct=neuronPct,\n",
    "                        neuronShrink=neuronShrink).to(device)\n",
    "        criterion = nn.CrossEntropyLoss()\n",
    "        optimizer = optim.Adam(model.parameters(), lr=learning_rate)\n",
    "\n",
    "        dataset_train = TensorDataset(x_train, y_train)\n",
    "        loader_train = DataLoader(dataset_train, batch_size=32, shuffle=True)\n",
    "\n",
    "        best_loss = float('inf')\n",
    "        patience_counter = 0\n",
    "\n",
    "        for epoch in range(EPOCHS):\n",
    "            model.train()\n",
    "            for batch_x, batch_y in loader_train:\n",
    "                optimizer.zero_grad()\n",
    "                outputs = model(batch_x)\n",
    "                loss = criterion(outputs, batch_y)\n",
    "                loss.backward()\n",
    "                optimizer.step()\n",
    "\n",
    "            model.eval()\n",
    "            with torch.no_grad():\n",
    "                outputs_test = model(x_test)\n",
    "                val_loss = criterion(outputs_test, y_test).item()\n",
    "\n",
    "            if val_loss < best_loss:\n",
    "                best_loss = val_loss\n",
    "                patience_counter = 0\n",
    "            else:\n",
    "                patience_counter += 1\n",
    "\n",
    "            if patience_counter >= PATIENCE:\n",
    "                epochs_needed.append(epoch)\n",
    "                break\n",
    "\n",
    "        # Evaluate\n",
    "        with torch.no_grad():\n",
    "            model.eval()\n",
    "            # Move predictions to CPU for evaluation\n",
    "            pred = model(x_test).cpu().numpy()\n",
    "            y_compare = y_test.cpu().numpy()\n",
    "            score = metrics.log_loss(y_compare, pred)\n",
    "            mean_benchmark.append(score)\n",
    "\n",
    "    return -statistics.mean(mean_benchmark)\n",
    "\n",
    "print(evaluate_network(learning_rate=1e-3, \n",
    "                       dropout=0.2,\n",
    "                      neuronPct=0.1, \n",
    "                      neuronShrink=0.25))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eYMNfmXPoUiN"
   },
   "source": [
    "You can try any combination of our three hyperparameters, plus the learning rate, to see how effective these four numbers are. Of course, our goal is not to manually choose different combinations of these four hyperparameters; we seek to automate.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZpyYvzKnoUiN",
    "outputId": "7b03f355-2a76-498d-a37a-fa06631f0a7f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-5.754028869647901\n"
     ]
    }
   ],
   "source": [
    "print(evaluate_network(\n",
    "    dropout=0.2,\n",
    "    neuronPct=0.1, \n",
    "    neuronShrink=0.25))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b05ssrH0qfdz"
   },
   "source": [
    "First, we must install the Bayesian optimization package if we are in Colab.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "S2iFisjIqjvO",
    "outputId": "a41238ba-deb5-47fe-fc98-6a505fac9021"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: bayesian-optimization in /Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages (1.4.3)\n",
      "Requirement already satisfied: numpy>=1.9.0 in /Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages (from bayesian-optimization) (1.25.1)\n",
      "Requirement already satisfied: scipy>=1.0.0 in /Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages (from bayesian-optimization) (1.11.1)\n",
      "Requirement already satisfied: scikit-learn>=0.18.0 in /Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages (from bayesian-optimization) (1.3.0)\n",
      "Requirement already satisfied: colorama>=0.4.6 in /Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages (from bayesian-optimization) (0.4.6)\n",
      "Requirement already satisfied: joblib>=1.1.1 in /Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages (from scikit-learn>=0.18.0->bayesian-optimization) (1.3.0)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages (from scikit-learn>=0.18.0->bayesian-optimization) (3.2.0)\n"
     ]
    }
   ],
   "source": [
    "# HIDE OUTPUT\n",
    "!pip install bayesian-optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Vh3nx_SDoUiO"
   },
   "source": [
    "We will now automate this process. We define the bounds for each of these four hyperparameters and begin the Bayesian optimization. Once the program finishes, the best combination of hyperparameters found is displayed. The **optimize** function accepts two parameters that will significantly impact how long the process takes to complete:\n",
    "\n",
    "- **n_iter** - How many steps of Bayesian optimization that you want to perform. The more steps, the more likely you will find a reasonable maximum.\n",
    "- **init_points**: How many steps of random exploration that you want to perform. Random exploration can help by diversifying the exploration space.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FiymzM63oUiO",
    "outputId": "278deaac-2c57-4929-b739-5685ddab06dd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|   iter    |  target   |  dropout  | learni... | neuronPct | neuron... |\n",
      "-------------------------------------------------------------------------\n",
      "| \u001b[0m1        \u001b[0m | \u001b[0m-9.167   \u001b[0m | \u001b[0m0.2081   \u001b[0m | \u001b[0m0.07203  \u001b[0m | \u001b[0m0.01011  \u001b[0m | \u001b[0m0.3093   \u001b[0m |\n",
      "| \u001b[95m2        \u001b[0m | \u001b[95m-8.29    \u001b[0m | \u001b[95m0.07323  \u001b[0m | \u001b[95m0.009234 \u001b[0m | \u001b[95m0.1944   \u001b[0m | \u001b[95m0.3521   \u001b[0m |\n",
      "| \u001b[0m3        \u001b[0m | \u001b[0m-9.167   \u001b[0m | \u001b[0m0.198    \u001b[0m | \u001b[0m0.05388  \u001b[0m | \u001b[0m0.425    \u001b[0m | \u001b[0m0.6884   \u001b[0m |\n",
      "| \u001b[0m4        \u001b[0m | \u001b[0m-9.167   \u001b[0m | \u001b[0m0.102    \u001b[0m | \u001b[0m0.08781  \u001b[0m | \u001b[0m0.03711  \u001b[0m | \u001b[0m0.6738   \u001b[0m |\n",
      "| \u001b[0m5        \u001b[0m | \u001b[0m-10.04   \u001b[0m | \u001b[0m0.2082   \u001b[0m | \u001b[0m0.05587  \u001b[0m | \u001b[0m0.149    \u001b[0m | \u001b[0m0.2061   \u001b[0m |\n",
      "| \u001b[0m6        \u001b[0m | \u001b[0m-12.71   \u001b[0m | \u001b[0m0.3996   \u001b[0m | \u001b[0m0.09683  \u001b[0m | \u001b[0m0.3203   \u001b[0m | \u001b[0m0.6954   \u001b[0m |\n",
      "| \u001b[0m7        \u001b[0m | \u001b[0m-8.29    \u001b[0m | \u001b[0m0.4373   \u001b[0m | \u001b[0m0.08946  \u001b[0m | \u001b[0m0.09419  \u001b[0m | \u001b[0m0.04866  \u001b[0m |\n",
      "| \u001b[0m8        \u001b[0m | \u001b[0m-11.84   \u001b[0m | \u001b[0m0.08475  \u001b[0m | \u001b[0m0.08781  \u001b[0m | \u001b[0m0.1074   \u001b[0m | \u001b[0m0.4269   \u001b[0m |\n",
      "| \u001b[0m9        \u001b[0m | \u001b[0m-8.29    \u001b[0m | \u001b[0m0.478    \u001b[0m | \u001b[0m0.05332  \u001b[0m | \u001b[0m0.695    \u001b[0m | \u001b[0m0.3224   \u001b[0m |\n",
      "| \u001b[0m10       \u001b[0m | \u001b[0m-10.04   \u001b[0m | \u001b[0m0.3426   \u001b[0m | \u001b[0m0.08346  \u001b[0m | \u001b[0m0.02811  \u001b[0m | \u001b[0m0.7526   \u001b[0m |\n",
      "| \u001b[0m11       \u001b[0m | \u001b[0m-10.04   \u001b[0m | \u001b[0m0.003852 \u001b[0m | \u001b[0m0.01619  \u001b[0m | \u001b[0m0.7123   \u001b[0m | \u001b[0m0.7671   \u001b[0m |\n",
      "| \u001b[0m12       \u001b[0m | \u001b[0m-12.08   \u001b[0m | \u001b[0m0.2208   \u001b[0m | \u001b[0m0.04135  \u001b[0m | \u001b[0m0.5523   \u001b[0m | \u001b[0m0.7468   \u001b[0m |\n",
      "| \u001b[0m13       \u001b[0m | \u001b[0m-8.29    \u001b[0m | \u001b[0m0.3111   \u001b[0m | \u001b[0m0.02232  \u001b[0m | \u001b[0m0.9595   \u001b[0m | \u001b[0m0.8066   \u001b[0m |\n",
      "| \u001b[0m14       \u001b[0m | \u001b[0m-12.71   \u001b[0m | \u001b[0m0.01058  \u001b[0m | \u001b[0m0.08079  \u001b[0m | \u001b[0m0.9652   \u001b[0m | \u001b[0m0.7051   \u001b[0m |\n",
      "| \u001b[0m15       \u001b[0m | \u001b[0m-8.29    \u001b[0m | \u001b[0m0.1818   \u001b[0m | \u001b[0m0.00224  \u001b[0m | \u001b[0m0.1609   \u001b[0m | \u001b[0m0.9718   \u001b[0m |\n",
      "| \u001b[0m16       \u001b[0m | \u001b[0m-9.167   \u001b[0m | \u001b[0m0.4967   \u001b[0m | \u001b[0m0.03252  \u001b[0m | \u001b[0m0.8187   \u001b[0m | \u001b[0m0.05419  \u001b[0m |\n",
      "| \u001b[0m17       \u001b[0m | \u001b[0m-9.167   \u001b[0m | \u001b[0m0.4896   \u001b[0m | \u001b[0m0.02704  \u001b[0m | \u001b[0m0.8107   \u001b[0m | \u001b[0m0.4521   \u001b[0m |\n",
      "| \u001b[0m18       \u001b[0m | \u001b[0m-11.84   \u001b[0m | \u001b[0m0.1621   \u001b[0m | \u001b[0m0.06358  \u001b[0m | \u001b[0m0.4065   \u001b[0m | \u001b[0m0.754    \u001b[0m |\n",
      "| \u001b[0m19       \u001b[0m | \u001b[0m-10.04   \u001b[0m | \u001b[0m0.1526   \u001b[0m | \u001b[0m0.06767  \u001b[0m | \u001b[0m0.5559   \u001b[0m | \u001b[0m0.3227   \u001b[0m |\n",
      "| \u001b[0m20       \u001b[0m | \u001b[0m-8.29    \u001b[0m | \u001b[0m0.1982   \u001b[0m | \u001b[0m0.01942  \u001b[0m | \u001b[0m0.06233  \u001b[0m | \u001b[0m0.3515   \u001b[0m |\n",
      "| \u001b[0m21       \u001b[0m | \u001b[0m-12.08   \u001b[0m | \u001b[0m0.004722 \u001b[0m | \u001b[0m0.05502  \u001b[0m | \u001b[0m0.1704   \u001b[0m | \u001b[0m0.483    \u001b[0m |\n",
      "| \u001b[0m22       \u001b[0m | \u001b[0m-8.29    \u001b[0m | \u001b[0m0.1658   \u001b[0m | \u001b[0m0.04848  \u001b[0m | \u001b[0m0.4227   \u001b[0m | \u001b[0m0.8226   \u001b[0m |\n",
      "| \u001b[0m23       \u001b[0m | \u001b[0m-12.87   \u001b[0m | \u001b[0m0.1168   \u001b[0m | \u001b[0m0.0447   \u001b[0m | \u001b[0m0.5546   \u001b[0m | \u001b[0m0.9497   \u001b[0m |\n",
      "| \u001b[0m24       \u001b[0m | \u001b[0m-9.167   \u001b[0m | \u001b[0m0.05882  \u001b[0m | \u001b[0m0.05505  \u001b[0m | \u001b[0m0.4966   \u001b[0m | \u001b[0m0.1545   \u001b[0m |\n",
      "| \u001b[0m25       \u001b[0m | \u001b[0m-9.167   \u001b[0m | \u001b[0m0.1311   \u001b[0m | \u001b[0m0.002183 \u001b[0m | \u001b[0m0.1758   \u001b[0m | \u001b[0m0.6452   \u001b[0m |\n",
      "| \u001b[0m26       \u001b[0m | \u001b[0m-9.167   \u001b[0m | \u001b[0m0.09784  \u001b[0m | \u001b[0m0.04004  \u001b[0m | \u001b[0m0.5202   \u001b[0m | \u001b[0m0.07939  \u001b[0m |\n",
      "| \u001b[0m27       \u001b[0m | \u001b[0m-8.29    \u001b[0m | \u001b[0m0.09021  \u001b[0m | \u001b[0m0.02718  \u001b[0m | \u001b[0m0.8055   \u001b[0m | \u001b[0m0.6516   \u001b[0m |\n",
      "| \u001b[0m28       \u001b[0m | \u001b[0m-9.167   \u001b[0m | \u001b[0m0.06672  \u001b[0m | \u001b[0m0.05078  \u001b[0m | \u001b[0m0.8902   \u001b[0m | \u001b[0m0.5715   \u001b[0m |\n",
      "| \u001b[0m29       \u001b[0m | \u001b[0m-9.167   \u001b[0m | \u001b[0m0.07567  \u001b[0m | \u001b[0m0.02043  \u001b[0m | \u001b[0m0.6758   \u001b[0m | \u001b[0m0.8761   \u001b[0m |\n",
      "| \u001b[0m30       \u001b[0m | \u001b[0m-8.29    \u001b[0m | \u001b[0m0.4082   \u001b[0m | \u001b[0m0.01635  \u001b[0m | \u001b[0m0.02488  \u001b[0m | \u001b[0m0.1694   \u001b[0m |\n",
      "=========================================================================\n",
      "Total runtime: 0:12:10.80\n",
      "{'target': -8.29004093276407, 'params': {'dropout': 0.07323118951773941, 'learning_rate': 0.009233859476879781, 'neuronPct': 0.19439760926389418, 'neuronShrink': 0.35210511977261727}}\n"
     ]
    }
   ],
   "source": [
    "from bayes_opt import BayesianOptimization\n",
    "import time\n",
    "\n",
    "# Supress NaN warnings\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=RuntimeWarning)\n",
    "\n",
    "# Bounded region of parameter space\n",
    "pbounds = {'dropout': (0.0, 0.499),\n",
    "           'learning_rate': (0.0, 0.1),\n",
    "           'neuronPct': (0.01, 1),\n",
    "           'neuronShrink': (0.01, 1)\n",
    "           }\n",
    "\n",
    "optimizer = BayesianOptimization(\n",
    "    f=evaluate_network,\n",
    "    pbounds=pbounds,\n",
    "    verbose=2,  # verbose = 1 prints only when a maximum\n",
    "    # is observed, verbose = 0 is silent\n",
    "    random_state=1,\n",
    ")\n",
    "\n",
    "start_time = time.time()\n",
    "optimizer.maximize(init_points=10, n_iter=20,)\n",
    "time_took = time.time() - start_time\n",
    "\n",
    "print(f\"Total runtime: {hms_string(time_took)}\")\n",
    "print(optimizer.max)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tZ7xW5UDqaOV"
   },
   "source": [
    "As you can see, the algorithm performed 30 total iterations. This total iteration count includes ten random and 20 optimization iterations.\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "name": "test.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.11 (torch)",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}