Deep Learning Basics_DNN_CNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "f69bf3ff-0d5b-485f-a3f4-62ce7cde524e",
     "showTitle": false,
     "title": ""
    },
    "id": "ef_x8mETjcF4"
   },
   "source": [
    "# Deep Learning Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "1007c308-bdfe-4413-a152-8c9677d73d5b",
     "showTitle": false,
     "title": ""
    },
    "id": "G-OjdVKFjcF6"
   },
   "source": [
    "This notebooks gives a step by step explainer on how to build an artificial and convolution neural network using [keras](https://keras.io/about/) deep learning library on the MNIST dataset.\n",
    "\n",
    "**MNIST Dataset**\n",
    "\n",
    "The MNIST problem is a dataset developed by Yann LeCun, Corinna Cortes and Christopher Burges for evaluating machine learning models on the handwritten digit classification problem. The dataset was constructed from a number of scanned document dataset available from the National Institute of Standards and Technology (NIST). Images of digits were taken from a variety of scanned documents, normalized in size and centered. This makes it an excellent dataset for evaluating models, allowing the developer to focus on the machine learning with very little data cleaning or preparation required.\n",
    "\n",
    "Each image is a 28 by 28 pixel square (784 pixels total). A standard split of the dataset is used to evaluate and compare models, where 60,000 images are used to train a model and a separate set of 10,000 images are used to test it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "76ffd926-da32-4520-b741-5613bb266c43",
     "showTitle": false,
     "title": ""
    },
    "id": "VdEUMmPnjcF_"
   },
   "source": [
    "## Deep Neural Network (DNN) Model\n",
    "\n",
    "The following section shows how to build a bsic deep neural network model with keras."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "9306daef-e568-413f-9137-713592d7d767",
     "showTitle": false,
     "title": ""
    },
    "id": "F7J83EUOjcGB"
   },
   "source": [
    "### I. Import Packages and Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "71dbdf9d-7966-405a-bb70-e927d2051519",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "n8GtZKhOjcGD",
    "outputId": "a49d289e-4c63-46e4-d94c-d82f1b7c1249"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.5.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "818df209-2e4f-41bc-8984-5e549872d341",
     "showTitle": false,
     "title": ""
    },
    "id": "kIDrR21xj59B"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "3b89d130-2f49-41e2-8f4d-93ff2e77a478",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gD8uiRv_jcGE",
    "outputId": "449e334b-54a1-4503-bb2c-0da1f0df5dc8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
      "11493376/11490434 [==============================] - 0s 0us/step\n"
     ]
    }
   ],
   "source": [
    "# Load mnist data and split into train and test sets\n",
    "mnist = tf.keras.datasets.mnist\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "5274ceef-b58b-4678-aa69-b70c0d10d608",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "W28XU8ahjcGF",
    "outputId": "2e180eac-2594-4a07-e5af-9543a8fd905c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MNIST Dataset Shape:\n",
      "x_train: (60000, 28, 28)\n",
      "y_train: (60000,)\n",
      "x_test: (10000, 28, 28)\n",
      "y_test:  (10000,)\n"
     ]
    }
   ],
   "source": [
    "# Check shape of data\n",
    "print('MNIST Dataset Shape:')\n",
    "print(f'x_train: {x_train.shape}')\n",
    "print(f'y_train: {y_train.shape}')\n",
    "print(f'x_test: {x_test.shape}')\n",
    "print(f'y_test:  {y_test.shape}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "182e4054-413d-45c8-92bf-4d6f356918da",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-Xn73uodjcGI",
    "outputId": "62dbb10b-77fe-42f2-ac01-d70a8aa08977"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique count of digits in train data:\n",
      " [[   0 5923]\n",
      " [   1 6742]\n",
      " [   2 5958]\n",
      " [   3 6131]\n",
      " [   4 5842]\n",
      " [   5 5421]\n",
      " [   6 5918]\n",
      " [   7 6265]\n",
      " [   8 5851]\n",
      " [   9 5949]]\n"
     ]
    }
   ],
   "source": [
    "# Check unique count of digits in training dataset\n",
    "unique, counts = np.unique(y_train, return_counts=True)\n",
    "print (\"Unique count of digits in train data:\\n\",np.asarray((unique, counts)).T)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "3224823b-9350-4dda-9fdd-c0992c3d28ca",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "oboeHeL8jcGK",
    "outputId": "998f8c01-6373-46b6-a7ee-cdbef25ce318",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique count of digits in test data:\n",
      " [[   0  980]\n",
      " [   1 1135]\n",
      " [   2 1032]\n",
      " [   3 1010]\n",
      " [   4  982]\n",
      " [   5  892]\n",
      " [   6  958]\n",
      " [   7 1028]\n",
      " [   8  974]\n",
      " [   9 1009]]\n"
     ]
    }
   ],
   "source": [
    "# Check unique count of digits in test dataset\n",
    "unique, counts = np.unique(y_test, return_counts=True)\n",
    "print (\"Unique count of digits in test data:\\n\",np.asarray((unique, counts)).T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "946c6e3a-d51a-4a8d-90df-6d7772c3353d",
     "showTitle": false,
     "title": ""
    },
    "id": "Dem0ObsWjcGM"
   },
   "source": [
    "### II. Normalize Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "c0411fb7-8567-4447-b153-cdcdd795b9b0",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 281
    },
    "id": "knU20X94jcGN",
    "outputId": "7137f172-f3b0-44c0-955f-c69e8a08ba6b",
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f71f6508350>"
      ]
     },
     "execution_count": 7,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAOZ0lEQVR4nO3dbYxc5XnG8euKbezamMQbB9chLjjgFAg0Jl0ZEBZQobgOqgSoCsSKIkJpnSY4Ca0rQWlV3IpWbpUQUUqRTHExFS+BBIQ/0CTUQpCowWWhBgwEDMY0NmaNWYENIX5Z3/2w42iBnWeXmTMv3vv/k1Yzc+45c24NXD5nznNmHkeEAIx/H+p0AwDag7ADSRB2IAnCDiRB2IEkJrZzY4d5ckzRtHZuEkjlV3pbe2OPR6o1FXbbiyVdJ2mCpH+LiJWl50/RNJ3qc5rZJICC9bGubq3hw3jbEyTdIOnzkk6UtMT2iY2+HoDWauYz+wJJL0TE5ojYK+lOSedV0xaAqjUT9qMk/WLY4621Ze9ie6ntPtt9+7Snic0BaEbLz8ZHxKqI6I2I3kma3OrNAaijmbBvkzRn2ONP1JYB6ELNhP1RSfNsz7V9mKQvSlpbTVsAqtbw0FtE7Le9TNKPNDT0tjoinq6sMwCVamqcPSLul3R/Rb0AaCEulwWSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiCJpmZxRffzxPJ/4gkfm9nS7T/3F8fUrQ1OPVBc9+hjdxTrU7/uYv3Vaw+rW3u893vFdXcOvl2sn3r38mL9uD9/pFjvhKbCbnuLpN2SBiXtj4jeKpoCUL0q9uy/FxE7K3gdAC3EZ3YgiWbDHpJ+bPsx20tHeoLtpbb7bPft054mNwegUc0exi+MiG22j5T0gO2fR8TDw58QEaskrZKkI9wTTW4PQIOa2rNHxLba7Q5J90paUEVTAKrXcNhtT7M9/eB9SYskbayqMQDVauYwfpake20ffJ3bI+KHlXQ1zkw4YV6xHpMnFeuvnPWRYv2d0+qPCfd8uDxe/JPPlMebO+k/fzm9WP/Hf1lcrK8/+fa6tZf2vVNcd2X/54r1j//k0PtE2nDYI2KzpM9U2AuAFmLoDUiCsANJEHYgCcIOJEHYgST4imsFBs/+bLF+7S03FOufmlT/q5jj2b4YLNb/5vqvFOsT3y4Pf51+97K6tenb9hfXnbyzPDQ3tW99sd6N2LMDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKMs1dg8nOvFOuP/WpOsf6pSf1VtlOp5dtPK9Y3v1X+Kepbjv1+3dqbB8rj5LP++b+L9VY69L7AOjr27EAShB1IgrADSRB2IAnCDiRB2IEkCDuQhCPaN6J4hHviVJ/Ttu11i4FLTi/Wdy0u/9zzhCcPL9af+Pr1H7ing67Z+TvF+qNnlcfRB994s1iP0+v/APGWbxZX1dwlT5SfgPdZH+u0KwZGnMuaPTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJME4exeYMPOjxfrg6wPF+ku31x8rf/rM1cV1F/zDN4r1I2/o3HfK8cE1Nc5ue7XtHbY3DlvWY/sB25tqtzOqbBhA9cZyGH+LpPfOen+lpHURMU/SutpjAF1s1LBHxMOS3nsceZ6kNbX7aySdX3FfACrW6G/QzYqI7bX7r0qaVe+JtpdKWipJUzS1wc0BaFbTZ+Nj6Axf3bN8EbEqInojoneSJje7OQANajTs/bZnS1Ltdkd1LQFohUbDvlbSxbX7F0u6r5p2ALTKqJ/Zbd8h6WxJM21vlXS1pJWS7rJ9qaSXJV3YyibHu8Gdrze1/r5djc/v/ukvPVOsv3bjhPILHCjPsY7uMWrYI2JJnRJXxwCHEC6XBZIg7EAShB1IgrADSRB2IAmmbB4HTrji+bq1S04uD5r8+9HrivWzvnBZsT79e48U6+ge7NmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnG2ceB0rTJr3/thOK6/7f2nWL9ymtuLdb/8sILivX43w/Xrc35+58V11Ubf+Y8A/bsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEUzYnN/BHpxfrt1397WJ97sQpDW/707cuK9bn3bS9WN+/eUvD2x6vmpqyGcD4QNiBJAg7kARhB5Ig7EAShB1IgrADSTDOjqI4Y36xfsTKrcX6HZ/8UcPbPv7BPy7Wf/tv63+PX5IGN21ueNuHqqbG2W2vtr3D9sZhy1bY3mZ7Q+3v3CobBlC9sRzG3yJp8QjLvxsR82t/91fbFoCqjRr2iHhY0kAbegHQQs2coFtm+8naYf6Mek+yvdR2n+2+fdrTxOYANKPRsN8o6VhJ8yVtl/Sdek+MiFUR0RsRvZM0ucHNAWhWQ2GPiP6IGIyIA5JukrSg2rYAVK2hsNuePezhBZI21nsugO4w6ji77TsknS1ppqR+SVfXHs+XFJK2SPpqRJS/fCzG2cejCbOOLNZfuei4urX1V1xXXPdDo+yLvvTSomL9zYWvF+vjUWmcfdRJIiJiyQiLb266KwBtxeWyQBKEHUiCsANJEHYgCcIOJMFXXNExd20tT9k81YcV67+MvcX6H3zj8vqvfe/64rqHKn5KGgBhB7Ig7EAShB1IgrADSRB2IAnCDiQx6rfekNuBheWfkn7xC+Upm0+av6VubbRx9NFcP3BKsT71vr6mXn+8Yc8OJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kwzj7OufekYv35b5bHum86Y02xfuaU8nfKm7En9hXrjwzMLb/AgVF/3TwV9uxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATj7IeAiXOPLtZfvOTjdWsrLrqzuO4fHr6zoZ6qcFV/b7H+0HWnFesz1pR/dx7vNuqe3fYc2w/afsb207a/VVveY/sB25tqtzNa3y6ARo3lMH6/pOURcaKk0yRdZvtESVdKWhcR8yStqz0G0KVGDXtEbI+Ix2v3d0t6VtJRks6TdPBayjWSzm9VkwCa94E+s9s+RtIpktZLmhURBy8+flXSrDrrLJW0VJKmaGqjfQJo0pjPxts+XNIPJF0eEbuG12JodsgRZ4iMiFUR0RsRvZM0ualmATRuTGG3PUlDQb8tIu6pLe63PbtWny1pR2taBFCFUQ/jbVvSzZKejYhrh5XWSrpY0sra7X0t6XAcmHjMbxXrb/7u7GL9or/7YbH+px+5p1hvpeXby8NjP/vX+sNrPbf8T3HdGQcYWqvSWD6znyHpy5Kesr2htuwqDYX8LtuXSnpZ0oWtaRFAFUYNe0T8VNKIk7tLOqfadgC0CpfLAkkQdiAJwg4kQdiBJAg7kARfcR2jibN/s25tYPW04rpfm/tQsb5ken9DPVVh2baFxfrjN5anbJ75/Y3Fes9uxsq7BXt2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUgizTj73t8v/2zx3j8bKNavOu7+urVFv/F2Qz1VpX/wnbq1M9cuL657/F//vFjveaM8Tn6gWEU3Yc8OJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0mkGWffcn7537XnT767Zdu+4Y1ji/XrHlpUrHuw3o/7Djn+mpfq1ub1ry+uO1isYjxhzw4kQdiBJAg7kARhB5Ig7EAShB1IgrADSTgiyk+w50i6VdIsSSFpVURcZ3uFpD+R9FrtqVdFRP0vfUs6wj1xqpn4FWiV9bFOu2JgxAszxnJRzX5JyyPicdvTJT1m+4Fa7bsR8e2qGgXQOmOZn327pO21+7ttPyvpqFY3BqBaH+gzu+1jJJ0i6eA1mMtsP2l7te0ZddZZarvPdt8+7WmqWQCNG3PYbR8u6QeSLo+IXZJulHSspPka2vN/Z6T1ImJVRPRGRO8kTa6gZQCNGFPYbU/SUNBvi4h7JCki+iNiMCIOSLpJ0oLWtQmgWaOG3bYl3Szp2Yi4dtjy2cOedoGk8nSeADpqLGfjz5D0ZUlP2d5QW3aVpCW252toOG6LpK+2pEMAlRjL2fifShpp3K44pg6gu3AFHZAEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IIlRf0q60o3Zr0l6ediimZJ2tq2BD6Zbe+vWviR6a1SVvR0dER8bqdDWsL9v43ZfRPR2rIGCbu2tW/uS6K1R7eqNw3ggCcIOJNHpsK/q8PZLurW3bu1LordGtaW3jn5mB9A+nd6zA2gTwg4k0ZGw215s+znbL9i+shM91GN7i+2nbG+w3dfhXlbb3mF747BlPbYfsL2pdjviHHsd6m2F7W21926D7XM71Nsc2w/afsb207a/VVve0feu0Fdb3re2f2a3PUHS85I+J2mrpEclLYmIZ9raSB22t0jqjYiOX4Bh+0xJb0m6NSJOqi37J0kDEbGy9g/ljIi4okt6WyHprU5P412brWj28GnGJZ0v6Svq4HtX6OtCteF968SefYGkFyJic0TslXSnpPM60EfXi4iHJQ28Z/F5ktbU7q/R0P8sbVent64QEdsj4vHa/d2SDk4z3tH3rtBXW3Qi7EdJ+sWwx1vVXfO9h6Qf237M9tJONzOCWRGxvXb/VUmzOtnMCEadxrud3jPNeNe8d41Mf94sTtC938KI+Kykz0u6rHa42pVi6DNYN42djmka73YZYZrxX+vke9fo9OfN6kTYt0maM+zxJ2rLukJEbKvd7pB0r7pvKur+gzPo1m53dLifX+umabxHmmZcXfDedXL6806E/VFJ82zPtX2YpC9KWtuBPt7H9rTaiRPZniZpkbpvKuq1ki6u3b9Y0n0d7OVdumUa73rTjKvD713Hpz+PiLb/STpXQ2fkX5T0V53ooU5fn5T0RO3v6U73JukODR3W7dPQuY1LJX1U0jpJmyT9l6SeLurtPyQ9JelJDQVrdod6W6ihQ/QnJW2o/Z3b6feu0Fdb3jculwWS4AQdkARhB5Ig7EAShB1IgrADSRB2IAnCDiTx/65XcTNOWsh5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the first image in the dataset\n",
    "plt.imshow(x_train[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "00cb5302-39d3-411c-a651-0112df019422",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4n-BJrYIjcGO",
    "outputId": "f351235c-5f44-43e4-b23a-fc302fb91872",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(28, 28)"
      ]
     },
     "execution_count": 8,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check shape - each image is a 28x28 pixel\n",
    "x_train[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "2ed04e90-cd08-4f01-9bdf-1bb0a110dc5b",
     "showTitle": false,
     "title": ""
    },
    "id": "GWj4YNK0jcGP"
   },
   "source": [
    "Normalize - The pixel values are between 0 and 255. Scaling the data to range 0 and 1 in order to make it easier for the network to learn. You can use the keras utility function `normalize` to do so or simply divide the enitre array by 255."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "d686c1fd-a566-474f-8d6c-1d3768bf6131",
     "showTitle": false,
     "title": ""
    },
    "id": "ICsrqOgQjcGP"
   },
   "outputs": [],
   "source": [
    "x_train = tf.keras.utils.normalize(x_train,axis=1)\n",
    "x_test = tf.keras.utils.normalize(x_test,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "b2b1d722-10c6-439f-841e-d5cb51221b52",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 281
    },
    "id": "upq90L3pjcGQ",
    "outputId": "17a309be-693b-4bd6-9a81-ad0712c7a6de"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f71f647d710>"
      ]
     },
     "execution_count": 10,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAOf0lEQVR4nO3dW4xd5XnG8eeZYXzANsQHGBxjYkockEvBtFOnBVTRoCRApEJygcIFohWqcxHURM1FEb2AS1TloFy0kZxixalSorQEYbWoQCwqStRQBmRsgxMwZ49sDwdjzGHs8fjtxSzQALO+Pewzfv8/abT3Xu9ee73Z8cPae317rc8RIQAnvoFeNwCgOwg7kARhB5Ig7EAShB1I4qRubmye58cCLermJoFUJvS2jsYRz1ZrKey2r5D0Q0mDkv45Im4vPX+BFunzvryVTQIoeCS21daa/hhve1DSP0q6UtI6SdfZXtfs6wHorFa+s2+QtCcinouIo5J+Lunq9rQFoN1aCfsqSS/PeLy3WvYBtjfaHrU9OqkjLWwOQCs6fjQ+IjZFxEhEjAxpfqc3B6BGK2Efk7R6xuMzq2UA+lArYX9U0lrbZ9ueJ+nrkra2py0A7db00FtEHLN9k6T7ND30tjkinmxbZwDaqqVx9oi4V9K9beoFQAfxc1kgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSaGkWV3wC2MXywPz55fUHWtsfHPvjc5te99DZC4r1Ff+5p1jffftnamtfuWBncd3zF+0t1r+25Oli/frVlxTrvdBS2G2/IOmwpClJxyJipB1NAWi/duzZ/zwiXm3D6wDoIL6zA0m0GvaQdL/tx2xvnO0JtjfaHrU9OqkjLW4OQLNa/Rh/aUSM2T5d0gO2fxsRD818QkRskrRJkk7xsmhxewCa1NKePSLGqttxSXdL2tCOpgC0X9Nht73I9pL37kv6kqRd7WoMQHu18jF+WNLdnh7HPUnSv0bEf7WlqxPM4GmnFes+abBYnzx7uFifOK1+rNzHi6vqzbPK/wSOLSqv38jAZKF2tLxuo22/dOPaYv3WS/69tnbvq39QXHfz+MXF+nef+oti/Rz9b7HeC02HPSKek3RhG3sB0EEMvQFJEHYgCcIOJEHYgSQIO5AEp7i2wcD55xXrT//Vp1p7/aPl01TnHaqvD73V4MV7+ZvGBtv+9EPl5qcWlP/5bnr+a7W1xS+9W1x3+eHyT7uX7uq/obVG2LMDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKMs7eBxw4U64PvLi3Wpxb27wV8Fo6Xexs8Wq6/c1r9/mTgWIP/3b/ZUd52eW0taVAvaXBm8CcSe3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJx9jaYOniwWF/zH28X6wfPLV8z+ZSXyudW7/98eWrjkgUHyyPKp/7b48V6TJavB710+PTa2sSFZxXXRXuxZweSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJBhn74YG52Uv33FysX78nXeK9WWnbKitHV5d/r946RPl3whMNRhHb2TqwHhtbej++hrar+Ge3fZm2+O2d81Ytsz2A7afqW7LV2cA0HNz+Rj/E0lXfGjZzZK2RcRaSduqxwD6WMOwR8RDkl7/0OKrJW2p7m+RdE2b+wLQZs1+Zx+OiH3V/f2ShuueaHujpI2StEDl76YAOqflo/ERESpM0RcRmyJiJCJGhjS/1c0BaFKzYT9ge6UkVbccVgX6XLNh3yrphur+DZLuaU87ADql4Xd223dKukzSCtt7Jd0q6XZJv7B9o6QXJV3bySZPdI3G0RsZPNL8decPrSvPHb/4yaZfGn2mYdgj4rqa0uVt7gVAB/FzWSAJwg4kQdiBJAg7kARhB5LgFNcTwPxtT9TWTl5yUXHdd04vT3x86ufOKdannn62WEf/YM8OJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kwzn4CKE2bfOqvni6uO3HtecX6vi/WXnFMkrT491cU6yePFU7f/b+dxXXRXuzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJxtlPcFMHy1MyDz9Ynt9j7CvlcfY31pbPh39zzZLa2hkDFxTXHfzti8X61BuHinV8EHt2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCcfbkGl33/czDbxfrr31hTbH+7un1+5PxP1pcXHfRmeVz7U/5n+eL9akD5d8QZNNwz257s+1x27tmLLvN9pjt7dXfVZ1tE0Cr5vIx/ieSrphl+Q8iYn31d2972wLQbg3DHhEPSXq9C70A6KBWDtDdZHtH9TF/ad2TbG+0PWp7dFJHWtgcgFY0G/YfSTpH0npJ+yR9r+6JEbEpIkYiYmRI85vcHIBWNRX2iDgQEVMRcVzSjyVtaG9bANqtqbDbXjnj4Vcl7ap7LoD+0HCc3fadki6TtML2Xkm3SrrM9npJIekFSd/oYI/ooWP79hfrS+96o1w/9+za2v5Lag/1SJJevaC8L3rt/M8W62fdxjj7TA3DHhHXzbL4jg70AqCD+LkskARhB5Ig7EAShB1IgrADSXCKK1pyfGKi/IQndteWfPGftrTto+e8W65/eaS2Nu++0Za2/UnEnh1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkmCcHUUnrTyjWJ9Yt6pcXz5UW4sBN9h6FKvev6BYn3f/bxq8fi7s2YEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcbZT3CDK5YX60cuXFOsH1hXnsVnamF5+wOThdrR8ro+Xh6Hn/dmeX1FeZw+G/bsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AE4+yfAIOfOrVYnzy/flrkg58tD4RPLiqPZTcaR2/FwteOF+tnPHyoWD9euCY9Pqrhnt32atsP2n7K9pO2v1UtX2b7AdvPVLflybYB9NRcPsYfk/SdiFgn6U8kfdP2Okk3S9oWEWslbaseA+hTDcMeEfsi4vHq/mFJuyWtknS1pC3V07ZIuqZTTQJo3cf6zm57jaSLJD0iaTgi9lWl/ZKGa9bZKGmjJC3Qyc32CaBFcz4ab3uxpLskfTsiPnAKQkSEaq4OGBGbImIkIkaGVD6pAkDnzCnstoc0HfSfRcQvq8UHbK+s6isljXemRQDt0PBjvG1LukPS7oj4/ozSVkk3SLq9ur2nIx2eAAaXlgcqYtWs34De9/KVy8rrD9bXht4qrtroas0NLRwvv8CKR16trU3tfqa4bnlgDh/XXL6zXyLpekk7bW+vlt2i6ZD/wvaNkl6UdG1nWgTQDg3DHhEPS6r75cXl7W0HQKfwc1kgCcIOJEHYgSQIO5AEYQeS4BTXOSpdknnP336uuK4bDBhPLSyPVQ8cLdfnHWo09XG9k18pN3fqfx8s1o//7tliferYsY/dEzqDPTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJJFmnH3s5ouL9aPryyd+n7ey/tocZ0zsL667d8/pxfrARPPj5JLkwlD2ip3vFtcd/PXOYp1x8hMHe3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSCLNOPuuv/mnYv3+d4aK9V+/XX/O+oP7y+ezz3+lcGF3Sct3TRXrA5Plc84X3bejtnZ8YqK4bouXjccnCHt2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUhiLvOzr5b0U0nDmh6W3RQRP7R9m6S/lvRK9dRbIuLeTjXaqi9/en3HXnuhni/Wz2pQbxXzmGMu5vKjmmOSvhMRj9teIukx2w9UtR9ExHc71x6AdpnL/Oz7JO2r7h+2vVvSqk43BqC9PtZ3dttrJF0k6ZFq0U22d9jebHtpzTobbY/aHp3UkZaaBdC8OYfd9mJJd0n6dkS8KelHks6RtF7Te/7vzbZeRGyKiJGIGBnS/Da0DKAZcwq77SFNB/1nEfFLSYqIAxExFRHHJf1Y0obOtQmgVQ3DbtuS7pC0OyK+P2P5yhlP+6qkXe1vD0C7zOVo/CWSrpe00/b2atktkq6zvV7Tw3EvSPpGRzoE0BZzORr/sKTZLmzet2PqAD6KX9ABSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeScET3Ju21/YqkF2csWiHp1a418PH0a2/92pdEb81qZ2+fiYjTZit0Newf2bg9GhEjPWugoF9769e+JHprVrd642M8kARhB5Loddg39Xj7Jf3aW7/2JdFbs7rSW0+/swPonl7v2QF0CWEHkuhJ2G1fYft3tvfYvrkXPdSx/YLtnba32x7tcS+bbY/b3jVj2TLbD9h+prqddY69HvV2m+2x6r3bbvuqHvW22vaDtp+y/aTtb1XLe/reFfrqyvvW9e/stgclPS3pi5L2SnpU0nUR8VRXG6lh+wVJIxHR8x9g2P4zSW9J+mlEnF8t+wdJr0fE7dV/KJdGxN/1SW+3SXqr19N4V7MVrZw5zbikayT9pXr43hX6ulZdeN96sWffIGlPRDwXEUcl/VzS1T3oo+9FxEOSXv/Q4qslbanub9H0P5auq+mtL0TEvoh4vLp/WNJ704z39L0r9NUVvQj7Kkkvz3i8V/0133tIut/2Y7Y39rqZWQxHxL7q/n5Jw71sZhYNp/Hupg9NM943710z05+3igN0H3VpRPyhpCslfbP6uNqXYvo7WD+Nnc5pGu9umWWa8ff18r1rdvrzVvUi7GOSVs94fGa1rC9ExFh1Oy7pbvXfVNQH3ptBt7od73E/7+unabxnm2ZcffDe9XL6816E/VFJa22fbXuepK9L2tqDPj7C9qLqwIlsL5L0JfXfVNRbJd1Q3b9B0j097OUD+mUa77ppxtXj967n059HRNf/JF2l6SPyz0r6+170UNPX70l6ovp7ste9SbpT0x/rJjV9bONGScslbZP0jKRfSVrWR739i6SdknZoOlgre9TbpZr+iL5D0vbq76pev3eFvrryvvFzWSAJDtABSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBL/D4j1Q5zjdSi3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the first image in the dataset after normalizing\n",
    "plt.imshow(x_train[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "53b2ac8c-25ed-4bb8-a167-0a378b44e961",
     "showTitle": false,
     "title": ""
    },
    "id": "Enrd7sJ9k1xU"
   },
   "source": [
    "Note: Normalizing does not change the shape of an image, it just normalizes the pixel values in the range of (0,1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "b74ab619-f7c6-463d-898a-d2da4128aa7d",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "t4lPWWimjcGQ",
    "outputId": "d19b0b77-ce16-42e6-8d06-8c93d9033230"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(28, 28)"
      ]
     },
     "execution_count": 11,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check shape after normalizing\n",
    "x_train[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "9dd3a6e5-e24f-46a5-9d92-754b4ff6bc8d",
     "showTitle": false,
     "title": ""
    },
    "id": "WF5kspuzjcGR"
   },
   "source": [
    "### III. Different layers of a DNN\n",
    "\n",
    "Before building a deep learning model, lets try to understand the individual layers (also referred as the building blocks) of a deep neural network model and their purpose."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "f2845420-3133-4040-8ed6-b632803339ab",
     "showTitle": false,
     "title": ""
    },
    "id": "92aY6V95jcGR"
   },
   "source": [
    "**Flatten Layer**\n",
    "\n",
    "Flattens the input. This layer converts the data into a 1-dimensional array which can be used an input by the following fully connected layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "7f415d2e-db90-45f3-b067-4d4171954e5b",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "eRrjI8NmjcGR",
    "outputId": "e10871ec-8adf-49d4-dd75-389a9b361b88"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(None, 6)"
      ]
     },
     "execution_count": 12,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a `Sequential` model and add a Flatten layer as the first layer. \n",
    "# Convert a 3x2 input into a 1-dimensional array of length 6\n",
    "model = tf.keras.models.Sequential()\n",
    "model.add(tf.keras.layers.Flatten(input_shape=(3,2)))\n",
    "model.output_shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "387f12fc-e48e-4984-97bc-1a3590faea3f",
     "showTitle": false,
     "title": ""
    },
    "id": "8boWZv2KjcGS"
   },
   "source": [
    "**Dense Layer**\n",
    "\n",
    "Dense class from keras is an implementation of the simplest neural network building block: the fully connected layer. In technical terms, dense layer is the implementation of the equation `output = activation(dot(input, kernel) + bias)`. This means that we take the dot product between our input tensor and whatever the weight kernel matrix is featured in our dense layer. Then, we add a bias vector (optional, if required) and take an element-wise activation of the output values (some function, linear or in most cases non-linear).\n",
    "\n",
    "*Activation function*\n",
    "\n",
    "An activation function is a function that is added into an artificial neural network in order to help the network learn complex patterns in the data. It takes in the output signal from the previous cell and converts it into some form that can be taken as input to the next cell. Some of the commonly used activation functions are shown below:\n",
    "\n",
    "<img src = 'https://miro.medium.com/max/1200/1*ZafDv3VUm60Eh10OeJu1vw.png' width = 600>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "d82f4f02-929b-453f-9847-d09e1161998b",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pGHpnX9VjcGS",
    "outputId": "4c91f081-bb04-4533-cbbf-f3d5828e9c7f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(None, 128)"
      ]
     },
     "execution_count": 13,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a `Sequential` model and add a Dense layer as the first layer\n",
    "model = tf.keras.models.Sequential()\n",
    "model.add(tf.keras.layers.InputLayer(input_shape=(16,)))\n",
    "model.add(tf.keras.layers.Dense(128,activation=tf.nn.relu))\n",
    "model.output_shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "143eb0d1-6ecb-4972-856d-1a559b8a3c75",
     "showTitle": false,
     "title": ""
    },
    "id": "QOPWa7q6jcGS"
   },
   "source": [
    "**Dropout Layer**\n",
    "\n",
    "A fully connected layer occupies most of the parameters, and hence, neurons develop co-dependency amongst each other during training which curbs the individual power of each neuron leading to over-fitting of training data.\n",
    "\n",
    "Hence, a [Dropout Layer](https://towardsdatascience.com/machine-learning-part-20-dropout-keras-layers-explained-8c9f6dc4c9ab?gi=e79b5385b6ad) layer is added to avoid overfitting. In passing 0.5, every hidden unit (neuron) is set to 0 with a probability of 0.5. In other words, there’s a 50% chance that the output of a given neuron will be forced to 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "240ca6f9-5ac3-4aba-a91a-45868bd10f58",
     "showTitle": false,
     "title": ""
    },
    "id": "QHwJiBfEjcGT"
   },
   "source": [
    "### IV. Build a DNN Model\n",
    "\n",
    "Now, we build a complete deep learning model. We will be using a `Sequential` model and then stack up the discussed layers so far using the `add()`function.\n",
    "\n",
    "- Flatten Layer\n",
    "    - This layer flattens the input data into a 1-dimensional array for inputting it to the next layer.\n",
    "- Dense Layer\n",
    "    - Two dense layers with 128 nodes and `relu` activation function\n",
    "    - Final dense layer with 10 nodes and `softmax` activation function. The number of nodes should be equal to the number of classes that we are trying to predict i.e. 10 in this data. The `softmax` activation function is used to turn the outputs into probability-like values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "e6b9ef75-99c7-438f-86a0-fb313476245e",
     "showTitle": false,
     "title": ""
    },
    "id": "8U7FEHZ5jcGT",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "model = tf.keras.models.Sequential([tf.keras.layers.Flatten(),\n",
    "                                    tf.keras.layers.Dense(128,activation=tf.nn.relu),\n",
    "                                    tf.keras.layers.Dense(128,activation=tf.nn.relu),\n",
    "                                    tf.keras.layers.Dense(10,activation=tf.nn.softmax)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "cf83be68-3d24-49be-99b7-5415518409b2",
     "showTitle": false,
     "title": ""
    },
    "id": "RkkgCbsdjcGU"
   },
   "source": [
    "Next, the model is compiled with the categorical cross entropy loss function and the efficient ADAM gradient descent algorithm is used to learn the weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "bacd34c3-3fda-4ca7-87b4-e42582ea5136",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mSU7aGCvjcGU",
    "outputId": "10b1b1d5-34df-4c84-b2f5-df2974e604f6",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.2632 - accuracy: 0.9239\n",
      "Epoch 2/5\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.1102 - accuracy: 0.9659\n",
      "Epoch 3/5\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.0744 - accuracy: 0.9766\n",
      "Epoch 4/5\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.0553 - accuracy: 0.9824\n",
      "Epoch 5/5\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.0423 - accuracy: 0.9861\n"
     ]
    }
   ],
   "source": [
    "# Complie the model\n",
    "model.compile(optimizer= 'adam'\n",
    "             ,loss= 'sparse_categorical_crossentropy'\n",
    "             ,metrics= ['accuracy'])\n",
    "model.fit(x_train,y_train,epochs=5);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UTcAYyUAvcjv"
   },
   "source": [
    "Note that the model is being trained on 1875 batches of 32 images each (1875*32 = 60,000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "0ef4c5d8-5803-4545-810e-10eb69bcc5d4",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HnOAzU3LjcGU",
    "outputId": "ff73b0e9-4fa4-4835-ceaa-9b8f5272b4ab"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_1 (Flatten)          (32, 784)                 0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (32, 128)                 100480    \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (32, 128)                 16512     \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (32, 10)                  1290      \n",
      "=================================================================\n",
      "Total params: 118,282\n",
      "Trainable params: 118,282\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Print model summary\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "288f9a18-1422-4f54-a4ec-a4688d9f07a0",
     "showTitle": false,
     "title": ""
    },
    "id": "kr-iGYyposNe"
   },
   "source": [
    "Now, lets try to undertsand some of the math related to the shape and # parameters:\n",
    "  1. Flatten layer shape is calculated by multiplying 28*28 = 784\n",
    "  2. The #parameters for the first fully-connected layer are calculated as follows (784*128) + 128 = 100,480. The input of the pervious layer multiplied by the neurons in the current layer and then adding the number of neurons in the current layer.\n",
    "  3. Similarly, the #parameters for the second fully-connected layer are calculated as follows (128*128) + 128 = 16,512\n",
    "  4. For the final layer, (128*10) + 10 = 1290"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "1526711f-543b-4afb-a4cf-30bcbd185068",
     "showTitle": false,
     "title": ""
    },
    "id": "PPaVQk80jcGU"
   },
   "source": [
    "Evaluate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "1326eeec-1255-4972-a305-dc430f8366bd",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lxOb6UARjcGV",
    "outputId": "936f6d02-b009-4d46-f4a3-8cd7f02cbb01",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 [==============================] - 0s 1ms/step - loss: 0.0922 - accuracy: 0.9728\n",
      "DNN Model evaluation \n",
      " Validation Loss: 0.09222272038459778\n",
      " Validation Accuracy: 0.9728000164031982\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the model\n",
    "val_loss, val_acc = model.evaluate(x_test,y_test)\n",
    "print(\"DNN Model evaluation \\n\",\n",
    "      f\"Validation Loss: {val_loss}\\n\",\n",
    "      f\"Validation Accuracy: {val_acc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "049b388a-e444-4a6f-9423-cb07aa05f7e9",
     "showTitle": false,
     "title": ""
    },
    "id": "gN5Sr_UzjcGV"
   },
   "source": [
    "Predict probability of classes on the test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "272ecf26-07a0-4a01-8f81-890dfc3e43df",
     "showTitle": false,
     "title": ""
    },
    "id": "XLMuXCE_jcGV"
   },
   "outputs": [],
   "source": [
    "pred_dnn = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "0e5a279f-0afc-490d-9db3-c316ab9a34f3",
     "showTitle": false,
     "title": ""
    },
    "id": "IhtlU7rvjcGV"
   },
   "source": [
    "Use argmax function to select the index of an array that results in the largest value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "ca6a97a1-b831-4120-a570-9db06967c611",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pTuR7bSJjcGW",
    "outputId": "a635f105-5f87-4243-9625-b4144bc1831d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7\n"
     ]
    }
   ],
   "source": [
    "print(np.argmax(pred_dnn[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "ac669166-e8a7-4fbc-90b1-75fa514c4258",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 265
    },
    "id": "0I12rH9jjcGW",
    "outputId": "cb43951c-0f32-483c-991c-a1c40944ff80",
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAANeklEQVR4nO3dbYxc5XnG8evyZr0GG4jNy7IxVsBAWxHSmnTrlAYVIpSIoFYmX1D8gboSykZVkJIKVUX0Q/iIqiYoUtNIm+DEVJQoEiCsCBVcKxKKWiEW5BiDAzauDXb8AgJqjGOvd/fuhz1GG9jzzDLv9v3/SauZPfecc26PfO05M8+ceRwRAnD2W9TrBgB0B2EHkiDsQBKEHUiCsANJfKKbO1vsoViipd3cJZDKCb2vyTjp+Wothd32LZK+L2lA0o8j4v7S45doqT7vm1vZJYCCZ2Nrba3p03jbA5J+IOkrkq6RtN72Nc1uD0BntfKafa2k3RGxJyImJf1M0rr2tAWg3VoJ+0pJb8z5fX+17PfYHrM9YXvilE62sDsArej4u/ERMR4RoxExOqihTu8OQI1Wwn5A0qo5v19WLQPQh1oJ+3OSrrZ9he3Fkr4maXN72gLQbk0PvUXElO27JD2l2aG3jRHxUts6A9BWLY2zR8STkp5sUy8AOoiPywJJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSaGnKZtt7Jb0naVrSVESMtqMpAO3XUtgrX4yIt9qwHQAdxGk8kESrYQ9JT9t+3vbYfA+wPWZ7wvbEKZ1scXcAmtXqafwNEXHA9iWSttj+TUQ8M/cBETEuaVySzveKaHF/AJrU0pE9Ig5Ut0ckPS5pbTuaAtB+TYfd9lLb552+L+nLkna0qzEA7dXKafywpMdtn97Of0TEf7alKwBt13TYI2KPpD9pYy8AOoihNyAJwg4kQdiBJAg7kARhB5Jox4UwKRz6+7+orZ28/r3iupPHFxfrcXygWL/qkVPF+uLdB2trUwcPFddFHhzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJxtkX6Nf/8G+1tfH/+1Rx3TVL9hXr706fW6xvvf4zxfpjT11fW1v2+uriuoumyl8eNHmBi3U1KGumtO8Gqzb439lo/alz6mvnHir/u1f85H/KGz8DcWQHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQYZ1+gz9/zd7W1ExeWB5vPe326WH/3qvL17L8bKQxWSxqcLKx7aXk8eeidcu/HV5b3HY2G4Qv/9IHJ8souX8avmfLXBGjgimO1ta9/dmtx3Ud/ckl542cgjuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATj7Av0yYc6d33z0hbXX7S0fgu+bKS4buzbX974H1zeREdzFIbpPdlgIP3Qm8XynruvbaKhWf/6yo3F+oh2Nr3tftXwyG57o+0jtnfMWbbC9hbbu6rb5Z1tE0CrFnIa/1NJt3xo2T2StkbE1ZK2Vr8D6GMNwx4Rz0h6+0OL10naVN3fJOm2NvcFoM2afc0+HBGnJxg7JGm47oG2xySNSdISlb9rDUDntPxufESEpNqrLSJiPCJGI2J0UEOt7g5Ak5oN+2HbI5JU3R5pX0sAOqHZsG+WtKG6v0HSE+1pB0CnNHzNbvsRSTdJusj2fknfkXS/pJ/bvlPSPkm3d7JJlM28/3598ZXdrW18+29aW78Vaz9bLE8Pla/Vn/lt/ecPVv+gfDJa/gaCM1PDsEfE+prSzW3uBUAH8XFZIAnCDiRB2IEkCDuQBGEHkuASV/TMwPnnF+uvrVtW3kCDr7G+fHP9JbTTu/aUVz4LcWQHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQYZ0fPHPviHxXrU+eUL2EdPFYeaB96453a2tl4CWsjHNmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnG2dFRA394VW3t0PUDDdYuj7Ovfrg8pXPGa9ZLOLIDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKMs6Ojjl57YW0tGnzv+3l7y8ei6V3/20xLaTU8stveaPuI7R1zlt1n+4DtbdXPrZ1tE0CrFnIa/1NJt8yz/IGIWFP9PNnetgC0W8OwR8Qzkt7uQi8AOqiVN+jusr29Os1fXvcg22O2J2xPnNLJFnYHoBXNhv2Hkq6UtEbSQUnfrXtgRIxHxGhEjA5qqMndAWhVU2GPiMMRMR0RM5J+JGlte9sC0G5Nhd32yJxfvyppR91jAfSHhuPsth+RdJOki2zvl/QdSTfZXqPZC473SvpGB3tEH/Pg4mL93avqr1n3TPl69U89daRYn57J+O3vzWsY9ohYP8/iBzvQC4AO4uOyQBKEHUiCsANJEHYgCcIOJMElrmjJ+391XbH+u+GZ2toFr5avcZ1+ZXdTPWF+HNmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnG2VHkP/1Msf7bG8tj5QMn6uuXbj1cXJcLWNuLIzuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJME4e3KLli4t1vf+9QXFerj+enVJOr9wSfr0rj3FddFeHNmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnG2c92Ll9vfviOPy7WJz9ZHkcfeqd8vBj+xWu1tanimmi3hkd226ts/9L2y7Zfsv2tavkK21ts76pul3e+XQDNWshp/JSkuyPiGkl/Lumbtq+RdI+krRFxtaSt1e8A+lTDsEfEwYh4obr/nqSdklZKWidpU/WwTZJu61STAFr3sV6z275c0nWSnpU0HBEHq9IhScM164xJGpOkJTq32T4BtGjB78bbXibpUUnfjoijc2sREZJivvUiYjwiRiNidFBDLTULoHkLCrvtQc0G/eGIeKxafNj2SFUfkXSkMy0CaIeGp/G2LelBSTsj4ntzSpslbZB0f3X7REc6REs+MXxJsX7i4vLQXM0J2wc+/YujxfrUofLXRaN7FvKa/QuS7pD0ou1t1bJ7NRvyn9u+U9I+Sbd3pkUA7dAw7BHxK0l1f/5vbm87ADqFj8sCSRB2IAnCDiRB2IEkCDuQBJe4ngUGLr64tvb631zZ0rZXPV2eODkmdrS0fXQPR3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJx9rPA0RtX19ZOLStfj77oVPl69nNffatYL4/Co59wZAeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJBhnPwPM3HhdsX74z+r/Zg+cbHc3OFNxZAeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJBYyP/sqSQ9JGtbsZN3jEfF92/dJ+rqkN6uH3hsRT3aq0cyOfO6cYn1m8UxtbeBk+Xr1wfL06vKJyfIDcMZYyIdqpiTdHREv2D5P0vO2t1S1ByLiXzrXHoB2Wcj87AclHazuv2d7p6SVnW4MQHt9rNfsti+XdJ2kZ6tFd9nebnuj7eU164zZnrA9cUp8dhPolQWH3fYySY9K+nZEHJX0Q0lXSlqj2SP/d+dbLyLGI2I0IkYHNdSGlgE0Y0Fhtz2o2aA/HBGPSVJEHI6I6YiYkfQjSWs71yaAVjUMu21LelDSzoj43pzlI3Me9lVJTOcJ9LGFvBv/BUl3SHrR9rZq2b2S1tteo9nhuL2SvtGRDtGSJW+Vh95GfrytWJ86fryd7aCHFvJu/K8kzfc/hjF14AzCJ+iAJAg7kARhB5Ig7EAShB1IgrADSfBV0meASx/4745tu/7iWJxtOLIDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKOiO7tzH5T0r45iy6S9FbXGvh4+rW3fu1LordmtbO3T0fExfMVuhr2j+zcnoiI0Z41UNCvvfVrXxK9NatbvXEaDyRB2IEkeh328R7vv6Rfe+vXviR6a1ZXeuvpa3YA3dPrIzuALiHsQBI9CbvtW2y/Ynu37Xt60UMd23ttv2h7m+2JHvey0fYR2zvmLFthe4vtXdXtvHPs9ai3+2wfqJ67bbZv7VFvq2z/0vbLtl+y/a1qeU+fu0JfXXneuv6a3faApFclfUnSfknPSVofES93tZEatvdKGo2Inn8Aw/ZfSjom6aGIuLZa9s+S3o6I+6s/lMsj4h/7pLf7JB3r9TTe1WxFI3OnGZd0m6S/VQ+fu0Jft6sLz1svjuxrJe2OiD0RMSnpZ5LW9aCPvhcRz0h6+0OL10naVN3fpNn/LF1X01tfiIiDEfFCdf89SaenGe/pc1foqyt6EfaVkt6Y8/t+9dd87yHpadvP2x7rdTPzGI6Ig9X9Q5KGe9nMPBpO491NH5pmvG+eu2amP28Vb9B91A0R8TlJX5H0zep0tS/F7Guwfho7XdA03t0yzzTjH+jlc9fs9Oet6kXYD0haNef3y6plfSEiDlS3RyQ9rv6bivrw6Rl0q9sjPe7nA/00jfd804yrD567Xk5/3ouwPyfpattX2F4s6WuSNvegj4+wvbR640S2l0r6svpvKurNkjZU9zdIeqKHvfyefpnGu26acfX4uev59OcR0fUfSbdq9h351yT9Uy96qOlrtaRfVz8v9bo3SY9o9rTulGbf27hT0oWStkraJem/JK3oo97+XdKLkrZrNlgjPertBs2eom+XtK36ubXXz12hr648b3xcFkiCN+iAJAg7kARhB5Ig7EAShB1IgrADSRB2IIn/B8MV+XYqrhXrAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(x_test[0]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "0fb8aa1f-deac-4dd7-8ed3-dd59984cb0bd",
     "showTitle": false,
     "title": ""
    },
    "id": "uPeCvkowu2Eg"
   },
   "source": [
    "Now, we include a `Dropout` layer to the previous model and try to check the model accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "55134dab-f25e-45c5-8427-5cc89e134719",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DHIBuBWGjcGW",
    "outputId": "41792cec-bb9e-4bc7-fe8f-c40964e39dfc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/3\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.3926 - accuracy: 0.8826\n",
      "Epoch 2/3\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.2172 - accuracy: 0.9356\n",
      "Epoch 3/3\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.1799 - accuracy: 0.9449\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.models.Sequential([tf.keras.layers.Flatten(),\n",
    "                                    tf.keras.layers.Dense(128,activation='relu'),\n",
    "                                    tf.keras.layers.Dropout(0.5),\n",
    "                                    tf.keras.layers.Dense(128, activation='relu'),\n",
    "                                    tf.keras.layers.Dense(10,activation='softmax')])\n",
    "# Complie the model\n",
    "model.compile(optimizer= 'adam'\n",
    "             ,loss= 'sparse_categorical_crossentropy'\n",
    "             ,metrics= ['accuracy'])\n",
    "model.fit(x_train,y_train,epochs=3);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "cebd39b4-f411-411e-b760-229fabd6f17a",
     "showTitle": false,
     "title": ""
    },
    "id": "bJGi8ni8jcGX"
   },
   "source": [
    "We can see that the model accuracy goes down slightly as the model tries to generalize itself by reducing overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "a847fc5f-3ea7-4261-92c9-e104c07359b1",
     "showTitle": false,
     "title": ""
    },
    "id": "guyJbEb2jcGY"
   },
   "source": [
    "### V. Build a CNN Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "13143bc3-c8f6-4cd9-bf77-73b6b9166e14",
     "showTitle": false,
     "title": ""
    },
    "id": "rB_-aiBljcGY"
   },
   "source": [
    "A CNN model is primarily used for computer vision applications. The covert a deep learning network to a convolution neural network, we add a convolution and pooling layer prior to a fully connected layer. Next, let's look at the individual layers to used in the model and their purpose.\n",
    "\n",
    "**Conv2D layer**\n",
    "\n",
    "The fundamental difference between a densely connected layer and a specialized layer in the convolution operation, which we will call the convolutional layer, is that the dense layer learns global patterns in its global input space, while the convolutional layers learn local patterns in small windows of two dimensions. \n",
    "\n",
    "In an intuitive way, we could say that the main purpose of a convolutional layer is to detect features or visual features in images such as edges, lines, color drops, etc. This is a very interesting property because, once it has learned a characteristic at a specific point in the image, it can recognize it later in any part of it. Instead, in a densely connected neural network it has to learn the pattern again if it appears in a new location of the image. This way CNN can be used to reduce the number of parameters we need to train without sacrificing performance — the power of combining signal processing and deep learning.\n",
    "\n",
    "\n",
    "Keras `Conv2D` is a 2D Convolution Layer, this layer creates a convolution kernel that is convolved with the layer input to produce a tensor of outputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "a18db311-024a-4183-959f-c87b95fa09e5",
     "showTitle": false,
     "title": ""
    },
    "id": "LcP-xy_TjcGZ"
   },
   "source": [
    "**Pooling Layer**\n",
    "\n",
    "Convolutional neural networks accompany the convolution layer with pooling layers, which are usually applied immediately after the convolutional layers. The pooling layers simplify the information collected by the convolutional layer and create a condensed version of the information contained in them. In technical terms, pooling layers reduce the dimension of the feature maps. Thus, it reduces the number of parameters to learn and the amount of computation performed in the network.\n",
    "\n",
    "As mentioned above, the result of using a pooling layer is a summarized version of the features detected in the input. So, further operations are performed on summarised features instead of precisely positioned features generated by the convolution layer. This makes the model more robust to variations in the position of the features in the input image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "401407a5-f29a-46c1-be0e-a50d8ae7ac6d",
     "showTitle": false,
     "title": ""
    },
    "id": "TD5UQ0t0jcGZ"
   },
   "source": [
    "Before building the CNN model, lets import the data and perform the required data transformation to prepare the input for the conv2D layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "35061aa7-726a-4ca7-8a84-361d4b51aa3c",
     "showTitle": false,
     "title": ""
    },
    "id": "c-dYrbHWjcGZ"
   },
   "outputs": [],
   "source": [
    "# Load mnist data and split into train and test sets\n",
    "mnist = tf.keras.datasets.mnist\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "bf11c2d7-8853-4f08-97c9-7a084356b9bb",
     "showTitle": false,
     "title": ""
    },
    "id": "jMQxjbKTjcGa"
   },
   "source": [
    "The training and test data needs to be reshaped because the first convolution layer expects an array with the number of images, height, width and number of channels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "07c6f851-887b-4303-8246-438915e48f3f",
     "showTitle": false,
     "title": ""
    },
    "id": "mf9AIxB4jcGa"
   },
   "outputs": [],
   "source": [
    "x_train = x_train.reshape(60000,28,28,1)\n",
    "x_train = x_train / 255.0\n",
    "x_test = x_test.reshape(10000, 28, 28, 1)\n",
    "x_test = x_test / 255.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "91eb2a79-d075-4a0c-9f69-f36d0d924452",
     "showTitle": false,
     "title": ""
    },
    "id": "5nnL4o89jcGa"
   },
   "source": [
    "**Complete CNN model**\n",
    "\n",
    "1. The first hidden layer is a convolutional layer called a `Convolution2D`. The layer has 32 feature maps, which with a 3x3 kernel and a rectifier activation function\n",
    "2. Next we define a pooling layer `MaxPooling2D` layer that takes the max, configured with a pool size of 2×2.\n",
    "3. `Dropout` layer configured to randomly exclude 20% of neurons in the layer in order to reduce overfitting\n",
    "4. Next is the `Flatten` layer (same as used before in DNN model) that converts the 2D matrix data to a vector. It allows the output to be processed by standard fully connected layers.\n",
    "5. Next a fully connected layer with 128 neurons and rectifier activation function.\n",
    "6. Finally, the output layer has 10 neurons for the 10 classes and a `softmax` activation function to output probability-like predictions for each class.\n",
    "\n",
    "To summarize, we can split the process of image detection via a CNN model into two parts:\n",
    "\n",
    "1. Feature extraction - Cov2D and pooling layers help to extract important features using the filters used \n",
    "2. Classification - After feature extraction we need to classify the data into various classes. Dense layer along with the activation function (FC - fully connected neural network) is used to classify image based on the output from convolution layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "3c8d2deb-304a-4b48-9b56-7d2fb1dd8ef5",
     "showTitle": false,
     "title": ""
    },
    "id": "4qp_4ExvjcGa"
   },
   "outputs": [],
   "source": [
    "model = tf.keras.Sequential([tf.keras.layers.Conv2D(32, kernel_size=3, input_shape=(28, 28, 1), activation='relu'),\n",
    "                             tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),\n",
    "                             tf.keras.layers.Flatten(),\n",
    "                             tf.keras.layers.Dense(128, activation='relu'),\n",
    "                             tf.keras.layers.Dense(128, activation='relu'),\n",
    "                             tf.keras.layers.Dense(10, activation='softmax')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "0d54bf37-11ce-4d75-9b1a-2abb86cfcd73",
     "showTitle": false,
     "title": ""
    },
    "id": "iut4nW6SjcGa"
   },
   "source": [
    "As before, the model is trained using logarithmic loss and the `adam` gradient descent algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "a1402ecb-7f7a-4643-bf9e-6cbc998eee6d",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mMCdXNE4jcGb",
    "outputId": "d4c1132e-95a7-42f6-ba9b-379892230643"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_4\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D) (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "flatten_3 (Flatten)          (None, 5408)              0         \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 128)               692352    \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 128)               16512     \n",
      "_________________________________________________________________\n",
      "dense_9 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 710,474\n",
      "Trainable params: 710,474\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Compile model using accuracy to measure model performance\n",
    "model.compile(optimizer='adam', \n",
    "              loss='sparse_categorical_crossentropy', \n",
    "              metrics=['accuracy'])\n",
    "# Print model summary\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "59fc401a-976f-4267-bbbe-61330c1a525e",
     "showTitle": false,
     "title": ""
    },
    "id": "qNMboyw9jcGb"
   },
   "source": [
    "The size of the ouput of a convolution layers in simplest case is `input_size - (filter_size - 1)`. If the input size is 28x28 and filter is 3x3, then the output shape will be 28-(3-1) = 26.\n",
    "\n",
    "If we don't want to downsample the output, we can zero pad the image. In this case, the size of the ouput is calculated using this formula `input_size + 2\\*padding_size - (filter_size - 1)`.\n",
    "\n",
    "If we explicitly want to downsample the image during convolution, then we define a stride. In this scenario, the output size is calculated as `((input_size = 2\\*padding - filter_size)/stride) + 1`.\n",
    "\n",
    "\n",
    "Now, lets try to understand the math related to the shape and #parameters for the above model:\n",
    "\n",
    "1. The convolution layer output shape with filter size 3x3, 0 padding and a stide of 1 is calculated as follows: (28 +(2\\*0) - 3)/1 + 1 = 26. The #parameters is calculated as follows: (3\\*3 + 1)\\*32 = 320\n",
    "2. The max pooling layer output shape is calculated using the same formula as that for calculating conv layer output: (26 + (2\\*0) -2)/2 +1 = 13. This layer has 0 parameters as the network is not learning any new features.\n",
    "3. The flatten layer shape is calculated as follows: 13\\*13*32 = 5408\n",
    "4. The #parameters for the first fully connected layer is 128*5408 + 128 = 692,352\n",
    "5. The #parameters for the second fully connected layer is 128*128 + 128 = 16,512\n",
    "5. The #parameters for the output layer is 128*10 + 10 = 1290\n",
    "\n",
    "To learn more about the convolution neural network and the math behind it, refer to this amazing free DeepLearning.AI [youtube video playlist](https://bit.ly/2UGa7uH) from Andrew  Ng."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "18a8a0f7-c95e-4236-98a2-50c63e25f863",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZqBv2k2ZjcGb",
    "outputId": "25dd625e-0b3c-4c7e-e687-21de7f0f8279",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1875/1875 [==============================] - 37s 19ms/step - loss: 0.1466 - accuracy: 0.9552 - val_loss: 0.0793 - val_accuracy: 0.9743\n",
      "Epoch 2/5\n",
      "1875/1875 [==============================] - 34s 18ms/step - loss: 0.0489 - accuracy: 0.9848 - val_loss: 0.0477 - val_accuracy: 0.9859\n",
      "Epoch 3/5\n",
      "1875/1875 [==============================] - 35s 19ms/step - loss: 0.0304 - accuracy: 0.9899 - val_loss: 0.0438 - val_accuracy: 0.9874\n",
      "Epoch 4/5\n",
      "1875/1875 [==============================] - 35s 19ms/step - loss: 0.0216 - accuracy: 0.9931 - val_loss: 0.0424 - val_accuracy: 0.9866\n",
      "Epoch 5/5\n",
      "1875/1875 [==============================] - 35s 19ms/step - loss: 0.0152 - accuracy: 0.9951 - val_loss: 0.0461 - val_accuracy: 0.9873\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "history = model.fit(x_train, y_train,validation_data=(x_test, y_test), epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "b43531f0-c136-45df-a17f-c5f7433f821c",
     "showTitle": false,
     "title": ""
    },
    "id": "WYKHJio7u2Em"
   },
   "source": [
    "We can see that the CNN model has lower loss and higher accuracy compared to the DNN model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "72410c4c-eef8-47b5-8a5e-30c0eb18df66",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "id": "JbHHLfORjcGb",
    "outputId": "15dce34a-90ba-4293-c999-bdedfac276e4",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXwV1fn48c+Tm30FQkggISRWENlkCaBSLa4FtVIVt1qRWhdsrVvdi4Co327WWr/VtqhVcSn61eqP1rUWrVurAiIKolKSSICEkEASErI/vz9mEi/hJrkJuZksz/v1yitzz5yZee4E7nPPOTNnRFUxxhhjWgrzOgBjjDE9kyUIY4wxAVmCMMYYE5AlCGOMMQFZgjDGGBOQJQhjjDEBWYIw3UJEXhaRi7q6rpdEJE9ETgzBflVEDnWX/ygitwVTtxPHuUBEXutsnG3sd6aIFHT1fk33C/c6ANNzichev5exQA3Q4L6+XFWfDHZfqjo7FHX7OlVd0BX7EZEsIBeIUNV6d99PAkH/DU3/YwnCtEpV45uWRSQPuERVX29ZT0TCmz50jDF9h3UxmQ5r6kIQkZtEpBB4REQGisjfRaRYRHa7yxl+27wpIpe4y/NF5B0RudutmysisztZN1tE3hKRChF5XUTuF5EnWok7mBjvEJF33f29JiKD/dZfKCL5IlIiIj9r4/xMF5FCEfH5lZ0hIuvd5Wki8m8R2SMiO0Tk9yIS2cq+HhWRO/1e3+Bus11ELm5R91QR+UhEykVkq4gs8Vv9lvt7j4jsFZGjms6t3/ZHi8iHIlLm/j462HPTFhE53N1+j4hsEJHT/dadIiIb3X1uE5Hr3fLB7t9nj4iUisjbImKfV93MTrjprDRgEDACuAzn39Ij7utMYB/w+za2nw58DgwGfgU8LCLSibpPAR8AycAS4MI2jhlMjN8DfgAMASKBpg+sMcAf3P0Pc4+XQQCq+j5QCRzfYr9PucsNwLXu+zkKOAH4URtx48Ywy43nJGAk0HL8oxKYBwwATgWuEJHvuuuOdX8PUNV4Vf13i30PAl4E7nPf2z3AiyKS3OI9HHBu2ok5Avgb8Jq73U+AJ0XkMLfKwzjdlQnAOGCVW/5ToABIAVKBWwGbF6ibWYIwndUILFbVGlXdp6olqvqcqlapagVwF/CtNrbPV9UHVbUBeAwYivNBEHRdEckEpgKLVLVWVd8BVrZ2wCBjfERVv1DVfcAzwES3fC7wd1V9S1VrgNvcc9CavwDnA4hIAnCKW4aqrlHV/6hqvarmAX8KEEcg57jxfaqqlTgJ0f/9vamqn6hqo6qud48XzH7BSShfqurjblx/ATYB3/Gr09q5acuRQDzwC/dvtAr4O+65AeqAMSKSqKq7VXWtX/lQYISq1qnq22oTx3U7SxCms4pVtbrphYjEisif3C6YcpwujQH+3SwtFDYtqGqVuxjfwbrDgFK/MoCtrQUcZIyFfstVfjEN89+3+wFd0tqxcFoLZ4pIFHAmsFZV8904RrndJ4VuHP+D05poz34xAPkt3t90EXnD7UIrAxYEud+mfee3KMsH0v1et3Zu2o1ZVf2Tqf9+z8JJnvki8i8ROcot/zWwGXhNRLaIyM3BvQ3TlSxBmM5q+W3up8BhwHRVTeTrLo3Wuo26wg5gkIjE+pUNb6P+wcS4w3/f7jGTW6usqhtxPghns3/3EjhdVZuAkW4ct3YmBpxuMn9P4bSghqtqEvBHv/229+17O07Xm79MYFsQcbW33+Etxg+a96uqH6rqHJzupxdwWiaoaoWq/lRVDwFOB64TkRMOMhbTQZYgTFdJwOnT3+P2Zy8O9QHdb+SrgSUiEul++/xOG5scTIzPAqeJyDfdAeWltP//5yngapxE9H8t4igH9orIaOCKIGN4BpgvImPcBNUy/gScFlW1iEzDSUxNinG6xA5pZd8vAaNE5HsiEi4i5wJjcLqDDsb7OK2NG0UkQkRm4vyNVrh/swtEJElV63DOSSOAiJwmIoe6Y01lOOM2bXXpmRCwBGG6yr1ADLAL+A/wSjcd9wKcgd4S4E7gaZz7NQLpdIyqugH4Mc6H/g5gN84galuaxgBWqeouv/LrcT68K4AH3ZiDieFl9z2swul+WdWiyo+ApSJSASzC/TbubluFM+byrntl0JEt9l0CnIbTyioBbgROaxF3h6lqLU5CmI1z3h8A5qnqJrfKhUCe29W2AOfvCc4g/OvAXuDfwAOq+sbBxGI6Tmzcx/QlIvI0sElVQ96CMaavsxaE6dVEZKqIfENEwtzLQOfg9GUbYw6S3Ulters04K84A8YFwBWq+pG3IRnTN1gXkzHGmICsi8kYY0xAfaaLafDgwZqVleV1GMYY06usWbNml6qmBFrXZxJEVlYWq1ev9joMY4zpVUSk5R30zayLyRhjTECWIIwxxgRkCcIYY0xAfWYMwhjT/erq6igoKKC6urr9ysZT0dHRZGRkEBEREfQ2liCMMZ1WUFBAQkICWVlZtP68J+M1VaWkpISCggKys7OD3s66mIwxnVZdXU1ycrIlhx5OREhOTu5wS88ShDHmoFhy6B0683fq9wmivr6MLVsWUlX1pdehGGNMj9LvE0RDwz4KCn5LXt4Sr0MxxnRQSUkJEydOZOLEiaSlpZGent78ura2ts1tV69ezVVXXdXuMY4++uguifXNN9/ktNNO65J9dZd+P0gdFZVGevpP2Lr1V2Rm3kJ8/DivQzLGBCk5OZl169YBsGTJEuLj47n++uub19fX1xMeHvhjLicnh5ycnHaP8d5773VNsL1Qv29BAGRm3ojPl0Be3iKvQzHGHKT58+ezYMECpk+fzo033sgHH3zAUUcdxaRJkzj66KP5/PPPgf2/0S9ZsoSLL76YmTNncsghh3Dfffc17y8+Pr65/syZM5k7dy6jR4/mggsuoGk27JdeeonRo0czZcoUrrrqqnZbCqWlpXz3u99lwoQJHHnkkaxfvx6Af/3rX80toEmTJlFRUcGOHTs49thjmThxIuPGjePtt9/u8nPWmn7fggCIiBjE8OE/JS9vMRUVa0hImOJ1SMb0Ol9+eQ17967r0n3Gx09k5Mh7O7xdQUEB7733Hj6fj/Lyct5++23Cw8N5/fXXufXWW3nuuecO2GbTpk288cYbVFRUcNhhh3HFFVcccM/ARx99xIYNGxg2bBgzZszg3XffJScnh8svv5y33nqL7Oxszj///HbjW7x4MZMmTeKFF15g1apVzJs3j3Xr1nH33Xdz//33M2PGDPbu3Ut0dDTLli3j29/+Nj/72c9oaGigqqqqw+ejs6wF4crIuIbw8GRycxd6HYox5iCdffbZ+Hw+AMrKyjj77LMZN24c1157LRs2bAi4zamnnkpUVBSDBw9myJAhFBUVHVBn2rRpZGRkEBYWxsSJE8nLy2PTpk0ccsghzfcXBJMg3nnnHS688EIAjj/+eEpKSigvL2fGjBlcd9113HfffezZs4fw8HCmTp3KI488wpIlS/jkk09ISEjo7GnpMGtBuMLDE8nMvIktW25kz553GDDgm16HZEyv0plv+qESFxfXvHzbbbdx3HHH8fzzz5OXl8fMmTMDbhMVFdW87PP5qK+v71Sdg3HzzTdz6qmn8tJLLzFjxgxeffVVjj32WN566y1efPFF5s+fz3XXXce8efO69LitsRaEn/T0HxMZmUZu7s+wJ+0Z0zeUlZWRnp4OwKOPPtrl+z/ssMPYsmULeXl5ADz99NPtbnPMMcfw5JNPAs7YxuDBg0lMTOS///0v48eP56abbmLq1Kls2rSJ/Px8UlNTufTSS7nkkktYu3Ztl7+H1oQ0QYjILBH5XEQ2i8jNAdYfKyJrRaReROYGWJ8oIgUi8vtQxtnE54slM/NnlJW9xe7dr3fHIY0xIXbjjTdyyy23MGnSpC7/xg8QExPDAw88wKxZs5gyZQoJCQkkJSW1uc2SJUtYs2YNEyZM4Oabb+axxx4D4N5772XcuHFMmDCBiIgIZs+ezZtvvskRRxzBpEmTePrpp7n66qu7/D20JmTPpBYRH/AFcBLOw+Q/BM5X1Y1+dbKAROB6YKWqPttiH78DUoBSVb2yrePl5ORoVzwwqLGxhvffH0VkZBqTJ//H7hI1pg2fffYZhx9+uNdheG7v3r3Ex8ejqvz4xz9m5MiRXHvttV6HdYBAfy8RWaOqAa/3DWULYhqwWVW3qGotsAKY419BVfNUdT3Q2HJjEZkCpAKvhTDGA4SFRZGVtYiKig8oKflbdx7aGNNLPfjgg0ycOJGxY8dSVlbG5Zdf7nVIXSKUCSId2Or3usAta5eIhAG/wWlZtFXvMhFZLSKri4uLOx1oS6mpFxETM5Lc3NtQPSB3GWPMfq699lrWrVvHxo0befLJJ4mNjfU6pC7RUwepfwS8pKoFbVVS1WWqmqOqOSkpAZ+53SlhYeFkZd1OZeV6iov/r8v2a4wxvUkoE8Q2YLjf6wy3LBhHAVeKSB5wNzBPRH7RteG1bciQc4mLG0du7iIaG7t+YMsYY3q6UCaID4GRIpItIpHAecDKYDZU1QtUNVNVs3C6mZar6gFXQYWSSBhZWXewb98XFBU90Z2HNsaYHiFkCUJV64ErgVeBz4BnVHWDiCwVkdMBRGSqiBQAZwN/EpHAtzh6ZPDgOSQk5JCffzuNjW3PDGmMMX1NSMcgVPUlVR2lqt9Q1bvcskWqutJd/lBVM1Q1TlWTVXVsgH082t4lrqEiImRn30l1dR47djzkRQjGmDYcd9xxvPrqq/uV3XvvvVxxxRWtbjNz5kyaLok/5ZRT2LNnzwF1lixZwt13393msV944QU2bmy+ap9Fixbx+usHf/9UT5oWvKcOUvcYAweeTFLSMeTn30lDwz6vwzHG+Dn//PNZsWLFfmUrVqwIaj4kcGZhHTBgQKeO3TJBLF26lBNPPLFT++qpLEG0o6kVUVu7g+3bH/A6HGOMn7lz5/Liiy82PxwoLy+P7du3c8wxx3DFFVeQk5PD2LFjWbx4ccDts7Ky2LVrFwB33XUXo0aN4pvf/GbzlODg3OMwdepUjjjiCM466yyqqqp47733WLlyJTfccAMTJ07kv//9L/Pnz+fZZ517ff/5z38yadIkxo8fz8UXX0xNTU3z8RYvXszkyZMZP348mzZtavP9eT0tuE3WF4QBA45l4MCT+eqrXzB06GWEh3ffbIrG9BrXXAPruna6byZOhHtbnwRw0KBBTJs2jZdffpk5c+awYsUKzjnnHESEu+66i0GDBtHQ0MAJJ5zA+vXrmTBhQsD9rFmzhhUrVrBu3Trq6+uZPHkyU6Y40/6feeaZXHrppQAsXLiQhx9+mJ/85CecfvrpnHbaacydu/8sQdXV1cyfP59//vOfjBo1innz5vGHP/yBa665BoDBgwezdu1aHnjgAe6++24eeqj17muvpwW3FkSQsrPvoK5uFwUFv/M6FGOMH/9uJv/upWeeeYbJkyczadIkNmzYsF93UEtvv/02Z5xxBrGxsSQmJnL66ac3r/v000855phjGD9+PE8++WSr04U3+fzzz8nOzmbUqFEAXHTRRbz11lvN688880wApkyZ0jzBX2u8nhbcWhBBSkycRnLyHLZuvZv09B8TETHQ65CM6Vna+KYfSnPmzOHaa69l7dq1VFVVMWXKFHJzc7n77rv58MMPGThwIPPnz6e6urpT+58/fz4vvPACRxxxBI8++ihvvvnmQcXbNGX4wUwX3l3TglsLogOys++goaGcrVvbvrrBGNN94uPjOe6447j44oubWw/l5eXExcWRlJREUVERL7/8cpv7OPbYY3nhhRfYt28fFRUV/O1vX8/DVlFRwdChQ6mrq2ueohsgISGBioqKA/Z12GGHkZeXx+bNmwF4/PHH+da3vtWp9+b1tODWguiA+PjxDBlyLgUFvyMj42oiI4d4HZIxBqeb6YwzzmjuamqaHnv06NEMHz6cGTNmtLn95MmTOffcczniiCMYMmQIU6dObV53xx13MH36dFJSUpg+fXpzUjjvvPO49NJLue+++5oHpwGio6N55JFHOPvss6mvr2fq1KksWLCgU++r6VnZEyZMIDY2dr9pwd944w3CwsIYO3Yss2fPZsWKFfz6178mIiKC+Ph4li9f3qlj+gvZdN/draum+25PVdUXfPDB4WRkXM2hh94T8uMZ05PZdN+9S0+a7rtPio0dRVraRWzb9gDV1W3OJWiMMb2aJYhOGDFiEdBIfv6dXodijDEhYwmiE2Jishg69DIKCx9m374tXodjjKf6Sjd1X9eZv5MliE4aMeJWRMLJy7vd61CM8Ux0dDQlJSWWJHo4VaWkpITo6OgObWdXMXVSVNQw0tOvZOvWe8jMvJm4OBuoM/1PRkYGBQUFdOUTHU1oREdHk5GR0aFtLEEchOHDb2L79j+Sl7eYsWOf8TocY7pdREQE2dnZXodhQsS6mA5CZORgMjKupbj4/6io+MjrcIwxpktZgjhIGRnXER4+kLy8RV6HYowxXcoSxEGKiBjA8OE3UFLyd8rK/uN1OMYY02UsQXSBjIyriIgYQm7uz7wOxRhjuowliC7g88WRmXkLe/asYvfuVV6HY4wxXcISRBcZNmwBUVEZ5OYutGvCjTF9giWILuLzRTNixG2Ul/+b0tK2pxY2xpjeIKQJQkRmicjnIrJZRG4OsP5YEVkrIvUiMtevfKKI/FtENojIehE5N5RxdpW0tB8QHX2I24po9DocY4w5KCFLECLiA+4HZgNjgPNFZEyLal8B84GnWpRXAfNUdSwwC7hXRAaEKtauEhYWQVbWEvbu/Yji4r96HY4xxhyUULYgpgGbVXWLqtYCK4A5/hVUNU9V1wONLcq/UNUv3eXtwE4gJYSxdpnU1O8RG3s4eXmLUG3wOhxjjOm0UCaIdGCr3+sCt6xDRGQaEAn8N8C6y0RktYis7ilzwYj4yMpaSlXVZxQVtWwYGWNM79GjB6lFZCjwOPADDdCpr6rLVDVHVXNSUnpOAyMl5Uzi4yeRl7eExsY6r8MxxphOCWWC2AYM93ud4ZYFRUQSgReBn6lqr7pFWSSM7Ow7qK7eQmHhI16HY4wxnRLKBPEhMFJEskUkEjgPWBnMhm7954Hlqvpse/V7okGDTiEx8Sjy8++goaHa63CMMabDQpYgVLUeuBJ4FfgMeEZVN4jIUhE5HUBEpopIAXA28CcR2eBufg5wLDBfRNa5PxNDFWsoiAjZ2XdSU1PAjh1/8jocY4zpMOkrd/3m5OTo6tWrvQ7jAOvWnUBl5acceeQWfL44r8Mxxpj9iMgaVc0JtK5HD1L3BdnZd1JXt5OCgv/1OhRjjOkQSxAhlpR0FIMGncrWrb+irm6P1+EYY0zQLEF0g+zsO6iv301BwT1eh2KMMUGzBNENEhImkZIyl4KC31Jbu8vrcIwxJiiWILpJVtZSGhqq2Lr1l16HYowxQbEE0U3i4g4nNfX7bNv2e2pqdngdjjHGtMsSRDfKylqMaj35+Xd5HYoxxrTLEkQ3iok5hLS0H7JjxzL27cvzOhxjjGmTJYhuNmLEQiCM/Pw7vA7FGGPaZAmim0VHZ5CefgWFhY9RVfWF1+EYY0yrLEF4IDPzFsLCosjLW+J1KMYY0ypLEB6IjBxCRsbV7Ny5gr17P/E6HGOMCcgShEeGD78Bny+R3NzbvA7FGGMCsgThkYiIgQwffj0lJf+P8vIPvQ7HGGMOYAnCQxkZVxMRMZjc3IVeh2KMMQewBOGh8PAEMjNvZvfu19iz5y2vwzHGmP1YgvDYsGE/IjJyKLm5C+krD28yxvQNliA85vPFMGLEQsrK3mb37te8DscYY5pZgugBhg69hKioEdaKMMb0KJYgeoCwsEiyshZTUbGaXbv+n9fhGGMMYAmix0hNvZCYmFHk5d2GaqPX4RhjTGgThIjMEpHPRWSziNwcYP2xIrJWROpFZG6LdReJyJfuz0WhjLMnCAsLJyvrdiorP2Xnzqe9DscYY0KXIETEB9wPzAbGAOeLyJgW1b4C5gNPtdh2ELAYmA5MAxaLyMBQxdpTDBlyDnFxE8jLW0xjY73X4Rhj+rlQtiCmAZtVdYuq1gIrgDn+FVQ1T1XXAy37VL4N/ENVS1V1N/APYFYIY+0RRMLIzr6Dffu+pKhoudfhGGP6uVAmiHRgq9/rAresy7YVkctEZLWIrC4uLu50oD1JcvJ3SEiYRl7e7TQ21ngdjjGmH+vVg9SqukxVc1Q1JyUlxetwuoSIkJ19JzU1X7Fjx0Neh2OM6cdCmSC2AcP9Xme4ZaHettcbOPBEkpKOJT//ThoaqrwOxxjTT4UyQXwIjBSRbBGJBM4DVga57avAySIy0B2cPtkt6xecVsRd1NYWsm3b/V6HY4zpp0KWIFS1HrgS54P9M+AZVd0gIktF5HQAEZkqIgXA2cCfRGSDu20pcAdOkvkQWOqW9RsDBnyTQYNm8dVXv6S+vtzrcIwx/ZD0lakdcnJydPXq1V6H0aXKy1ezdu1UsrJuJytrkdfhGGP6IBFZo6o5gdb16kHqvi4xMYfBg89g69bfUFfXrxpQxpgewBJED5eVtZSGhgq2bv2116EYY/oZSxA9XHz8OIYMOZ+CgvuoqSn0OhxjTD9iCaIXyMpaQmNjDV999QuvQzHG9COWIHqB2NiRpKXNZ/v2P1BdvbX9DYwxpgtYguglmq5iys+/0+NIjDH9hSWIXiI6OpNhwy6jsPDP7Nv3X6/DMcb0A5YgepHMzJ8hEkFe3hKvQzHG9AOWIHqRqKg00tN/QlHRk1RWbvQ6HGNMH2cJopfJzLwRny+e3Fy7s9oYE1qWIHqZiIhkMjKuY9eu56ioWOt1OMaYPswSRC80fPi1hIcPJDf3Nq9DMcb0YZYgeqHw8CQyM2+itPQlysre8zocY0wfZQmil0pPv5KIiFRycxd6HYoxpo+yBNFL+XxxjBhxK3v2vMHu3f/0OhxjTB8UVIIQkTgRCXOXR4nI6SISEdrQTHuGDbucqKjh5OYupK8818MY03ME24J4C4gWkXTgNeBC4NFQBWWCExYWxYgRt1Fe/h9KSl70OhxjTB8TbIIQVa0CzgQeUNWzgbGhC8sEKy1tPtHR33BbEY1eh2OM6UOCThAichRwAdD0VdUXmpBMR4SFRZCdfTuVlR9TXPyc1+EYY/qQYBPENcAtwPOqukFEDgHeCF1YpiOGDDmP2Ngx5OUtQrXB63CMMX1EUAlCVf+lqqer6i/dwepdqnpViGMzQRLxkZ19B1VVmygqetLrcIwxfUSwVzE9JSKJIhIHfApsFJEbgthuloh8LiKbReTmAOujRORpd/37IpLllkeIyGMi8omIfCYit3TsbfU/gwefQXz8ZPLyltDYWOt1OMaYPiDYLqYxqloOfBd4GcjGuZKpVSLiA+4HZgNjgPNFZEyLaj8EdqvqocBvgV+65WcDUao6HpgCXN6UPExgIkJ29p1UV+eyY8efvQ7HGNMHBJsgItz7Hr4LrFTVOqC9C++nAZtVdYuq1gIrgDkt6swBHnOXnwVOEBFx9x0nIuFADFALlAcZa781aNAsEhNnkJ9/Bw0N+7wOxxjTywWbIP4E5AFxwFsiMoL2P7DTAf8HKBe4ZQHrqGo9UAYk4ySLSmAH8BVwt6qWtjyAiFwmIqtFZHVxcXGQbyWAZcug9IDd9zpNrYja2u1s3/5Hr8MxxvRywQ5S36eq6ap6ijrygeNCGNc0oAEYhtOd9VP3yqmWcS1T1RxVzUlJSenckb74An70Izj8cHjqKejldyQPHDiTgQNP5Kuvfk59/V6vwzHG9GLBDlInicg9Td/WReQ3OK2JtmwDhvu9znDLAtZxu5OSgBLge8ArqlqnqjuBd4GcYGLtsFGjYM0ayM6GCy6Ak0+GzZtDcqjukp19J3V1xWzbdp/XoRhjerFgu5j+DFQA57g/5cAj7WzzITBSRLJFJBI4D1jZos5K4CJ3eS6wSp1Jhb4CjgdnHijgSGBTkLF23BFHwLvvwv33wwcfwLhxcNddUNs7rwZKTJxOcvJ32Lr119TV7fE6HGNMLxVsgviGqi52B5y3qOrtwAFdPv7cMYUrgVeBz4Bn3JvslorI6W61h4FkEdkMXAc0XQp7PxAvIhtwEs0jqrq+Y2+tg3w+p6vps89gzhxYuBAmToS33w7pYUMlO/sO6uv3UFDwG69DMcb0UhLMLKAi8m/gBlV9x309A2fg+KgQxxe0nJwcXb16ddft8KWXnISRnw+XXAK//CUMGtR1++8GGzacS2npS0yfvoXIyE6O0Rhj+jQRWaOqAbvwg21BLADuF5E8EckDfg9c3kXx9UynnAIbNsCNN8Ijj8Do0fDEE71qEDsr63YaGqr46qtfeB2KMaYXCvYqpo9V9QhgAjBBVSfhjhH0aXFxTsth7Vr4xjfgwgvhpJPgyy+9jiwocXGjSUubx/btD1BT0/L6AGOMaVuHniinquXuHdXgjBn0DxMmOIPYf/gDrF4N48fDnXdCTY3XkbVrxAhnAr/8/Lu8DsUY08sczCNHpcui6A3CwmDBAmcQ+7vfhdtucwax33rL68jaFBOTzdChl7Bjx0Ps25frdTjGmF7kYBJE7+mM70pDh8KKFfDyy04L4lvfgh/+EEpKvI6sVSNG/AwRH/n5S70OxRjTi7SZIESkQkTKA/xU4Nzl3H/NmgWffgo33wzLlzuD2MuX98hB7KiodIYN+xGFhcuprAzd7STGmL6lzQShqgmqmhjgJ0FVw7sryB4rNhZ+/nNnEHvkSLjoIjjxRGf6jh4mM/NmwsJiyMtb4nUoxphe4mC6mEyT8ePhnXfgj390pu0YPx6WLu1Rg9iRkSlkZFxDcfHT7N37sdfhGGN6AUsQXSUsDC6/HDZtgrPOgsWLnSk8/vUvryNrNnz49YSHDyA3d5HXoRhjegFLEF0tLc2ZFfaVV5y5nGbOhB/8AHbt8joyIiIGMHz49ZSUrKS8/H2vwzHG9HCWIELl2992BrFvucW5A3v0aHjsMc8HsdPTryYiIoXc3IWexmGM6fksQYRSbCz8z//ARx/BYYfB/Plw/PHw+eeehRQeHk9m5i3s3v06u3e/6VkcxjjHCtwAABk0SURBVJiezxJEdxg3zpkVdtkyWLfOuTP79ts9G8QeNmwBkZHDyM1dSDCTNRpj+idLEN0lLAwuvdQZxJ47F5YscRLFm292eyg+XwwjRtxGefm7lJa+2u3HN8b0DpYgultqKjz5JLz6KtTXw3HHOV1P3TyIPXToxURHZ1krwhjTKksQXjn5ZGcQ+9ZbnYQxejQ8+mi3DWKHhUWSlbWEvXvXsGvX891yTGNM72IJwksxMc6jTdetcxLED37gtCg2dc90GKmp3yc2djS5uc6Mr8YY488SRE8wdqwzK+yDD8LHHztjE4sXQ3V1SA8r4iMr63aqqjawc+eKkB7LGNP7WILoKcLCnEebbtoE55zjTNUxYQKsWhXSw6akzCUu7gjy8pbQ2FgX0mMZY3oXSxA9TWqqc2Pda69BYyOccALMmwfFxSE5nEgY2dl3sG/fZgoLHwvJMYwxvZMliJ7qpJPgk09g4ULn+ROjR8Of/xySQezk5NNISJhOfv5SGht7zgSDxhhvhTRBiMgsEflcRDaLyM0B1keJyNPu+vdFJMtv3QQR+beIbBCRT0QkOpSx9kgxMXDHHc4g9pgxzoOJZs50nmrXhUSEQw65i5qarWzfvqxL922M6b1CliBExAfcD8wGxgDni8iYFtV+COxW1UOB3wK/dLcNB54AFqjqWGAm0H87yMeMcWaFfeghp1VxxBGwaFGXDmIPGHA8AwbMJD//LhoaKrtsv8aY3iuULYhpwGZV3aKqtcAKYE6LOnOApo7vZ4ETRESAk4H1qvoxgKqWaH+/DjMszGlBbNoE557rtCzGj4fXX++S3YsI2dl3UldXxLZt93fJPo0xvVsoE0Q6sNXvdYFbFrCOqtYDZUAyMApQEXlVRNaKyI2BDiAil4nIahFZXRyiQdweZ8gQePxx+Mc/nNcnnQQXXgg7dx70rpOSZjBo0Gy++uqX1NeXHfT+jDG9W08dpA4Hvglc4P4+Q0ROaFlJVZepao6q5qSkpHR3jN468USnu2nRInj6aWcQ++GHnSufDkJ29p3U15eydetvuyhQY0xvFcoEsQ0Y7vc6wy0LWMcdd0gCSnBaG2+p6i5VrQJeAiaHMNbeKTramRX244+d7qZLLnEGsTdu7PQuExImM3jwWRQU3ENdXUnXxWqM6XVCmSA+BEaKSLaIRALnAStb1FkJXOQuzwVWqTNz3KvAeBGJdRPHt4DOf+r1dYcf7swK++c/w4YNMHGic3nsvn2d2l129u00NOzlq69+1bVxGmN6lZAlCHdM4UqcD/vPgGdUdYOILBWR091qDwPJIrIZuA642d12N3APTpJZB6xV1RdDFWufIOLM5bRpE5x/vjPH0/jxX49VdEBc3FhSUy9g27b/paamMATBGmN6A+krUz3n5OTo6tWrvQ6j51i1ChYsgC+/hO99D+65x7lLO0hVVZv54IPRpKf/iJEj7wthoMYYL4nIGlXNCbSupw5Sm4N1/PGwfr0z6d+zzzqD2A8+GPQgdmzsoQwdejHbt/+J6uqvQhysMaYnsgTRl0VHO0+u+/hj5+a6yy6DY491ximCMGLEbQDk598RwiCNMT2VJYj+YPRoeOMNeOQRZ4xi4kTnQUXtDGJHRw9n2LAF7NjxCFVVX3ZTsMaYnsISRH8h4jzadNMm+P734ec/h3HjnEeftmHEiFsJC4siL+/27onTGNNjWILobwYPdloSq1ZBeDjMmuUMYhcGvlopMjKVjIyr2LnzKfbu/bSbgzXGeMkSRH913HHOIPaSJfDcc869FMuWBRzEHj78Bny+BPLyFnV/nMYYz1iC6M+iopyrnNavd8YlLr8cjjkGPt2/pRARMYjhw3/Krl3PU1GxxqNgjTHdzRKEgcMOc7qcHnsMPv8cJk2CW26BqqrmKhkZ1xAenkxu7kIPAzXGdCdLEMYh4jzadNMmZ3bYX/zCGcR+5RUAwsMTycy8idLSV9iz5x2PgzXGdAdLEGZ/gwc7czq98QZERsLs2c7UHYWFpKf/mMjINHJzF9JX7sA3xrTOEoQJbOZM5wa722+Hv/4VRo/G9+ByMjNuoazsX2zd+ht7ZoQxfZwlCNO6qCjneROffAJTpsAVV5B+3l8YUjSBLVtu4L330ti48XxKSl6isbHe62iNMV3MJuszwVGFJ56A665D9+yhccwhVCfsozKmiNqEWhoHxRE9fDoJ2d8mOiMHSUmBlBRIToaICK+jN8a0oq3J+ixBmI4pKXEGsDdtgl270F3F6M4dhJVXtb5NUpIztpGS4vz2/2lZlpLi1A+zxq0x3aGtBBHe3cGYXi45GX796+aX4v5QV0dd4WZKv3iKss1/pW7HRiLKIKH2GyTWfIOYyiTCSvbAtm3O2EZxMVRXBz6Gz+ccp60k0jLRxMV1x7s3xnuNjbBnD5SWOl/YSkud7uDjj+/yQ1kLwoREVdWXFBU9TlHR41RX5xEWFkdKylmkpc1jwICZCGHOfRa7djk/xcVfL7dV1tp05TEx7ScR/zLr+jJeU4Xy8q8/5P0/8Nv6vXu3s62/adPg/fc7FYZ1MRnPqDZSVvYuRUWPs3PnMzQ0lBEZmU5q6vdJS7uQuLixwe+s6ZtTawkkUEIpa+NKqwEDOpZUrOvLBKIKlZXBfbj7/y4thYaG1vebmAiDBjlfZtr7nZoK2dmdCt8ShOkRGhr2UVLyN4qKHqek5GWggfj4yaSlzWPIkPOJjBzS9QetrXX+QwabVIqLoaYm8L58vgMTSHtdYLGxXf+eTOhUVbX/oR5oXV1d6/uMi2v/Q75l2cCB3dbCtQRhepza2p3s3PkXCgsfZ+/eNYCPQYNmkZY2j+Tk0/H5or0JrOnbYHtJxP91SUn7XV/x8U4/cdNPZOT+r0Nd5vN173n0Wk1Nx77NN/1ubVwMnAdwBfNtvuUHf1RU973vTrAEYXq0ysoNFBY+TlHRE9TWbsPnS2LIkLNJTZ1HUtIMRHp4t05Dw/5dXy2TSHGxk3Rqa50PLv+ftsqCfDxsUHw+bxJTe2UibcddV9ex/vmm31VtXFUXEeF8gHfkQz452Un2fZAlCNMrqDawZ8+bFBYup7j4ORobK4mOziY19fukpl5IbOxIr0PsXvX17SeRUJW1VacrRUQETiQ1Nc4HfUVF69v6fB37Nt/0Oy6u/cTUj3iWIERkFvA7wAc8pKq/aLE+ClgOTAFKgHNVNc9vfSawEViiqne3dSxLEH1LQ0MlxcXPU1S0nN27XweUxMSjSE29kCFDziUiYpDXIfZPqs63+lAmoJoaJ0m011efmGgf9F3AkwQhIj7gC+AkoAD4EDhfVTf61fkRMEFVF4jIecAZqnqu3/pnAQXetwTRf9XUbKOo6CmKipZTWfkpIpEkJ59GauqFJCefQlhYpNchGtNreXWj3DRgs6pucYNYAczBaRE0mQMscZefBX4vIqKqKiLfBXKByhDGaHqBqKh0MjNvYPjw69m792OKipZTVPQku3b9lfDwZIYMOY+0tHkkJExF7BulMV0mlAkiHdjq97oAmN5aHVWtF5EyIFlEqoGbcFof14cwRtOLiAgJCRNJSJjIIYf8it27/0FR0XIKCx9m+/b7iYkZRVraPFJTv0909AivwzWm1+upU20sAX6rqnvb+kYoIpcBlwFkZmZ2T2SmRwgLCyc5eTbJybOpry+juPg5CguXk5u7kNzchSQlfYu0tHmkpMwlPDzR63CN6ZVCef3gNmC43+sMtyxgHREJB5JwBqunA78SkTzgGuBWEbmy5QFUdZmq5qhqTkpKSte/A9MrhIcnMXToxUya9CbTp+eSnX0ntbU7+PzzH/Lee6nulOQv25TkxnRQKAepw3EGqU/ASQQfAt9T1Q1+dX4MjPcbpD5TVc9psZ8lwF4bpDYdoapUVHxAYeHj7Nz5F+rrS4mISCU19Xukps4jPv4IG68wBo8Gqd0xhSuBV3Euc/2zqm4QkaXAalVdCTwMPC4im4FS4LxQxWP6FxEhMXE6iYnTOfTQeygtfZnCwuVs2/Z7Cgp+S1zceFJTLyQ19QKiooZ5Ha4xPZLdKGf6lbq6EnbufIaioscpL/83EMbAgSeSmnohKSln4PPZtOGmf7E7qY0JoN0pyaWfzV9k+iVLEMa04espyZe7U5KXExWV0TzFR1zcGK9DNCZkLEEYE6SmKckLC5dTWvoKzpTkU9wpyc8LzZTkxnjIEoQxnRBoSvLk5NnuFB8eTkluTBeyBGHMQQo8Jfk5flOS2yWzpneyBGFMF2l9SvIL3SnJD/U6RGM6xBKEMSHQ+pTk8xgy5Bybktz0CpYgjAmx1qYkT0ubx6BBs21KctNjWYIwppuo6n5TktfV7SQ8PJmBA48jLm58809MTLbdZ2F6BK+eB2FMv3PglOSvUVT0JOXl71Nc/BzO868gLCyGuLixxMWN80sc44iMTLMBb9NjWIIwJkScKclPITn5FMAZs6is3Ehl5adUVn5CZeUnlJa+QmHho83bhIcnEx8/vkXiGGtTlhtPWIIwppv4fHEkJk4lMXHqfuW1tcVu0vg6cRQWPkpDw97mOlFRI4iLG+cmDyeBxMaOtrENE1KWIIzxWGRkCpGRxzFw4HHNZapKdXW+mzA+bf69e/drqNYBIBJOTMwo4uLG79fqiI7OQiSUj3ox/YUlCGN6IBEhJiaLmJgsBg/+TnN5Y2MtVVVf7NfaqKj4gOLip5vrhIXFueMb4/drddg0IaajLEEY04uEhUUSHz+O+Phx+D8+pb6+gqqqjezd+0lza6OkZCWFhQ8314mISNlvQDw+fjyxsWMJD4/34J2Y3sAShDF9QHh4QvMDkvzV1hZRWfnpfoljx44HaWysaq4THZ3dnDSauqtiYkYRFhbR3W/D9DCWIIzpwyIjU4mMTGXgwBOay1Qbqa7O9UscTndVScmLQAMAIhHExo7eL3HExY0jOnqEXYbbj1iCMKafEQkjJuYbxMR8g8GD5zSXNzbWUFX1OZWVnzQnjrKyd9i586nmOj5fgpswWt6/MdiLt2JCzBKEMQaAsLAo4uMnEB8/gdTUr8vr68uorNzQ3EW1d+8nFBc/x44dDzbXiYxMOyBpxMWNxeeL9eCdmK5iCcIY06bw8CSSko4mKeno5jJVpba2sPlKqqbEsX37H2hsrHZrCdHRh+x374YzzchIwsLso6c3sL+SMabDRISoqKFERQ1l0KCTm8tVG9i3b8t+92/s3fsJu3atBBrdbSOJjT38gMQRFZVh4xs9TEgThIjMAn4H+ICHVPUXLdZHAcuBKUAJcK6q5onIScAvgEigFrhBVVeFMlZjzMET8REbO5LY2JGkpJzZXN7QUE1V1Wf7JY49e96kqOiJ5jo+X1Lz5bcxMYcRHZ1FTEw20dFZhIcnefF2+r2QJQhxpqq8HzgJKAA+FJGVqrrRr9oPgd2qeqiInAf8EjgX2AV8R1W3i8g44FUgPVSxGmNCy+eLJiFhEgkJk/Yrr6vbfcA0Izt3rqC+fs9+9cLDBxId7SSLpt9NySM6OgufL647306/EcoWxDRgs6puARCRFcAcwD9BzAGWuMvPAr8XEVHVj/zqbABiRCRKVWtCGK8xpptFRAxkwIBjGDDgmOYyVaWuroTq6jyqq3P3+11VtZHS0pf8xjma9pPSagKJihphzw/vpFAmiHRgq9/rAmB6a3VUtV5EyoBknBZEk7OAtZYcjOkfRITIyMFERg4mMfHAxxQ4A+RFARPI3r1r2bXr+eb5qppERg5tI4Fk2k2BrejRg9QiMhan2+nkVtZfBlwGkJmZ2Y2RGWO84gyQpxEVlUZS0pEHrFdtoKZmR8AEUl7+Hjt3Pk3TDYGOMKKi0ltJINlERaX324c7hTJBbAOG+73OcMsC1SkQkXAgCWewGhHJAJ4H5qnqfwMdQFWXAcvAeaJcl0ZvjOmVRHxER2cQHZ0BfPOA9Y2N9dTUFByQPKqrc9mzZxU1NdtoerCTs79woqIyWxn/yHYf8tQ3Z88NZYL4EBgpItk4ieA84Hst6qwELgL+DcwFVqmqisgA4EXgZlV9N4QxGmP6mbCw8OaZcgNpbKyhunprwARSWvoitbWF+9UXiSI6ekSrCSQiIqXXXr4bsgThjilciXMFkg/4s6puEJGlwGpVXQk8DDwuIpuBUr6envJK4FBgkYgscstOVtWdoYrXGGPAuaM8NvZQYmMPDbi+oWEf1dX5ARPIrl1rqavb1WJ/sc1XWwVKIOHhA3tsAhHVvtEzk5OTo6tXr/Y6DGNMP1dfX+GXQPZPIvv25dLQULZffZ8vsdXk4dwDEtrHzYrIGlU98GoAevggtTHG9Dbh4Ql+z+w4UF3dnoCtj+rq/7J79+s0Nla22N+gNhLIiJDeA2IJwhhjulFExAAiIg68aRD87wE5MIFUVW2gtPTFAPeADGHAgOMYO3ZFl8dqCcIYY3qI/e8BmXrAetVGamt3HpBAIiJSQhKPJQhjjOklRML87gE5KuTH65sX7xpjjDloliCMMcYEZAnCGGNMQJYgjDHGBGQJwhhjTECWIIwxxgRkCcIYY0xAliCMMcYE1Gcm6xORYiD/IHYxmP2fZNdTWFwdY3F1jMXVMX0xrhGqGvBW7D6TIA6WiKxubUZDL1lcHWNxdYzF1TH9LS7rYjLGGBOQJQhjjDEBWYL42jKvA2iFxdUxFlfHWFwd06/isjEIY4wxAVkLwhhjTECWIIwxxgTUrxKEiPxZRHaKyKetrBcRuU9ENovIehGZ3EPimikiZSKyzv1Z1E1xDReRN0Rko4hsEJGrA9Tp9nMWZFzdfs5EJFpEPhCRj924bg9QJ0pEnnbP1/siktVD4povIsV+5+uSUMfld2yfiHwkIn8PsK7bz1cQMXl5rvJE5BP3uKsDrO/a/4+q2m9+gGOBycCnraw/BXgZEOBI4P0eEtdM4O8enK+hwGR3OQH4Ahjj9TkLMq5uP2fuOYh3lyOA94EjW9T5EfBHd/k84OkeEtd84Pfd/W/MPfZ1wFOB/l5enK8gYvLyXOUBg9tY36X/H/tVC0JV3wJK26gyB1iujv8AA0RkaA+IyxOqukNV17rLFcBnQHqLat1+zoKMq9u552Cv+zLC/Wl5Fcgc4DF3+VngBBGRHhCXJ0QkAzgVeKiVKt1+voKIqSfr0v+P/SpBBCEd2Or3uoAe8MHjOsrtInhZRMZ298Hdpv0knG+f/jw9Z23EBR6cM7drYh2wE/iHqrZ6vlS1HigDkntAXABnud0Sz4rI8FDH5LoXuBFobGW9F+ervZjAm3MFTmJ/TUTWiMhlAdZ36f9HSxC9w1qc+VKOAP4XeKE7Dy4i8cBzwDWqWt6dx25LO3F5cs5UtUFVJwIZwDQRGdcdx21PEHH9DchS1QnAP/j6W3vIiMhpwE5VXRPqYwUryJi6/Vz5+aaqTgZmAz8WkWNDeTBLEPvbBvh/G8hwyzylquVNXQSq+hIQISKDu+PYIhKB8yH8pKr+NUAVT85Ze3F5ec7cY+4B3gBmtVjVfL5EJBxIAkq8jktVS1S1xn35EDClG8KZAZwuInnACuB4EXmiRZ3uPl/txuTRuWo69jb3907geWBaiypd+v/REsT+VgLz3CsBjgTKVHWH10GJSFpTv6uITMP5u4X8Q8U95sPAZ6p6TyvVuv2cBROXF+dMRFJEZIC7HAOcBGxqUW0lcJG7PBdYpe7oopdxteinPh1nXCekVPUWVc1Q1SycAehVqvr9FtW69XwFE5MX58o9bpyIJDQtAycDLa987NL/j+GdjrYXEpG/4FzdMlhECoDFOAN2qOofgZdwrgLYDFQBP+ghcc0FrhCRemAfcF6oP1RcM4ALgU/c/muAW4FMv9i8OGfBxOXFORsKPCYiPpyE9Iyq/l1ElgKrVXUlTmJ7XEQ241yYcF6IYwo2rqtE5HSg3o1rfjfEFVAPOF/txeTVuUoFnne/94QDT6nqKyKyAELz/9Gm2jDGGBOQdTEZY4wJyBKEMcaYgCxBGGOMCcgShDHGmIAsQRhjjAnIEoQx7RCRBr+ZO9eJyM1duO8saWUWX2O81q/ugzCmk/a501QY069YC8KYTnLn5v+VOz//ByJyqFueJSKr3Mnc/ikimW55qog8704g+LGIHO3uyiciD4rzrIbX3LudEZGrxHnmxXoRWeHR2zT9mCUIY9oX06KL6Vy/dWWqOh74Pc4soOBMDviYO5nbk8B9bvl9wL/cCQQnAxvc8pHA/ao6FtgDnOWW3wxMcvezIFRvzpjW2J3UxrRDRPaqanyA8jzgeFXd4k4eWKiqySKyCxiqqnVu+Q5VHSwixUCG30RvTdOV/0NVR7qvbwIiVPVOEXkF2IszE+0Lfs90MKZbWAvCmIOjrSx3RI3fcgNfjw2eCtyP09r40J3N1JhuYwnCmINzrt/vf7vL7/H1pHIXAG+7y/8EroDmB/gktbZTEQkDhqvqG8BNONNcH9CKMSaU7BuJMe2L8Zs1FuAVVW261HWgiKzHaQWc75b9BHhERG4Aivl6Rs2rgWUi8kOclsIVQGtTMfuAJ9wkIsB97rMcjOk2NgZhTCe5YxA5qrrL61iMCQXrYjLGGBOQtSCMMcYEZC0IY4wxAVmCMMYYE5AlCGOMMQFZgjDGGBOQJQhjjDEB/X9EXFCeF4ypcgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot training and validation loss over epochs\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "epochs = range(1, len(loss) + 1)\n",
    "plt.plot(epochs, loss, 'y', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'r', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "a9fdd283-2bb1-43cf-bf99-b2bc3782960b",
     "showTitle": false,
     "title": ""
    },
    "id": "hW0lz_M6jcGc"
   },
   "source": [
    "Evaluate the CNN model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "8297da48-7a0b-421e-a307-3ca91ec98dd3",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "IthhctpljcGc",
    "outputId": "feb80270-1d7d-4cdb-9157-792f05bc9c4b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CNN Model evaluation \n",
      " Validation Loss: 0.046058375388383865\n",
      " Validation Accuracy: 0.9872999787330627\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the model\n",
    "val_loss, val_acc = model.evaluate(x_test,y_test,verbose=0)\n",
    "print(\"CNN Model evaluation \\n\",\n",
    "      f\"Validation Loss: {val_loss}\\n\",\n",
    "      f\"Validation Accuracy: {val_acc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "385018af-b395-421e-92e9-4e84fd124587",
     "showTitle": false,
     "title": ""
    },
    "id": "QdDj-zntjcGc"
   },
   "source": [
    "We can see that the CNN model performs better on the test data as well compared to the DNN model.\n",
    "\n",
    "Next we get predict classes(digits) on the test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "24e284bf-99ca-4ac1-a7ec-87a73839ca3e",
     "showTitle": false,
     "title": ""
    },
    "id": "2uQ2hd7_jcGd"
   },
   "outputs": [],
   "source": [
    "pred_cnn = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "75aa722d-9b4e-4ba9-bdbe-6cd96c34fc77",
     "showTitle": false,
     "title": ""
    },
    "id": "l0C4fRFTjcGe"
   },
   "source": [
    "Use argmax function to select the index of an array that results in the largest value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "02653b91-99a1-4793-98d2-5690a538672c",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SpFJEBjyjcGe",
    "outputId": "6cbcfc0b-bac9-47b9-d19b-a804b8c08bde",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7\n"
     ]
    }
   ],
   "source": [
    "print (np.argmax(pred_cnn[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "inputWidgets": {},
     "nuid": "583d024e-0ec3-4e21-b0cf-168b30c13971",
     "showTitle": false,
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 281
    },
    "id": "QUW2ZW0yjcGe",
    "outputId": "3eef4e06-0368-4c58-ccc3-6e60a738eb25"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f71ed2b9f10>"
      ]
     },
     "execution_count": 31,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAANiklEQVR4nO3df4wc9XnH8c8n/kV8QGtDcF3j4ISQqE4aSHWBRNDKESUFImSiJBRLtVyJ5lALElRRW0QVBalVSlEIok0aySluHESgaQBhJTSNa6W1UKljg4yxgdaEmsau8QFOaxPAP/DTP24cHXD7vWNndmft5/2SVrs7z87Oo/F9PLMzO/t1RAjA8e9tbTcAoD8IO5AEYQeSIOxAEoQdSGJ6Pxc207PiBA31c5FAKq/qZzoYBzxRrVbYbV8s6XZJ0yT9bUTcXHr9CRrSeb6wziIBFGyIdR1rXe/G254m6auSLpG0WNIy24u7fT8AvVXnM/u5kp6OiGci4qCkeyQtbaYtAE2rE/YFkn4y7vnOatrr2B6xvcn2pkM6UGNxAOro+dH4iFgZEcMRMTxDs3q9OAAd1An7LkkLxz0/vZoGYADVCftGSWfZfpftmZKulLSmmbYANK3rU28Rcdj2tZL+SWOn3lZFxLbGOgPQqFrn2SPiQUkPNtQLgB7i67JAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJGoN2Wx7h6T9kl6TdDgihptoCkDzaoW98rGIeKGB9wHQQ+zGA0nUDXtI+oHtR2yPTPQC2yO2N9nedEgHai4OQLfq7sZfEBG7bJ8maa3tpyJi/fgXRMRKSSsl6WTPjZrLA9ClWlv2iNhV3Y9Kul/SuU00BaB5XYfd9pDtk44+lvRxSVubagxAs+rsxs+TdL/to+/zrYj4fiNdAWhc12GPiGcknd1gLwB6iFNvQBKEHUiCsANJEHYgCcIOJNHEhTApvPjZj3asvXP508V5nxqdV6wfPDCjWF9wd7k+e+dLHWtHNj9RnBd5sGUHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQ4zz5Ff/xH3+pY+9TQT8szn1lz4UvK5R2HX+5Yu/35j9Vc+LHrR6NndKwN3foLxXmnr3uk6XZax5YdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JwRP8GaTnZc+M8X9i35TXpZ58+r2PthQ+W/8+c82R5Hf/0V1ysz/zg/xbrt3zgvo61i97+SnHe7718YrH+idmdr5Wv65U4WKxvODBUrC854VDXy37P964u1t87srHr927ThlinfbF3wj8otuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATXs0/R0Hc2FGr13vvkerPrr39pScfan5+/qLzsfy3/5v0tS97TRUdTM/2VI8X60Jbdxfop6+8t1n91Zuff25+9o/xb/MejSbfstlfZHrW9ddy0ubbX2t5e3c/pbZsA6prKbvw3JF38hmk3SFoXEWdJWlc9BzDAJg17RKyXtPcNk5dKWl09Xi3p8ob7AtCwbj+zz4uIox+onpPUcTAz2yOSRiTpBM3ucnEA6qp9ND7GrqTpeKVHRKyMiOGIGJ6hWXUXB6BL3YZ9j+35klTdjzbXEoBe6DbsayStqB6vkPRAM+0A6JVJP7Pbvltjv1x+qu2dkr4g6WZJ37Z9laRnJV3RyyZRdvi5PR1rQ/d2rknSa5O899B3Xuyio2bs+b2PFuvvn1n+8/3S3vd1rC36u2eK8x4uVo9Nk4Y9IpZ1KB2bv0IBJMXXZYEkCDuQBGEHkiDsQBKEHUiCS1zRmulnLCzWv3LjV4r1GZ5WrP/D7b/ZsXbK7oeL8x6P2LIDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKcZ0drnvrDBcX6h2eVh7LedrA8HPXcJ15+yz0dz9iyA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASnGdHTx34xIc71h799G2TzF0eQej3r7uuWH/7v/1okvfPhS07kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBeXb01H9f0nl7cqLL59GX/ddFxfrs7z9WrEexms+kW3bbq2yP2t46btpNtnfZ3lzdLu1tmwDqmspu/DckXTzB9Nsi4pzq9mCzbQFo2qRhj4j1kvb2oRcAPVTnAN21trdUu/lzOr3I9ojtTbY3HdKBGosDUEe3Yf+apDMlnSNpt6RbO70wIlZGxHBEDM+Y5MIGAL3TVdgjYk9EvBYRRyR9XdK5zbYFoGldhd32/HFPPylpa6fXAhgMk55nt323pCWSTrW9U9IXJC2xfY7GTmXukHR1D3vEAHvbSScV68t//aGOtX1HXi3OO/rFdxfrsw5sLNbxepOGPSKWTTD5jh70AqCH+LoskARhB5Ig7EAShB1IgrADSXCJK2rZftP7i/Xvnvo3HWtLt3+qOO+sBzm11iS27EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBOfZUfR/v/ORYn3Lb/9Vsf7jw4c61l76y9OL887S7mIdbw1bdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgvPsyU1f8MvF+vWf//tifZbLf0JXPra8Y+0d/8j16v3Elh1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkuA8+3HO08v/xGd/d2ex/pkTXyzW79p/WrE+7/OdtydHinOiaZNu2W0vtP1D20/Y3mb7umr6XNtrbW+v7uf0vl0A3ZrKbvxhSZ+LiMWSPiLpGtuLJd0gaV1EnCVpXfUcwICaNOwRsTsiHq0e75f0pKQFkpZKWl29bLWky3vVJID63tJndtuLJH1I0gZJ8yLi6I+EPSdpXod5RiSNSNIJmt1tnwBqmvLReNsnSrpX0vURsW98LSJCUkw0X0SsjIjhiBieoVm1mgXQvSmF3fYMjQX9roi4r5q8x/b8qj5f0mhvWgTQhEl3421b0h2SnoyIL48rrZG0QtLN1f0DPekQ9Zz9vmL5z067s9bbf/WLnynWf/Gxh2u9P5ozlc/s50taLulx25uraTdqLOTftn2VpGclXdGbFgE0YdKwR8RDktyhfGGz7QDoFb4uCyRB2IEkCDuQBGEHkiDsQBJc4nocmLb4vR1rI/fU+/rD4lXXFOuL7vz3Wu+P/mHLDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJcJ79OPDUH3T+Yd/LZu/rWJuK0//lYPkFMeEPFGEAsWUHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQ4z34MePWyc4v1dZfdWqgy5BbGsGUHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSSmMj77QknflDRPUkhaGRG3275J0mclPV+99MaIeLBXjWb2P+dPK9bfOb37c+l37T+tWJ+xr3w9O1ezHzum8qWaw5I+FxGP2j5J0iO211a12yLiS71rD0BTpjI++25Ju6vH+20/KWlBrxsD0Ky39Jnd9iJJH5K0oZp0re0ttlfZnvC3kWyP2N5ke9MhHajVLIDuTTnstk+UdK+k6yNin6SvSTpT0jka2/JP+AXtiFgZEcMRMTxDsxpoGUA3phR22zM0FvS7IuI+SYqIPRHxWkQckfR1SeWrNQC0atKw27akOyQ9GRFfHjd9/riXfVLS1ubbA9CUqRyNP1/SckmP295cTbtR0jLb52js7MsOSVf3pEPU8hcvLi7WH/6tRcV67H68wW7QpqkcjX9IkicocU4dOIbwDTogCcIOJEHYgSQIO5AEYQeSIOxAEo4+Drl7sufGeb6wb8sDstkQ67Qv9k50qpwtO5AFYQeSIOxAEoQdSIKwA0kQdiAJwg4k0dfz7Lafl/TsuEmnSnqhbw28NYPa26D2JdFbt5rs7YyIeMdEhb6G/U0LtzdFxHBrDRQMam+D2pdEb93qV2/sxgNJEHYgibbDvrLl5ZcMam+D2pdEb93qS2+tfmYH0D9tb9kB9AlhB5JoJey2L7b9H7aftn1DGz10YnuH7cdtb7a9qeVeVtketb113LS5ttfa3l7dTzjGXku93WR7V7XuNtu+tKXeFtr+oe0nbG+zfV01vdV1V+irL+ut75/ZbU+T9J+SLpK0U9JGScsi4om+NtKB7R2ShiOi9S9g2P4NSS9J+mZEfKCadoukvRFxc/Uf5ZyI+JMB6e0mSS+1PYx3NVrR/PHDjEu6XNLvqsV1V+jrCvVhvbWxZT9X0tMR8UxEHJR0j6SlLfQx8CJivaS9b5i8VNLq6vFqjf2x9F2H3gZCROyOiEerx/slHR1mvNV1V+irL9oI+wJJPxn3fKcGa7z3kPQD24/YHmm7mQnMi4jd1ePnJM1rs5kJTDqMdz+9YZjxgVl33Qx/XhcH6N7sgoj4NUmXSLqm2l0dSDH2GWyQzp1OaRjvfplgmPGfa3PddTv8eV1thH2XpIXjnp9eTRsIEbGruh+VdL8GbyjqPUdH0K3uR1vu5+cGaRjviYYZ1wCsuzaHP28j7BslnWX7XbZnSrpS0poW+ngT20PVgRPZHpL0cQ3eUNRrJK2oHq+Q9ECLvbzOoAzj3WmYcbW87lof/jwi+n6TdKnGjsj/WNKfttFDh77eLemx6rat7d4k3a2x3bpDGju2cZWkUyStk7Rd0j9LmjtAvd0p6XFJWzQWrPkt9XaBxnbRt0jaXN0ubXvdFfrqy3rj67JAEhygA5Ig7EAShB1IgrADSRB2IAnCDiRB2IEk/h9BCfQTVPflJQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(x_test[0].reshape(28,28))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "application/vnd.databricks.v1+notebook": {
   "dashboards": [],
   "language": "python",
   "notebookMetadata": {
    "pythonIndentUnit": 2
   },
   "notebookName": "Deep Learning Basics_DNN_CNN (1)",
   "notebookOrigID": 4235718099986368,
   "widgets": {}
  },
  "colab": {
   "name": "Deep Learning Basics_DNN_CNN.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}