MLP Model for Diabetes Prediction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "T9PTH4vn5gND"
   },
   "source": [
    "# Diabetes Prediction Using Ensembling of Different Machine Learning Classifiers \n",
    "\n",
    "**Authors:** Md. Kamrul Hasan and Md. Ashraful Alam\n",
    "\n",
    "\n",
    "Corresponding Author- <br>\n",
    "**Md. Kamrul Hasan** <br>\n",
    "Erasmus Scholar [2017-2019] <br>\n",
    "M.Sc. in Medical Imaging and Applications (MAIA)(https://maiamaster.udg.edu/ ) <br>\n",
    "& <br>\n",
    "Assistant Professor <br>\n",
    "Department of Electrical and Electronic Engineering (EEE) <br>\n",
    "Khulna University of Engineering & Technology (KUET) <br>\n",
    "Khulna-9203, Bangladesh <br>\n",
    "\n",
    "\n",
    "E-mail: kamruleeekuet@gmail.com or m.k.hasan@eee.kuet.ac.bd<br>\n",
    "G.Scholar: https://scholar.google.com/citations?user=36WXELIAAAAJ&hl=en\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9KrtXQ1F7ry2"
   },
   "source": [
    "### Mount the drive with the colab notebook to access the content in the drive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "78a3956e-f633-aa99-53ec-6fe742674cd9",
    "colab": {},
    "colab_type": "code",
    "id": "0LVeW0lotXjP"
   },
   "outputs": [],
   "source": [
    "## Make 1 or True  if you run colab\n",
    "## Other wise 0 or False\n",
    "##if you run in  colab\n",
    "\n",
    "colab =0                                                     \n",
    "\n",
    "if colab ==True:\n",
    "  data_dir='/content/drive/My Drive/PIMA_Journal/diabetes.csv'\n",
    "  from google.colab import drive\n",
    "  drive.mount('/content/drive')\n",
    "else:\n",
    "  data_dir='diabetes.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "D7MsAIp0Y3Nc"
   },
   "outputs": [],
   "source": [
    "# !pip uninstall tensorflow\n",
    "# !pip install tensorflow==1.1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9byrg2P_q0xh"
   },
   "source": [
    "## The python Library we used are :\n",
    "\n",
    "<ul>\n",
    "<li>Numpy</li>\n",
    "<li>Pandas</li>\n",
    "<li>scipy</li>\n",
    "<li>Seaborn</li>\n",
    "<li>Keras</li>\n",
    "<li>Sklearn</li>\n",
    " \n",
    "</ul>\n",
    "\n",
    "<b>Numpy : </b> NumPy is a general-purpose array-processing package. It provides a high-performance multidimensional array object, and tools for working with these arrays. It is the fundamental package for scientific computing with Python. <br>\n",
    "<b>Pandas : </b> In Python programming language for data manipulation and analysis Pandas is widely Used . In particular, it offers data structures and operations for manipulating numerical tables and time series.<br>\n",
    "<b>SciPy : </b>It  is a free and open-source Python library used for scientific computing and technical computing. SciPy contains modules for optimization, linear algebra, integration, interpolation, special functions, FFT, signal and image processing, ODE solvers and other tasks common in science and engineering.<br>\n",
    "<b>Seaborn : </b> Seaborn is a Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics.<br>\n",
    "<b> Keras : </b> Keras is an open-source neural-network library written in Python. It is capable of running on top of TensorFlow, Microsoft Cognitive Toolkit, R, Theano, or PlaidML. Designed to enable fast experimentation with deep neural networks, it focuses on being user-friendly, modular, and extensible. <br>\n",
    "<b> Sklearn : </b>  Scikit-learn is a library in Python that provides many unsupervised and supervised learning algorithms. It’s built upon some of the technology you might already be familiar with, like NumPy, pandas, and  Matplotlib! The functionality that scikit-learn provides include:\n",
    "<ul>\n",
    "<li>Regression</li>\n",
    "<li>Classification</li>\n",
    "<li>Model selection\n",
    "Preprocessing</li>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dujqAogI8bQZ"
   },
   "source": [
    "### Loading of different packagaes and APIs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6344,
     "status": "ok",
     "timestamp": 1585596171655,
     "user": {
      "displayName": "__phiniks_ _",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj7KQ5WRKqZl709NEi7W2pySAoctrhlmXZfwrGq=s64",
      "userId": "17328933659108214149"
     },
     "user_tz": -360
    },
    "id": "Xz80SgiJtd5o",
    "outputId": "250d1f32-5fc6-47dd-da9d-5be6a8dd2139"
   },
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "np.random.seed(6)\n",
    "import random\n",
    "random.seed(6)\n",
    "# from keras import backend as K\n",
    "from tensorflow import set_random_seed\n",
    "set_random_seed(6)\n",
    "import pandas as pd \n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "# sns.set(style=\"whitegrid\")\n",
    "import warnings \n",
    "from sklearn.svm import SVC, NuSVC\n",
    "warnings.filterwarnings('ignore')\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.neighbors  import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "import xgboost as xgb\n",
    "from scipy import stats\n",
    "from scipy.stats import uniform, randint\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.model_selection import KFold, StratifiedKFold, RepeatedStratifiedKFold\n",
    "from sklearn.metrics import roc_curve, auc, accuracy_score\n",
    "# from tflearn.data_utils import to_categorical\n",
    "from sklearn import preprocessing\n",
    "from sklearn.model_selection import GridSearchCV, RandomizedSearchCV\n",
    "from sklearn.metrics import classification_report\n",
    "from scipy import interp\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.decomposition import FastICA\n",
    "from keras.utils import to_categorical\n",
    "from sklearn.ensemble import VotingClassifier\n",
    "from sklearn.metrics import roc_auc_score, accuracy_score\n",
    "from sklearn.model_selection import GridSearchCV, KFold\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.wrappers.scikit_learn import KerasClassifier\n",
    "from keras.optimizers import Adam\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import KFold\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.wrappers.scikit_learn import KerasClassifier\n",
    "from keras.optimizers import Adam\n",
    "from keras.callbacks import ModelCheckpoint\n",
    "from keras.layers import Activation, Dense, Dropout, BatchNormalization, Input\n",
    "from keras.models import Model\n",
    "from keras.optimizers import Adam\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "d272noMSoezZ"
   },
   "source": [
    "### Utility Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "B5ce3ePzKLDP"
   },
   "outputs": [],
   "source": [
    "Renamed_feature= []               #list of names that will rename to feature column\n",
    "all_clf_res=[]                    #every classifier auc values are stored in it\n",
    "random_initializer=20            #random initializer\n",
    "n_dots=50\n",
    "##########################################################\n",
    "\n",
    "for i in range(8):\n",
    "  #for renaming dataset of columns features F1 -- F8 \n",
    "  Renamed_feature.append('F'+str(i+1)) \n",
    "############################################################\n",
    "\n",
    "# Pairs plots are just showing all variables paired with all the other variables\n",
    "def pair_plot(data):\n",
    "  '''\n",
    "  This function will create a grid of Axes such that each variable\n",
    "  in data will by shared in the y-axis across a single row and in the x-axis\n",
    "  across a single column.The diagonal Axes are treated differently, drawing \n",
    "  a plot to show the univariate distribution of the data for the variable in\n",
    "  that column.\n",
    "  Parameters :\n",
    "  Input - data is the pandas type variable for \n",
    "  plotting pair plot of features in this\n",
    "  dataframe \n",
    "  \n",
    "  Output :\n",
    "  This function Plot pairwise relationships in a dataset.\n",
    "  \n",
    "  '''\n",
    "  plt.figure()\n",
    "  \n",
    "  pair_plot =sns.pairplot(data=data,            \n",
    "                          height=3,\n",
    "                          hue='Outcome',  \n",
    "                          diag_kind='kde')\n",
    "  # fig.suptitle(\"Pairplot of all features\")\n",
    "  pair_plot.fig.suptitle(\"Pairplot of all features\")\n",
    "  plt.show()\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "###################################################################\n",
    "# this function for Gaussian distribution plot \n",
    "# and box plot simultaneously in a figure\n",
    "def Box_Gaussian(data):\n",
    "  '''\n",
    "  Parameters :\n",
    "  Input - data is the pandas type variable\n",
    "\n",
    "  Output - The Gaussian distribution plot for\n",
    "  eight feature of input data \n",
    "\n",
    "  '''\n",
    "  \n",
    "  fig=plt.figure(figsize=(15,6))                                #define figure size\n",
    "  fig.suptitle(\"Box Gaussian plot of all features\")\n",
    "  \n",
    "  n_scaler = preprocessing.StandardScaler()                 #standardization function\n",
    "  temp_Data = n_scaler.fit_transform(data)                  #pass into function for standrd.\n",
    "  for i in range(8):                                        #loop for all 8 feature \n",
    "    \n",
    "    plt.subplot(2, 4, i+1)                                  #subplot for 2 rows in 4 columns\n",
    "    Data = temp_Data[:,i]                                   #data for every feature\n",
    "    sns.kdeplot(Data, shade=True,color='red', alpha=0.3)    #kernel density function under red shaded arae\n",
    "    ax = sns.boxplot(Data, saturation=0.9, color=\"green\")   #boxplot  with green shaded area\n",
    "                                                            # https://seaborn.pydata.org/generated/seaborn.kdeplot.html\n",
    "                                                            # https://seaborn.pydata.org/generated/seaborn.boxplot.html\n",
    "    plt.gca().invert_yaxis()                                #Reverse Y-Axis in PyPlot\n",
    "    # plt.title('F'+str(i+1))\n",
    "    frame1 = plt.gca()\n",
    "    frame1.axes.xaxis.set_ticklabels([])                    #removing xlabel data \n",
    "    plt.ylim((-0.5,0.65))                                   #y axis  limit\n",
    "    plt.tight_layout()                                      #This module provides routines to adjust subplot params so that subplots are nicely fit in the figure.\n",
    "                                                            # https://matplotlib.org/api/tight_layout_api.html\n",
    "    # plt.grid('on')\n",
    "    \n",
    "    for patch in ax.artists:\n",
    "      r, g, b, a = patch.get_facecolor()                     #Get the facecolor of the Axes.\n",
    "      patch.set_facecolor((r, g, b, 0.3))                    #set colour intensity\n",
    "#############################################################\n",
    "\n",
    "\n",
    "def plot_confusionMatrix(data):\n",
    "  '''\n",
    "  Parameters :\n",
    "  Input - data is the pandas type variable\n",
    "\n",
    "  Output -visualization of correalation matrix of\n",
    "  input data \n",
    "\n",
    "  '''\n",
    "  sns.set(font_scale=1.15)                                    # Set aesthetic parameters in one step.\n",
    "  ax = plt.figure(figsize=(10, 8))                            #set figure size   https://seaborn.pydata.org/generated/seaborn.set.html\n",
    "  plt.title(\"Confusion Matrix of all features\")\n",
    "  sns.heatmap(data.corr(),                                    # input correlation matrix  of dataset\n",
    "              vmax=1.0,                                       #Values to anchor the colormap, otherwise they are inferred from\n",
    "                                                              #the data and other keyword arguments.\n",
    "              vmin=0.0,\n",
    "              linewidths=0.01,\n",
    "              square=False,                                   #If True, set the Axes aspect to “equal” so each cell will be square-shaped.\n",
    "              annot=True,                                     #If True, write the data value in each cell. \n",
    "              linecolor=\"black\")                              #Color of the lines that will divide each cell.            \n",
    "                                                              #https://seaborn.pydata.org/generated/seaborn.heatmap.html\n",
    "  b, t = plt.ylim()                                           # discover the values for bottom and top\n",
    "  b += 0.5                                                    # Add 0.5 to the bottom\n",
    "  t -= 0.5                                                    # Subtract 0.5 from the top\n",
    "  plt.ylim(b, t)                                              # update the ylim(bottom, top) values\n",
    "  plt.show() \n",
    "\n",
    "\n",
    "\n",
    "############################################################\n",
    "# this function plot univariate distribution of \n",
    "# every feature\n",
    "\n",
    "def dist_Plot(data):\n",
    "  '''\n",
    "  Parameters :\n",
    "  Input - data is the pandas type variable\n",
    "\n",
    "  Output - The distribution plot for\n",
    "  eight feature of input data \n",
    "\n",
    "  '''\n",
    "  fig, ax = plt.subplots(2,4, figsize=(12,5))                 #set numbers of rows and columns of subplot\n",
    "  sns.set()\n",
    "  sns.distplot(data.F1, bins = 10, ax=ax[0,0])                #Flexibly plot a univariate distribution of observations.\n",
    "  sns.distplot(data.F2, bins = 10, ax=ax[0,1]) \n",
    "  sns.distplot(data.F3, bins = 10, ax=ax[0,2]) \n",
    "  sns.distplot(data.F4, bins = 10, ax=ax[0,3])\n",
    "  sns.distplot(data.F5, bins = 10, ax=ax[1,0]) \n",
    "  sns.distplot(data.F6, bins = 10, ax=ax[1,1]) \n",
    "  sns.distplot(data.F7, bins = 10, ax=ax[1,2]) \n",
    "  sns.distplot(data.F8, bins = 10, ax=ax[1,3]) \n",
    "  fig.suptitle(\"Gaussian Distribution of all features\")\n",
    "  fig.tight_layout()                                          #This module provides routines to adjust subplot params\n",
    "                                                              #  so that subplots are nicely fit in the figure.\n",
    "                                                        \n",
    "\n",
    "############################################################\n",
    "# this function plot violin plot  of \n",
    "# every feature\n",
    "\n",
    "\n",
    "def plot_violinplot (data):\n",
    "\n",
    "  '''\n",
    "  Parameters :\n",
    "  Input - data is the pandas type variable\n",
    "\n",
    "  Output - The violinplot plot for\n",
    "  eight feature of input data \n",
    "\n",
    "  '''\n",
    "\n",
    "  # A violin plot is a method of plotting numeric data.\n",
    "  # It is similar to box plot with a rotated kernel \n",
    "  # density plot on each side. Violin plots are similar\n",
    "  # to box plots, except that they also show the probability\n",
    "  # density of the data at different values (in the simplest\n",
    "  # case this could be a histogram).\n",
    "  fig, ax = plt.subplots(2,4, figsize=(12,6))          \n",
    "  # #set numbers of rows and columns of subplot and figure size \n",
    "  sns.set()\n",
    "  sns.violinplot(x = data.Outcome, y=data.F1,  ax=ax[0,0])    #violine plot for F1 feature\n",
    "  sns.violinplot(x = data.Outcome, y=data.F2,  ax=ax[0,1])    #violine plot for F2 feature \n",
    "  sns.violinplot(x = data.Outcome, y=data.F3,  ax=ax[0,2])    #violine plot for F3 feature \n",
    "  sns.violinplot(x = data.Outcome, y=data.F4,  ax=ax[0,3])    #violine plot for F4 feature\n",
    "  sns.violinplot(x = data.Outcome, y=data.F5,  ax=ax[1,0])    #violine plot for F5 feature \n",
    "  sns.violinplot(x = data.Outcome, y=data.F6,  ax=ax[1,1])    #violine plot for F6 feature \n",
    "  sns.violinplot(x = data.Outcome, y=data.F7,  ax=ax[1,2])    #violine plot for F7 feature \n",
    "  sns.violinplot(x = data.Outcome, y=data.F8,  ax=ax[1,3])    #violine plot for F8 feature \n",
    "  fig.suptitle(\"Violin plot of all features\")\n",
    "  fig.tight_layout()\n",
    "\n",
    "                                                              # https://seaborn.pydata.org/generated/seaborn.violinplot.html\n",
    "\n",
    "############################################################\n",
    "\n",
    "\n",
    "#this function  is for manual outleir rejection\n",
    "def Manual (data):\n",
    "\n",
    "    '''\n",
    "    Parameters :\n",
    "    Input - data is the pandas type variable\n",
    "\n",
    "    Return - dataframe with outleir rejection\n",
    "    of input data \n",
    "\n",
    "    '''\n",
    "    # input dataset is data \n",
    "    max_Pregnancies = data.F1.max()                         #maximum feature of F1\n",
    "    data = data[data.F1!=max_Pregnancies]                   #find  where extreme value is absent and remove extreme\n",
    "    max_Glucose = data.F2.max()                             #maximum feature of F2  \n",
    "    data = data[data.F2!=max_Glucose]                       #find  where extreme value is absent and remove\n",
    "    for i in range(4):                                      #in this loop we succesively remove 4 minimum element \n",
    "      min_Glucose = data.F2.min()                           #find minimum\n",
    "      data = data[data.F2!=min_Glucose]                     #reject minimum\n",
    "    max_BloodPressure = data.F3.max()                       #maximum feature of F3\n",
    "    data = data[data.F3!=max_BloodPressure]                 #find  where extreme value is absent and remove\n",
    "    for i in range(2):                                      #in this loop we succesively remove 2 extreme element  \n",
    "      max_skinthickness = data.F4.max() \n",
    "      data = data[data.F4!=max_skinthickness]\n",
    "    for i in range(25):                                     #in this loop we succesively remove 25 extreme element  \n",
    "      max_Insulin = data.F5.max() \n",
    "      data = data[data.F5!=max_Insulin]\n",
    "    max_bmi = data.F6.max()\n",
    "    data = data[data.F6!=max_bmi]\n",
    "    for i in range(4):                                      #in this loop we succesively remove 4 minimum element  \n",
    "      min_bmi = data.F6.min() \n",
    "      data = data[data.F6!=min_bmi]\n",
    "    for i in range(20):                                     #in this loop we succesively remove 20 extreme element \n",
    "      max_DiabetesPedigreeF = data.F7.max()\n",
    "      data = data[data.F7!=max_DiabetesPedigreeF]\n",
    "    for i in range(20):                                     #in this loop we succesively remove 20 extreme element  \n",
    "      max_age = data.F8.max() \n",
    "      data = data[data.F8!=max_age]\n",
    "      df =data\n",
    "    return data\n",
    "\n",
    "############################################################\n",
    "\n",
    "# this function if for outlair rejection with\n",
    "# respect to mean value\n",
    "def IQR_Mean (data):\n",
    "\n",
    "\n",
    "  '''\n",
    "  Parameters :\n",
    "  Input - data is the pandas type variable\n",
    "\n",
    "  Return - dataframe with outleir rejection\n",
    "  filled with mean\n",
    "  of input data \n",
    "\n",
    "  '''\n",
    "  for i in range(8): \n",
    "    x = data[Renamed_feature[i]]\n",
    "    Q1 = x.quantile(0.25)                                   # Q1 is the \"middle\" value in the first half of the rank-ordered data set.\n",
    "    Q3 = x.quantile(0.75)                                   # Q3 is the \"middle\" value in the second half of the rank-ordered data set.\n",
    "    IQR = Q3-Q1                                             # The interquartile range is equal to Q3 minus Q1.\n",
    "    mean = x.mean()                                         #mean of feature \n",
    "    for j in range(569):                                    # loop for first 569 elements of feature\n",
    "      temp = x[j]                                           # every feature value\n",
    "      LW = (Q1 - 1.5 * IQR)                                 #lower considerable range of gaussian distribution\n",
    "      UW = (Q3 + 1.5 * IQR)                                 #upper considerable range of gaussian distribution\n",
    "      if temp < LW:                                         #replace upper value with mean\n",
    "        x[j] = mean\n",
    "      if temp > UW:                                         #replace lower value with mean\n",
    "        x[j] = mean\n",
    "    data[Renamed_feature[i]] = x\n",
    "  return data\n",
    "\n",
    "############################################################\n",
    "# this function if for outlair rejection with\n",
    "# respect to median value same as previous function\n",
    "def IQR_Median (data): \n",
    "  '''\n",
    "  Parameters :\n",
    "  Input - data is the pandas type variable\n",
    "\n",
    "  Return - dataframe with outleir rejection\n",
    "  filled with median\n",
    "  of input data \n",
    "\n",
    "  '''\n",
    "  for i in range(8):\n",
    "    x = data[Renamed_feature[i]]\n",
    "    Q1 = x.quantile(0.25)\n",
    "    Q3 = x.quantile(0.75)\n",
    "    IQR = Q3-Q1\n",
    "    median = x.quantile(0.5)                                # find the median\n",
    "    for j in range(569):                                    #replace the first 569 values with respect to median\n",
    "      temp = x[j]\n",
    "      LW = (Q1 - 1.5 * IQR)\n",
    "      UW = (Q3 + 1.5 * IQR)\n",
    "      if temp < LW:                                         #replace upper value with median\n",
    "        x[j] = median\n",
    "      if temp > UW:\n",
    "        x[j] = median                                       #replace upper value with median\n",
    "    data[Renamed_feature[i]] = x\n",
    "  return data\n",
    "\n",
    "############################################################\n",
    "\n",
    "\n",
    "\n",
    "# this function if for outlair rejection with\n",
    "# 1.5 times of IQR that means that are\n",
    "# significant in gaussian distribution\n",
    "def IQR (data):\n",
    "\n",
    "  '''\n",
    "  Parameters :\n",
    "  Input - data is the pandas type variable\n",
    "\n",
    "  Return - dataframe with outleir rejection\n",
    "  of input data \n",
    "\n",
    "  '''\n",
    "  #input dataset as data\n",
    "  for i in range(8):                                        # for every feature\n",
    "    Q1 = data[Renamed_feature[i]].quantile(0.25)\n",
    "    Q3 = data[Renamed_feature[i]].quantile(0.75)\n",
    "    IQR = Q3-Q1                                             #find IQR\n",
    "    LW = (Q1 - 1.5 * IQR)                                   #find lower boundary\n",
    "          # print(LW)\n",
    "    UW = (Q3 + 1.5 * IQR)                                   #find upper boundary\n",
    "          # print(UW)\n",
    "    data = data[data[Renamed_feature[i]]<UW]                #drop greater than upper limit\n",
    "    data = data[data[Renamed_feature[i]]>LW]                #drop smaller than lower limit\n",
    "\n",
    "  return data\n",
    "\n",
    "\n",
    "############################################################\n",
    "#outlier rejection with different condition\n",
    "\n",
    "def outlier_Rejection (data, iqr_Mean, iqr_Medain, iqr, manual):\n",
    "  '''\n",
    "  Parameters :\n",
    "  Input - \n",
    "  data is the pandas type variable\n",
    "  iqr_Mean - for outleir rejection with Mean\n",
    "  iqr_Medain- for outleir rejection with Medain\n",
    "  iqr- for drop the outleir \n",
    "  manual -for manual rejection \n",
    "  Return - dataframe with outleir rejection\n",
    "  filled with Input parameter\n",
    "\n",
    "  '''\n",
    "  \n",
    "  # outlier_Rejection with conditional input\n",
    "  if iqr_Mean == True:                                     #reject outleir with Mean\n",
    "    data = IQR_Mean (data)\n",
    "  if iqr_Medain == True:                                   #reject outleir with Median\n",
    "    data = IQR_Medain (data)\n",
    "  if iqr == True:                                          #reject outleir in IQR range\n",
    "    data = IQR (data)\n",
    "  if manual == True:                                       #reject outleir with manual\n",
    "    data = Manual (data)\n",
    "\n",
    "  return data\n",
    "\n",
    "############################################################\n",
    "\n",
    "#data plot on different input condition \n",
    "def data_plot (data,\n",
    "               Pair_plot,\n",
    "               Dist_Plot,\n",
    "               Plot_violinplot,\n",
    "               Plot_confusionMatrix,\n",
    "               box_Gaussian):\n",
    "  \n",
    "  '''\n",
    "  Parameters :\n",
    "  Input - \n",
    "  data - It is the pandas type variable\n",
    "  Pair_plot - for pair plot visualization of input  data  \n",
    "  Dist_Plot- for gaussian distribution plot visualization of input  data  \n",
    "  Plot_violinplot- for violin plot visualization of input  data  \n",
    "  Plot_confusionMatrix -for confusion matrix visualization of input  data   \n",
    "  \n",
    "  Output - dataframe with outleir rejection\n",
    "  filled with Input parameter\n",
    "\n",
    "  '''\n",
    "  if Pair_plot ==True:\n",
    "    pair_plot(data)\n",
    "\n",
    "  if Dist_Plot ==True:\n",
    "    dist_Plot(data)\n",
    "\n",
    "  if Plot_violinplot ==True:\n",
    "    plot_violinplot (data)\n",
    "\n",
    "  if Plot_confusionMatrix ==True:\n",
    "    plot_confusionMatrix(data)\n",
    "\n",
    "  if box_Gaussian ==True:\n",
    "    Box_Gaussian(data)\n",
    "\n",
    "\n",
    "############################################################\n",
    "def replace_zero(data, field, target):\n",
    "    \n",
    "    mean_by_target = data.loc[data[field] != 0, [field, target]].groupby(target).mean()\n",
    "    data.loc[(data[field] == 0)&(data[target] == 0), field] = mean_by_target.iloc[0][0]\n",
    "    data.loc[(data[field] == 0)&(data[target] == 1), field] = mean_by_target.iloc[1][0]\n",
    "\n",
    "\n",
    "############################################################\n",
    "def metrics (y_true, y_pred, probas_):\n",
    "\n",
    "\n",
    "  '''\n",
    "  Parameters :\n",
    "  Input - \n",
    "  y_true - true  value of input data    \n",
    "  y_pred- predicted  value of input data  \n",
    "  probas_- probability/confidence of predicted output\n",
    "\n",
    "  return -True Negative(tn),False Positive(fp),False Negative(fn)\n",
    "  True positive(tp),AUC(roc_auc),False Positive Rate(fpr),\n",
    "  True positive rate(tpr)\n",
    "\n",
    "  '''\n",
    "\n",
    "\n",
    "  points=n_dots*'-'\n",
    "  print(points)\n",
    "#    print(\"Best parameters set found on development set:\")\n",
    "#    print(clf.best_params_)\n",
    "  fpr, tpr, thresholds = roc_curve(y_true, probas_[:, 1])\n",
    "  tprs.append(interp(mean_fpr, fpr, tpr))\n",
    "  tprs[-1][0] = 0.0\n",
    "  roc_auc = auc(fpr, tpr)\n",
    "  #  aucs.append(roc_auc)\n",
    "  print(\"Detailed classification report for current fold:\")\n",
    "  print()\n",
    "  print(classification_report(y_true, y_pred))\n",
    "  print()\n",
    "  print(\"Area Under ROC (AUC): {}\".format(roc_auc))\n",
    "  print()\n",
    "  print('Confusion Matrix for current fold: ')\n",
    "  print(confusion_matrix(y_true, y_pred))\n",
    "  print()\n",
    "  print(\"Accuracy for Current Fold: {}\".format(accuracy_score(y_true, y_pred)))\n",
    "  print()\n",
    "  tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()\n",
    "\n",
    "  return  tn, fp, fn, tp, roc_auc, fpr, tpr\n",
    "\n",
    "############################################################\n",
    "\n",
    "\n",
    "def average_ROC(mean_fpr,tprs,aucs,TP,TN,FP,FN):\n",
    "\n",
    "  '''\n",
    "  Parameters :\n",
    "  mean_fpr - Mean False positive rate\n",
    "  tprs -values of true positive rate\n",
    "  aucs  - values of auc\n",
    "  TP    - True positive \n",
    "  TN    - True Negative\n",
    "  FP    - False Positiv\n",
    "  FN    - False Negative\n",
    "\n",
    "  Output - \n",
    "  Visalization of TPR vs FPR plot\n",
    "  '''\n",
    "  sen = (np.sum(TP))/(np.sum(TP)+np.sum(FN))\n",
    "  spe = (np.sum(TN))/(np.sum(TN)+np.sum(FP))\n",
    "\n",
    "  mean_tpr = np.mean(tprs, axis=0)\n",
    "  mean_tpr[-1] = 1.0\n",
    "  # mean_auc = auc(mean_fpr, mean_tpr)\n",
    "  mean_auc = np.mean(aucs)\n",
    "  std_auc = np.std(aucs)\n",
    "  # plt.figure(figsize=(8, 5))\n",
    "  # plt.grid(True)\n",
    "  ax = plt.axes()\n",
    "  ax.grid(color='lightgray', linestyle='-', linewidth=.5)\n",
    "  # Setting the background color\n",
    "  ax.set_facecolor(\"white\")\n",
    "  \n",
    "  ax.spines['bottom'].set_color('#000000')\n",
    "  ax.spines['top'].set_color('#000000') \n",
    "  ax.spines['right'].set_color('#000000')\n",
    "  ax.spines['left'].set_color('#000000')\n",
    "\n",
    "  plt.plot(mean_fpr, mean_tpr, color='blue',\n",
    "          label=r'Avg. ROC (AUC (avg $\\pm$ std) = %0.3f $\\pm$ %0.3f)' % (mean_auc, std_auc),\n",
    "          lw=2, alpha=.8)\n",
    "  \n",
    "  plt.scatter((1-spe), sen, s=80, c='r', marker='x',)\n",
    "  plt.scatter(0, sen, s=80, c='r', marker='x',)\n",
    "  plt.scatter((1-spe),0, s=80, c='r', marker='x',)\n",
    "  plt.axhline(y=sen, color='r', linestyle='--')\n",
    "  plt.axvline(x=(1-spe), color='r', linestyle='--')\n",
    "  plt.text((1-spe), 0.02, \"FPR={:0.3f}\".format((1-spe)))\n",
    "  plt.text(0.009, sen+0.05, \"TPR={:0.3f}\".format(sen))\n",
    "\n",
    "  std_tpr = np.std(tprs, axis=0)\n",
    "  tprs_upper = np.minimum(mean_tpr + std_tpr, 1)\n",
    "  tprs_lower = np.maximum(mean_tpr - std_tpr, 0)\n",
    "  plt.fill_between(mean_fpr, tprs_lower, tprs_upper, color='darkgray', alpha=0.5,\n",
    "                  label=r'$\\pm$ 1 Standard deviation')\n",
    "\n",
    "  plt.xticks(np.arange(0.0, 1.01, step=0.1))\n",
    "  plt.yticks(np.arange(0.0, 1.01, step=0.1))\n",
    "  left=0.0\n",
    "  right=1.0\n",
    "  plt.xlim(left, right)\n",
    "  plt.ylim(left, right)\n",
    "  plt.xlabel('False Positive Rate (FPR)')\n",
    "  plt.ylabel('True Positive Rate (TPR)')\n",
    "  plt.legend(loc=\"lower right\")\n",
    "  # plt.grid(True)\n",
    "  plt.show()\n",
    "   \n",
    "############################################################\n",
    "def plot_Current_ROC(fpr,tpr,iterator,roc_auc):\n",
    "  \n",
    "  '''\n",
    "  Parameters :\n",
    "  Input - \n",
    "  fpr - False positive rate\n",
    "  tpr - True positive rate\n",
    "  roc_auc -auc values of roc curve\n",
    "\n",
    "  Output - \n",
    "  Visalization of current roc curve\n",
    "\n",
    "  '''\n",
    "  plt.plot(fpr,\n",
    "          \n",
    "          tpr,\n",
    "          # Color[iterator],\n",
    "          alpha=0.35,\n",
    "          # label='macro-average ROC (AUC = {0:0.3f})'.format(roc_auc)\n",
    "          # +FOLD[iterator],\n",
    "          linewidth=1)\n",
    "   \n",
    "############################################################\n",
    "def creat_Model (classifier, X_Train, Y_Train, tuned_parameters, verbose):\n",
    "\n",
    "  '''\n",
    "  Parameters :\n",
    "  Input - \n",
    "  X_Train -train data \n",
    "  Y_Train - label/output of train data\n",
    "  tuned_parameters =parameters of models\n",
    "  verbose = condition about model\n",
    "\n",
    "  Output - \n",
    "  Returned a tuned classifier using grid search\n",
    "  '''\n",
    "  clf = GridSearchCV(classifier,\n",
    "                    tuned_parameters,\n",
    "                    verbose=verbose,\n",
    "                    cv=5,\n",
    "                    scoring='roc_auc',\n",
    "                    n_jobs=-1)\n",
    "  clf.fit(X_Train, Y_Train)\n",
    "  return clf\n",
    "############################################################\n",
    "\n",
    "def average_performance(aucs,Accuracy,TP,TN,FP,FN): \n",
    "\n",
    "  '''\n",
    "  Parameters :\n",
    "  Input - \n",
    "  aucs= values of aucs\n",
    "  Accuracy - value of accuracy\n",
    "  TP  - True Positive\n",
    "  TN  - True Negative\n",
    "  FP  - False Positive\n",
    "  FN  - False Negative\n",
    "\n",
    "\n",
    "  Output - \n",
    "  It prints the average aucs,accuarcy,confusion matrix\n",
    "  '''\n",
    "\n",
    "  print()\n",
    "  n_dotsav=(n_dots-len('Average'))//2\n",
    "    \n",
    "  print('-'*n_dotsav+'Average'+'-'*n_dotsav)\n",
    "  print(\"AUC (Avg. +/- Std.) is  %0.3f +/- %0.3f\" %(np.mean(aucs),np.std(aucs)))\n",
    "  print(\"Accuracy (Avg. +/- Std.) is  %0.3f +/- %0.3f\" %(np.mean(Accuracy),np.std(Accuracy)))\n",
    "  cm = [[int(np.mean(TP)), int(np.mean(FP))],[int(np.mean(FN)), int(np.mean(TN))]]\n",
    "  print ('Avg. CM is '+str(cm))\n",
    "  cm = [[int(np.sum(TP)), int(np.sum(FP))],[int(np.sum(FN)), int(np.sum(TN))]]\n",
    "  print ('Total for all folds CM is '+str(cm))\n",
    "  re_auc=str(round(np.mean(aucs), 3))+'+/-'+str(round(np.std(aucs),3))\n",
    "  all_clf_res.append(re_auc)\n",
    "    \n",
    "############################################################\n",
    "#this  function is for algorithm based feature selection \n",
    "def feature_Selector(data, algo, n_feature):\n",
    "    '''\n",
    "    Parameters :\n",
    "    Input - \n",
    "    data - It is the pandas type variable\n",
    "    algo - type of algorith PCA,ICA,Correlation\n",
    "\n",
    "\n",
    "    Output - \n",
    "    It prints the average aucs,accuarcy,confusion matrix\n",
    "    '''\n",
    "    if algo=='PCA':                                                   #for pca algorithm\n",
    "        X_Data= data.iloc[:,:8].values\n",
    "        pca = PCA(n_components=n_feature)                             #number of feature\n",
    "        X_Data = pca.fit_transform(X_Data)\n",
    "        return X_Data , data.iloc[:,8:].values\n",
    "\n",
    "    if algo == 'ICA':\n",
    "        X_Data= data.iloc[:,:8].values\n",
    "        ICA = FastICA(n_components=n_feature, random_state=12) \n",
    "        X_Data = ICA.fit_transform(X_Data)\n",
    "        return X_Data , data.iloc[:,8:].values\n",
    "    \n",
    "    if algo =='corr':                                                   #for ica algorithm\n",
    "        if n_feature ==4:\n",
    "            data = data[['F2','F5','F4','F6','Outcome']]                #for 4 feature\n",
    "            return data.iloc[:,:4].values, data.iloc[:,4:].values\n",
    "        if n_feature ==6:\n",
    "            data = data[['F1','F2','F4','F5','F6','F8','Outcome']]       #for 6 feature\n",
    "            return data.iloc[:,:6].values, data.iloc[:,6:].values\n",
    "        \n",
    "    if algo == 'None':\n",
    "        return data.iloc[:,:8].values, data.iloc[:,8:].values            #if feature selection is off all features are counted\n",
    "    \n",
    "    \n",
    "def run (hLayer,\n",
    "         batch_size = [32],           \n",
    "         epochs = [100],              \n",
    "         learn_rate = [0.001],         \n",
    "         dropout_rate = [0.3],       \n",
    "         activation = ['relu'],   \n",
    "         init =['normal']):\n",
    "    \"\"\"\n",
    "    Input : hidden layer number \n",
    "    and others are alwalys constant\n",
    "    \n",
    "    Output : optimized model with best number of neuron \n",
    "    in every layer \n",
    "    \"\"\"\n",
    "    \n",
    "    \n",
    "    if hLayer==1:  \n",
    "        \n",
    "        ###########################################\n",
    "        #  number of neurons  for every layer\n",
    "        #  optimization\n",
    "        neuron1 = [16, 32, 64] \n",
    "        neuron2 = [16, 32, 64]\n",
    "        ############################################\n",
    "        \n",
    "        \n",
    "        #############################################\n",
    "        # parameter dictionary for grid search#\n",
    "        param_grid = dict(batch_size=batch_size,\n",
    "                          epochs=epochs,\n",
    "                          learn_rate=learn_rate,\n",
    "                          dropout_rate=dropout_rate,\n",
    "                          activation=activation,\n",
    "                          init=init,\n",
    "                          neuron1=neuron1,\n",
    "                          neuron2=neuron2)\n",
    "        ###############################################\n",
    "        \n",
    "        ###############################################\n",
    "        # model building function for given parameters#\n",
    "        def nn_opt_1(activation,\n",
    "                   dropout_rate,\n",
    "                   init,\n",
    "                   learn_rate,\n",
    "                   neuron1,\n",
    "                   neuron2):\n",
    "            \"\"\"\n",
    "            Input : activation ,dropout_rate,init,learn_rate,neuron1,neuron2\n",
    "            \n",
    "            Output : Using input hyper_parameters build model and return it\n",
    "            \"\"\"\n",
    "            \n",
    "            #####################################################################\n",
    "            ##               model initialization and building block           ##\n",
    "            model = Sequential()\n",
    "            model.add(Dense(neuron1, input_dim = 6, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron2, input_dim = neuron1, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dropout(dropout_rate))\n",
    "            model.add(Dense(2, activation='softmax'))\n",
    "            optimizer = Adam(lr = learn_rate)\n",
    "            model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "            \n",
    "            #####################################################################\n",
    "            \n",
    "            return model\n",
    "        \n",
    "        ########################### grid search for optimisaion #############################  \n",
    "        grid = GridSearchCV(estimator = KerasClassifier(build_fn = nn_opt_1, verbose = 0),\n",
    "                            param_grid = param_grid,\n",
    "                            cv = 5,\n",
    "                            n_jobs = -1,\n",
    "                            verbose = 1)\n",
    "        grid_results = grid.fit(X_Data, to_categorical(Y_Lavel,2))\n",
    "        \n",
    "        ########################################################################################\n",
    "        \n",
    "        ########################################################################################\n",
    "        ##                  build model with optimized parameters                             ##\n",
    "        model =nn_opt_1(grid_results.best_params_['activation'],\n",
    "                grid_results.best_params_['dropout_rate'],\n",
    "                grid_results.best_params_['init'],\n",
    "                grid_results.best_params_['learn_rate'],\n",
    "                grid_results.best_params_['neuron1'],\n",
    "                grid_results.best_params_['neuron2'])\n",
    "        ########################################################################################\n",
    "        \n",
    "        #  return best parameters and model\n",
    "        return grid_results.best_params_['batch_size'], grid_results.best_params_['epochs'], model, grid_results\n",
    "    \n",
    "    \n",
    "    if hLayer==2:       \n",
    "        neuron1 = [16, 32, 64]             \n",
    "        neuron2 = [16, 32, 64]                \n",
    "        neuron3 = [16, 32, 64]   \n",
    "        param_grid = dict(batch_size=batch_size,\n",
    "                          epochs=epochs,\n",
    "                          learn_rate=learn_rate,\n",
    "                          dropout_rate=dropout_rate,\n",
    "                          activation=activation,\n",
    "                          init=init,\n",
    "                          neuron1=neuron1,\n",
    "                          neuron2=neuron2,\n",
    "                          neuron3=neuron3)\n",
    "        \n",
    "        def nn_opt_2(activation,\n",
    "                   dropout_rate,\n",
    "                   init,\n",
    "                   learn_rate,\n",
    "                   neuron1,\n",
    "                   neuron2,\n",
    "                   neuron3):\n",
    "            model = Sequential()\n",
    "            model.add(Dense(neuron1, input_dim = 6, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron2, input_dim = neuron1, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron3, input_dim = neuron2, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dropout(dropout_rate))\n",
    "            model.add(Dense(2, activation='softmax'))\n",
    "            optimizer = Adam(lr = learn_rate)\n",
    "            model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "            return model\n",
    "        \n",
    "        grid = GridSearchCV(estimator = KerasClassifier(build_fn = nn_opt_2, verbose = 0),\n",
    "                            param_grid = param_grid,\n",
    "                            cv = 5,\n",
    "                            n_jobs = -1,\n",
    "                            verbose = 1)\n",
    "        grid_results = grid.fit(X_Data, to_categorical(Y_Lavel,2))\n",
    "        \n",
    "        model =nn_opt_2(grid_results.best_params_['activation'],\n",
    "                grid_results.best_params_['dropout_rate'],\n",
    "                grid_results.best_params_['init'],\n",
    "                grid_results.best_params_['learn_rate'],\n",
    "                grid_results.best_params_['neuron1'],\n",
    "                grid_results.best_params_['neuron2'],\n",
    "                grid_results.best_params_['neuron3'])\n",
    "        return grid_results.best_params_['batch_size'], grid_results.best_params_['epochs'], model, grid_results\n",
    "    \n",
    "    if hLayer==3:       \n",
    "        neuron1 = [16, 32, 64]              \n",
    "        neuron2 = [16, 32, 64]                \n",
    "        neuron3 = [16, 32, 64]  \n",
    "        neuron4 = [16, 32, 64]  \n",
    "        param_grid = dict(batch_size=batch_size,\n",
    "                          epochs=epochs,\n",
    "                          learn_rate=learn_rate,\n",
    "                          dropout_rate=dropout_rate,\n",
    "                          activation=activation,\n",
    "                          init=init,\n",
    "                          neuron1=neuron1,\n",
    "                          neuron2=neuron2,\n",
    "                          neuron3=neuron3,\n",
    "                          neuron4=neuron4)\n",
    "        \n",
    "        def nn_opt_3(activation,\n",
    "                   dropout_rate,\n",
    "                   init,\n",
    "                   learn_rate,\n",
    "                   neuron1,\n",
    "                   neuron2,\n",
    "                   neuron3,\n",
    "                   neuron4):\n",
    "            model = Sequential()\n",
    "            model.add(Dense(neuron1, input_dim = 6, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron2, input_dim = neuron1, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron3, input_dim = neuron2, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron4, input_dim = neuron3, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dropout(dropout_rate))\n",
    "            model.add(Dense(2, activation='softmax'))\n",
    "            optimizer = Adam(lr = learn_rate)\n",
    "            model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "            return model\n",
    "        \n",
    "        grid = GridSearchCV(estimator = KerasClassifier(build_fn = nn_opt_3, verbose = 0),\n",
    "                            param_grid = param_grid,\n",
    "                            cv = 5,\n",
    "                            n_jobs = -1,\n",
    "                            verbose = 1)\n",
    "        grid_results = grid.fit(X_Data, to_categorical(Y_Lavel,2))\n",
    "        \n",
    "        model =nn_opt_3(grid_results.best_params_['activation'],\n",
    "                grid_results.best_params_['dropout_rate'],\n",
    "                grid_results.best_params_['init'],\n",
    "                grid_results.best_params_['learn_rate'],\n",
    "                grid_results.best_params_['neuron1'],\n",
    "                grid_results.best_params_['neuron2'],\n",
    "                grid_results.best_params_['neuron3'],\n",
    "                grid_results.best_params_['neuron4'])\n",
    "        return grid_results.best_params_['batch_size'], grid_results.best_params_['epochs'], model, grid_results\n",
    "    \n",
    "    if hLayer==4:       \n",
    "        neuron1 = [16, 32, 64]              \n",
    "        neuron2 = [16, 32, 64]                \n",
    "        neuron3 = [16, 32, 64]   \n",
    "        neuron4 = [16, 32, 64]  \n",
    "        neuron5 = [16, 32, 64] \n",
    "        param_grid = dict(batch_size=batch_size,\n",
    "                          epochs=epochs,\n",
    "                          learn_rate=learn_rate,\n",
    "                          dropout_rate=dropout_rate,\n",
    "                          activation=activation,\n",
    "                          init=init,\n",
    "                          neuron1=neuron1,\n",
    "                          neuron2=neuron2,\n",
    "                          neuron3=neuron3,\n",
    "                          neuron4=neuron4,\n",
    "                          neuron5=neuron5)\n",
    "        \n",
    "        def nn_opt_4(activation,\n",
    "                   dropout_rate,\n",
    "                   init,\n",
    "                   learn_rate,\n",
    "                   neuron1,\n",
    "                   neuron2,\n",
    "                   neuron3,\n",
    "                   neuron4,\n",
    "                   neuron5):\n",
    "            model = Sequential()\n",
    "            model.add(Dense(neuron1, input_dim = 6, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron2, input_dim = neuron1, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron3, input_dim = neuron2, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron4, input_dim = neuron3, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron5, input_dim = neuron4, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dropout(dropout_rate))\n",
    "            model.add(Dense(2, activation='softmax'))\n",
    "            optimizer = Adam(lr = learn_rate)\n",
    "            model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "            return model\n",
    "        \n",
    "        grid = GridSearchCV(estimator = KerasClassifier(build_fn = nn_opt_4, verbose = 0),\n",
    "                            param_grid = param_grid,\n",
    "                            cv = 5,\n",
    "                            n_jobs = -1,\n",
    "                            verbose = 1)\n",
    "        grid_results = grid.fit(X_Data, to_categorical(Y_Lavel,2))\n",
    "        \n",
    "        model =nn_opt_4(grid_results.best_params_['activation'],\n",
    "                grid_results.best_params_['dropout_rate'],\n",
    "                grid_results.best_params_['init'],\n",
    "                grid_results.best_params_['learn_rate'],\n",
    "                grid_results.best_params_['neuron1'],\n",
    "                grid_results.best_params_['neuron2'],\n",
    "                grid_results.best_params_['neuron3'],\n",
    "                grid_results.best_params_['neuron4'],\n",
    "                grid_results.best_params_['neuron5'])\n",
    "        return grid_results.best_params_['batch_size'], grid_results.best_params_['epochs'], model, grid_results\n",
    "    \n",
    "    if hLayer==5:       \n",
    "        neuron1 = [16, 32, 64]             \n",
    "        neuron2 = [16, 32, 64]                \n",
    "        neuron3 = [16, 32, 64] \n",
    "        neuron4 = [16, 32, 64]  \n",
    "        neuron5 = [16, 32, 64] \n",
    "        neuron6 = [16, 32, 64] \n",
    "        param_grid = dict(batch_size=batch_size,\n",
    "                          epochs=epochs,\n",
    "                          learn_rate=learn_rate,\n",
    "                          dropout_rate=dropout_rate,\n",
    "                          activation=activation,\n",
    "                          init=init,\n",
    "                          neuron1=neuron1,\n",
    "                          neuron2=neuron2,\n",
    "                          neuron3=neuron3,\n",
    "                          neuron4=neuron4,\n",
    "                          neuron5=neuron5,\n",
    "                          neuron6=neuron6)\n",
    "        \n",
    "        def nn_opt_5(activation,\n",
    "                   dropout_rate,\n",
    "                   init,\n",
    "                   learn_rate,\n",
    "                   neuron1,\n",
    "                   neuron2,\n",
    "                   neuron3,\n",
    "                   neuron4,\n",
    "                   neuron5,\n",
    "                   neuron6):\n",
    "            model = Sequential()\n",
    "            model.add(Dense(neuron1, input_dim = 6, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron2, input_dim = neuron1, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron3, input_dim = neuron2, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron4, input_dim = neuron3, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron5, input_dim = neuron4, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron6, input_dim = neuron5, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dropout(dropout_rate))\n",
    "            model.add(Dense(2, activation='softmax'))\n",
    "            optimizer = Adam(lr = learn_rate)\n",
    "            model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "            return model\n",
    "        \n",
    "        grid = GridSearchCV(estimator = KerasClassifier(build_fn = nn_opt_5, verbose = 0),\n",
    "                            param_grid = param_grid,\n",
    "                            cv = 5,\n",
    "                            n_jobs = -1,\n",
    "                            verbose = 1)\n",
    "        grid_results = grid.fit(X_Data, to_categorical(Y_Lavel,2))\n",
    "        \n",
    "        model =nn_opt_5(grid_results.best_params_['activation'],\n",
    "                grid_results.best_params_['dropout_rate'],\n",
    "                grid_results.best_params_['init'],\n",
    "                grid_results.best_params_['learn_rate'],\n",
    "                grid_results.best_params_['neuron1'],\n",
    "                grid_results.best_params_['neuron2'],\n",
    "                grid_results.best_params_['neuron3'],\n",
    "                grid_results.best_params_['neuron4'],\n",
    "                grid_results.best_params_['neuron5'],\n",
    "                grid_results.best_params_['neuron6'])\n",
    "        return grid_results.best_params_['batch_size'], grid_results.best_params_['epochs'], model, grid_results\n",
    "    \n",
    "\n",
    "    if hLayer==6:       \n",
    "        neuron1 = [16, 32, 64]              \n",
    "        neuron2 = [16, 32, 64]               \n",
    "        neuron3 = [16, 32, 64]  \n",
    "        neuron4 = [16, 32, 64]  \n",
    "        neuron5 = [16, 32, 64] \n",
    "        neuron6 = [16, 32, 64] \n",
    "        neuron7 = [16, 32, 64] \n",
    "        param_grid = dict(batch_size=batch_size,\n",
    "                          epochs=epochs,\n",
    "                          learn_rate=learn_rate,\n",
    "                          dropout_rate=dropout_rate,\n",
    "                          activation=activation,\n",
    "                          init=init,\n",
    "                          neuron1=neuron1,\n",
    "                          neuron2=neuron2,\n",
    "                          neuron3=neuron3,\n",
    "                          neuron4=neuron4,\n",
    "                          neuron5=neuron5,\n",
    "                          neuron6=neuron6,\n",
    "                          neuron7=neuron7)\n",
    "        \n",
    "        def nn_opt_6(activation,\n",
    "                   dropout_rate,\n",
    "                   init,\n",
    "                   learn_rate,\n",
    "                   neuron1,\n",
    "                   neuron2,\n",
    "                   neuron3,\n",
    "                   neuron4,\n",
    "                   neuron5,\n",
    "                   neuron6,\n",
    "                   neuron7):\n",
    "            model = Sequential()\n",
    "            model.add(Dense(neuron1, input_dim = 6, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron2, input_dim = neuron1, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron3, input_dim = neuron2, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron4, input_dim = neuron3, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron5, input_dim = neuron4, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron6, input_dim = neuron5, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron7, input_dim = neuron6, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dropout(dropout_rate))\n",
    "            model.add(Dense(2, activation='softmax'))\n",
    "            optimizer = Adam(lr = learn_rate)\n",
    "            model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "            return model\n",
    "        \n",
    "        grid = GridSearchCV(estimator = KerasClassifier(build_fn = nn_opt_6, verbose = 0),\n",
    "                            param_grid = param_grid,\n",
    "                            cv = 5,\n",
    "                            n_jobs = -1,\n",
    "                            verbose = 1)\n",
    "        grid_results = grid.fit(X_Data, to_categorical(Y_Lavel,2))\n",
    "        \n",
    "        model =nn_opt_6(grid_results.best_params_['activation'],\n",
    "                grid_results.best_params_['dropout_rate'],\n",
    "                grid_results.best_params_['init'],\n",
    "                grid_results.best_params_['learn_rate'],\n",
    "                grid_results.best_params_['neuron1'],\n",
    "                grid_results.best_params_['neuron2'],\n",
    "                grid_results.best_params_['neuron3'],\n",
    "                grid_results.best_params_['neuron4'],\n",
    "                grid_results.best_params_['neuron5'],\n",
    "                grid_results.best_params_['neuron6'],\n",
    "                grid_results.best_params_['neuron7'])\n",
    "        return grid_results.best_params_['batch_size'], grid_results.best_params_['epochs'], model, grid_results\n",
    "    \n",
    "    if hLayer==7:       \n",
    "        neuron1 = [16, 32, 64]               \n",
    "        neuron2 = [16, 32, 64]                \n",
    "        neuron3 = [16, 32, 64]   \n",
    "        neuron4 = [16, 32, 64] \n",
    "        neuron5 = [16, 32, 64] \n",
    "        neuron6 = [16, 32, 64] \n",
    "        neuron7 = [16, 32, 64] \n",
    "        neuron8 = [16, 32, 64] \n",
    "        param_grid = dict(batch_size=batch_size,\n",
    "                          epochs=epochs,\n",
    "                          learn_rate=learn_rate,\n",
    "                          dropout_rate=dropout_rate,\n",
    "                          activation=activation,\n",
    "                          init=init,\n",
    "                          neuron1=neuron1,\n",
    "                          neuron2=neuron2,\n",
    "                          neuron3=neuron3,\n",
    "                          neuron4=neuron4,\n",
    "                          neuron5=neuron5,\n",
    "                          neuron6=neuron6,\n",
    "                          neuron7=neuron7,\n",
    "                          neuron8=neuron8)\n",
    "        \n",
    "        def nn_opt_7(activation,\n",
    "                   dropout_rate,\n",
    "                   init,\n",
    "                   learn_rate,\n",
    "                   neuron1,\n",
    "                   neuron2,\n",
    "                   neuron3,\n",
    "                   neuron4,\n",
    "                   neuron5,\n",
    "                   neuron6,\n",
    "                   neuron7,\n",
    "                   neuron8):\n",
    "            model = Sequential()\n",
    "            model.add(Dense(neuron1, input_dim = 6, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron2, input_dim = neuron1, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron3, input_dim = neuron2, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron4, input_dim = neuron3, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron5, input_dim = neuron4, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron6, input_dim = neuron5, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron7, input_dim = neuron6, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron8, input_dim = neuron7, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dropout(dropout_rate))\n",
    "            model.add(Dense(2, activation='softmax'))\n",
    "            optimizer = Adam(lr = learn_rate)\n",
    "            model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "            return model\n",
    "        \n",
    "        grid = GridSearchCV(estimator = KerasClassifier(build_fn = nn_opt_7, verbose = 0),\n",
    "                            param_grid = param_grid,\n",
    "                            cv = 5,\n",
    "                            n_jobs = -1,\n",
    "                            verbose = 1)\n",
    "        grid_results = grid.fit(X_Data, to_categorical(Y_Lavel,2))\n",
    "        \n",
    "        model =nn_opt_7(grid_results.best_params_['activation'],\n",
    "                grid_results.best_params_['dropout_rate'],\n",
    "                grid_results.best_params_['init'],\n",
    "                grid_results.best_params_['learn_rate'],\n",
    "                grid_results.best_params_['neuron1'],\n",
    "                grid_results.best_params_['neuron2'],\n",
    "                grid_results.best_params_['neuron3'],\n",
    "                grid_results.best_params_['neuron4'],\n",
    "                grid_results.best_params_['neuron5'],\n",
    "                grid_results.best_params_['neuron6'],\n",
    "                grid_results.best_params_['neuron7'],\n",
    "                grid_results.best_params_['neuron8'])\n",
    "        return grid_results.best_params_['batch_size'], grid_results.best_params_['epochs'], model, grid_results\n",
    "    \n",
    "    if hLayer==8:       \n",
    "        neuron1 = [16, 32, 64]              \n",
    "        neuron2 = [16, 32, 64]                \n",
    "        neuron3 = [16, 32, 64]   \n",
    "        neuron4 = [16, 32, 64]  \n",
    "        neuron5 = [16, 32, 64] \n",
    "        neuron6 = [16, 32, 64] \n",
    "        neuron7 = [16, 32, 64] \n",
    "        neuron8 = [16, 32, 64] \n",
    "        neuron9 = [16, 32, 64] \n",
    "        param_grid = dict(batch_size=batch_size,\n",
    "                          epochs=epochs,\n",
    "                          learn_rate=learn_rate,\n",
    "                          dropout_rate=dropout_rate,\n",
    "                          activation=activation,\n",
    "                          init=init,\n",
    "                          neuron1=neuron1,\n",
    "                          neuron2=neuron2,\n",
    "                          neuron3=neuron3,\n",
    "                          neuron4=neuron4,\n",
    "                          neuron5=neuron5,\n",
    "                          neuron6=neuron6,\n",
    "                          neuron7=neuron7,\n",
    "                          neuron8=neuron8,\n",
    "                          neuron9 =neuron9)\n",
    "        \n",
    "        def nn_opt_8(activation,\n",
    "                   dropout_rate,\n",
    "                   init,\n",
    "                   learn_rate,\n",
    "                   neuron1,\n",
    "                   neuron2,\n",
    "                   neuron3,\n",
    "                   neuron4,\n",
    "                   neuron5,\n",
    "                   neuron6,\n",
    "                   neuron7,\n",
    "                   neuron8,\n",
    "                   neuron9):\n",
    "            model = Sequential()\n",
    "            model.add(Dense(neuron1, input_dim = 6, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron2, input_dim = neuron1, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron3, input_dim = neuron2, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron4, input_dim = neuron3, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron5, input_dim = neuron4, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron6, input_dim = neuron5, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron7, input_dim = neuron6, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron8, input_dim = neuron7, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron9, input_dim = neuron8, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dropout(dropout_rate))\n",
    "            model.add(Dense(2, activation='softmax'))\n",
    "            optimizer = Adam(lr = learn_rate)\n",
    "            model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "            return model\n",
    "        \n",
    "        grid = GridSearchCV(estimator = KerasClassifier(build_fn = nn_opt_8, verbose = 0),\n",
    "                            param_grid = param_grid,\n",
    "                            cv = 5,\n",
    "                            n_jobs = -1,\n",
    "                            verbose = 1)\n",
    "        grid_results = grid.fit(X_Data, to_categorical(Y_Lavel,2))\n",
    "        \n",
    "        model =nn_opt_8(grid_results.best_params_['activation'],\n",
    "                grid_results.best_params_['dropout_rate'],\n",
    "                grid_results.best_params_['init'],\n",
    "                grid_results.best_params_['learn_rate'],\n",
    "                grid_results.best_params_['neuron1'],\n",
    "                grid_results.best_params_['neuron2'],\n",
    "                grid_results.best_params_['neuron3'],\n",
    "                grid_results.best_params_['neuron4'],\n",
    "                grid_results.best_params_['neuron5'],\n",
    "                grid_results.best_params_['neuron6'],\n",
    "                grid_results.best_params_['neuron7'],\n",
    "                grid_results.best_params_['neuron8'],\n",
    "                grid_results.best_params_['neuron9'])\n",
    "        return grid_results.best_params_['batch_size'], grid_results.best_params_['epochs'], model, grid_results\n",
    "    \n",
    "    if hLayer==9:       \n",
    "        neuron1 = [16, 32, 64]               \n",
    "        neuron2 = [16, 32, 64]               \n",
    "        neuron3 = [16, 32, 64]   \n",
    "        neuron4 = [16, 32, 64]  \n",
    "        neuron5 = [16, 32, 64] \n",
    "        neuron6 = [16, 32, 64] \n",
    "        neuron7 = [16, 32, 64] \n",
    "        neuron8 = [16, 32, 64] \n",
    "        neuron9 = [16, 32, 64] \n",
    "        neuron10 = [16, 32, 64] \n",
    "        param_grid = dict(batch_size=batch_size,\n",
    "                          epochs=epochs,\n",
    "                          learn_rate=learn_rate,\n",
    "                          dropout_rate=dropout_rate,\n",
    "                          activation=activation,\n",
    "                          init=init,\n",
    "                          neuron1=neuron1,\n",
    "                          neuron2=neuron2,\n",
    "                          neuron3=neuron3,\n",
    "                          neuron4=neuron4,\n",
    "                          neuron5=neuron5,\n",
    "                          neuron6=neuron6,\n",
    "                          neuron7=neuron7,\n",
    "                          neuron8=neuron8,\n",
    "                          neuron9=neuron9,\n",
    "                          neuron10=neuron10)\n",
    "        \n",
    "        def nn_opt_9(activation,\n",
    "                   dropout_rate,\n",
    "                   init,\n",
    "                   learn_rate,\n",
    "                   neuron1,\n",
    "                   neuron2,\n",
    "                   neuron3,\n",
    "                   neuron4,\n",
    "                   neuron5,\n",
    "                   neuron6,\n",
    "                   neuron7,\n",
    "                   neuron8,\n",
    "                   neuron9,\n",
    "                   neuron10 ):\n",
    "            model = Sequential()\n",
    "            model.add(Dense(neuron1, input_dim = 6, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron2, input_dim = neuron1, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron3, input_dim = neuron2, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron4, input_dim = neuron3, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron5, input_dim = neuron4, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron6, input_dim = neuron5, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron7, input_dim = neuron6, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron8, input_dim = neuron7, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron9, input_dim = neuron8, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron10, input_dim = neuron9, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dropout(dropout_rate))\n",
    "            model.add(Dense(2, activation='softmax'))\n",
    "            optimizer = Adam(lr = learn_rate)\n",
    "            model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "            return model\n",
    "        \n",
    "        grid = GridSearchCV(estimator = KerasClassifier(build_fn = nn_opt_9, verbose = 0),\n",
    "                            param_grid = param_grid,\n",
    "                            cv = 5,\n",
    "                            n_jobs = -1,\n",
    "                            verbose = 1)\n",
    "        grid_results = grid.fit(X_Data, to_categorical(Y_Lavel,2))\n",
    "        \n",
    "        model =nn_opt_9(grid_results.best_params_['activation'],\n",
    "                grid_results.best_params_['dropout_rate'],\n",
    "                grid_results.best_params_['init'],\n",
    "                grid_results.best_params_['learn_rate'],\n",
    "                grid_results.best_params_['neuron1'],\n",
    "                grid_results.best_params_['neuron2'],\n",
    "                grid_results.best_params_['neuron3'],\n",
    "                grid_results.best_params_['neuron4'],\n",
    "                grid_results.best_params_['neuron5'],\n",
    "                grid_results.best_params_['neuron6'],\n",
    "                grid_results.best_params_['neuron7'],\n",
    "                grid_results.best_params_['neuron8'],\n",
    "                grid_results.best_params_['neuron9'],\n",
    "                grid_results.best_params_['neuron10'])\n",
    "        return grid_results.best_params_['batch_size'], grid_results.best_params_['epochs'], model, grid_results\n",
    "    \n",
    "    if hLayer==10:       \n",
    "        neuron1 = [16, 32, 64]             \n",
    "        neuron2 = [16, 32, 64]                \n",
    "        neuron3 = [16, 32, 64]   \n",
    "        neuron4 = [16, 32, 64]  \n",
    "        neuron5 = [16, 32, 64] \n",
    "        neuron6 = [16, 32, 64] \n",
    "        neuron7 = [16, 32, 64] \n",
    "        neuron8 = [16, 32, 64] \n",
    "        neuron9 = [16, 32, 64] \n",
    "        neuron10 = [16, 32, 64] \n",
    "        neuron11 = [16, 32, 64] \n",
    "        param_grid = dict(batch_size=batch_size,\n",
    "                          epochs=epochs,\n",
    "                          learn_rate=learn_rate,\n",
    "                          dropout_rate=dropout_rate,\n",
    "                          activation=activation,\n",
    "                          init=init,\n",
    "                          neuron1=neuron1,\n",
    "                          neuron2=neuron2,\n",
    "                          neuron3=neuron3,\n",
    "                          neuron4=neuron4,\n",
    "                          neuron5=neuron5,\n",
    "                          neuron6=neuron6,\n",
    "                          neuron7=neuron7,\n",
    "                          neuron8=neuron8,\n",
    "                          neuron9 =neuron9,\n",
    "                          neuron10=neuron10,\n",
    "                          neuron11 =neuron11)\n",
    "        \n",
    "        def nn_opt_10(activation,\n",
    "                   dropout_rate,\n",
    "                   init,\n",
    "                   learn_rate,\n",
    "                   neuron1,\n",
    "                   neuron2,\n",
    "                   neuron3,\n",
    "                   neuron4,\n",
    "                   neuron5,\n",
    "                   neuron6,\n",
    "                   neuron7,\n",
    "                   neuron8,\n",
    "                   neuron9,\n",
    "                   neuron10,\n",
    "                   neuron11):\n",
    "            model = Sequential()\n",
    "            model.add(Dense(neuron1, input_dim = 6, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron2, input_dim = neuron1, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron3, input_dim = neuron2, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron4, input_dim = neuron3, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron5, input_dim = neuron4, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron6, input_dim = neuron5, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron7, input_dim = neuron6, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron8, input_dim = neuron7, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron9, input_dim = neuron8, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron10, input_dim = neuron9, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dense(neuron11, input_dim = neuron10, kernel_initializer= init, activation= activation))\n",
    "            model.add(Dropout(dropout_rate))\n",
    "            model.add(Dense(2, activation='softmax'))\n",
    "            optimizer = Adam(lr = learn_rate)\n",
    "            model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "            return model\n",
    "        \n",
    "        grid = GridSearchCV(estimator = KerasClassifier(build_fn = nn_opt_10, verbose = 0),\n",
    "                            param_grid = param_grid,\n",
    "                            cv = 5,\n",
    "                            n_jobs = -1,\n",
    "                            verbose = 1)\n",
    "        grid_results = grid.fit(X_Data, to_categorical(Y_Lavel,2))\n",
    "        \n",
    "        model =nn_opt_10(grid_results.best_params_['activation'],\n",
    "                grid_results.best_params_['dropout_rate'],\n",
    "                grid_results.best_params_['init'],\n",
    "                grid_results.best_params_['learn_rate'],\n",
    "                grid_results.best_params_['neuron1'],\n",
    "                grid_results.best_params_['neuron2'],\n",
    "                grid_results.best_params_['neuron3'],\n",
    "                grid_results.best_params_['neuron4'],\n",
    "                grid_results.best_params_['neuron5'],\n",
    "                grid_results.best_params_['neuron6'],\n",
    "                grid_results.best_params_['neuron7'],\n",
    "                grid_results.best_params_['neuron8'],\n",
    "                grid_results.best_params_['neuron9'],\n",
    "                grid_results.best_params_['neuron10'],\n",
    "                grid_results.best_params_['neuron11'])\n",
    "        return grid_results.best_params_['batch_size'], grid_results.best_params_['epochs'], model, grid_results\n",
    "    \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "################################################################ \n",
    "#           this function optimize four hyper-parameters are\n",
    "#           activation,dropout_rate,init,learn_rate            #\n",
    "def nn_opt(activation,dropout_rate,init,learn_rate):\n",
    "    \"\"\"\n",
    "  Parameters :\n",
    "  Input - list of 4 hyper-parameter that are need to\n",
    "  be optimized\n",
    "\n",
    "  Output - Best optimized model \n",
    "    \"\"\"\n",
    "    \n",
    "    #define the optimmized neuron  number from experiment \n",
    "    neuron1,neuron2,neuron3,neuron4=64,16,64,64\n",
    "    ###############################################################################################\n",
    "    # the model building block \n",
    "    \n",
    "    model = Sequential()\n",
    "    np.random.seed(6)\n",
    "    model.add(Dense(neuron1, input_dim =4 , kernel_initializer= init, activation= activation))\n",
    "    model.add(Dense(neuron2, input_dim = neuron1, kernel_initializer= init, activation= activation))\n",
    "    model.add(Dense(neuron3, input_dim = neuron2, kernel_initializer= init, activation= activation))\n",
    "    model.add(Dense(neuron4, input_dim = neuron3, kernel_initializer= init, activation= activation))\n",
    "    model.add(Dropout(dropout_rate))\n",
    "    model.add(Dense(2, activation='softmax'))\n",
    "    \n",
    "    optimizer = Adam(lr = learn_rate)               #optimizer of Neural network \n",
    "    model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy']) #compile model \n",
    "    #################################################################################################\n",
    "    return model     \n",
    "\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-40s9dkHssNu"
   },
   "source": [
    "### Helper function for Ensembling model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a_BXPZol8pem"
   },
   "source": [
    "### Read the data from the drive using pandas (Python Data Analysis Library)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 12334,
     "status": "ok",
     "timestamp": 1585596177659,
     "user": {
      "displayName": "__phiniks_ _",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj7KQ5WRKqZl709NEi7W2pySAoctrhlmXZfwrGq=s64",
      "userId": "17328933659108214149"
     },
     "user_tz": -360
    },
    "id": "n-q5jMU4z4GA",
    "outputId": "1bd1988b-844f-4abb-f47f-eb4eeeebfe8a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(768, 9)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "if colab ==True:\n",
    "  data = pd.read_csv(data_dir)\n",
    "else:\n",
    "  data = pd.read_csv(data_dir)\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 238
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 12326,
     "status": "ok",
     "timestamp": 1585596177666,
     "user": {
      "displayName": "__phiniks_ _",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj7KQ5WRKqZl709NEi7W2pySAoctrhlmXZfwrGq=s64",
      "userId": "17328933659108214149"
     },
     "user_tz": -360
    },
    "id": "KluoAwn49hGG",
    "outputId": "dad57c21-5203-449a-c3e1-61c0a4b74c91"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>116</td>\n",
       "      <td>74</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>25.6</td>\n",
       "      <td>0.201</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0            6      148             72             35        0  33.6   \n",
       "1            1       85             66             29        0  26.6   \n",
       "2            8      183             64              0        0  23.3   \n",
       "3            1       89             66             23       94  28.1   \n",
       "4            0      137             40             35      168  43.1   \n",
       "5            5      116             74              0        0  25.6   \n",
       "\n",
       "   DiabetesPedigreeFunction  Age  Outcome  \n",
       "0                     0.627   50        1  \n",
       "1                     0.351   31        0  \n",
       "2                     0.672   32        1  \n",
       "3                     0.167   21        0  \n",
       "4                     2.288   33        1  \n",
       "5                     0.201   30        0  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(n=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nwaLj2vN_RmG"
   },
   "source": [
    "### Renaming the Features by F1, F2, and so on ......\n",
    "---\n",
    "New Name | Original Name | Comments| \n",
    "---|---|---|\n",
    "F1|Pregnancies|Number of times pregnant|\n",
    "F2|Glucose|Plasma glucose concentration a 2 hours in an oral glucose tolerance test|\n",
    "F3|BloodPressure|Diastolic blood pressure (mm Hg)|\n",
    "F4|SkinThickness|Triceps skin fold thickness (mm)|\n",
    "F5|Insulin|2-Hour serum insulin (mu U/ml)|\n",
    "F6|BMI|Body mass index (weight in kg/(height in m)^2)|\n",
    "F7|DiabetesPedigreeFunction|Diabetes pedigree function|\n",
    "F8|Age|Age (years)|\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VvHHG7dC_jzd"
   },
   "outputs": [],
   "source": [
    "data = pd.DataFrame({'F1':data.iloc[:,:8].values[:,0],\n",
    "                     'F2':data.iloc[:,:8].values[:,1],\n",
    "                     'F3':data.iloc[:,:8].values[:,2],\n",
    "                     'F4':data.iloc[:,:8].values[:,3],\n",
    "                     'F5':data.iloc[:,:8].values[:,4],\n",
    "                     'F6':data.iloc[:,:8].values[:,5],\n",
    "                     'F7':data.iloc[:,:8].values[:,6],\n",
    "                     'F8':data.iloc[:,:8].values[:,7],\n",
    "                     'Outcome':data.iloc[:,8:].values[:,0]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oRoQ8jmU-f2A"
   },
   "source": [
    "### Show the statistical description of the data which sumarize the central tendency, dispersion, and shape of a data distribution.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 300
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 12308,
     "status": "ok",
     "timestamp": 1585596177675,
     "user": {
      "displayName": "__phiniks_ _",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj7KQ5WRKqZl709NEi7W2pySAoctrhlmXZfwrGq=s64",
      "userId": "17328933659108214149"
     },
     "user_tz": -360
    },
    "id": "PdYuOfaHtpyn",
    "outputId": "f6ef6f1c-21d7-43c6-866e-010229784ee2"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>F1</th>\n",
       "      <th>F2</th>\n",
       "      <th>F3</th>\n",
       "      <th>F4</th>\n",
       "      <th>F5</th>\n",
       "      <th>F6</th>\n",
       "      <th>F7</th>\n",
       "      <th>F8</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.845052</td>\n",
       "      <td>120.894531</td>\n",
       "      <td>69.105469</td>\n",
       "      <td>20.536458</td>\n",
       "      <td>79.799479</td>\n",
       "      <td>31.992578</td>\n",
       "      <td>0.471876</td>\n",
       "      <td>33.240885</td>\n",
       "      <td>0.348958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3.369578</td>\n",
       "      <td>31.972618</td>\n",
       "      <td>19.355807</td>\n",
       "      <td>15.952218</td>\n",
       "      <td>115.244002</td>\n",
       "      <td>7.884160</td>\n",
       "      <td>0.331329</td>\n",
       "      <td>11.760232</td>\n",
       "      <td>0.476951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078000</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>27.300000</td>\n",
       "      <td>0.243750</td>\n",
       "      <td>24.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>117.000000</td>\n",
       "      <td>72.000000</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>30.500000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>0.372500</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6.000000</td>\n",
       "      <td>140.250000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>127.250000</td>\n",
       "      <td>36.600000</td>\n",
       "      <td>0.626250</td>\n",
       "      <td>41.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>17.000000</td>\n",
       "      <td>199.000000</td>\n",
       "      <td>122.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>846.000000</td>\n",
       "      <td>67.100000</td>\n",
       "      <td>2.420000</td>\n",
       "      <td>81.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               F1          F2          F3          F4          F5          F6  \\\n",
       "count  768.000000  768.000000  768.000000  768.000000  768.000000  768.000000   \n",
       "mean     3.845052  120.894531   69.105469   20.536458   79.799479   31.992578   \n",
       "std      3.369578   31.972618   19.355807   15.952218  115.244002    7.884160   \n",
       "min      0.000000    0.000000    0.000000    0.000000    0.000000    0.000000   \n",
       "25%      1.000000   99.000000   62.000000    0.000000    0.000000   27.300000   \n",
       "50%      3.000000  117.000000   72.000000   23.000000   30.500000   32.000000   \n",
       "75%      6.000000  140.250000   80.000000   32.000000  127.250000   36.600000   \n",
       "max     17.000000  199.000000  122.000000   99.000000  846.000000   67.100000   \n",
       "\n",
       "               F7          F8     Outcome  \n",
       "count  768.000000  768.000000  768.000000  \n",
       "mean     0.471876   33.240885    0.348958  \n",
       "std      0.331329   11.760232    0.476951  \n",
       "min      0.078000   21.000000    0.000000  \n",
       "25%      0.243750   24.000000    0.000000  \n",
       "50%      0.372500   29.000000    0.000000  \n",
       "75%      0.626250   41.000000    1.000000  \n",
       "max      2.420000   81.000000    1.000000  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "3dad9d1d-07d2-48fe-e5a4-022dfb83d33a",
    "colab_type": "text",
    "id": "dRMSzvK-tXjg"
   },
   "source": [
    "# Raw Data Plot and Presenation\n",
    "**Plot pairwise relationships in a dataset:**\n",
    "* The diagonal shows the distribution of the dataset with the kernel density plots of both the classes.\n",
    "\n",
    "* The scatter-plots shows the relation between each and every attribute or features taken pairwise where the scatter-plots shows that no attributes are able to clearly distinguish the two outcome. \n",
    "\n",
    "**Distribution Plot:**\n",
    "\n",
    "Dist Plot helps us to flexibly plot a univariate distribution of observations.\n",
    "\n",
    "**Violin Plots:** \n",
    "A violin plot is a method of plotting numeric data. It is similar to box plot with a rotated kernel density plot on each side. Violin plots are similar to box plots, except that they also show the probability density of the data at different values (in the simplest case this could be a histogram). \n",
    "\n",
    "**Correlation:** \n",
    "A correlation could be positive, meaning both variables move in the same direction, or negative, meaning that when one variable’s value increases, the other variables’ values decrease. Correlation can also be neural or zero, meaning that the variables are unrelated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "_cell_guid": "415da1b4-d02b-a94e-d289-87ff0b84d6c3",
    "colab": {},
    "colab_type": "code",
    "id": "_wpbciDvtXji"
   },
   "outputs": [],
   "source": [
    "# data_plot (data,\n",
    "#           Pair_plot=True,\n",
    "#           Dist_Plot=True,\n",
    "#           Plot_violinplot=True,\n",
    "#           Plot_confusionMatrix=True,\n",
    "#           box_Gaussian=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "7790af18-672d-658b-9d0a-58648dd63b13",
    "colab_type": "text",
    "id": "nluf1OUetXjy"
   },
   "source": [
    "# Data Preprocessing  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Zr2EoFiZrUvm"
   },
   "source": [
    "### Preprocessing Selection\n",
    "---\n",
    "Process Selector | Preprocessing | \n",
    "---|---|\n",
    "P|Outlier Rejection|\n",
    "Q|Filling Missing Value|\n",
    "R|Standardization|\n",
    "\n",
    "**Each P, Q, and R process has four techniques for feature selection which are N/A, PCA, ICA, and correlation-based feature selection**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "_cell_guid": "048157d7-1cfc-13b7-feca-5c73b00a03ee",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 72
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 12287,
     "status": "ok",
     "timestamp": 1585596177680,
     "user": {
      "displayName": "__phiniks_ _",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj7KQ5WRKqZl709NEi7W2pySAoctrhlmXZfwrGq=s64",
      "userId": "17328933659108214149"
     },
     "user_tz": -360
    },
    "id": "jbTObgkctXj1",
    "outputId": "567a24a4-7e9c-41b1-d7d6-138ca0530737"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape After outlier Removed: (636, 9)\n",
      "Shape After Filling Missing Value: (636, 9)\n",
      "Shape After Feature Selection: (636, 4)\n"
     ]
    }
   ],
   "source": [
    "# print('Shape Before Process: ' + str(data.shape))\n",
    "\n",
    "# The process for the outlier rejection (P)\n",
    "data = outlier_Rejection (data,\n",
    "                  iqr_Mean=False,\n",
    "                  iqr_Medain=False,\n",
    "                  iqr=True,\n",
    "                  manual=False)\n",
    "print('Shape After outlier Removed: ' + str(data.shape))\n",
    "\n",
    "## The process for the filling missing values (Q)\n",
    "for col in ['F2', 'F3', 'F4', 'F5', 'F6']:   \n",
    "    replace_zero(data, col, 'Outcome')              \n",
    "print('Shape After Filling Missing Value: ' + str(data.shape))\n",
    "     \n",
    "                        \n",
    "X_Data,Y_Lavel = feature_Selector(data, algo='corr', n_feature=4)    \n",
    "print('Shape After Feature Selection: ' + str(X_Data.shape))\n",
    "\n",
    "\n",
    "# scaler =  preprocessing.StandardScaler()\n",
    "# X_Data,Y_Lavel= scaler.fit_transform(X_Data), Y_Lavel\n",
    "# print('Shape After Standardization: ' + str(X_Data.shape))          \n",
    "\n",
    "kf = StratifiedKFold(n_splits=5,\n",
    "                     shuffle=False,\n",
    "                     random_state=random_initializer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_uTtoPjo8ypS"
   },
   "source": [
    "# Processed Data Plot and Presenation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hj0btC5h831I"
   },
   "outputs": [],
   "source": [
    "# data_plot (data,\n",
    "#           Pair_plot=True,\n",
    "#           Dist_Plot=True,\n",
    "#           Plot_violinplot=True,\n",
    "#           Plot_confusionMatrix=True,\n",
    "#           box_Gaussian=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "L9kt1_ZOnSWm"
   },
   "source": [
    "### MLP Experiment Block\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TAXmal3xwbP_"
   },
   "source": [
    "A  neural network (NN) is an artificial neural network (ANN) with multiple layers between the input and output layers. The NN finds the correct mathematical manipulation to turn the input into the output, whether it be a linear relationship or a non-linear relationship. For the \n",
    "comparison of performance between ML classifiers we find the best peformance using Neural Network.At first we find out the optimum number of hidden layer and number of neuron in every layer. Then, Using grid search technique we optimze the hyperparameters(Number of epoch, batch size, learning rate, droupt out rate , activation afunction , kernel initializer) of NN model ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pO-9nWCUwz--"
   },
   "source": [
    "#### Find the optimum number of hidden layers and neurons in number every layer "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Vc7-YnJowbQR"
   },
   "outputs": [],
   "source": [
    "# ################################\n",
    "# # list for store accuracy and auc value and plot\n",
    "# accuracy = []\n",
    "# AUC = []\n",
    "# ##############################\n",
    "\n",
    "\n",
    "# ################################################################################\n",
    "# # in this block we  optimize number of hiiden layer and neuron number \n",
    "# for hLayer in range(1,11):\n",
    "#     print ('------------------------------Model Running with # of Hidden = '+str(hLayer)+'-----------------------------')\n",
    "#     batch_size, epochs, model, grid_results = run (hLayer,   \n",
    "#                                      batch_size = [32],           \n",
    "#                                      epochs = [100],              \n",
    "#                                      learn_rate = [0.001],         \n",
    "#                                      dropout_rate = [0.3],       \n",
    "#                                      activation = ['relu'],   \n",
    "#                                      init =['normal'])\n",
    "    \n",
    "#     print(grid_results.best_params_)\n",
    "    \n",
    "#     Y_Train_1Hot = to_categorical(Y_Lavel,2)\n",
    "#     model.fit(x=X_Data,\n",
    "#             y=Y_Train_1Hot,\n",
    "#             batch_size=batch_size,\n",
    "#             epochs=epochs,\n",
    "#             verbose=0)\n",
    "#     probas_ = model.predict(X_Data)\n",
    "#     y_pred = np.argmax(model.predict(X_Data), axis=1)\n",
    "    \n",
    "#     #############################################################################\n",
    "#     #   print and stroe the optimized model accuracy and  auc  \n",
    "#     print(accuracy_score(Y_Lavel, y_pred))\n",
    "#     print(roc_auc_score(Y_Train_1Hot, probas_))\n",
    "#     accuracy.append(accuracy_score(Y_Lavel, y_pred))\n",
    "#     AUC.append(roc_auc_score(Y_Train_1Hot, probas_))\n",
    "#     #############################################################################\n",
    "    \n",
    "\n",
    "# ######   plot the value of auc and accuracy ####\n",
    "# plt.plot(AUC)\n",
    "# plt.plot(accuracy)\n",
    "# ################################################"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LyVnsEj3RE_M"
   },
   "outputs": [],
   "source": [
    "##########################\n",
    "# Define a random seed\n",
    "seed = 6\n",
    "np.random.seed(seed)\n",
    "###########################\n",
    "\n",
    "\n",
    "\n",
    "# create the model for optimization \n",
    "model = KerasClassifier(build_fn = nn_opt, verbose = 0)\n",
    "\n",
    "##########################################################\n",
    "##List of hyper-parameters of MLP model for optimization##\n",
    "\n",
    "batch_size = [8, 16, 32]            \n",
    "epochs = [100, 150, 200]              \n",
    "learn_rate =[0.001,.05, 0.1]         \n",
    "dropout_rate = [0.0, 0.3, 0.6]       \n",
    "activation = ['relu', 'tanh']   \n",
    "init =['uniform', 'normal']         \n",
    "\n",
    "###########################################################\n",
    "\n",
    "###########################################################\n",
    "## parameter dictionary for grid ssearch   ##\n",
    "param_grid = dict(batch_size=batch_size,\n",
    "                  epochs=epochs,\n",
    "                  learn_rate=learn_rate,\n",
    "                  dropout_rate=dropout_rate,\n",
    "                  activation=activation,\n",
    "                  init=init)\n",
    "\n",
    "#build and fit the GridSearchCV\n",
    "grid = GridSearchCV(estimator = model,\n",
    "                    param_grid = param_grid,\n",
    "                    cv = 5,\n",
    "                    n_jobs = -1,\n",
    "                    verbose = 1)\n",
    "###########################################################\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 620
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 13166,
     "status": "error",
     "timestamp": 1585596178598,
     "user": {
      "displayName": "__phiniks_ _",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj7KQ5WRKqZl709NEi7W2pySAoctrhlmXZfwrGq=s64",
      "userId": "17328933659108214149"
     },
     "user_tz": -360
    },
    "id": "UPZJ_TGUh9Q_",
    "outputId": "feef9548-25d3-423a-9775-e0a3a75201a4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "relu 8 200 0.001 0.6 normal 64 16 64 64\n",
      "------------------->>>>>>>>>>Fold no =  1\n",
      "Epoch 1/200\n",
      "508/508 [==============================] - 0s 981us/step - loss: 0.6518 - acc: 0.6811\n",
      "Epoch 2/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.6164 - acc: 0.6929\n",
      "Epoch 3/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.5779 - acc: 0.7126\n",
      "Epoch 4/200\n",
      "508/508 [==============================] - 0s 140us/step - loss: 0.5623 - acc: 0.6988\n",
      "Epoch 5/200\n",
      "508/508 [==============================] - 0s 142us/step - loss: 0.5470 - acc: 0.7303\n",
      "Epoch 6/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.5336 - acc: 0.7559\n",
      "Epoch 7/200\n",
      "508/508 [==============================] - 0s 142us/step - loss: 0.5304 - acc: 0.7421\n",
      "Epoch 8/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.5092 - acc: 0.7736\n",
      "Epoch 9/200\n",
      "508/508 [==============================] - 0s 140us/step - loss: 0.5063 - acc: 0.7657\n",
      "Epoch 10/200\n",
      "508/508 [==============================] - 0s 156us/step - loss: 0.4983 - acc: 0.7795\n",
      "Epoch 11/200\n",
      "508/508 [==============================] - 0s 142us/step - loss: 0.4736 - acc: 0.8091\n",
      "Epoch 12/200\n",
      "508/508 [==============================] - 0s 158us/step - loss: 0.4625 - acc: 0.8169\n",
      "Epoch 13/200\n",
      "508/508 [==============================] - 0s 150us/step - loss: 0.4530 - acc: 0.8110\n",
      "Epoch 14/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.4362 - acc: 0.8189\n",
      "Epoch 15/200\n",
      "508/508 [==============================] - 0s 148us/step - loss: 0.4089 - acc: 0.8307\n",
      "Epoch 16/200\n",
      "508/508 [==============================] - 0s 148us/step - loss: 0.4004 - acc: 0.8465\n",
      "Epoch 17/200\n",
      "508/508 [==============================] - 0s 142us/step - loss: 0.3824 - acc: 0.8425\n",
      "Epoch 18/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.4035 - acc: 0.8346\n",
      "Epoch 19/200\n",
      "508/508 [==============================] - 0s 142us/step - loss: 0.3853 - acc: 0.8445\n",
      "Epoch 20/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3714 - acc: 0.8622\n",
      "Epoch 21/200\n",
      "508/508 [==============================] - 0s 140us/step - loss: 0.3784 - acc: 0.8504\n",
      "Epoch 22/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3970 - acc: 0.8209\n",
      "Epoch 23/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3741 - acc: 0.8386\n",
      "Epoch 24/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3662 - acc: 0.8465\n",
      "Epoch 25/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3909 - acc: 0.8327\n",
      "Epoch 26/200\n",
      "508/508 [==============================] - 0s 130us/step - loss: 0.3705 - acc: 0.8406\n",
      "Epoch 27/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3746 - acc: 0.8425\n",
      "Epoch 28/200\n",
      "508/508 [==============================] - 0s 140us/step - loss: 0.3706 - acc: 0.8366\n",
      "Epoch 29/200\n",
      "508/508 [==============================] - 0s 148us/step - loss: 0.3761 - acc: 0.8307\n",
      "Epoch 30/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3643 - acc: 0.8445\n",
      "Epoch 31/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3643 - acc: 0.8563\n",
      "Epoch 32/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3690 - acc: 0.8484\n",
      "Epoch 33/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3731 - acc: 0.8445\n",
      "Epoch 34/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3634 - acc: 0.8504\n",
      "Epoch 35/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3467 - acc: 0.8445\n",
      "Epoch 36/200\n",
      "508/508 [==============================] - 0s 140us/step - loss: 0.3564 - acc: 0.8701\n",
      "Epoch 37/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3459 - acc: 0.8465\n",
      "Epoch 38/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3595 - acc: 0.8524\n",
      "Epoch 39/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3586 - acc: 0.8524\n",
      "Epoch 40/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3461 - acc: 0.8524\n",
      "Epoch 41/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3548 - acc: 0.8386\n",
      "Epoch 42/200\n",
      "508/508 [==============================] - 0s 148us/step - loss: 0.3569 - acc: 0.8524\n",
      "Epoch 43/200\n",
      "508/508 [==============================] - 0s 140us/step - loss: 0.3403 - acc: 0.8583\n",
      "Epoch 44/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3707 - acc: 0.8406\n",
      "Epoch 45/200\n",
      "508/508 [==============================] - 0s 146us/step - loss: 0.3402 - acc: 0.8504\n",
      "Epoch 46/200\n",
      "508/508 [==============================] - 0s 146us/step - loss: 0.3711 - acc: 0.8327\n",
      "Epoch 47/200\n",
      "508/508 [==============================] - 0s 146us/step - loss: 0.3418 - acc: 0.8543\n",
      "Epoch 48/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3430 - acc: 0.8524\n",
      "Epoch 49/200\n",
      "508/508 [==============================] - 0s 148us/step - loss: 0.3307 - acc: 0.8465\n",
      "Epoch 50/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3519 - acc: 0.8543\n",
      "Epoch 51/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3509 - acc: 0.8327\n",
      "Epoch 52/200\n",
      "508/508 [==============================] - 0s 154us/step - loss: 0.3405 - acc: 0.8406\n",
      "Epoch 53/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3402 - acc: 0.8406\n",
      "Epoch 54/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3461 - acc: 0.8563\n",
      "Epoch 55/200\n",
      "508/508 [==============================] - 0s 130us/step - loss: 0.3409 - acc: 0.8425\n",
      "Epoch 56/200\n",
      "508/508 [==============================] - 0s 130us/step - loss: 0.3434 - acc: 0.8524\n",
      "Epoch 57/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3718 - acc: 0.8287\n",
      "Epoch 58/200\n",
      "508/508 [==============================] - 0s 130us/step - loss: 0.3451 - acc: 0.8465\n",
      "Epoch 59/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3560 - acc: 0.8465\n",
      "Epoch 60/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3356 - acc: 0.8563\n",
      "Epoch 61/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3091 - acc: 0.8563\n",
      "Epoch 62/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3269 - acc: 0.8425\n",
      "Epoch 63/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3537 - acc: 0.8366\n",
      "Epoch 64/200\n",
      "508/508 [==============================] - 0s 140us/step - loss: 0.3422 - acc: 0.8465\n",
      "Epoch 65/200\n",
      "508/508 [==============================] - 0s 142us/step - loss: 0.3490 - acc: 0.8386\n",
      "Epoch 66/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3451 - acc: 0.8484\n",
      "Epoch 67/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3390 - acc: 0.8563\n",
      "Epoch 68/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3344 - acc: 0.8366\n",
      "Epoch 69/200\n",
      "508/508 [==============================] - 0s 142us/step - loss: 0.3423 - acc: 0.8445\n",
      "Epoch 70/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3333 - acc: 0.8504\n",
      "Epoch 71/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3323 - acc: 0.8543\n",
      "Epoch 72/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3235 - acc: 0.8524\n",
      "Epoch 73/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3284 - acc: 0.8465\n",
      "Epoch 74/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3428 - acc: 0.8406\n",
      "Epoch 75/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3463 - acc: 0.8366\n",
      "Epoch 76/200\n",
      "508/508 [==============================] - 0s 130us/step - loss: 0.3334 - acc: 0.8524\n",
      "Epoch 77/200\n",
      "508/508 [==============================] - 0s 128us/step - loss: 0.3347 - acc: 0.8484\n",
      "Epoch 78/200\n",
      "508/508 [==============================] - 0s 128us/step - loss: 0.3293 - acc: 0.8445\n",
      "Epoch 79/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3399 - acc: 0.8465\n",
      "Epoch 80/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3476 - acc: 0.8425\n",
      "Epoch 81/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3432 - acc: 0.8346\n",
      "Epoch 82/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3495 - acc: 0.8465\n",
      "Epoch 83/200\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "508/508 [==============================] - 0s 136us/step - loss: 0.3644 - acc: 0.8622\n",
      "Epoch 84/200\n",
      "508/508 [==============================] - 0s 140us/step - loss: 0.3364 - acc: 0.8445\n",
      "Epoch 85/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3357 - acc: 0.8563\n",
      "Epoch 86/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3338 - acc: 0.8583\n",
      "Epoch 87/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3477 - acc: 0.8504\n",
      "Epoch 88/200\n",
      "508/508 [==============================] - 0s 126us/step - loss: 0.3297 - acc: 0.8701\n",
      "Epoch 89/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3359 - acc: 0.8524\n",
      "Epoch 90/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3450 - acc: 0.8563\n",
      "Epoch 91/200\n",
      "508/508 [==============================] - 0s 142us/step - loss: 0.3298 - acc: 0.8425\n",
      "Epoch 92/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3176 - acc: 0.8465\n",
      "Epoch 93/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3328 - acc: 0.8524\n",
      "Epoch 94/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3408 - acc: 0.8622\n",
      "Epoch 95/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3262 - acc: 0.8642\n",
      "Epoch 96/200\n",
      "508/508 [==============================] - 0s 126us/step - loss: 0.3320 - acc: 0.8504\n",
      "Epoch 97/200\n",
      "508/508 [==============================] - 0s 128us/step - loss: 0.3216 - acc: 0.8661\n",
      "Epoch 98/200\n",
      "508/508 [==============================] - 0s 128us/step - loss: 0.3236 - acc: 0.8642\n",
      "Epoch 99/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3233 - acc: 0.8681\n",
      "Epoch 100/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3211 - acc: 0.8661\n",
      "Epoch 101/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3312 - acc: 0.8445\n",
      "Epoch 102/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3336 - acc: 0.8563\n",
      "Epoch 103/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3355 - acc: 0.8445\n",
      "Epoch 104/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3178 - acc: 0.8622\n",
      "Epoch 105/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3137 - acc: 0.8543\n",
      "Epoch 106/200\n",
      "508/508 [==============================] - 0s 142us/step - loss: 0.3241 - acc: 0.8563\n",
      "Epoch 107/200\n",
      "508/508 [==============================] - 0s 140us/step - loss: 0.3226 - acc: 0.8504\n",
      "Epoch 108/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3419 - acc: 0.8327\n",
      "Epoch 109/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3288 - acc: 0.8465\n",
      "Epoch 110/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3214 - acc: 0.8622\n",
      "Epoch 111/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3423 - acc: 0.8406\n",
      "Epoch 112/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3187 - acc: 0.8642\n",
      "Epoch 113/200\n",
      "508/508 [==============================] - 0s 142us/step - loss: 0.3413 - acc: 0.8406\n",
      "Epoch 114/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3398 - acc: 0.8504\n",
      "Epoch 115/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3449 - acc: 0.8465\n",
      "Epoch 116/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3297 - acc: 0.8484\n",
      "Epoch 117/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3469 - acc: 0.8425\n",
      "Epoch 118/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3270 - acc: 0.8583\n",
      "Epoch 119/200\n",
      "508/508 [==============================] - 0s 130us/step - loss: 0.3353 - acc: 0.8720\n",
      "Epoch 120/200\n",
      "508/508 [==============================] - 0s 128us/step - loss: 0.3330 - acc: 0.8484\n",
      "Epoch 121/200\n",
      "508/508 [==============================] - 0s 126us/step - loss: 0.3393 - acc: 0.8504\n",
      "Epoch 122/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3238 - acc: 0.8563\n",
      "Epoch 123/200\n",
      "508/508 [==============================] - 0s 126us/step - loss: 0.3192 - acc: 0.8602\n",
      "Epoch 124/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3149 - acc: 0.8406\n",
      "Epoch 125/200\n",
      "508/508 [==============================] - 0s 140us/step - loss: 0.3142 - acc: 0.8602\n",
      "Epoch 126/200\n",
      "508/508 [==============================] - 0s 146us/step - loss: 0.3248 - acc: 0.8583\n",
      "Epoch 127/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3105 - acc: 0.8504\n",
      "Epoch 128/200\n",
      "508/508 [==============================] - 0s 128us/step - loss: 0.3272 - acc: 0.8524\n",
      "Epoch 129/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3156 - acc: 0.8366\n",
      "Epoch 130/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3506 - acc: 0.8484\n",
      "Epoch 131/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3211 - acc: 0.8425\n",
      "Epoch 132/200\n",
      "508/508 [==============================] - 0s 130us/step - loss: 0.3148 - acc: 0.8563\n",
      "Epoch 133/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3250 - acc: 0.8543\n",
      "Epoch 134/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3392 - acc: 0.8563\n",
      "Epoch 135/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3180 - acc: 0.8484\n",
      "Epoch 136/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3144 - acc: 0.8504\n",
      "Epoch 137/200\n",
      "508/508 [==============================] - 0s 128us/step - loss: 0.3196 - acc: 0.8720\n",
      "Epoch 138/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3079 - acc: 0.8543\n",
      "Epoch 139/200\n",
      "508/508 [==============================] - 0s 130us/step - loss: 0.3124 - acc: 0.8543\n",
      "Epoch 140/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3143 - acc: 0.8583\n",
      "Epoch 141/200\n",
      "508/508 [==============================] - 0s 128us/step - loss: 0.3374 - acc: 0.8602\n",
      "Epoch 142/200\n",
      "508/508 [==============================] - 0s 128us/step - loss: 0.3198 - acc: 0.8622\n",
      "Epoch 143/200\n",
      "508/508 [==============================] - 0s 140us/step - loss: 0.3235 - acc: 0.8386\n",
      "Epoch 144/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3270 - acc: 0.8622\n",
      "Epoch 145/200\n",
      "508/508 [==============================] - 0s 140us/step - loss: 0.3148 - acc: 0.8504\n",
      "Epoch 146/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3232 - acc: 0.8524\n",
      "Epoch 147/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3304 - acc: 0.8386\n",
      "Epoch 148/200\n",
      "508/508 [==============================] - 0s 128us/step - loss: 0.3108 - acc: 0.8504\n",
      "Epoch 149/200\n",
      "508/508 [==============================] - 0s 126us/step - loss: 0.3197 - acc: 0.8425\n",
      "Epoch 150/200\n",
      "508/508 [==============================] - 0s 130us/step - loss: 0.3177 - acc: 0.8642\n",
      "Epoch 151/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3225 - acc: 0.8504\n",
      "Epoch 152/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3156 - acc: 0.8602\n",
      "Epoch 153/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3091 - acc: 0.8583\n",
      "Epoch 154/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3168 - acc: 0.8465\n",
      "Epoch 155/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3573 - acc: 0.8465\n",
      "Epoch 156/200\n",
      "508/508 [==============================] - 0s 140us/step - loss: 0.3375 - acc: 0.8386\n",
      "Epoch 157/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3130 - acc: 0.8524\n",
      "Epoch 158/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3209 - acc: 0.8563\n",
      "Epoch 159/200\n",
      "508/508 [==============================] - 0s 142us/step - loss: 0.3201 - acc: 0.8563\n",
      "Epoch 160/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3051 - acc: 0.8563\n",
      "Epoch 161/200\n",
      "508/508 [==============================] - 0s 142us/step - loss: 0.3352 - acc: 0.8524\n",
      "Epoch 162/200\n",
      "508/508 [==============================] - 0s 140us/step - loss: 0.3214 - acc: 0.8504\n",
      "Epoch 163/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3330 - acc: 0.8504\n",
      "Epoch 164/200\n",
      "508/508 [==============================] - 0s 128us/step - loss: 0.3000 - acc: 0.8583\n",
      "Epoch 165/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.3097 - acc: 0.8583\n",
      "Epoch 166/200\n",
      "508/508 [==============================] - 0s 140us/step - loss: 0.3180 - acc: 0.8720\n",
      "Epoch 167/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3082 - acc: 0.8642\n",
      "Epoch 168/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3261 - acc: 0.8484\n",
      "Epoch 169/200\n",
      "508/508 [==============================] - 0s 126us/step - loss: 0.3076 - acc: 0.8681\n",
      "Epoch 170/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3155 - acc: 0.8583\n",
      "Epoch 171/200\n",
      "508/508 [==============================] - 0s 126us/step - loss: 0.2970 - acc: 0.8642\n",
      "Epoch 172/200\n",
      "508/508 [==============================] - 0s 128us/step - loss: 0.3375 - acc: 0.8661\n",
      "Epoch 173/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3231 - acc: 0.8642\n",
      "Epoch 174/200\n",
      "508/508 [==============================] - 0s 128us/step - loss: 0.3124 - acc: 0.8661\n",
      "Epoch 175/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3164 - acc: 0.8681\n",
      "Epoch 176/200\n",
      "508/508 [==============================] - 0s 130us/step - loss: 0.3044 - acc: 0.8583\n",
      "Epoch 177/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3323 - acc: 0.8484\n",
      "Epoch 178/200\n",
      "508/508 [==============================] - 0s 124us/step - loss: 0.3269 - acc: 0.8661\n",
      "Epoch 179/200\n",
      "508/508 [==============================] - 0s 126us/step - loss: 0.3042 - acc: 0.8642\n",
      "Epoch 180/200\n",
      "508/508 [==============================] - 0s 124us/step - loss: 0.3143 - acc: 0.8504\n",
      "Epoch 181/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3371 - acc: 0.8504\n",
      "Epoch 182/200\n",
      "508/508 [==============================] - 0s 132us/step - loss: 0.3308 - acc: 0.8642\n",
      "Epoch 183/200\n",
      "508/508 [==============================] - 0s 130us/step - loss: 0.3146 - acc: 0.8720\n",
      "Epoch 184/200\n",
      "508/508 [==============================] - 0s 140us/step - loss: 0.3171 - acc: 0.8720\n",
      "Epoch 185/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3347 - acc: 0.8543\n",
      "Epoch 186/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3144 - acc: 0.8504\n",
      "Epoch 187/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.2911 - acc: 0.8701\n",
      "Epoch 188/200\n",
      "508/508 [==============================] - 0s 134us/step - loss: 0.3300 - acc: 0.8504\n",
      "Epoch 189/200\n",
      "508/508 [==============================] - 0s 128us/step - loss: 0.3098 - acc: 0.8661\n",
      "Epoch 190/200\n",
      "508/508 [==============================] - 0s 130us/step - loss: 0.3240 - acc: 0.8524\n",
      "Epoch 191/200\n",
      "508/508 [==============================] - 0s 130us/step - loss: 0.2973 - acc: 0.8642\n",
      "Epoch 192/200\n",
      "508/508 [==============================] - 0s 130us/step - loss: 0.3176 - acc: 0.8661\n",
      "Epoch 193/200\n",
      "508/508 [==============================] - 0s 144us/step - loss: 0.3103 - acc: 0.8622\n",
      "Epoch 194/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3200 - acc: 0.8602\n",
      "Epoch 195/200\n",
      "508/508 [==============================] - 0s 138us/step - loss: 0.2937 - acc: 0.8701\n",
      "Epoch 196/200\n",
      "508/508 [==============================] - 0s 136us/step - loss: 0.3187 - acc: 0.8583\n",
      "Epoch 197/200\n",
      "508/508 [==============================] - 0s 130us/step - loss: 0.2939 - acc: 0.8720\n",
      "Epoch 198/200\n",
      "508/508 [==============================] - 0s 146us/step - loss: 0.3199 - acc: 0.8602\n",
      "Epoch 199/200\n",
      "508/508 [==============================] - 0s 130us/step - loss: 0.3018 - acc: 0.8681\n",
      "Epoch 200/200\n",
      "508/508 [==============================] - 0s 128us/step - loss: 0.2917 - acc: 0.8740\n",
      "--------------------------------------------------\n",
      "Detailed classification report for current fold:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.82      0.87        88\n",
      "           1       0.69      0.88      0.77        40\n",
      "\n",
      "    accuracy                           0.84       128\n",
      "   macro avg       0.81      0.85      0.82       128\n",
      "weighted avg       0.86      0.84      0.84       128\n",
      "\n",
      "\n",
      "Area Under ROC (AUC): 0.875284090909091\n",
      "\n",
      "Confusion Matrix for current fold: \n",
      "[[72 16]\n",
      " [ 5 35]]\n",
      "\n",
      "Accuracy for Current Fold: 0.8359375\n",
      "\n",
      "31.5\n",
      "------------------->>>>>>>>>>Fold no =  2\n",
      "Epoch 1/200\n",
      "509/509 [==============================] - 1s 1ms/step - loss: 0.6545 - acc: 0.6739\n",
      "Epoch 2/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.6282 - acc: 0.6798\n",
      "Epoch 3/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.6050 - acc: 0.6994\n",
      "Epoch 4/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.5903 - acc: 0.7230\n",
      "Epoch 5/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.5749 - acc: 0.7367\n",
      "Epoch 6/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.5600 - acc: 0.7387\n",
      "Epoch 7/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.5381 - acc: 0.7230\n",
      "Epoch 8/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.5473 - acc: 0.7289\n",
      "Epoch 9/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.5261 - acc: 0.7407\n",
      "Epoch 10/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.5148 - acc: 0.7642\n",
      "Epoch 11/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.5193 - acc: 0.7466\n",
      "Epoch 12/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.4976 - acc: 0.7642\n",
      "Epoch 13/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.4940 - acc: 0.7741\n",
      "Epoch 14/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.4911 - acc: 0.7898\n",
      "Epoch 15/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.4840 - acc: 0.7741\n",
      "Epoch 16/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.4813 - acc: 0.7957\n",
      "Epoch 17/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.4604 - acc: 0.8114\n",
      "Epoch 18/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.4511 - acc: 0.8114\n",
      "Epoch 19/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.4544 - acc: 0.7917\n",
      "Epoch 20/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.4433 - acc: 0.8134\n",
      "Epoch 21/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.4385 - acc: 0.8153\n",
      "Epoch 22/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.4297 - acc: 0.8193\n",
      "Epoch 23/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.4193 - acc: 0.8291\n",
      "Epoch 24/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.4091 - acc: 0.8212\n",
      "Epoch 25/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.4146 - acc: 0.8369\n",
      "Epoch 26/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.4062 - acc: 0.8291\n",
      "Epoch 27/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3896 - acc: 0.8330\n",
      "Epoch 28/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3903 - acc: 0.8546\n",
      "Epoch 29/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.4073 - acc: 0.8134\n",
      "Epoch 30/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3817 - acc: 0.8605\n",
      "Epoch 31/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3864 - acc: 0.8271\n",
      "Epoch 32/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3901 - acc: 0.8487\n",
      "Epoch 33/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3755 - acc: 0.8468\n",
      "Epoch 34/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3972 - acc: 0.8409\n",
      "Epoch 35/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3701 - acc: 0.8389\n",
      "Epoch 36/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3767 - acc: 0.8527\n",
      "Epoch 37/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3908 - acc: 0.8369\n",
      "Epoch 38/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3825 - acc: 0.8428\n",
      "Epoch 39/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3732 - acc: 0.8251\n",
      "Epoch 40/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3608 - acc: 0.8428\n",
      "Epoch 41/200\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "509/509 [==============================] - 0s 140us/step - loss: 0.3914 - acc: 0.8428\n",
      "Epoch 42/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3621 - acc: 0.8448\n",
      "Epoch 43/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3704 - acc: 0.8546\n",
      "Epoch 44/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3676 - acc: 0.8546\n",
      "Epoch 45/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3642 - acc: 0.8428\n",
      "Epoch 46/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3572 - acc: 0.8546\n",
      "Epoch 47/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3751 - acc: 0.8271\n",
      "Epoch 48/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3587 - acc: 0.8527\n",
      "Epoch 49/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3942 - acc: 0.8389\n",
      "Epoch 50/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3694 - acc: 0.8389\n",
      "Epoch 51/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3642 - acc: 0.8566\n",
      "Epoch 52/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3449 - acc: 0.8625\n",
      "Epoch 53/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3824 - acc: 0.8566\n",
      "Epoch 54/200\n",
      "509/509 [==============================] - 0s 147us/step - loss: 0.3774 - acc: 0.8409\n",
      "Epoch 55/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3629 - acc: 0.8546\n",
      "Epoch 56/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3356 - acc: 0.8625\n",
      "Epoch 57/200\n",
      "509/509 [==============================] - 0s 126us/step - loss: 0.3597 - acc: 0.8625\n",
      "Epoch 58/200\n",
      "509/509 [==============================] - 0s 124us/step - loss: 0.3629 - acc: 0.8566\n",
      "Epoch 59/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3368 - acc: 0.8644\n",
      "Epoch 60/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3873 - acc: 0.8409\n",
      "Epoch 61/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3373 - acc: 0.8644\n",
      "Epoch 62/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3662 - acc: 0.8625\n",
      "Epoch 63/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3510 - acc: 0.8507\n",
      "Epoch 64/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3331 - acc: 0.8684\n",
      "Epoch 65/200\n",
      "509/509 [==============================] - 0s 126us/step - loss: 0.3549 - acc: 0.8527\n",
      "Epoch 66/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3881 - acc: 0.8644\n",
      "Epoch 67/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3868 - acc: 0.8487\n",
      "Epoch 68/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3622 - acc: 0.8625\n",
      "Epoch 69/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3454 - acc: 0.8428\n",
      "Epoch 70/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3403 - acc: 0.8546\n",
      "Epoch 71/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3386 - acc: 0.8605\n",
      "Epoch 72/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3449 - acc: 0.8625\n",
      "Epoch 73/200\n",
      "509/509 [==============================] - 0s 153us/step - loss: 0.3519 - acc: 0.8644\n",
      "Epoch 74/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3611 - acc: 0.8448\n",
      "Epoch 75/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3751 - acc: 0.8389\n",
      "Epoch 76/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3691 - acc: 0.8605\n",
      "Epoch 77/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3467 - acc: 0.8743\n",
      "Epoch 78/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3608 - acc: 0.8585\n",
      "Epoch 79/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3564 - acc: 0.8487\n",
      "Epoch 80/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3478 - acc: 0.8625\n",
      "Epoch 81/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3497 - acc: 0.8625\n",
      "Epoch 82/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3565 - acc: 0.8605\n",
      "Epoch 83/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3431 - acc: 0.8585\n",
      "Epoch 84/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3479 - acc: 0.8743\n",
      "Epoch 85/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3410 - acc: 0.8743\n",
      "Epoch 86/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.3667 - acc: 0.8527\n",
      "Epoch 87/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3604 - acc: 0.8585\n",
      "Epoch 88/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3266 - acc: 0.8703\n",
      "Epoch 89/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3445 - acc: 0.8625\n",
      "Epoch 90/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3494 - acc: 0.8684\n",
      "Epoch 91/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3389 - acc: 0.8625\n",
      "Epoch 92/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3466 - acc: 0.8644\n",
      "Epoch 93/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3231 - acc: 0.8644\n",
      "Epoch 94/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3415 - acc: 0.8625\n",
      "Epoch 95/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3375 - acc: 0.8644\n",
      "Epoch 96/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3501 - acc: 0.8625\n",
      "Epoch 97/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3352 - acc: 0.8605\n",
      "Epoch 98/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3321 - acc: 0.8664\n",
      "Epoch 99/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3324 - acc: 0.8644\n",
      "Epoch 100/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3369 - acc: 0.8644\n",
      "Epoch 101/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.3334 - acc: 0.8684\n",
      "Epoch 102/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3487 - acc: 0.8605\n",
      "Epoch 103/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3498 - acc: 0.8684\n",
      "Epoch 104/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3406 - acc: 0.8644\n",
      "Epoch 105/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3734 - acc: 0.8448\n",
      "Epoch 106/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3358 - acc: 0.8527\n",
      "Epoch 107/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3371 - acc: 0.8625\n",
      "Epoch 108/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3273 - acc: 0.8644\n",
      "Epoch 109/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3339 - acc: 0.8762\n",
      "Epoch 110/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3489 - acc: 0.8566\n",
      "Epoch 111/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3490 - acc: 0.8684\n",
      "Epoch 112/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.3285 - acc: 0.8605\n",
      "Epoch 113/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3273 - acc: 0.8703\n",
      "Epoch 114/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3370 - acc: 0.8723\n",
      "Epoch 115/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3274 - acc: 0.8605\n",
      "Epoch 116/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3421 - acc: 0.8605\n",
      "Epoch 117/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.3347 - acc: 0.8703\n",
      "Epoch 118/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3221 - acc: 0.8684\n",
      "Epoch 119/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3357 - acc: 0.8684\n",
      "Epoch 120/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3307 - acc: 0.8703\n",
      "Epoch 121/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3425 - acc: 0.8644\n",
      "Epoch 122/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3310 - acc: 0.8821\n",
      "Epoch 123/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3301 - acc: 0.8605\n",
      "Epoch 124/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3337 - acc: 0.8625\n",
      "Epoch 125/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3404 - acc: 0.8664\n",
      "Epoch 126/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3386 - acc: 0.8605\n",
      "Epoch 127/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3303 - acc: 0.8625\n",
      "Epoch 128/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3177 - acc: 0.8664\n",
      "Epoch 129/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3219 - acc: 0.8605\n",
      "Epoch 130/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3241 - acc: 0.8723\n",
      "Epoch 131/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3402 - acc: 0.8743\n",
      "Epoch 132/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3133 - acc: 0.8723\n",
      "Epoch 133/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3319 - acc: 0.8605\n",
      "Epoch 134/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3140 - acc: 0.8703\n",
      "Epoch 135/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3360 - acc: 0.8684\n",
      "Epoch 136/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3164 - acc: 0.8625\n",
      "Epoch 137/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3466 - acc: 0.8703\n",
      "Epoch 138/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3411 - acc: 0.8762\n",
      "Epoch 139/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3301 - acc: 0.8664\n",
      "Epoch 140/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3253 - acc: 0.8743\n",
      "Epoch 141/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3229 - acc: 0.8703\n",
      "Epoch 142/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3074 - acc: 0.8802\n",
      "Epoch 143/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3262 - acc: 0.8782\n",
      "Epoch 144/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3268 - acc: 0.8703\n",
      "Epoch 145/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3117 - acc: 0.8684\n",
      "Epoch 146/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3285 - acc: 0.8507\n",
      "Epoch 147/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3251 - acc: 0.8743\n",
      "Epoch 148/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.3270 - acc: 0.8743\n",
      "Epoch 149/200\n",
      "509/509 [==============================] - 0s 124us/step - loss: 0.3252 - acc: 0.8802\n",
      "Epoch 150/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3239 - acc: 0.8684\n",
      "Epoch 151/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3307 - acc: 0.8723\n",
      "Epoch 152/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3356 - acc: 0.8762\n",
      "Epoch 153/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3298 - acc: 0.8900\n",
      "Epoch 154/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3340 - acc: 0.8625\n",
      "Epoch 155/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3302 - acc: 0.8468\n",
      "Epoch 156/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3340 - acc: 0.8723\n",
      "Epoch 157/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3088 - acc: 0.8762\n",
      "Epoch 158/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.2997 - acc: 0.8861\n",
      "Epoch 159/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3154 - acc: 0.8703\n",
      "Epoch 160/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3228 - acc: 0.8644\n",
      "Epoch 161/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.3257 - acc: 0.8782\n",
      "Epoch 162/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.3362 - acc: 0.8762\n",
      "Epoch 163/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3186 - acc: 0.8723\n",
      "Epoch 164/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3267 - acc: 0.8821\n",
      "Epoch 165/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3312 - acc: 0.8743\n",
      "Epoch 166/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3158 - acc: 0.8743\n",
      "Epoch 167/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3318 - acc: 0.8585\n",
      "Epoch 168/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3433 - acc: 0.8625\n",
      "Epoch 169/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3089 - acc: 0.8782\n",
      "Epoch 170/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3205 - acc: 0.8644\n",
      "Epoch 171/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3315 - acc: 0.8703\n",
      "Epoch 172/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3289 - acc: 0.8743\n",
      "Epoch 173/200\n",
      "509/509 [==============================] - 0s 149us/step - loss: 0.3328 - acc: 0.8703\n",
      "Epoch 174/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3289 - acc: 0.8723\n",
      "Epoch 175/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3376 - acc: 0.8684\n",
      "Epoch 176/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3220 - acc: 0.8723\n",
      "Epoch 177/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.2974 - acc: 0.8762\n",
      "Epoch 178/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3093 - acc: 0.8802\n",
      "Epoch 179/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3191 - acc: 0.8762\n",
      "Epoch 180/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3023 - acc: 0.8762\n",
      "Epoch 181/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3079 - acc: 0.8723\n",
      "Epoch 182/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.2956 - acc: 0.8802\n",
      "Epoch 183/200\n",
      "509/509 [==============================] - 0s 126us/step - loss: 0.3037 - acc: 0.8743\n",
      "Epoch 184/200\n",
      "509/509 [==============================] - 0s 126us/step - loss: 0.2992 - acc: 0.8743\n",
      "Epoch 185/200\n",
      "509/509 [==============================] - 0s 126us/step - loss: 0.3085 - acc: 0.8762\n",
      "Epoch 186/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3244 - acc: 0.8743\n",
      "Epoch 187/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3185 - acc: 0.8605\n",
      "Epoch 188/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3205 - acc: 0.8900\n",
      "Epoch 189/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3318 - acc: 0.8743\n",
      "Epoch 190/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3586 - acc: 0.8546\n",
      "Epoch 191/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3197 - acc: 0.8703\n",
      "Epoch 192/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3411 - acc: 0.8684\n",
      "Epoch 193/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3255 - acc: 0.8605\n",
      "Epoch 194/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3183 - acc: 0.8762\n",
      "Epoch 195/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.3055 - acc: 0.8743\n",
      "Epoch 196/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3174 - acc: 0.8644\n",
      "Epoch 197/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3147 - acc: 0.8841\n",
      "Epoch 198/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3216 - acc: 0.8841\n",
      "Epoch 199/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3345 - acc: 0.8723\n",
      "Epoch 200/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3067 - acc: 0.8762\n",
      "--------------------------------------------------\n",
      "Detailed classification report for current fold:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.94      0.91        87\n",
      "           1       0.85      0.72      0.78        40\n",
      "\n",
      "    accuracy                           0.87       127\n",
      "   macro avg       0.87      0.83      0.85       127\n",
      "weighted avg       0.87      0.87      0.87       127\n",
      "\n",
      "\n",
      "Area Under ROC (AUC): 0.8847701149425288\n",
      "\n",
      "Confusion Matrix for current fold: \n",
      "[[82  5]\n",
      " [11 29]]\n",
      "\n",
      "Accuracy for Current Fold: 0.8740157480314961\n",
      "\n",
      "43.236363636363635\n",
      "------------------->>>>>>>>>>Fold no =  3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/200\n",
      "509/509 [==============================] - 1s 1ms/step - loss: 0.6581 - acc: 0.6739\n",
      "Epoch 2/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.6272 - acc: 0.6739\n",
      "Epoch 3/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.6131 - acc: 0.6837\n",
      "Epoch 4/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.5976 - acc: 0.6994\n",
      "Epoch 5/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.5781 - acc: 0.7073\n",
      "Epoch 6/200\n",
      "509/509 [==============================] - 0s 147us/step - loss: 0.5708 - acc: 0.7033\n",
      "Epoch 7/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.5586 - acc: 0.7485\n",
      "Epoch 8/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.5557 - acc: 0.7289\n",
      "Epoch 9/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.5281 - acc: 0.7387\n",
      "Epoch 10/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.5228 - acc: 0.7485\n",
      "Epoch 11/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.5179 - acc: 0.7603\n",
      "Epoch 12/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.4982 - acc: 0.7485\n",
      "Epoch 13/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.5046 - acc: 0.7662\n",
      "Epoch 14/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.4889 - acc: 0.7800\n",
      "Epoch 15/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.4837 - acc: 0.7780\n",
      "Epoch 16/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.4770 - acc: 0.7819\n",
      "Epoch 17/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.4682 - acc: 0.7800\n",
      "Epoch 18/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.4488 - acc: 0.7996\n",
      "Epoch 19/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.4297 - acc: 0.8075\n",
      "Epoch 20/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.4380 - acc: 0.8153\n",
      "Epoch 21/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.4353 - acc: 0.8016\n",
      "Epoch 22/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.4324 - acc: 0.7996\n",
      "Epoch 23/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.4150 - acc: 0.8232\n",
      "Epoch 24/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.4167 - acc: 0.7996\n",
      "Epoch 25/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.4077 - acc: 0.8350\n",
      "Epoch 26/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.4045 - acc: 0.8310\n",
      "Epoch 27/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.4038 - acc: 0.8350\n",
      "Epoch 28/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3896 - acc: 0.8448\n",
      "Epoch 29/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3994 - acc: 0.8330\n",
      "Epoch 30/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3917 - acc: 0.8546\n",
      "Epoch 31/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.4000 - acc: 0.8369\n",
      "Epoch 32/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3863 - acc: 0.8527\n",
      "Epoch 33/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3828 - acc: 0.8310\n",
      "Epoch 34/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3844 - acc: 0.8350\n",
      "Epoch 35/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3671 - acc: 0.8389\n",
      "Epoch 36/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3664 - acc: 0.8527\n",
      "Epoch 37/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3716 - acc: 0.8527\n",
      "Epoch 38/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3807 - acc: 0.8350\n",
      "Epoch 39/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3840 - acc: 0.8527\n",
      "Epoch 40/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3660 - acc: 0.8409\n",
      "Epoch 41/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3841 - acc: 0.8585\n",
      "Epoch 42/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3681 - acc: 0.8546\n",
      "Epoch 43/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3755 - acc: 0.8507\n",
      "Epoch 44/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3766 - acc: 0.8389\n",
      "Epoch 45/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3642 - acc: 0.8625\n",
      "Epoch 46/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3754 - acc: 0.8487\n",
      "Epoch 47/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.3687 - acc: 0.8546\n",
      "Epoch 48/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3741 - acc: 0.8566\n",
      "Epoch 49/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.4017 - acc: 0.8487\n",
      "Epoch 50/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3684 - acc: 0.8546\n",
      "Epoch 51/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3799 - acc: 0.8448\n",
      "Epoch 52/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3845 - acc: 0.8507\n",
      "Epoch 53/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3669 - acc: 0.8546\n",
      "Epoch 54/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3628 - acc: 0.8487\n",
      "Epoch 55/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3668 - acc: 0.8605\n",
      "Epoch 56/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3487 - acc: 0.8585\n",
      "Epoch 57/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3624 - acc: 0.8625\n",
      "Epoch 58/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3799 - acc: 0.8546\n",
      "Epoch 59/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3442 - acc: 0.8743\n",
      "Epoch 60/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3725 - acc: 0.8605\n",
      "Epoch 61/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3722 - acc: 0.8527\n",
      "Epoch 62/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3801 - acc: 0.8585\n",
      "Epoch 63/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3535 - acc: 0.8625\n",
      "Epoch 64/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3502 - acc: 0.8585\n",
      "Epoch 65/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3575 - acc: 0.8527\n",
      "Epoch 66/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3649 - acc: 0.8625\n",
      "Epoch 67/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3866 - acc: 0.8546\n",
      "Epoch 68/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3601 - acc: 0.8625\n",
      "Epoch 69/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.3388 - acc: 0.8605\n",
      "Epoch 70/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3819 - acc: 0.8566\n",
      "Epoch 71/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3650 - acc: 0.8743\n",
      "Epoch 72/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3546 - acc: 0.8684\n",
      "Epoch 73/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3560 - acc: 0.8625\n",
      "Epoch 74/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3551 - acc: 0.8448\n",
      "Epoch 75/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3625 - acc: 0.8625\n",
      "Epoch 76/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3628 - acc: 0.8527\n",
      "Epoch 77/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3649 - acc: 0.8703\n",
      "Epoch 78/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3636 - acc: 0.8546\n",
      "Epoch 79/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3706 - acc: 0.8566\n",
      "Epoch 80/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3568 - acc: 0.8664\n",
      "Epoch 81/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3471 - acc: 0.8664\n",
      "Epoch 82/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3700 - acc: 0.8546\n",
      "Epoch 83/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3820 - acc: 0.8585\n",
      "Epoch 84/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3518 - acc: 0.8684\n",
      "Epoch 85/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3568 - acc: 0.8605\n",
      "Epoch 86/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3713 - acc: 0.8507\n",
      "Epoch 87/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3749 - acc: 0.8585\n",
      "Epoch 88/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3480 - acc: 0.8723\n",
      "Epoch 89/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3773 - acc: 0.8527\n",
      "Epoch 90/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3509 - acc: 0.8684\n",
      "Epoch 91/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3662 - acc: 0.8585\n",
      "Epoch 92/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3607 - acc: 0.8625\n",
      "Epoch 93/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3590 - acc: 0.8625\n",
      "Epoch 94/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3439 - acc: 0.8644\n",
      "Epoch 95/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3652 - acc: 0.8585\n",
      "Epoch 96/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3764 - acc: 0.8684\n",
      "Epoch 97/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3478 - acc: 0.8625\n",
      "Epoch 98/200\n",
      "509/509 [==============================] - 0s 147us/step - loss: 0.3514 - acc: 0.8644\n",
      "Epoch 99/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3430 - acc: 0.8546\n",
      "Epoch 100/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3508 - acc: 0.8644\n",
      "Epoch 101/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3680 - acc: 0.8625\n",
      "Epoch 102/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3702 - acc: 0.8585\n",
      "Epoch 103/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3578 - acc: 0.8566\n",
      "Epoch 104/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3588 - acc: 0.8566\n",
      "Epoch 105/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3599 - acc: 0.8566\n",
      "Epoch 106/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3428 - acc: 0.8782\n",
      "Epoch 107/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3491 - acc: 0.8625\n",
      "Epoch 108/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3495 - acc: 0.8625\n",
      "Epoch 109/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3660 - acc: 0.8684\n",
      "Epoch 110/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3551 - acc: 0.8703\n",
      "Epoch 111/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3663 - acc: 0.8605\n",
      "Epoch 112/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3537 - acc: 0.8546\n",
      "Epoch 113/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3482 - acc: 0.8625\n",
      "Epoch 114/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3685 - acc: 0.8585\n",
      "Epoch 115/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3419 - acc: 0.8566\n",
      "Epoch 116/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3609 - acc: 0.8684\n",
      "Epoch 117/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3495 - acc: 0.8644\n",
      "Epoch 118/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3501 - acc: 0.8703\n",
      "Epoch 119/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3525 - acc: 0.8644\n",
      "Epoch 120/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3678 - acc: 0.8703\n",
      "Epoch 121/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3591 - acc: 0.8566\n",
      "Epoch 122/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3499 - acc: 0.8664\n",
      "Epoch 123/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3370 - acc: 0.8625\n",
      "Epoch 124/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3598 - acc: 0.8585\n",
      "Epoch 125/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3636 - acc: 0.8546\n",
      "Epoch 126/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3616 - acc: 0.8546\n",
      "Epoch 127/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3591 - acc: 0.8507\n",
      "Epoch 128/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3586 - acc: 0.8625\n",
      "Epoch 129/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3560 - acc: 0.8684\n",
      "Epoch 130/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3611 - acc: 0.8664\n",
      "Epoch 131/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3546 - acc: 0.8527\n",
      "Epoch 132/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3520 - acc: 0.8585\n",
      "Epoch 133/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3359 - acc: 0.8605\n",
      "Epoch 134/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3644 - acc: 0.8409\n",
      "Epoch 135/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3569 - acc: 0.8625\n",
      "Epoch 136/200\n",
      "509/509 [==============================] - 0s 151us/step - loss: 0.3469 - acc: 0.8684\n",
      "Epoch 137/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3423 - acc: 0.8605\n",
      "Epoch 138/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3603 - acc: 0.8566\n",
      "Epoch 139/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3532 - acc: 0.8644\n",
      "Epoch 140/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3477 - acc: 0.8585\n",
      "Epoch 141/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3588 - acc: 0.8546\n",
      "Epoch 142/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3347 - acc: 0.8723\n",
      "Epoch 143/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3682 - acc: 0.8585\n",
      "Epoch 144/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3543 - acc: 0.8684\n",
      "Epoch 145/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3576 - acc: 0.8546\n",
      "Epoch 146/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3533 - acc: 0.8566\n",
      "Epoch 147/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3644 - acc: 0.8468\n",
      "Epoch 148/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3551 - acc: 0.8664\n",
      "Epoch 149/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3495 - acc: 0.8703\n",
      "Epoch 150/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3454 - acc: 0.8684\n",
      "Epoch 151/200\n",
      "509/509 [==============================] - 0s 147us/step - loss: 0.3522 - acc: 0.8566\n",
      "Epoch 152/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3452 - acc: 0.8703\n",
      "Epoch 153/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3570 - acc: 0.8585\n",
      "Epoch 154/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3610 - acc: 0.8625\n",
      "Epoch 155/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3412 - acc: 0.8605\n",
      "Epoch 156/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3516 - acc: 0.8644\n",
      "Epoch 157/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3565 - acc: 0.8566\n",
      "Epoch 158/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3323 - acc: 0.8546\n",
      "Epoch 159/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3240 - acc: 0.8684\n",
      "Epoch 160/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3340 - acc: 0.8625\n",
      "Epoch 161/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3328 - acc: 0.8625\n",
      "Epoch 162/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3474 - acc: 0.8605\n",
      "Epoch 163/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3527 - acc: 0.8684\n",
      "Epoch 164/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3411 - acc: 0.8684\n",
      "Epoch 165/200\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "509/509 [==============================] - 0s 134us/step - loss: 0.3910 - acc: 0.8487\n",
      "Epoch 166/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3486 - acc: 0.8664\n",
      "Epoch 167/200\n",
      "509/509 [==============================] - 0s 149us/step - loss: 0.3528 - acc: 0.8723\n",
      "Epoch 168/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3317 - acc: 0.8566\n",
      "Epoch 169/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3457 - acc: 0.8625\n",
      "Epoch 170/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3434 - acc: 0.8644\n",
      "Epoch 171/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3514 - acc: 0.8664\n",
      "Epoch 172/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3686 - acc: 0.8546\n",
      "Epoch 173/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3616 - acc: 0.8409\n",
      "Epoch 174/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3521 - acc: 0.8605\n",
      "Epoch 175/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3441 - acc: 0.8625\n",
      "Epoch 176/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3402 - acc: 0.8762\n",
      "Epoch 177/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3435 - acc: 0.8664\n",
      "Epoch 178/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3306 - acc: 0.8743\n",
      "Epoch 179/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3361 - acc: 0.8743\n",
      "Epoch 180/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3399 - acc: 0.8605\n",
      "Epoch 181/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3440 - acc: 0.8585\n",
      "Epoch 182/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3330 - acc: 0.8841\n",
      "Epoch 183/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3447 - acc: 0.8566\n",
      "Epoch 184/200\n",
      "509/509 [==============================] - 0s 126us/step - loss: 0.3310 - acc: 0.8743\n",
      "Epoch 185/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3363 - acc: 0.8625\n",
      "Epoch 186/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3609 - acc: 0.8644\n",
      "Epoch 187/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3462 - acc: 0.8625\n",
      "Epoch 188/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3472 - acc: 0.8743\n",
      "Epoch 189/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3347 - acc: 0.8625\n",
      "Epoch 190/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3290 - acc: 0.8644\n",
      "Epoch 191/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3437 - acc: 0.8644\n",
      "Epoch 192/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3343 - acc: 0.8566\n",
      "Epoch 193/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3305 - acc: 0.8703\n",
      "Epoch 194/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3237 - acc: 0.8644\n",
      "Epoch 195/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3502 - acc: 0.8684\n",
      "Epoch 196/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3576 - acc: 0.8684\n",
      "Epoch 197/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3402 - acc: 0.8762\n",
      "Epoch 198/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3241 - acc: 0.8703\n",
      "Epoch 199/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3425 - acc: 0.8762\n",
      "Epoch 200/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3513 - acc: 0.8664\n",
      "--------------------------------------------------\n",
      "Detailed classification report for current fold:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.93      0.91        87\n",
      "           1       0.83      0.72      0.77        40\n",
      "\n",
      "    accuracy                           0.87       127\n",
      "   macro avg       0.85      0.83      0.84       127\n",
      "weighted avg       0.86      0.87      0.86       127\n",
      "\n",
      "\n",
      "Area Under ROC (AUC): 0.928735632183908\n",
      "\n",
      "Confusion Matrix for current fold: \n",
      "[[81  6]\n",
      " [11 29]]\n",
      "\n",
      "Accuracy for Current Fold: 0.8661417322834646\n",
      "\n",
      "35.59090909090909\n",
      "------------------->>>>>>>>>>Fold no =  4\n",
      "Epoch 1/200\n",
      "509/509 [==============================] - 1s 1ms/step - loss: 0.6651 - acc: 0.6758\n",
      "Epoch 2/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.6301 - acc: 0.6778\n",
      "Epoch 3/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.6197 - acc: 0.7407\n",
      "Epoch 4/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.6011 - acc: 0.7112\n",
      "Epoch 5/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.5747 - acc: 0.7367\n",
      "Epoch 6/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.5723 - acc: 0.7230\n",
      "Epoch 7/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.5519 - acc: 0.7446\n",
      "Epoch 8/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.5491 - acc: 0.7525\n",
      "Epoch 9/200\n",
      "509/509 [==============================] - 0s 147us/step - loss: 0.5276 - acc: 0.7741\n",
      "Epoch 10/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.5186 - acc: 0.7642\n",
      "Epoch 11/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.5076 - acc: 0.7878\n",
      "Epoch 12/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.4959 - acc: 0.8035\n",
      "Epoch 13/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.5097 - acc: 0.7760\n",
      "Epoch 14/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.4741 - acc: 0.8173\n",
      "Epoch 15/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.4791 - acc: 0.7996\n",
      "Epoch 16/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.4602 - acc: 0.8173\n",
      "Epoch 17/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.4564 - acc: 0.8114\n",
      "Epoch 18/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.4406 - acc: 0.8232\n",
      "Epoch 19/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.4338 - acc: 0.8212\n",
      "Epoch 20/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.4380 - acc: 0.8330\n",
      "Epoch 21/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.4473 - acc: 0.8212\n",
      "Epoch 22/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.4262 - acc: 0.8291\n",
      "Epoch 23/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.4230 - acc: 0.8350\n",
      "Epoch 24/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3978 - acc: 0.8566\n",
      "Epoch 25/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.4232 - acc: 0.8369\n",
      "Epoch 26/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.4064 - acc: 0.8487\n",
      "Epoch 27/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.4209 - acc: 0.8409\n",
      "Epoch 28/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3930 - acc: 0.8566\n",
      "Epoch 29/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3990 - acc: 0.8527\n",
      "Epoch 30/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3866 - acc: 0.8546\n",
      "Epoch 31/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3890 - acc: 0.8605\n",
      "Epoch 32/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.3897 - acc: 0.8527\n",
      "Epoch 33/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3878 - acc: 0.8605\n",
      "Epoch 34/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3778 - acc: 0.8507\n",
      "Epoch 35/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3820 - acc: 0.8605\n",
      "Epoch 36/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3944 - acc: 0.8330\n",
      "Epoch 37/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3694 - acc: 0.8625\n",
      "Epoch 38/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3917 - acc: 0.8546\n",
      "Epoch 39/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3714 - acc: 0.8644\n",
      "Epoch 40/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3524 - acc: 0.8605\n",
      "Epoch 41/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3787 - acc: 0.8546\n",
      "Epoch 42/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3564 - acc: 0.8723\n",
      "Epoch 43/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3711 - acc: 0.8644\n",
      "Epoch 44/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3758 - acc: 0.8546\n",
      "Epoch 45/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3788 - acc: 0.8527\n",
      "Epoch 46/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3626 - acc: 0.8546\n",
      "Epoch 47/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3691 - acc: 0.8625\n",
      "Epoch 48/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3658 - acc: 0.8605\n",
      "Epoch 49/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3761 - acc: 0.8625\n",
      "Epoch 50/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3650 - acc: 0.8585\n",
      "Epoch 51/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3588 - acc: 0.8703\n",
      "Epoch 52/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3658 - acc: 0.8644\n",
      "Epoch 53/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3550 - acc: 0.8527\n",
      "Epoch 54/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3453 - acc: 0.8664\n",
      "Epoch 55/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3514 - acc: 0.8743\n",
      "Epoch 56/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3290 - acc: 0.8625\n",
      "Epoch 57/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3525 - acc: 0.8644\n",
      "Epoch 58/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3653 - acc: 0.8664\n",
      "Epoch 59/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3482 - acc: 0.8605\n",
      "Epoch 60/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3594 - acc: 0.8664\n",
      "Epoch 61/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3516 - acc: 0.8585\n",
      "Epoch 62/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3596 - acc: 0.8644\n",
      "Epoch 63/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3409 - acc: 0.8664\n",
      "Epoch 64/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3500 - acc: 0.8644\n",
      "Epoch 65/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3588 - acc: 0.8585\n",
      "Epoch 66/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3530 - acc: 0.8644\n",
      "Epoch 67/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3518 - acc: 0.8664\n",
      "Epoch 68/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3649 - acc: 0.8703\n",
      "Epoch 69/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3406 - acc: 0.8644\n",
      "Epoch 70/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3684 - acc: 0.8703\n",
      "Epoch 71/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3471 - acc: 0.8644\n",
      "Epoch 72/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3311 - acc: 0.8703\n",
      "Epoch 73/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3375 - acc: 0.8782\n",
      "Epoch 74/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3462 - acc: 0.8644\n",
      "Epoch 75/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3576 - acc: 0.8546\n",
      "Epoch 76/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3374 - acc: 0.8703\n",
      "Epoch 77/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3322 - acc: 0.8664\n",
      "Epoch 78/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3499 - acc: 0.8703\n",
      "Epoch 79/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.3429 - acc: 0.8684\n",
      "Epoch 80/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3403 - acc: 0.8762\n",
      "Epoch 81/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3278 - acc: 0.8703\n",
      "Epoch 82/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3453 - acc: 0.8585\n",
      "Epoch 83/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3526 - acc: 0.8585\n",
      "Epoch 84/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3465 - acc: 0.8723\n",
      "Epoch 85/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3325 - acc: 0.8743\n",
      "Epoch 86/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3462 - acc: 0.8664\n",
      "Epoch 87/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3493 - acc: 0.8644\n",
      "Epoch 88/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3375 - acc: 0.8664\n",
      "Epoch 89/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3435 - acc: 0.8664\n",
      "Epoch 90/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3240 - acc: 0.8723\n",
      "Epoch 91/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3515 - acc: 0.8703\n",
      "Epoch 92/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3407 - acc: 0.8684\n",
      "Epoch 93/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3217 - acc: 0.8762\n",
      "Epoch 94/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3518 - acc: 0.8664\n",
      "Epoch 95/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3451 - acc: 0.8625\n",
      "Epoch 96/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3435 - acc: 0.8605\n",
      "Epoch 97/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3359 - acc: 0.8664\n",
      "Epoch 98/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3164 - acc: 0.8703\n",
      "Epoch 99/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3227 - acc: 0.8644\n",
      "Epoch 100/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3406 - acc: 0.8566\n",
      "Epoch 101/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3291 - acc: 0.8703\n",
      "Epoch 102/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3269 - acc: 0.8644\n",
      "Epoch 103/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3349 - acc: 0.8625\n",
      "Epoch 104/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3263 - acc: 0.8625\n",
      "Epoch 105/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3348 - acc: 0.8703\n",
      "Epoch 106/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3298 - acc: 0.8625\n",
      "Epoch 107/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3352 - acc: 0.8625\n",
      "Epoch 108/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3302 - acc: 0.8625\n",
      "Epoch 109/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3322 - acc: 0.8703\n",
      "Epoch 110/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3384 - acc: 0.8644\n",
      "Epoch 111/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3353 - acc: 0.8566\n",
      "Epoch 112/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3143 - acc: 0.8605\n",
      "Epoch 113/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3339 - acc: 0.8664\n",
      "Epoch 114/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3308 - acc: 0.8644\n",
      "Epoch 115/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3427 - acc: 0.8684\n",
      "Epoch 116/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3461 - acc: 0.8644\n",
      "Epoch 117/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3463 - acc: 0.8684\n",
      "Epoch 118/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3311 - acc: 0.8585\n",
      "Epoch 119/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3209 - acc: 0.8723\n",
      "Epoch 120/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3371 - acc: 0.8625\n",
      "Epoch 121/200\n",
      "509/509 [==============================] - 0s 147us/step - loss: 0.3345 - acc: 0.8585\n",
      "Epoch 122/200\n",
      "509/509 [==============================] - 0s 161us/step - loss: 0.3455 - acc: 0.8566\n",
      "Epoch 123/200\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "509/509 [==============================] - 0s 140us/step - loss: 0.3369 - acc: 0.8625\n",
      "Epoch 124/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3273 - acc: 0.8566\n",
      "Epoch 125/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3542 - acc: 0.8585\n",
      "Epoch 126/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3290 - acc: 0.8664\n",
      "Epoch 127/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3340 - acc: 0.8546\n",
      "Epoch 128/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3283 - acc: 0.8703\n",
      "Epoch 129/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3266 - acc: 0.8684\n",
      "Epoch 130/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3190 - acc: 0.8723\n",
      "Epoch 131/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3306 - acc: 0.8605\n",
      "Epoch 132/200\n",
      "509/509 [==============================] - 0s 157us/step - loss: 0.3670 - acc: 0.8527\n",
      "Epoch 133/200\n",
      "509/509 [==============================] - 0s 149us/step - loss: 0.3182 - acc: 0.8664\n",
      "Epoch 134/200\n",
      "509/509 [==============================] - 0s 147us/step - loss: 0.3433 - acc: 0.8703\n",
      "Epoch 135/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3274 - acc: 0.8644\n",
      "Epoch 136/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3206 - acc: 0.8664\n",
      "Epoch 137/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3292 - acc: 0.8644\n",
      "Epoch 138/200\n",
      "509/509 [==============================] - 0s 147us/step - loss: 0.3367 - acc: 0.8644\n",
      "Epoch 139/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3253 - acc: 0.8664\n",
      "Epoch 140/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3240 - acc: 0.8644\n",
      "Epoch 141/200\n",
      "509/509 [==============================] - 0s 147us/step - loss: 0.3292 - acc: 0.8625\n",
      "Epoch 142/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3163 - acc: 0.8625\n",
      "Epoch 143/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3284 - acc: 0.8644\n",
      "Epoch 144/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3228 - acc: 0.8684\n",
      "Epoch 145/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3285 - acc: 0.8605\n",
      "Epoch 146/200\n",
      "509/509 [==============================] - 0s 149us/step - loss: 0.3388 - acc: 0.8546\n",
      "Epoch 147/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3369 - acc: 0.8723\n",
      "Epoch 148/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3318 - acc: 0.8605\n",
      "Epoch 149/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3403 - acc: 0.8585\n",
      "Epoch 150/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3257 - acc: 0.8684\n",
      "Epoch 151/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3352 - acc: 0.8684\n",
      "Epoch 152/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3118 - acc: 0.8723\n",
      "Epoch 153/200\n",
      "509/509 [==============================] - 0s 155us/step - loss: 0.3338 - acc: 0.8585\n",
      "Epoch 154/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3165 - acc: 0.8684\n",
      "Epoch 155/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3322 - acc: 0.8605\n",
      "Epoch 156/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3116 - acc: 0.8703\n",
      "Epoch 157/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3223 - acc: 0.8664\n",
      "Epoch 158/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3097 - acc: 0.8684\n",
      "Epoch 159/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3075 - acc: 0.8743\n",
      "Epoch 160/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3204 - acc: 0.8625\n",
      "Epoch 161/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3077 - acc: 0.8703\n",
      "Epoch 162/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3085 - acc: 0.8723\n",
      "Epoch 163/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3426 - acc: 0.8625\n",
      "Epoch 164/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3253 - acc: 0.8625\n",
      "Epoch 165/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3250 - acc: 0.8684\n",
      "Epoch 166/200\n",
      "509/509 [==============================] - 0s 126us/step - loss: 0.3191 - acc: 0.8743\n",
      "Epoch 167/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3103 - acc: 0.8723\n",
      "Epoch 168/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3169 - acc: 0.8527\n",
      "Epoch 169/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3117 - acc: 0.8762\n",
      "Epoch 170/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.3194 - acc: 0.8684\n",
      "Epoch 171/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3263 - acc: 0.8605\n",
      "Epoch 172/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.3223 - acc: 0.8684\n",
      "Epoch 173/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3417 - acc: 0.8605\n",
      "Epoch 174/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3182 - acc: 0.8585\n",
      "Epoch 175/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3200 - acc: 0.8703\n",
      "Epoch 176/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3187 - acc: 0.8703\n",
      "Epoch 177/200\n",
      "509/509 [==============================] - 0s 128us/step - loss: 0.3191 - acc: 0.8743\n",
      "Epoch 178/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3065 - acc: 0.8703\n",
      "Epoch 179/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3255 - acc: 0.8585\n",
      "Epoch 180/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3294 - acc: 0.8605\n",
      "Epoch 181/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3252 - acc: 0.8644\n",
      "Epoch 182/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3326 - acc: 0.8684\n",
      "Epoch 183/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3095 - acc: 0.8684\n",
      "Epoch 184/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.2934 - acc: 0.8762\n",
      "Epoch 185/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3129 - acc: 0.8743\n",
      "Epoch 186/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3166 - acc: 0.8723\n",
      "Epoch 187/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3148 - acc: 0.8703\n",
      "Epoch 188/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3086 - acc: 0.8762\n",
      "Epoch 189/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3053 - acc: 0.8664\n",
      "Epoch 190/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3057 - acc: 0.8802\n",
      "Epoch 191/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3260 - acc: 0.8644\n",
      "Epoch 192/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3262 - acc: 0.8664\n",
      "Epoch 193/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3230 - acc: 0.8625\n",
      "Epoch 194/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3184 - acc: 0.8527\n",
      "Epoch 195/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3209 - acc: 0.8644\n",
      "Epoch 196/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3359 - acc: 0.8605\n",
      "Epoch 197/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3124 - acc: 0.8723\n",
      "Epoch 198/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3229 - acc: 0.8684\n",
      "Epoch 199/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3339 - acc: 0.8585\n",
      "Epoch 200/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3130 - acc: 0.8841\n",
      "--------------------------------------------------\n",
      "Detailed classification report for current fold:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      0.92      0.91        88\n",
      "           1       0.81      0.74      0.77        39\n",
      "\n",
      "    accuracy                           0.87       127\n",
      "   macro avg       0.85      0.83      0.84       127\n",
      "weighted avg       0.86      0.87      0.86       127\n",
      "\n",
      "\n",
      "Area Under ROC (AUC): 0.9192890442890442\n",
      "\n",
      "Confusion Matrix for current fold: \n",
      "[[81  7]\n",
      " [10 29]]\n",
      "\n",
      "Accuracy for Current Fold: 0.8661417322834646\n",
      "\n",
      "33.55714285714286\n",
      "------------------->>>>>>>>>>Fold no =  5\n",
      "Epoch 1/200\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "509/509 [==============================] - 1s 1ms/step - loss: 0.6637 - acc: 0.6739\n",
      "Epoch 2/200\n",
      "509/509 [==============================] - 0s 147us/step - loss: 0.6347 - acc: 0.6896\n",
      "Epoch 3/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.6208 - acc: 0.7132\n",
      "Epoch 4/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.5856 - acc: 0.7289\n",
      "Epoch 5/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.5707 - acc: 0.6916\n",
      "Epoch 6/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.5516 - acc: 0.7485\n",
      "Epoch 7/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.5503 - acc: 0.7485\n",
      "Epoch 8/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.5369 - acc: 0.7564\n",
      "Epoch 9/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.5062 - acc: 0.7642\n",
      "Epoch 10/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.5088 - acc: 0.7623\n",
      "Epoch 11/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.4952 - acc: 0.7800\n",
      "Epoch 12/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.4990 - acc: 0.7701\n",
      "Epoch 13/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.4772 - acc: 0.8114\n",
      "Epoch 14/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.4603 - acc: 0.8035\n",
      "Epoch 15/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.4702 - acc: 0.8134\n",
      "Epoch 16/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.4430 - acc: 0.8330\n",
      "Epoch 17/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.4388 - acc: 0.8232\n",
      "Epoch 18/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.4138 - acc: 0.8330\n",
      "Epoch 19/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.4327 - acc: 0.8271\n",
      "Epoch 20/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3947 - acc: 0.8546\n",
      "Epoch 21/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3977 - acc: 0.8585\n",
      "Epoch 22/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.4123 - acc: 0.8330\n",
      "Epoch 23/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3987 - acc: 0.8468\n",
      "Epoch 24/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3953 - acc: 0.8507\n",
      "Epoch 25/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3976 - acc: 0.8546\n",
      "Epoch 26/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3861 - acc: 0.8644\n",
      "Epoch 27/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3799 - acc: 0.8664\n",
      "Epoch 28/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3869 - acc: 0.8566\n",
      "Epoch 29/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3791 - acc: 0.8605\n",
      "Epoch 30/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3927 - acc: 0.8605\n",
      "Epoch 31/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3934 - acc: 0.8684\n",
      "Epoch 32/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3863 - acc: 0.8605\n",
      "Epoch 33/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3723 - acc: 0.8703\n",
      "Epoch 34/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3695 - acc: 0.8585\n",
      "Epoch 35/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3731 - acc: 0.8625\n",
      "Epoch 36/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3628 - acc: 0.8684\n",
      "Epoch 37/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3581 - acc: 0.8644\n",
      "Epoch 38/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3676 - acc: 0.8527\n",
      "Epoch 39/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3518 - acc: 0.8723\n",
      "Epoch 40/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3587 - acc: 0.8684\n",
      "Epoch 41/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3658 - acc: 0.8684\n",
      "Epoch 42/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3605 - acc: 0.8644\n",
      "Epoch 43/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3719 - acc: 0.8507\n",
      "Epoch 44/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3573 - acc: 0.8664\n",
      "Epoch 45/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3629 - acc: 0.8664\n",
      "Epoch 46/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3441 - acc: 0.8762\n",
      "Epoch 47/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3606 - acc: 0.8546\n",
      "Epoch 48/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3596 - acc: 0.8644\n",
      "Epoch 49/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3561 - acc: 0.8703\n",
      "Epoch 50/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3483 - acc: 0.8684\n",
      "Epoch 51/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3600 - acc: 0.8625\n",
      "Epoch 52/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3382 - acc: 0.8802\n",
      "Epoch 53/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3445 - acc: 0.8605\n",
      "Epoch 54/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3468 - acc: 0.8723\n",
      "Epoch 55/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3407 - acc: 0.8703\n",
      "Epoch 56/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3589 - acc: 0.8684\n",
      "Epoch 57/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3742 - acc: 0.8605\n",
      "Epoch 58/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3417 - acc: 0.8703\n",
      "Epoch 59/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3349 - acc: 0.8684\n",
      "Epoch 60/200\n",
      "509/509 [==============================] - 0s 147us/step - loss: 0.3638 - acc: 0.8664\n",
      "Epoch 61/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3474 - acc: 0.8743\n",
      "Epoch 62/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3472 - acc: 0.8723\n",
      "Epoch 63/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3323 - acc: 0.8782\n",
      "Epoch 64/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3410 - acc: 0.8802\n",
      "Epoch 65/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3382 - acc: 0.8625\n",
      "Epoch 66/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3371 - acc: 0.8625\n",
      "Epoch 67/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3426 - acc: 0.8723\n",
      "Epoch 68/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3452 - acc: 0.8684\n",
      "Epoch 69/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3362 - acc: 0.8684\n",
      "Epoch 70/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3474 - acc: 0.8703\n",
      "Epoch 71/200\n",
      "509/509 [==============================] - 0s 146us/step - loss: 0.3305 - acc: 0.8723\n",
      "Epoch 72/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3472 - acc: 0.8605\n",
      "Epoch 73/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3376 - acc: 0.8782\n",
      "Epoch 74/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3480 - acc: 0.8802\n",
      "Epoch 75/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3426 - acc: 0.8703\n",
      "Epoch 76/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3387 - acc: 0.8605\n",
      "Epoch 77/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3439 - acc: 0.8743\n",
      "Epoch 78/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3443 - acc: 0.8703\n",
      "Epoch 79/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3408 - acc: 0.8703\n",
      "Epoch 80/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3382 - acc: 0.8703\n",
      "Epoch 81/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3345 - acc: 0.8743\n",
      "Epoch 82/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3262 - acc: 0.8723\n",
      "Epoch 83/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3340 - acc: 0.8585\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 84/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3250 - acc: 0.8782\n",
      "Epoch 85/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3356 - acc: 0.8743\n",
      "Epoch 86/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3451 - acc: 0.8703\n",
      "Epoch 87/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3298 - acc: 0.8861\n",
      "Epoch 88/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3390 - acc: 0.8605\n",
      "Epoch 89/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3430 - acc: 0.8703\n",
      "Epoch 90/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3204 - acc: 0.8841\n",
      "Epoch 91/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3284 - acc: 0.8703\n",
      "Epoch 92/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3307 - acc: 0.8762\n",
      "Epoch 93/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3220 - acc: 0.8723\n",
      "Epoch 94/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3244 - acc: 0.8861\n",
      "Epoch 95/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3230 - acc: 0.8644\n",
      "Epoch 96/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3486 - acc: 0.8723\n",
      "Epoch 97/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3248 - acc: 0.8821\n",
      "Epoch 98/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3219 - acc: 0.8821\n",
      "Epoch 99/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3140 - acc: 0.8743\n",
      "Epoch 100/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3384 - acc: 0.8762\n",
      "Epoch 101/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3233 - acc: 0.8782\n",
      "Epoch 102/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3249 - acc: 0.8723\n",
      "Epoch 103/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3316 - acc: 0.8762\n",
      "Epoch 104/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3138 - acc: 0.8723\n",
      "Epoch 105/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3252 - acc: 0.8782\n",
      "Epoch 106/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3218 - acc: 0.8743\n",
      "Epoch 107/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3240 - acc: 0.8782\n",
      "Epoch 108/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3226 - acc: 0.8762\n",
      "Epoch 109/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3273 - acc: 0.8684\n",
      "Epoch 110/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3317 - acc: 0.8743\n",
      "Epoch 111/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3350 - acc: 0.8743\n",
      "Epoch 112/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3134 - acc: 0.8664\n",
      "Epoch 113/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3256 - acc: 0.8743\n",
      "Epoch 114/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3181 - acc: 0.8782\n",
      "Epoch 115/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3216 - acc: 0.8684\n",
      "Epoch 116/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3294 - acc: 0.8782\n",
      "Epoch 117/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3348 - acc: 0.8644\n",
      "Epoch 118/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3261 - acc: 0.8743\n",
      "Epoch 119/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3110 - acc: 0.8821\n",
      "Epoch 120/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3263 - acc: 0.8802\n",
      "Epoch 121/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3184 - acc: 0.8802\n",
      "Epoch 122/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3136 - acc: 0.8664\n",
      "Epoch 123/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3180 - acc: 0.8841\n",
      "Epoch 124/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3088 - acc: 0.8762\n",
      "Epoch 125/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3205 - acc: 0.8762\n",
      "Epoch 126/200\n",
      "509/509 [==============================] - 0s 130us/step - loss: 0.3109 - acc: 0.8821\n",
      "Epoch 127/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3438 - acc: 0.8723\n",
      "Epoch 128/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3167 - acc: 0.8841\n",
      "Epoch 129/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3029 - acc: 0.8821\n",
      "Epoch 130/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3395 - acc: 0.8664\n",
      "Epoch 131/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.3488 - acc: 0.8605\n",
      "Epoch 132/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3209 - acc: 0.8743\n",
      "Epoch 133/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3313 - acc: 0.8605\n",
      "Epoch 134/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3340 - acc: 0.8782\n",
      "Epoch 135/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3256 - acc: 0.8703\n",
      "Epoch 136/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3196 - acc: 0.8762\n",
      "Epoch 137/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3131 - acc: 0.8664\n",
      "Epoch 138/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3179 - acc: 0.8802\n",
      "Epoch 139/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3164 - acc: 0.8841\n",
      "Epoch 140/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3057 - acc: 0.8900\n",
      "Epoch 141/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3112 - acc: 0.8723\n",
      "Epoch 142/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.2948 - acc: 0.8802\n",
      "Epoch 143/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3084 - acc: 0.8802\n",
      "Epoch 144/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3197 - acc: 0.8762\n",
      "Epoch 145/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3034 - acc: 0.8841\n",
      "Epoch 146/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3170 - acc: 0.8703\n",
      "Epoch 147/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3092 - acc: 0.8762\n",
      "Epoch 148/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.3136 - acc: 0.8703\n",
      "Epoch 149/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3129 - acc: 0.8821\n",
      "Epoch 150/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.2972 - acc: 0.8841\n",
      "Epoch 151/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3038 - acc: 0.8880\n",
      "Epoch 152/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.2919 - acc: 0.8782\n",
      "Epoch 153/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3013 - acc: 0.8743\n",
      "Epoch 154/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.3140 - acc: 0.8664\n",
      "Epoch 155/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3024 - acc: 0.8841\n",
      "Epoch 156/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3086 - acc: 0.8762\n",
      "Epoch 157/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.3152 - acc: 0.8821\n",
      "Epoch 158/200\n",
      "509/509 [==============================] - 0s 147us/step - loss: 0.2994 - acc: 0.8802\n",
      "Epoch 159/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.2980 - acc: 0.8762\n",
      "Epoch 160/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.2858 - acc: 0.8900\n",
      "Epoch 161/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3040 - acc: 0.8782\n",
      "Epoch 162/200\n",
      "509/509 [==============================] - 0s 134us/step - loss: 0.2979 - acc: 0.8861\n",
      "Epoch 163/200\n",
      "509/509 [==============================] - 0s 132us/step - loss: 0.2993 - acc: 0.8684\n",
      "Epoch 164/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3061 - acc: 0.8743\n",
      "Epoch 165/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3235 - acc: 0.8821\n",
      "Epoch 166/200\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "509/509 [==============================] - 0s 142us/step - loss: 0.2983 - acc: 0.8782\n",
      "Epoch 167/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.2998 - acc: 0.8762\n",
      "Epoch 168/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3017 - acc: 0.8782\n",
      "Epoch 169/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.2846 - acc: 0.8861\n",
      "Epoch 170/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.2870 - acc: 0.8782\n",
      "Epoch 171/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.2942 - acc: 0.8841\n",
      "Epoch 172/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.3034 - acc: 0.8782\n",
      "Epoch 173/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.2935 - acc: 0.8900\n",
      "Epoch 174/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.2992 - acc: 0.8861\n",
      "Epoch 175/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3157 - acc: 0.8782\n",
      "Epoch 176/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.2899 - acc: 0.8821\n",
      "Epoch 177/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.2965 - acc: 0.8782\n",
      "Epoch 178/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.2955 - acc: 0.8782\n",
      "Epoch 179/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.2892 - acc: 0.8841\n",
      "Epoch 180/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.3161 - acc: 0.8821\n",
      "Epoch 181/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.2917 - acc: 0.8861\n",
      "Epoch 182/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.2905 - acc: 0.8821\n",
      "Epoch 183/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.2923 - acc: 0.8861\n",
      "Epoch 184/200\n",
      "509/509 [==============================] - 0s 138us/step - loss: 0.2802 - acc: 0.8959\n",
      "Epoch 185/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.3164 - acc: 0.8880\n",
      "Epoch 186/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.2924 - acc: 0.8900\n",
      "Epoch 187/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.2828 - acc: 0.8743\n",
      "Epoch 188/200\n",
      "509/509 [==============================] - 0s 144us/step - loss: 0.2681 - acc: 0.8880\n",
      "Epoch 189/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.2900 - acc: 0.8861\n",
      "Epoch 190/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.2828 - acc: 0.8841\n",
      "Epoch 191/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.2988 - acc: 0.8664\n",
      "Epoch 192/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.2766 - acc: 0.8841\n",
      "Epoch 193/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.2970 - acc: 0.8821\n",
      "Epoch 194/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.2915 - acc: 0.8802\n",
      "Epoch 195/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.2932 - acc: 0.8821\n",
      "Epoch 196/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.2907 - acc: 0.8900\n",
      "Epoch 197/200\n",
      "509/509 [==============================] - 0s 140us/step - loss: 0.2883 - acc: 0.8802\n",
      "Epoch 198/200\n",
      "509/509 [==============================] - 0s 142us/step - loss: 0.2969 - acc: 0.8821\n",
      "Epoch 199/200\n",
      "509/509 [==============================] - 0s 136us/step - loss: 0.2985 - acc: 0.8821\n",
      "Epoch 200/200\n",
      "509/509 [==============================] - 0s 147us/step - loss: 0.2814 - acc: 0.8880\n",
      "--------------------------------------------------\n",
      "Detailed classification report for current fold:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.89      0.88        88\n",
      "           1       0.74      0.72      0.73        39\n",
      "\n",
      "    accuracy                           0.83       127\n",
      "   macro avg       0.81      0.80      0.80       127\n",
      "weighted avg       0.83      0.83      0.83       127\n",
      "\n",
      "\n",
      "Area Under ROC (AUC): 0.8997668997668997\n",
      "\n",
      "Confusion Matrix for current fold: \n",
      "[[78 10]\n",
      " [11 28]]\n",
      "\n",
      "Accuracy for Current Fold: 0.8346456692913385\n",
      "\n",
      "19.854545454545455\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOy9d3hc1bm3fa9pGo0kS5ZkuUg27tjGRKLahGYCBkMMJsBJDCQHEt5jIJAETshHKEn4SEJIIZRznOIkHBIOYHghlBDTS0hoNkUGF9wNlsuot9G0vffz/rEleaQZaUbSjDS2131dvjyz99pr/2Y8nmfWepoSETQajUaj6QvHSAvQaDQaTXajDYVGo9Fo+kUbCo1Go9H0izYUGo1Go+kXbSg0Go1G0y/aUGg0Go2mXzJmKJRS9yulapVS6/o4r5RS9ymltiqlPlJKHZ0pLRqNRqMZPJlcUTwALOrn/NnAjM4/y4DfZlCLRqPRaAZJxgyFiLwBNPYzZAnwF7F5ByhSSo3PlB6NRqPRDA7XCN67HNgV87ym89je3gOVUsuwVx34fL5jpk+fPiwC+8K1bRsAxrRpI6pDRFBKjaiGbNFxIGtoN9oZaIUECdu/8QQBXVwhDkc4MiLvS8Rw0RFyI5Lez2LUcBE1nEObxLGpIxwO5w3m0pE0FIneyYT/tCKyAlgBUFlZKWvXrs2kruQsWEAkEsHz1lsjKsPv9zN27NgR1ZAtOg5kDau2r+KcqecM6JqP/vkpnjFBWlpa4oyT2+0mGo0OWEc6GWkNzrfexjrx87jdbj791MH69XlEIpnbaa+rc/PBBwW0tLhQEYsCT+y9FOmwWrnSxMwq4ZhjopSU5DHQ3yTf/OaUXclHJWYkDUUNMDHmeQWwZ4S0aDQjSigUorW1tccxl8uF2+3G7XYjIkQiEcLhMM3NzdTU1FGWm4PP50u4inG73cMlvU9GSoMIfNZSxttPl7N6dR67d3uG7d5lZVFmTWvi+BnpNUqlpVHm5/8VZ+UXmTFjBi7XwL+6v/71YPtg7z+ShuIZ4Fql1EpgHtAiInHbTlnJrbcSaGpi+D5+moMVEaG1tZWPGz7Gsqwe55RSPYxA1/aUw+Egx+MhFMpjyxb9KezCNGH9+lzefTcf/7YyyC8AID/f5KijOhg92sjYvXNzhaqqAFOmRHh9exvzJ+z/anW5XBjG0O5tWRbFzR7GzpyJ0znELahBkDFDoZR6BFgAlCqlaoAfAW4AEfkdsAo4B9gKdABfz5SWtHPGGUT8/pFWoRlmwuFw3Jd5LIZhEAwGU57PsixqamqoratldtnslL4AGhudrF6dx+sv5LJzb17a98IPFgrzWjh+ocW8eQFmzw4yiB/gA8YwDDo6wlhiMWbMmO7j4XCYnJycIc3tdrsZ5/0UxwgYCcigoRCRi5OcF+CaTN0/o1RX42pogCzwD2gyi4gQCATYs2cPzc3N/TqrXS4Xe/akvnva5fzO8eT0ayT8fherV+exenU+W7bkIKIwwhbePJg2LYTb3bfxOtQoL48yf347s+tfRp1yEmC/z8FgaMABAwPF4/EwdepU9rhDTJ68f1c9bf6zxpHLjx7JracDl+uuoyASgdNPH2klBxwvffoSUXPgTs7AJgeWCdFoBMMw484b0SiuDOyJuzfuxIxEUAocypE4BGOIjMJBrbdngMaexnw+3DaW6m3jqGko6Dxq4aGJueU1HDd3B2fOXE1eTjj9gg50NoNyOnB9FkQQIuEIdQFFbl4+jgxGxkWdTj6tUUxxKPh4dPdxb0sL1BYO/QbOkfM7aUOhGTba29vZW7uXBWMXdB/Lycnpdti6XC6cTidOpxOHw9H9692yLKo/3YG7tIOOjo6Ec2cqysaztwbXaWdkJPQ2FFKsXevj3XfzWLvWRyi0/xejYey/X/5Yi6OP7mDevHZOKX6WyMzTgLnAXBK/GxoA0zQxTZPRo0fT0uTiC0dPTH5RBgj5/RQe4LsP2lBo+kREMM39v94dDgcOx8CXv4ZhsHv3bvbt20dLawt+9vt3YrcDlFJ9bg/U74syrjC3zygfsJf+6UZ53Oz1u/nLX0rZsWNo+8y9aWtzEo0mfi35+SbHHhtg3rwARx4ZxO2235fczwwinWN6//t04XA4+vWlDAcjpcGyrG7HsVKKkpISJk+ezEctOqByKGhDoUmIZVls2bKFlpaWHse9Xi/5+fnk5+d376sHg0EaGhr6nKempoZoNIrP58MT9pCbm5v0/r0NhjfHkRFD0B+WBc+tnsHK30/MWAz+9Okh5s0LcNxxAUpL90fGOJ1Cb5ssIkSj0e5VlVKKnJychMZ7MAY93YyEBqfTSWFhIQUFBfh8Purr60c8EfNgQBsKTUK6nLexv+C7fsE2NjZSX1/fPdbtdlNbW9vnXG63G5/PN6D7J/rPvWOHh7//vajHFk0mqa118en6Msh3cOKJbfzbvzWl1XHs9Qr5+cnnExHC4TCmaVLszWHWrFl4vV7cbnfC9+lATj7UZCfaUAyGO+6gvbGR4pHWkSGe2vAUn+3+DI/bgwqn8GsslPp5lxr4Ry4aVTz13Bhee2cMljW8vw5LClq54nvtnDb276ioAel0g3TQZzW0TQ1hLMtekZmmiSfHQ35ePrtzRrF3exvQ1ue0LS0tFO7NXM5AKmSDhlgdHufIr7AOZLShGAyf/zzRAzSPovWFF5F+nL7RaITo5tc4Ne9wmtrzsTIQp19L3yVYTFOxessEmtttf4CgWL1pAv7mPNyudhYdvY25h9WlXVMinA6L2VObyD3mONRnBi2TFg7LfQE2mQHmT3BhmiaTJ09mzJgxKW+h+P2uEf81nw0asknHgY42FIPhrbdwNzbC4sUjrWTASDRK4eIvdm9nNDc394gkamtrwz9qN7PHnoq1OUTZTO+wadu+3cPvflfGp5/GO40PmxPhyisbOfzwAqAg/uIMIyJ9Rlx1MdjIqy5/TKwhCIfDuFxeZs2aRV7eoOq4aTRpQxuKwXDzzeRHIgekoTBNg71791JXV0coZO8JxTodHQ4HbtfwxmuHw4rHHx/Ns88WYVmKsrIoJ5zQ3u3MHT3a4AtfaOuO/BluDMMgEokwbty4fr+029raKCgYmBETEUQEwzB6OPBLO5o54oiZg6rpo9GkG/0pPEjpHToZjUbx+/00ffop0fHjcbvd5ObmJt7OGEY3wLp1uaxYMQa/343DIXzxi818+cuNeL3ZUTu7q2zHuHHjyJ80qd+xhmFQUlKSlvsW1oo2EpqsQX8SD0JEhJ07d1JX13MvXymF1+XGPcAIpHTR0uIkFLKtkGkqnnmmiNdeGwXApElhrryyjunT+840tksxBBMat3QUXkuE2+1m9uzZ+LZ9mva5NZoDBW0oDjJaX3iR93a+TX1jvf2LtNd3qjid1CepGZMsMkmE7i/8ZNTXu1m9Oo93381L6HtwuYSLLmpk8eJmYitwvPlZgNhKHYJwWO1r5Hqc9vZOr9u3hkJ4ven3p0SVgw01q7GUm73m7n7HpjPSR0fpaLIJbSgOMsxwiE9mjOGE0ef12Q/g8EHMa1nwySde3n03n9Wr82hsHPhHx+u1GDVq/7d/eXmEr32tgfLyeAewYcKpU/I6723R0dHBtLwiShZcmTCRKxvi9nWEjeZgJaOGQim1CLgXcAJ/FJE7e50/DLgfGIMdUf5VEanJpKa0cM89tDU0kJ7d6PTS0NCImWsOuWnMyy+P4i/3TwKXnX1tmqpH/SG3W3A6k/sRvF6LqqqOuFIU/REOhwmHwwQC9nOlFJMnT6a0dhcqCzKONZpDjUz2o3ACy4GF2N3s1iilnhGRDTHDfgX8RUT+rJT6AvAz4GuZ0pQ2qqowsjCPorW1lZaWFjxTh7YFs369lz/9qZRIUHDl7P9iHjcuyvHHtzNvXoBp08IDbsWYCl2F3ComVnDccZOBmAY+dboUg0YzEmRyRXE8sFVEtgN0drJbAsQaijnA9Z2PXwOeyqCe9PHyy3iamuDf/m3Ybx0IBGhvbycQCBAIBNi1a38bXNM08bqcQ4paampycu+947AsxZkL6rnsKjsDWCnweCQjxqELESEUCjF58mT27jWyol6RRqPJrKEoB2KbeddgtzyNZS1wIfb21JeAAqVUiYj0qDCnlFoGLAMoLy/HP8K/5kf/8Id4LQv/KacM2z1f3/M6bYE2wpEwCvsXdqIv0jGhbbjU3EHdwzTh3nvH0tLiZM6cIOedVZsRB3Eso2peQ1lG5/1NRrtdeKPrGL2njZb2njkJ4nAT7uPfPhAIjPjnIhs0ZIuObNCQLTqyQcNQyaShSPTbs/cG9Q3AfyulLgfeAHYDcWEjIrICWAFQWVkpI+4w9HiIRCJDclzGltIQsQiFQnFlme38K7tiqNq9hqNzp+HxeGLCQ+P3+6XoCBjd2x7H0+WcfusVN+GwbXDqGjx8vDGHwoIIlyz+lOHIu1OWXRojEomglGLu3Lm43W6aqndTWFWe8jzZ4cweeQ3ZoiMbNGSLjmzQMFQyaShqgNhOIRVAj6LwIrIHuABAKZUPXCgiPetaH6RINILrtNNoaGigrq4uYV+BWBonBvCUdrZ2HMJ9d+1y8+KLhaxenUdzswsjbPXwQ3hyhe/dXMeM2W46W5xnHMuyiEajzJkzZ8hOeI1Gk34yaSjWADOUUlOwVwpLgUtiByilSoFGEbGAm7AjoA4Jmpub8a9fj1IKj8eTtPm6SjFvoS+iUcWTTxbx1FOjMU17rrKyKHOmNDNx5v65Z88OMW3a8LXXFOwkuoqKigGXv9BoNMNDxgyFiBhKqWuBF7DDY+8XkfVKqduB90TkGWAB8DOllGBvPV2TKT3ZRG1tLfX1DXg/582Iw1bE7t3QtaUUCDh45JESamrsxj+nn97KGWe0MGVKhLotw1v4rzdG1KCgoIDx48ePmAaNRtM/Gc2jEJFVwKpex34Y8/hx4PFMasgIv/89rfX1lA7i0sbGRnbs2IHP485IToBhwH33jeXdd/Pjzo0fH2HZsjrmzEnWQGJ46CqEN3XqVB3hpNFkMTozezAcfjhmUdGALml94UUCLS3s3buHXJeLza3Cvh2B7vPbI+9jSt/lH5zKhdES6PM8gGkonv5tOVs25eLxRCgr228QJk1qZ/7n/dQ1Cv/4V8xFDtjo7t8/0htBiEQi3c9dTidGEh9LHxNxPLms+yS+jaouYaHRZA/aUAyGv/0NT3MzfC313MBQexs1UybjnjkDh8vFvh2B7hIVAK5GJycWDz7cNhpV/PrXY6nZnsv4ccItt9QydWpvX8OoQc/fhWmaBINBxo8/jKJOY9nS0kJhYeGA53I4HOTv8KM+l3p0k0ajGX60oRgMd91FXiSSsqEIhULs3bsX55QpQy4d3RXW+s47+Xz4oY9g0P7lbRiKYNBBni/KD36wj8mTI0lmGjjRaJRoNMq0adMoLS3tDtMNhUKDMhQAGc3g02g0aUEbigwTjUbZvHkzliV4PJ5BzWEYsGFDLqtX57F6dT4tLc6E44qLDZYt/ZTJk4f+5ZuopLfT6WT27Nk6OkmjOcTQhiKDiAjbtm0jHA7j87gHnP/Q3u7gsceKefPNfNrb9xuHsrIo8+e3M39+gNLS/ZVX8/IsGreHgaFHMYXDYfLz85k6dWq3sXA6nbqZjkZzCKL/12eI5z7eS1tHkJpdtXhyPKg9n7Dlk/3bQU4HuBr350707gHx3ns+/vjHMTQ12ccrKiLMm9fOcccFmDw5ktEdm2g0ilKKadOmxed3bHgazJ5lwb0tLVA7yK0np06w02iyHW0oMoTnrdc51htlenMTOTk5fDLaxamzvpBw7ObNOaxb5+OJziXHp596usNbDz88xBVX1HHYYfuNTN3WEGIlmgnUEIOFLMsiEokwa9asxEmAZhSOvKjHoZDfT+EBXqJAo9H0zQFtKBoaGjj99NMB2LdvH06nkzFjxgCwdu1aKisrMQyD2bNn8+c//xmfz4fT6eTII4/EMAymTJnCgw8+2B29k4wdO3awdOlSGmtrOWLGDB6LROL8Dg899BC//OUvaa9vwpnnZcuWLTz22GM0lCnu/PrXqa+v7/4CvueeFTz//ExWrWpl9+5LsKxmRCzKym6npORMLr64kUWLWuidYiAWaUmSE7HrSMW2ELUsi4kTJw7eOa3RaA46Duhg9ZKSEqqrq6muruaqq67i+uuv736el5dHdXU169atw+Px8Lvf/Q6A3Nzc7uPFxcUsX7485fvdeOONXH/99WzZsYPC8eP505/+FDfm0ksvpbq6muU3/Igf/ehHTJgwgVmzZnWfv/POO3n88ce59da/c8cdR/HSS4U0NNzJvHnncMMNz3H55f9FW9u3+eUvd3HOOfFGIh1Eo1ECgQDBYBCv10t5eXn3nylTpugsaY1G04MDekWRKieffDIfffRR3PETTjgh4fFEiAivvvoqDz/8MDz6KJeOG8ddTz3F1VdfnXB8KBjipZf+yTnnnBN37vnnR/E//2OvfKZNCzFpUgezZu3jK19ppLq6hnfeKWHcuPT0Xo7TFQrhcDiYPHkyRUVFSWtMaTQazUFvKAzD4LnnnmPRokU9jpumySuvvMIVV1wBQFtbGyeffHLCOR5++GHKysooKiqyo35++1umtbWxO5y4eJ5lWXwW2MizLz7L1T+9mjcb3+x2Vl933Y9oavIwatT5XHPNMs4/v5nGxitZtmwZDz/8MMFgkD/84Q9pfAdsusJdc3NzmTlzZv8GIoHDuhvtfNZoDjkOWkMRDAapqqoC7BVFl0HoOr5z506OOeYYFi5cCEBBQQHV1dV9zldXVxd3TPURetTW1sb2mj0U+Yr48rFf7j6+5fTP8be/zaWwsA2Ri3C7vTid57Fq1SrOP/98LrvsMqqrq7n55pt58sknu+sf9XZeJ3NYiwiGYcT5HkaPHs3UqVOTh7gmcFhrNJpDl4PWUHT5Ivo63tLSwuLFi1m+fDnf/va3k64oZs+eTXNzM4Zh4AJ2RyJMmDAh4fi6ujo+3Li1e9spFFI88kgJzz9vO4ivuipEe/uZfPzxx5x33nk8+eST3T6UqqoqwuEwTU1NlJSUAKk7r0WEcDiMaZr4fL7u6wE8Hg9lZWW6+J5GoxkwGTUUSqlF2G1OncAfReTOXucnAX8GijrHfL+z4mzGKSws5L777mPJkiVcffXVSVcUAKeddhqPP/44S4EH6+pYksA/YRgGDQ0NVG/cxo233cnHH+eyYsUY/H6FSB3f/KZw0kmN3HjjG8yfPx+AcePG8c4773D++eezfft2IpEIxcXF/WoxTbO74U8sRUVFjBs3joKCgj5XPBqNRjMQMmYolFJOYDmwELvb3Rql1DMisiFm2K3AYyLyW6XUHOyS5JMzpak3Rx11FJWVlaxcuZKvpVC36ec//zlLly7l1o8/ptLn697OeuaZZ1izZg3f/e532b17Nx988AFFo/J59dUqVq2yQ28rKhrZvv0c/uu/Itx7r8X8+fO58MILAfje977HbbfdxoMPPohSip/85Cc9vuRFBMuyuv8YhoHT6aSoqIiKiooeqwTdIU6j0aSbTK4ojge2ish2AKXUSmAJEGsohP0lTQvp1Sp1INx22209nre3tycc1/v43/72t5TvMXXqVFavXg319dTW1pKTk4OIcOKJJ3LYYYexefNmnE4nJ510Ep984fusWlWEyyVcdFEj557bjMu1MuG806ZN48EHH0x4LhqNEo5EMAy7fEZOTg7FxcWUlpb2yMnQaDSaTJFJQ1EO7Ip5XgPM6zXmNuBFpdS3gDzgjEQTKaWWAcsAysvL8fv9aRc7UAJeL69XP0ZLewuRSIRtTQaW2KsAI+zkH+8sBDdcemkD55wzuDbgIoKIMLqoiPLynm2S6uvrCQQCg3ovcra9gLL6iGoCxOEmPIB5B6sjnWgN2aUjGzRki45s0DBUMmkoEm2Q966LdzHwgIjcpZQ6AXhQKTW3s4f2/otEVgArACorK2XsMJSLsCyL9vZ2fD5fd5RQ6wsvItEo7pdfRrW345yZxzE5U/B4PIytg9lldpb2Q6/NxYqUMXlGmLPOGryRCAQCTJo0iXbTQaLX7Pf7Ex5PSq0vrVFNg9aRRrSG7NKRDRqyRUc2aBgqmTQUNcDEmOcVxG8tXQEsAhCRt5VSXqAUqM2grn7pckbv2bOHSCSC2+2mvLyc0tJSJBrBceqp8OPbyTVN9l3ydQ4vOxUA/44As6bksWuXh2c3VqDy4T/+YzfOxBXBkxIKhSguLmbcuHFs3T1ib4dGo9Fk1FCsAWYopaYAu4GlwCW9xnwGnA48oJSajV0fOz5hYZjo6Ohg48aNmKaJx+MhLy8PwzD49NNP2bNnD+5du+jYuJHZhoFyOOLyEaJR+OMfSzFNxcKFLUyfnjghrzciQigUQkS6n3s8HqZMmaLDWTUazYiTMUMhIoZS6lrgBezQ1/tFZL1S6nbgPRF5Bvgu8Ael1PXY21KXS9e3ZWo3gddfh9NOS4dedu3ahYjg8/m6j7tcLlwuF9EdOzBNi9zc3B5f3oYBH36Yx7NPevjzvgkEAk4KC00uvrgx5ft2dHRQXFzcI+/B5/PpCCaNRpMVZDSPojMnYlWvYz+MebwBOHGQk8P118O998Krrw7ZWLS3t/PKZ6/g9Diho9fJujp445+UG1GaJmxkTHAPEVNo2fYRP3zoWD7ZWABRB968ZqZUdHD1RWuY0FALDUleAkIkHKFsVAFjcsfg2N3H6uEzD3jjW5sOug+ELsOh0WgGwIGZmR1rJL7zHViwYIjTCTU1NVgOi1NKTokfUAzszMX3j5V07JhCQe4EmkMmrdtuYdsmB8VGLYu/sId5V0xh/PgoUEkyF7ZlWXR0dDBhwgTKJk7sPzkutA+OHBd/WPeB0Gg0w0BSQ6GU8gDnACcDE4AgsA5YJSKfZFZeH8QaibvvZqjt3tra2mhtbcXt6ueX9llnwabX4Z132HnBBfw5ciZ//6MDR6Cd6y9byxFXHAX0HXIai2maBINBJk6cyIQJE9j+YR2m2UcnIsDp1H4KjUYzcvRrKJRStwIXAm8A7wMvYTucZwJ3K/tn8A0isi7TQrtwbdwIH30E5eVQXW1vOX35y/DNb0JHByQo683ll9t/6uvhop5hoQI0L16M67TTyKtrZMoNX4+7vP6yy2hbsAB3bgFTXnmXyKtvcmHH37jIMBk7Oog151sEAO8nnzD+5z+Pu37fd75DsKqK3Opqxt5zD5Zl4c3NxdPpg3D9x61Mu/QMePll+MlP4vX//vdAGfztb3DXXd2HR0ci4PHAgw/CxInw6KPw29/GX//441BaCg88YP/pzapV4PPBb34Djz0Wf/711+2/f/UrePbZnudyc/fP+eMfwyuv9DxfUgJPPGE/vukmePvtnucrKuB//9d+fN119r9pLDNnwooV9uNly2Dz5p7nq6rgnnvsx1/9KtTU9Dx/wgnws5/Zjy+8EBp67Qeefjr84Af247PPhmCw5/nFi+GGG+zHiVauMZ+90V/6kv3vEUs/nz0Arr4avvIV2LULElUH+O534dxzYdMmuPLK+PO33gpnnGG/b9ddB8R8LgDuuAM+/3l46y24+eb46++5x34P+/vsHX543Gevmz4+e90aMv3Ze+45+3Ffn73f/MZ+fJB/9gb6vTdQkq0oPhaRBJ8eAH6hlBpPzxDY4WP69LRMYxoG4VAorlNdIpqDPqAM1VLDGPYR8eRRcpg3eZhWZyc5ZziM1eksT1rBVaPRaLIENZAgox4XKlUhIjXJR6aXyrIyWVtXl7Ztp23bttHU1ITX6+XNxjc5sTjet75nj5vf/a6MLe+2Yjly+J99S3B4FNaZM/CdVGVvS/XCNE3Cnf0qlFLk5ubi8/koKysjPz+/x9jNa/Yx87h4H0QysiWRJxt0aA3ZpSMbNGSLjmzQAKCUel9Ejh3Mtan4KI7DLsfxLxGpV0odAdwIfAE7iW54GT8eLrnE9lHA0IzFhqfxfLqdUsPE6XTiC22jsL13yBM8+OhxbP24CLcR5MjCjygvDWAV59ByUhW88w4AcuaZmKZJNBrFsiycTifjx4+npKSE3NxcXclVo9EcsCTzUfwM20exFrhVKfUk8B3g58BVmZfXB3ffbf89RGPR+mY1jeESHA4vDocir91Na35ujzEi8O6HFZgtIW6ffhfTL5xF0cpcmkNm90qiY8sW8HjIXbCAoqIiRo0aRWFhIc5+0rK3fVDb7cDWzmqNRpPNJFtRLAEqRSSolCrGLsFRKSKbMi+tH5TqaSyWLBlUHoUVNQgffzy5ubmIUtQ1Kmb22nravdtNXbCIAvUZU780yzYOK/dXgY2cdhpepTji+utxvvQSHHlkSvc2TWtQ200ajUYz3CQzFCERCQKISKNS6pMRNxJddBmLQRoJsHMZlFL9bgtVV/vAk0PlaV4cZ/f0RUink3raN76B86ij0pIhrtFoNNlGMkMxVSn1187HCpgc8xwRuSBjylJBqSF9OVuWRTJn/tq1djmPylM8gJ0dveN//od/7AgwPxKhoKCAUYWF2khoNJqDlmSG4sJez/87U0JGAtMy+z0fiSg2bLB9FpWVQd78LIDReYnTYVeanTRpknZUazSag5p+DYWIvKKUOhKYBqwXkS3DI2t4sPrJhgbYsMFLNKqYOjVMYaGJ0QinTsmj5IEH7BIcV18dF+qq0Wg0BxvJop5uxu4Z8QFwnFLqdhG5P9XJlVKLgHuxq8f+UUTu7HX+bqBrz8YHlIlI0QD0DwnTMPpdDXRvO1XaIbOCYBgGea+9hhkSmi68ns3N+wZ1bx3ppNFoDhSSbT1dCnxORAJKqTHYlWBTMhRKKSewHFiI3cRojVLqmc6KsQCIyPUx478FHDVA/UPCMI1+Q1irq21DUVXVQTgcJhKJ4HD48LjdhKMmc08cmaR0jUajGU6S/awNi0gAQETqUhgfy/HAVhHZLiIRYCV2uG1fXAw8MoD5h4xpWn02BqqtdbFnjwefz2LGjBCmaTJ27Fg+97nP4fV6UUqvCDQazaHBQKOepg0g6qkc2BXzvAaYl2igUuowYArwalLFaUJEME2zzxVF17bTkUd2dLcz7be6rEaj0RykZDLqKdHmf6nT/wIAACAASURBVF+xqEuBx0UkYRiSUmoZsAygvLwcv98/ABmJsSx7NdGXj+LDD3v6J9xuNx3NHfj9foocDgyXOy06hkIgEBhxDdmiQ2vILh3ZoCFbdGSDhqGSzFBcIiJXDHLuGnpWlq3AzuxOxFLgmr4mEpEVwAqAyspKSUeBrY6ODtqsxFFP77/v4/338zAjFhWjmvBvihKJRPC4RtPymdDy84dpa2/jGF1sLGt0aA3ZpSMbNGSLjmzQMFSSGYqhOJfXADOUUlOA3djG4JLeg5RShwOjgbd7n8sk0WiURAuc2loXv/lNGQBLFtVy+HEuTFMh4iZCITOr7LIbfr/OndBoNIcGyQyFrzOPIuG3ooh81NeFImIopa4FXsAOj71fRNYrpW4H3hORZzqHXgyslMHWOx8k0Wg0zk4YUQf33DuW9nYnRx8dYOGpDYAX0zTx+XwQ6hz44x+T194OCZoUaTQazcFGMkNRjh3i2pe/IUGD6ZgBIquwQ2pjj/2w1/PbkqrMAKFQKO5VvfJ/j2TbNi9lZVGuuaaWjs6NMtM0yc3N3W8oXnkFTyQyrHo1Go1mpEhmKLaKSL/G4EAlFAr1CHHdsMHL+6+PpShHuO46P/n5Fl2dKSzLsg1FU2o9sTUajeZg4pBNBrANxf4lxbPP2gnhS5Y0MW1auMdYpVRKrVI1Go3mYCSZoUjQjf3gIBwO4+g0FHv3uvngAx8ul8lZZ7XGjdWGQqPRHMok23q6svNX90siYsSe6EySuwyoGUj9p2zANE1M0+z2UaxaVYiI4oj5uygszI0bLyK43THJdiUlWOFw3DiNRqM5GElmKK4BvgssV0r5gTrAi51FvQtYLiJPZFZi+rFDYwEU7e0O/vGPAgCOX7gF+FyPsSKCw+HA5Yp5q554gha/H+/wyNVoNJoRJVmZ8d3AfwL/qZSaDowHgsAmEWkbBn0ZIRqNopRiU7SOlU/7aegYw9Q5fsaXB+PGmqZJTk6O7jmh0WgOWZKtKLoRka3A1gxqGTai0SgiQsRUbPvnseQ7XVxxQStVo+NLUZmmSV5eXs+DN91EfiAA9903TIo1Go1m5EjZUBxMhDv9Cx9u/xxNTS4qKiJUVsavJiAmhyKWt9/GrfMoNBrNIcIhGR4bCoVwOBx8vHMuAGed1UJ/O0ter/ZGaDSaQ5eUDYVSytPppzjg6TIUe5vsuk0zZ4b6HKtDYzUazaFOSoZCKfVF4GPgpc7nVUqpJzMpLJOEw2FM00V9azEOh1Be3n/GtTYUGo3mUCZVH8Xt2E2HXgMQkeoDcnWx4WmMSJD8nTvZVTsWkVMYPz6K2524HqGIxOdQAFRUYIb6XoVoNBrNwUSqhiIqIs29QkSHtdprOrCMMFu9VbSVT2PjnjGI00lFRd9OaQFcLld8F7z//V9a/X7iU/M0Go3m4CNVQ7FRKfVlwNHZX+I7wDvJLlJKLQLuxS4z/kcRuTPBmC8Dt2F/L68VkbieFelARGhoaKDF14LP52PXLg+W6aDIE6B2c1+rAwuvN6+PcxqNRnNokKqhuBb4IWABf8XuMXFTfxcopZzYJcoXYne7W6OUekZENsSMmdE5z4ki0qSUKhv4S0iN+vp6mpua8ZX6UEpRU+MBDOYcK5TNTBzVFAqF4kNjAa67jvyODlixIlNyNRqNJmtI1VCcJSI3Ajd2HVBKXYBtNPrieOwy5ds7x68ElgAbYsb8B3YZkCYAEakdgPaU6ejoYMeOHYz1eAh3bp/t2mUbiokT+956siwrcWhsdbXOo9BoNIcMqRqKW4k3CrckOBZLOXY9qC5qsB3iscwEUEq9ib09dZuIPN97IqXUMmAZQHl5+YAblQcCAVwuFw6HHeQVDCrq6tw4HRbjxvUd8eR2uwmHw933a2lpwe93MToSwbSsEW+Yni1N27NBh9aQXTqyQUO26MgGDUOlX0OhlDoLWASUK6V+HXNqFPY2VL+XJzjW2wHuAmYAC4AK4J9Kqbki0tzjIpEVwAqAyspKGWij8p07d9rVYjuxt52gtKiF3n7qWAzDoLS0lPz8fAAK9xp2k3SPh0gkMuIN07OlaXs26NAasktHNmjIFh3ZoGGoJFtR1ALrsJuAro853gZ8P8m1NcDEmOcVwJ4EY94RkSiwQym1CdtwrEky94AIhUI9IpfsbScYO7oZKEh4TVdorM6h0Gg0hzrJqsd+CHyolHpIRAaaOLAGmNEZJbUbWAr0jmh6CrgYeEApVYq9FbV9gPdJSpeh2LOnkEAoxMYPCjHCFmXFTfRlKMLhMMXFxYkNxcyZGMEg2oRoNJpDgVR9FOVKqZ8Cc2B/GwYRmdnXBSJiKKWuxY6QcgL3i8h6pdTtwHsi8kznuTOVUhsAE/ieiDQM8rUkxLIsIpEIubm5WJaibKaXppAPV46DieXbgEmJtGNZFuPHj0886YoVtPn9+NIpVKPRaLKUVA3FA8BPgF8BZwNfJ7mPAhFZBazqdeyHMY+Fzn4XKeoYMF1NimKTBbt8FONKEtukSCRCQUEBPp82BRqNRpOqofCJyAtKqV+JyDbgVqXUPzMpLF28/NbPqK3bh8vlImJMom3XanbVj8XtMSgtCMSNFxEMw6C8vLzvZkXLllEQDMKDD2ZYvUaj0Yw8qRqKsLK/Nbcppa7C9jlkLDkunYQjQY6Y/A18Ph+1hGhoLyLfmc+0ySFmjpoWF4YVjUbx+XwUFCT2XQCweTMunUeh0WgOEVI1FNcD+cC3gZ8ChcA3MiUqnZiW2eN5V8RTX4l20WiUyZMn69anGo1G00lKhkJE3u182AZ8DUApVZEpUenEiBrdiXaw3z+RqBigYRh4PB6KioqGTZ9Go9FkO0n7USiljlNKnd8ZvopS6gil1F9IoShgNmCYRo8cis8+63tFEYlEGD9+fA/DotFoNIc6/X4jKqV+BjwEXAo8r5S6BbsnxVo6y29kO6ZpdhsKkf0rit6Gwg7AguLi4uSTVlURnTs3vUI1Go0mS0m29bQEqBSRoFKqGDuzulJENmVe2tAxTROxpNvfsGlrHm1tTkaPNigu7um7CIfDFBUVpZaJfc89tPv96ALkGo3mUCDZHktIRIIAItIIfHKgGAmwt5Jgfw7FK/+0VwtnntlKb1+1ZVkHfD0WjUajyQTJVhRTlVJdFWIVMDnmOSJyQcaUpYENf34F77ZCaretxd+Ux7rq8eTmtLAw/wXUvyKIy375pmnicrn6D4mN5atfZVQoBI8/nkH1Go1Gkx0kMxQX9nr+35kSkgmi4Sih41oom1PJM38sxVXo45TTWyk48/ge+RPhcJjy8vLUndg1NTh1HoVGozlESFYU8JXhEpIJDMMAoL3dwRtv2KuFs8/uUcG824ldUlIyvOI0Go3mACHVhLsDhw1Pg2nXd3I0bYfDXLz88ijCYQeVlR1MnNizUVE0GiU/Pz9xJzuNRqPRHISGwozCkRcBEHzmfwkUFfLCC4UAnHNOc9xwwzC0E1uj0Wj6YUCZZUqpnAGOX6SU2qSU2qqUimt0pJS6XClVp5Sq7vzzfwYyf3+ICKZlsn1dOY2NLioqIlRWBuPGAIwaNWpgk59wAtFjj02XVI1Go8lqUlpRKKWOB/6EXeNpklKqEvg/IvKtfq5xAsuBhdid7NYopZ4RkQ29hj4qItcOSn0/GIYBItTvtX0TRx3VERcSG41GKSgowO12D2zyn/1M51FoNJpDhlRXFPcBi4EGABFZC5yW5Jrjga0isl1EIsBK7AS+YSESiYBStLfYvofRo424MYZhaCe2RqPRJCFVH4VDRD7tVVHV7GtwJ+XArpjnNcC8BOMuVEqdAmwGrheRXb0HKKWWAcsAysvL8fv9fd7U29JCyO8nFArhUI4YQ9FTrojgdruJRqN9zvfqliai5v7+TG6nA7/fT+EVV5BvGPj//Oc+dQwHgUCg3/fiUNKhNWSXjmzQkC06skHDUEnVUOzq3H6Szi2lb2F/sfdHojrdvds//A14RETCnX0u/gx8Ie4ikRXACoDKykrp1/lcW0jh2LH4/X5MyyTQaSiKinquKLpqQE2YMKHPkuK+vQZLqsrjT7S3E4lERtwJ7vf7R1xDtujQGrJLRzZoyBYd2aBhqKS69XQ1drvSSYAfmN95rD9qgIkxzyuwa0V1IyINIhLufPoH4JgU9SQlGAyiemw99VxRRCIRSkpKdN8JjUajSUKqKwpDRJYOcO41wAyl1BTsjnhLgUtiByilxovI3s6n5wEbB3iPPgkGgyhsQ+ElfkUBUFhYmK7baTQazUFLqoZijVJqE/Ao8FcRaUt2gYgYSqlrgRcAJ3C/iKxXSt0OvCcizwDfVkqdBxhAI3D5YF5EIkKhEMGIB8Nw4s2zyM3dv+vVte3k8/nSdTuNRqM5aEm1w900pdTnsVcF/79SqhpYKSIrk1y3CljV69gPYx7fBNw0YNVJsCyLaDRKazBxxFMkEqG0tHTwDYpOP51IezspFCTXaDSaA56UM7NF5C3gLaXUbcA92A2N+jUUI0VXefHWji5HdnzEU6IGRc99vJdITJSTx9mHIfnBDwj4/eSnSa9Go9FkM6km3OVj50AsBWYDTwOfz6CuIRGN2vWcWjrsRPJYR7aI3cgoPz/+az5iWomjnDQajeYQJtUVxTrsUNZfiMg/M6gnLUQiEUSEloBtKGId2ZFIhMLCwh59tAfM2WdTFA7Dq68OVapGo9FkPakaiqkiYiUflh2EQiGUUt0riti2p6ZpptYXuz+CQZTuR6HRaA4R+jUUSqm7ROS7wBNKqd7Jclnb4a6jowOn0xm3ohARRGTgRQA1Go3mECbZiuLRzr8PqM52oVCo01D0TLYzDIO8vDw8Hh2vpNFoNKmSrMPd6s6Hs0Wkh7HozJHIug54IkIoHMLr9XZvPXWtKKLRKOPGjRtJeRqNRnPAkWoiwTcSHLsinULShWmZ3ZFNXVtPsT6KtGRjL15MeOHCoc+j0Wg0BwDJfBRfwQ6JnaKU+mvMqQIgvl1cFmAYBkopgkFFxHDickfJzbUwTROXy0Vubu7Qb3LDDXT4/RQMfSaNRqPJepL5KFZj96CowG5C1EUb8GGmRA0FwzAQh9DcbL+0/MIQStlhsWVlZboIoEaj0QyQZD6KHcAO4OXhkTN0TMMEDzQ12XkS+YUhwPZdjB49usfYlDOxe7NgAaMjEXjrrfSI1mg0miwm2dbTP0TkVKVUEz17SShARGSICQnpJxwO4/A6ulcUBUVBLMuJUoq8vJ7NS3Umtkaj0SQn2dZTV7vT0kwLSRfRaBSn09m9osgrDBGNeigqKhpaNrZGo9EcovS71xKTjT0RcIqICZwAXAnk9XlhJ0qpRUqpTUqprUqp7/cz7iKllCiljh2A9oREIhEcDgdNTft9FKZpxq0mNBqNRpMaqYbHPoXdBnUa8BfswoAP93dBZ8vU5cDZwBzgYqXUnATjCoBvA+8OQHdCTNMkunY7zrfeovkju5lel4/C7XYPdXqNRqM5JEnVUFgiEgUuAO4RkW8ByTb3jwe2ish2EYlglyRfkmDcj4FfAKEUtfRJNBoF04STT6apYAp4vRQU2dO6XClXVE/Ol79M6Lzz0jefRqPRZDEpt0JVSv0b8DXg/M5jyX6ilwO7Yp7XAPNiByiljgImisizSqkb+ppIKbUMWAZQXl6O3+/vPvfGvjcwrM7MayPKaLOBPY1vsrV2ISGzkMKiKG63m7a2Np5f7ycaE+Xkdjp6zJUyF15IIBAgOJhr00ggEBic/oNQh9aQXTqyQUO26MgGDUMlVUPxDeCb2GXGt3f2wX4kyTWJEha6I6eUUg7gblJofyoiK4AVAJWVlTJ27Njuc3mBPM6Zeg4A9fX1NDr/xZTiE/lD+1i8Tlhw2FwMI0BJSQm+Rmd6opw6OvB3dBCrYyTw+/0jriFbdGgN2aUjGzRki45s0DBUUm2Fuk4p9W1gulJqFvaW0k+TXFaD7QTvogLYE/O8AJgLvN6ZBDcOeEYpdZ6IvJfqC4glGAwCEA4rgkEHTqdJfr5FKJTmradzztF5FBqN5pAh1Q53JwMPAruxVwrjlFJfE5E3+7lsDTCjc/WxG7sUyCVdJ0WkhZiwW6XU68ANgzUSYBsKpRzdobGFBQZK2cl2OjRWo9FoBkeqP7PvBs4RkQ0ASqnZ2Iajz3BWETE6K8y+ADiB+0VkvVLqduA9EXlmaNLjCYVC+ByqO9mucJSBiL3bpQ2FRqPRDI5UDYWny0gAiMhGpVTSpg4isgpY1evYD/sYuyBFLX3di3A4TL5SPVYUIoLL5dI1njQajWaQpGooPlBK/R57FQFwKVlUFFBEaG1t7Vw9qO5ku1GjDCzL0jkUGo1GMwRSNRRXYSfF/X/YPoo3gP/KlKiBEI6E2bJlC01NTd0GoWtFUTQqimVZeL3e9N708ssJtrSg++RpNJpDgaSGQil1JDANeFJEfpF5SanT2NjIrl27mDh6Ij6fr3t7qbePIu0rissvJ+T3k4YWSBqNRpP19JuZrZS6Gbt8x6XAS0qpRJ3uRoxAIIBSCq/X28MHEeujyMjWU309qqEhvXNqNBpNlpJsRXEp8DkRCSilxmA7pu/PvKzUME0z7pgI1NTYm0KlxfbWk8eT5k2iiy6iSOdRaDSaQ4RkhiIsIgEAEanrzKYeUdqNdlZttwOp/LV+3I6eq4WGQAFNTS7y803GlEZQSmlntkaj0QyBZIZiakyvbAVMi+2dLSIXZExZH4hId8mOzcZm2txtPc5/ss8u0TF9ehhHp1nTORQajUYzeJIZigt7Pf/vTAkZDJZlxeVHbPJPAGD6dLtqrFJKGwqNRqMZAsl6Zr8yXEIGg2macYZic61tKGbO3F+1PK11njQajeYQ44D+Bu1tKKJRxba6sVBgbz0FdmeoztPVV9PR3KzzKDQazSHBAW0oem897dzpIWo6qaiIkJdn8UF9mKCY7Hb5cTgceJxp8sV/5SuED/D68hqNRpMqAzIUSqkcEQlnSsxAMU2zR0TT5s1ewOr2T5iWcPKkXI49qiK9tZ527cJRVwcHeI15jUajSYWUfmIrpY5XSn0MbOl8XqmUSlrCQym1SCm1SSm1VSn1/QTnr1JKfayUqlZK/StRT+3+6L2i2LrVLtXR5Z/oyspOe0HAr32NwmuvTe+cGo1Gk6WkuhdzH7AYaAAQkbXAaf1doJRyAsuBs4E5wMUJDMHDInKkiFRh983+darCRaS7hHgXmzfnADBzZrhrkHZkazQazRBJ9VvUISKf9vplHp8W3ZPjsTvhbQdQSq0ElgCx5cpbY8bnEdMqtS+cgRAtz/4dy7Jw7diOI8c2Do1tXup3lpHniVJeHrHnh/RnZWs0Gs0hRqqGYpdS6nhAOlcK3wI2J7mmHNgV87wGmNd7kFLqGuA/AQ/whaRKRChc/EWi0SjGhx/i8fkA2PxuHuQXMGvyXhwOZ+fQDBQE1Gg0mkOMVA3F1djbT5MAP/By57H+SOQYiFsxiMhyYLlS6hLgVuCyuImUWgYsAzhsTD4t//oTlmVR1tLSHfpas6YKZ8RH4Zha/rFjlP3inA5M08Sf5gil0ZEIpmWlfd6BEggERlxDtujQGrJLRzZoyBYd2aBhqKRkKESkFrvn9UCoASbGPK8A9vQzfiXw2z7uvwJYATCzvEQKT7qCYDDIlo8/xte5oviobgKmJxf37AmcOsW2R5+1G4waNYqysrIBSk/CTTfR1NzM2BGOevL7/SOuIVt0aA3ZpSMbNGSLjmzQMFRSMhRKqT+QeDWwrJ/L1gAzlFJTgN3YhuaSXvPOEJEtnU+/SGdUVSpYltX92DBg2zY74mnCYUHAfqzIUFb2uecSOcB/IWg0Gk2qpPot+nLMYy/wJXr6H+IQEUMpdS3wAuAE7heR9Uqp24H3ROQZ4Fql1BlAFGgiwbZTP/N3P163LpdoVDFhQoTcvJ4+9ozUedq0CWd9vc6j0Gg0hwSpbj09GvtcKfUg8FIK163C7mERe+yHMY+/k5rMeGJXFKtWFQGwYEFb3LiMrCiuvJJRuh+FRqM5RBjst+gU4LB0ChkoXcl2T71r8K81ObhdBjnT9+GKWUDoyrEajUYzdFL1UTSx30fhABqBuEzr4cSyLESE1a+XUOBxsnBhC4uO8PYYk5GCgBqNRnOIkdRQKDvLrhLbIQ1gSe+U6BHAsiza252sf68IrwMWLWrpcb5LojYUGo1GMzSSlvDoNApPiojZ+WfEjQTYhuK114qIGoqqqg4qKqI9zosIDocDh2PEu7dqNBrNAU2qPorVSqmjReSDjKoZAOGwycsvlwBw9tnNcecty8rcauLWWwk0Nel+FBqN5pCgX0OhlHKJiAGcBPyHUmobEMBOURAROXoYNPbAsISnq3fzxosRamrzKRsfprIyGDdORHBmqiDgGWfoPAqNRnPIkOybdDVwNHD+MGhJmSVV5bz7WBP5HgfnnxUkURVxy7JwOjK0oqiuxtXQoPMoNBrNIUEyQ6EARGTbMGgZEG1toBSMHp24iK1lWbhcOZm5+XXXURCJwOmnZ2Z+jUajySKSGYoxSqn/7OukiKTcPyLdtLfbf+flWQnPi+5FodFoNGkh2TepE8gncSXYEaW93Zbk8yVeUYgILh0aq9FoNEMmmaHYKyK3D4uSAWIbCtXnikIppUNjNRqNJg0k+ybNupVEF+3ttnRtKDQajSazJFtRZK23NhBQKJXYUHT1085YeOwdd9De2EhxZmbXaDSarKLfn9wi0jiUyZVSi5RSm5RSW5VScbWhlFL/qZTaoJT6SCn1ilIqpUKDpgmhkG0ovN54QxEKhSguLsaTqTaon/880eOOy8zcGo1Gk2VkbG+ms7f2cuBsYA5wsVJqTq9hHwLHisjngMeBX6Qyd1fEk89n0Xt3yTRNlFJMmjQpcxtnb72Fe82aDE2u0Wg02UUmN/GPB7aKyHYRiWC3Ol0SO0BEXhORjs6n72C3S01Ka6sgYhuK3oRCISZOnIjHk8ECGzffTP4dd2Rufo1Go8kiMploUE7PLng1wLx+xl8BPJfohFJqGbAMYGJpATt21KNUXpx/wjAMfD4fIoLf76elpRW/P/3LitGRCKZljXjD9La2NpqamjAMY0R1iAh1dXVaQxZoyBYd2aAhW3QMtwaXy8WoUaPSWusuk4Yi0Td0wsqzSqmvAscCpyY6LyIrgBUAU8cXi8s1GpFwD0MhIkQiEebMmUN+fj4ALZ9JZpqaezxEIpERb5je1NREaWkpJSUlqER1TIaJaDSKO1P+IK3hgNSRDRqyRcdwahARGhoaaGtrY8qUKWmbN5NbTzXAxJjnFcCe3oM6e2bfApwnIuFUJm5tte1NXkx/7EgkQmFhYbeROBQwDGPEjYRGo8kelFKUlJQQCoXSOm8mDcUaYIZSaopSygMsBZ6JHaCUOgr4PbaRqE114i5DEeujME2ToqKiNMg+sNBGQqPRxJKJ74SMbT2JiKGUuhZ4AbsUyP0isl4pdTvwnog8A/wSu0TI/+18cZ+JyHnJ5k5kKIDhW03ccw9tDQ2UDM/dNBqNZkTJaNU8EVkFrOp17Icxj88YzLxtbV1bT7ahsCwLpRS5ubmD1jogqqowdD8KjUZziHBA1rhoa7P/zs+3fRTRaJSCgoLhK9nx8st43nhjeO6l0Wg0I8wBbSi6tp6G3T/xk5+Qd/fdw3e/A4Ann3wSpRSffPJJRuZ3Op1UVVUxd+5czj33XJqb97e/ramp4YILLmDGjBlMmzaN73znO0Qike7z+/btY+nSpUybNo05c+ZwzjnnsHnz5rh7BINBTj31VExzf5BEote1c+dO5s6d2+Pa2267jV//+tdDvl+20dzczG9+85s+z99222386le/AuyAklNOOSUt4drPP/88hx9+ONOnT+fOO+/sc9y9997L3LlzOeKII7jnnnvirp89e3aP63ft2sVpp53G7NmzOeKII7j33nuHrDVVvX2N6U9TomvS+T6nykFhKAAKCgpGSI0G4JFHHuGkk05i5cqVGZk/NzeX6upq1q1bR3FxMcuXLwfscMALLriAJUuWsGXLFjZv3kx7ezu33HJL9/kvfelLLFiwgG3btrFhwwbuuOOOhDkw999/PxdccEGP+POBvq6h3i+TvP7661x++eUDuiaZoYjF4/Fw+umn8+ijjw5C3X5M0+Saa67hueeeY8OGDTzyyCNs2LAhbty6dev4wx/+wOrVq1m7di3PPvssW7Zs6XH92rVre1zvcrm466672LhxI++88w7Lly9POHcXqbxnqejtb0xfmvq6Jl3v80A4IA1FbNOiYfdPZCnHHpuZP6nQ3t7Om2++yZ/+9KfuL9Qbb7yxxxfMbbfdxl133QXAj3/8Y2bNmsXChQu5+OKLu3+RpsoJJ5zA7t27AXj11Vfxer1cdtllgL3yuPvuu7n//vvp6Ojgtddew+12c9VVV3VfX1VVxcknnxw370MPPcSSJfuLByR6XckYyv3OP/98jjnmGI444ghWrFgBZPZ97E0gEOCLX/wilZWVzJ07l0cffZTvf//7bNu2jaqqKr73ve8B8NOf/pTDDz+cM844g02bNvWY4/zzz+ehhx4ako7Vq1czffp0pk6disfjYenSpTz99NNx4zZu3Mj8+fPx+Xy4XC5OPfVUnnzyyX6vHz9+PEcffTRg/7icPXt292cpk3oHo6m/a9LxPg+EA7IFnL2iEPLyLAzDGF7/hCaOp59+mkWLFjFz5kyKi4v54IMPWLp0Kddddx3f/OY3AXjsscd4/vnnee+993jiiSf48MMPMQyDo48+mmOOOSble5mmySuvvMIVV1wBwPr16+OuHzVqFJMmTWLr1q2sW7cupfkjkQjbt29n8uTJ3ceeeuqpuNfV9R+6L4Zyv/vvv5/i4mKCwSDHHXccF154YVrex3nz5hEOh2lvb6exsZGqqioAfv7zn3PWWWd1j3v++eeZMGECf//73wFoaWlh3rx5gxaQQAAAHPFJREFUrFu3jurqagDef/99Vq5c2ed9586dy5o+6qCdfPLJtHVtB8Twq1/9ijPO2B/Xsnv3biZO3J+CVVFRwbvvvht33dy5c7nllltoaGggNzeXVatWceyxx6Z8/c6dO/nwww+ZNy++YESq71mqegej6cUXX+zzmv7e50xwgBqK/d3tDMM4JPMnevPeeyN378cee4zrr78egKVLl/LII4/wy1/+ktraWvbs2UNdXR2jR49m0qRJ/PWvf2XJkiXdK8Bzzz03pXsEg0GqqqrYuXMnxxxzDAsXLgTsrZ5EceN9He+L+vr6uM/RI488wnXXXdfjdR199NF9zjvU+9133308+eSTgL1vvWXLFubPnz/k97Hry+X111/ngQce4IEHHkg47sgjj+SGG27gxhtvZPHixZx88sk0NTX1GPPPf/6TL33pS/h8PgDOO69nNLvT6cTj8dDW1kZxcXHctakgEl/AIdF7O3v2bG688UYWLlxIfn4+lZWVuFyulK5vb2/nwgsv5J577mHUqFFx41N9z1LVOxhN/V0T+z4Px7b7AWko7K2nru52avizsX//e1rr6ykd3rtmJQ0NDbz22musX78epVR39d5f/OIXXHTRRTz++OPdzl1I/B8mFbp8FC0tLSxevJjly5fz7W9/myOOOIInnniix9jW1lZ27drFtGnTqK2t5fHHH09p/ths1oaGBl599VXWrVsX97pKSkrivkAbGxuZOHEikydPHtT9Xn/9dV5++WXefvttfD4fCxYs6D6fzvexP2bOnMn777/PqlWruOmmmzjzzDP593//97hxyQxiOBzG6/XGHU91RVFRUcGuXfvLxNXU1PD/2jv38CirO49/fiQEkiBIuagQjEFCzI0k3FaLhrAuCJQuBbFA1UKBUqgXVpd91rbbrXaRYtutpatdymofUatYeRaKqHS9JCsK4ZKIEEDAhaCAQhIh5EYSMr/9431nmEwmk0kyt+D5PM88zzvnPe853zkzc37vey6/36BBg7zWtXDhQtfT5Y9//GMSEhLavL6xsZE777yTu+++m5kzZ/r8LP7gj96OaGrrmtbaOSg4g/x0ldcN1/TVzMyLmpZWpR99tE937dqlTU1N6o3Duz73mh4Ivvjii6CV7S/79+8PtwRds2aNLlq0qFlabm6uvvfee1pSUqK33HKLJicn6+nTp1VVddeuXZqTk6N1dXVaVVWlw4cP11/96ldt1hMfH+86Li4u1iFDhmhDQ4M6HA4dNWqUPvvss6qqeunSJV20aJE+/PDDqqrqcDh07NixunbtWtf1u3bt0oKCghZ1JCQkaF1dnetzLV682OvnUlUdNWqUvv3226qqWlFRocnJyXro0KEO17dp0yadNm2aqqoeOnRIe/Toofn5+aqq7W7HhoaGNtvTG6dOnXLp2bhxo06fPl3Ly8v1+uuvd+UpKirSzMxMra2t1QsXLuiwYcOafX/l5eV60003dViDqmpjY6MmJSXpsWPHtL6+XkeMGKElJSVe8545c0ZVVU+cOKEpKSn65ZdfNru+urq62fUOh0PvvfdeXbZsWYf1tVdvQ0ODzzytafJ1jbOdW+PgwYMt0rA2Oneo3w17x9/eV+LAfpqRcVFHj67UoqIiPXToUKuNFTRDsXmzfvn888Epux1EgqEYP368vvbaa83SVq9erUuWLFFV1YyMDM3Ly2t2/mc/+5kOHz5cJ06cqN/5zneadapTpkzRU6dOtajH3VCoqk6bNk2ft7+DTz/9VKdOnarDhg3ToUOH6v33368XL1505T116pTeddddOnToUE1LS9OpU6fqkSNHWtSxYMECfeutt1yf680332z1cx04cEDz8vI0KytLs7Ky9MUXX3R1jh2p7+LFizp58mTNzMzUWbNm6fjx412Gor3t6NlJjx071qXT/bV169Zm+bZu3aqZmZmalZWlo0eP1t27d6uq6ty5czU9PV2XL1+uqqorVqxw1fu9732vmaF49dVX9eGHH+6UoVBVff311zU5OVmHDh2qK1asaHbO/Tdy6623ampqqo4YMcJluH1dv23bNgVcnzMrK0tff/31FvX722Zt1TdlyhQtLS3tsKbWrnG2c2t85Q3FkAEDNCOjTsePP6c7d+503WF5I2iGYvx4rb/lluCU3Q4iwVCotv8OtqqqSlVVa2pqdNSoUVpUVBRyDd4oLi7We+65J2QaOltfa+0YiLboKDNmzNCPP/44rBrciQQdwdDgbOfWCLSh6HJzFA4VVC97ji375CJVJ7/wmjcqyqyEikQWL17MwYMHuXjxIvPmzWtzJVGoyMnJYcKECTQ1NYVkb0Nn64u0dmxoaOBb3/oWKSkpNDY2hlXLlYx7O4cKsQxN12Fw/+u077XHSEur5p/+qZSeFweSeZtfobYDR14eDQ0NxGzfHtp6PSgpKWmxQzgcfNV8/keyhkjREQkaIkVHODQcOnSI1NTUZmkiUqSqfu6Oak6Xu+V2OLqhdhhUVSUquss9FBkMBkOXIqiGQkQmi8hhEflERB7xcj5XRIpF5JKIzPKnTIcKzs12AFFmo53BYDAElaDdjotIFPA0MBEr2t1uEdmsqu5OUD4F5gPL/S1X1flE0URMTAzSGIbAPS+8QGVZGQNCX7PBYDCEnGCO24wFPlHVYwAish6YDrgMhaqW2ucc3grwhkO7AUps7CV69OgB4ZgzGzIER0xMGCo2GAyG0BNMQzEY+Mzt/UmgpVMVPxCRxcBigD5xiVi7saFbt25UVl7gzJnQPlX02LQJ6us5M3t2SOv1RFUjYnWJw+EIuw6jIbJ0RIKGSNERDg1NTU1ePRZ3lGAaCm+9d4eWWKnqWmAtwNW9hqmqEhNTT+/evZE+wjXXXNMZne3n5ZetVU8PPhjaej0oKysL+4oO+OquLIlEDZGiIxI0RIqOcGiIiooKaL8YzJngk8AQt/cJwOnOFqp62SFgjx49OlucwWAwGNogmIZiN5AsIkkiEgPMATZ3tlBrjsLacBfuOwWDwWD4KhC0oSdVvSQi9wN/BaKAP6rqARH5OdZW8s0iMgbYCPQFvikij6lquq9yHQ7LUMTFOYyh8KCkpISampqAlRcfHx/wDX0LFixgy5YtDBw4kJKSklbzPf7447z00ktERUXRrVs3/vCHP5CSksJLL73kis3QWR599FF69erF8uV+L7rrUBmdrefrX/86231s7jx//nyLdmnrGoOhPQR1E4KqvqGqw1X1RlV93E77V1XdbB/vVtUEVY1X1X5tGQnrGmvoKTb2kjEUHtTU1BAfHx+wV3uNjj9hI+fPn8/WrVt95tmxYwdbtmyhuLiYffv28fbbbzNkyJB2heUMBqqKw+H3Ar2A0VaH761djJEwBJIut1vNuTy2d28JWazhFmzYwPlnnglP3V2c3NzcFgFtPPn888/p37+/aw6qf//+DBo0yGtYTmf40KysLFf40NLSUlJTU/n+979Peno6kyZNoq6uDmg7jKdnKFJnWT/84Q8ZOXIkn332mc8ynPjK8+KLLzJ27Fiys7P5wQ9+QFNTk8+Qp+7xVrxp9NYuzmt+85vfkJGRQUZGBr/97W9d5fhqI4OhBR31JhiuV0x0hqalXdB9+yzPqcGMOeGLSIxHUVhYqPv37w/Yq7Cw0C8dY8aM0aysLL3xxhu1b9++bbpkPn78uKanp7daXlVVlWZlZWlycrIuXbrUFcvB23UVFRWqqlpZWanp6elaXl6ux48f16ioKP3www9VVfWuu+7SF154Qffs2aMZGRlaU1OjlZWVeuONNzZzke0sq7a2tllZIqI7duxQVfVZhtNLqK88Bw8e1GnTprnyLl26VNetW6fFxcWam5vr0pKamqonTpxQ1eYu1lvT6N4uDQ0NGh8f79JRXV2tVVVVmpaWpsXFxa629NZGgSISvLaqRoaOcGj4ynuPBYiLu0TPnmFc8fTcc/SsrIRly8KnIYL44IMP6N69u19hI/2hV69eFBUVsW3bNvLz85k9ezarVq0iLy+vRV5n+FBVdYUPvfbaa0lKSnLFOR41ahSlpaWUl5f7DOPpLRTptddeS2JiIjfffDPQdijQtvK88847FBUVMWbMGMAK8Tpw4EC++93veg152trn9dTojffff58ZM2YQHx8PwMyZM9m2bRs5OTkAXtvIYPBGlzQUsbGXQhcC0BvPPUdsQ4MxFEEkKiqKvLw88vLyyMzMZN26dS0MhXv40O7duzNx4kRX+FD3pdNRUVGuYZXWwnj6CkXq7Gid+BMbu7U8qsq8efP4xS9+0eKct5Cn/mpsrS5ftNZGBoMnXW6OAqw9FGE1FDZRUVFkZ2e7XqWlpRQUFNCnTx9ycnJITU3lscceA2iWftNNN7V7BczWrVtJSUlh2LBhrFq1ymuexsZGysrKOH36dIs/fW1tLWfPnuXs2bPU1ta2uObs2bNUVla2swWak5eX1+mnCYDDhw9z9OhR1/u9e/eSmJjIVVdd1SzmcmVlJX379iUuLo6PP/6YwsJCn+Xm5uayceNG6urqqKqq4rXXXmt3Wb7K8CfP7bffzoYNGzh79ixgxdo+ceIEAHPmzGH9+vVs2LCBWbNa+shsTaNnu7jr2LRpE7W1tdTU1LBx40Zuu+02n21kMHijSz5RxMU10b17+A1FbGwse/fubZZWWlrKbbfdxpYtW6ipqSE7O5tp06YBuNLr6urIyclhxowZjBs3rs16mpqauO+++3jrrbdISEhgzJgxXoc8evXqRU1NDQ6Hg7q6OtcKHVXl/PnzXH311YDVOTkcDkSEyspK4uPjiY6O5sKFC6iqazWZ5510a4wbN46GhoYW6U888QR33HFHs7S5c+dSUFBAeXk5CQkJPPbYYyxcuLBZnurqah544AHOnz9PdHQ0w4YNY+3atfTr149x48aRkZHBlClTWLFiBWvWrGHEiBEkJye7hodaY+TIkcyePZvs7GwSExObdZqTJ092lZWSktJqWb7K8CdPWloaK1asYNKkSTgc1hLvp59+msTERNLT06mqqmLw4MFcd911LcptTaNnu6xcudKlY/78+YwdOxaARYsWuYadDIZ20dHJjXC9YqIz9O67P9Xq6mpVDdNkth0K1TOOs6pqfn6+fuMb33C9nz17tr7yyiutpvvD9u3bddKkSa73K1eu1JUrV7YaCvXYsWOuSU9VKxC7M26vqmppaamWl5drfX19szI88/nLV3XCMBI1qEaGjkjQoBoZOsxkdpjo1UsjYg9FXV2dazIwKSnJNcnopKKigsLCQn76059SVlbmSj937hxHjx4lNzcXgPz8fB566KEW5cfFxbF9+3ZOnTrFkCGXvaEkJCSwc+dOv3V6+prp3r07jY2NLdJjYmK8PhkYDIavNl3SUMTHh9lQvPEG586cITYzs8XQE+BaWdKtWzceeeQR0tPTKSgoYNu2bYwYMYLDhw/zyCOPuFarTJgwwWs5TtTLpKQ/E6oGg8EQCLqkoejdW8LbUcbFWa9WcM5FtJZ+5MgRbr31VmbMmEF2dnabTxQJCQl89tllj+0nT55k0KBBgGVE2mqL7t27U11d7Xrf2NhIr169XE8WThoaGogxcTYMhi6NtxvLztIFDYVA7SWO7P4CgKioMCzc+v3vib1wocOXDx8+nB/96Ec88cQTvPzyy20+UYwZM4ajR49y/PhxBg8ezPr1611+kCoqKujXr59PY9GnTx9OnTrFpUuXAGv1zODBg4mOjqZbt25UV1cTHx9PRUVF6F22GwyGgKGqVFRUBHxVaBc0FJCUHsfwMd43GQWN/HzIywMR+POf6ek+lq8KBQUwYYLfxS1ZsoRf//rXHD9+nKSkJJ95o6Ojeeqpp7jjjjtoampiwYIFpKenc/r0afbs2cOAAQOIi4ujvr6esrIy14qmqKgo15NHfX09BQUFgGU4nMtPGxoa2LlzJ6pKbGwsTU1NnD7dPm/wTU1N4XOnYjREpI5I0BApOkKtoWfPniQkJAS20I7OgofrFROdqX/5S1lHFgJ0nHffVQXVZctUHQ7XqidVtd4vW2adf/fdkMqKBDciqpGhw2i4TCToiAQNqpGhIxI0qEbwqicRmQysxnIz/oyqrvI43wN4HhgFVACz1Y6j7YuvfS3EE9l5edYu7NWrm6erwkMPWenLlln5DAaD4QojaIZCRKKAp4GJWNHudovIZlU96JZtIXBOVYeJyBzgCaDNQNRXXx3iR0kRePJJ63j1ahg8GK6/vrmRePJJK5/BYDBcYQTziWIs8ImqHgMQkfXAdMDdUEwHHrWPNwBPiYjYj0mt0rdvGKZWPI2FKuzYYYyEwWC44glmjzsY+Mzt/Ungb1rLo1ZEvEqgH1DunklEFgOL7ePGhITY/UDg14D5ySgYVXb6NAOAotWri1oMSYWO/ni0VZiIBB1Gw2UiQUckaIDI0BEJGgBSOnphMA2Ft1tsz87dnzyo6lpgLYCI7FHV0Z2X1zlEZM+JMOuIpLYItw6jIbJ0RIKGSNERCRqcOjp6bTA3IZwEhri9TwA811268ohINNAH+DKImgwGg8HQToJpKHYDySKSJCIxwBxgs0eezcA8+3gW8G5b8xMGg8FgCC1BG3qy5xzuB/6KtTz2j6p6QER+jrWedzPwLPCCiHyC9STRMlpLS9YGS3M7iQQdkaABIkOH0XCZSNARCRogMnREggbohA4xN/AGg8Fg8EWXjHBnMBgMhtBhDIXBYDAYfBKxhkJEJovIYRH5REQe8XK+h4i8Yp/fKSI3hEFDrogUi8glEWkZ5Dh0Oh4WkYMisk9E3hGRxDBoWCIi+0Vkr4i8LyJpgdbgjw63fLNEREUk4MsS/WiL+SJSZrfFXhFZFGgN/uiw83zb/m0cEJGXQq1BRJ50a4cjInI+0Br81HG9iOSLyIf2/2RqGDQk2v/PfSJSICIB9twHIvJHETkrIiWtnBcR+Z2tcZ+IjPSr4I46iQrmC2vy+/+AoUAM8BGQ5pHnh8Aa+3gO8EoYNNwAjMDyVzUrjG0xAYizj5eGqS16ux3/PbA1HG1h57sKeA8oBEaHoS3mA08F4/fQTh3JwIdAX/v9wHB8H275H8Ba1BKOtlgLLLWP04DSMGh4FZhnH/8t8EIQ2iIXGAmUtHJ+KvAm1h62m4Gd/pQbqU8ULvcfqtoAON1/uDMdWGcfbwBuFwmoH402NahqqaruAxwBrLcjOvJVtdZ+W4i1ZyXUGtwDdMQTnJ3z/vwuAP4N+CVwMYwago0/Or4PPK2q5wBU9WwYNLgzF3g5wBr81aFAb/u4Dy33dIVCQxrwjn2c7+V8p1HV9/C9F2068LxaFAJXi8h1bZUbqYbCm/uPwa3lUdVLgNP9Ryg1hIL26liIdccQcg0icp+I/B9WJ/1ggDX4pUNEcoAhqtoyxGCINNjcaT/abxCRIV7Oh0LHcGC4iHwgIoVieXMOtQbAGnYBkoB3A6zBXx2PAveIyEngDaynm1Br+Ai40z6eAVwlIoHss/yhQ/1apBqKgLn/CLKGUOC3DhG5BxgN/CocGlT1aVW9Efhn4F8CrKFNHSLSDXgS+Mcg1O2XBpvXgBtUdQTwNpeffEOtIxpr+CkP627+GRG5OsQanMwBNqhqUwDrb4+OucBzqpqANfzygv17CaWG5cB4EfkQGA+cAi4FUIM/dKhfi1RDEQnuP/zREAr80iEifwf8BPh7Va0PhwY31gPfCrAGf3RcBWQABSJSijUGuznAE9pttoWqVrh9B/+FFW8l0Pj7H/mLqjaq6nHgMJbhCKUGJ3MIzrCTvzoWAn8GUNUdQE8sZ30h06Cqp1V1pqrmYP1XUdXKAGrwh471a4GeTAnQhEw0cAzrUdU5MZTukec+mk9m/znUGtzyPkfwJrP9aYscrIm05DBqSHY7/iadiKYViO/Ezl9A4Cez/WmL69yOZwCFYfpOJgPr7OP+WEMO/UL9fWB5LS3F3uAbprZ4E5hvH6didY4B0+Onhv5AN/v4ceDnQWqPG2h9MvsbNJ/M3uVXmcEQGqAPOxU4YneAP7HTfo51xwzWHcGrwCfALmBoGDSMwbLQNVgR+g6EqS3eBs4Ae+3X5jBoWA0csOvP99ZhhEKHR94CAmwo/GyLX9ht8ZHdFjeF6XchwG+wYsDsB+aE4/vAmh9YFYw2aEdbpAEf2N/JXmBSGDTMAo7aeZ4BegRBw8vA50Cj3TctBJYAS9x+E0/bGvf7+/8wLjwMBoPB4JNInaMwGAwGQ4RgDIXBYDAYfGIMhcFgMBh8YgyFwWAwGHxiDIXBYDAYfGIMhSFoiEiTm+fQvb48/IrIDa15vGxnnQW2B8+PbNcVKR0oY4mIfNc+ni8ig9zOPRMIz7geOneLSLYf1/yDiMR1oK7fikiuR73O72SWne78rkpE5FVnPR7przl3dovIABHZ2l4thq6JMRSGYFKnqtlur9IQ1Xu3qmZhuc5otzsTVV2jqs/bb+cDg9zOLVLVgwFReVnn7/FP5z8A7TIUIvI14Ga1nMW51+v8TjbYac7vKgNowFp775n+JdZGV1S1DPhcRMa1R4+ha2IMhSGk2E8O28SK41EsIl/3kiddRHbZd7L7RCTZTr/HLf0PIhLVRnXvAcPsa2+3YxHst33297DTV8nlWB6/ttMeFZHl9t32aOBPdp2x9h35aBFZKiK/dNM8X0T+o4M6d+DmmE1E/lNE9ogVQ+IxO+1BLIOVLyL5dtokEdlht+OrItLLS9mzgPbe+W9ztpsvncAm4O52lm3oghhDYQgmsW5DHBvttLPARFUdCcwGfufluiXAalXNxuqoT4pIqp1/nJ3eRNud1DeB/SLSE8vNymxVzcRyt7DUvtuegbWLfASwwv1i+257D5fvwOvcTm8AZrq9nw280kGdk7E6XSc/UdXRWLFOxovICFX9HZbbiQmqOkFE+mM5Xvw7uy33AA97KXscUOSR9ie376WZ91Lbb9oUrF277ulRwO3AZrfkPcBtbXw2wxVAdLgFGK5o6uzO0p3uwFP2mHwTlitsT3YAPxErAth/q+pREbkdy7nebrHCjsRiGR1v/ElE6rD8Cz2A5WvouKoesc+vwxpCeQorZsUzIvI64LdrclUtE5FjInIzlluGFCwXEfe1U2c8VtAb90hj3xaRxVj/z+uw3E/s87j2Zjv9A7ueGKx28+Q6oMwj7W5V3eORFisie+3jbcCzHuk3YBmct9yuOYvbsJzhysUYCkOoeQjLL1UW1hNti+BCqvqSiOzEcmD2V7FCiQqWg7sf+VFHs47Q867ZrZ5LIjIW6055DnA/VuQxf3kF+DbwMbBRVVWsXttvnVi+h1Zh+d+ZKSJJWO6ox6jqORF5DsuvmScCvKWqc9uoo66V61vk82LUXeki0gfLkN7H5afAnnb5hiscM/RkCDV9gM9V1QHci3U33QwRGQocs4dbNmMNwbwDzBKRgXaer4n/scE/Bm4QEee4+73A/9pj+n1U9Q2siWJvHWUVlvtyb/w3ljv1uVhGg/bqVNVGrCGkm+1hq95YTiYrReQarGEgb1oKgXHOzyQicSLi7ensEN7nG9qFWu6wHwSWi0h3O3k40OmVaobIxxgKQ6j5PTBPRAqxOpoaL3lmAyX2kMdNWKEbD2J1qP8jIvuwhkDaDOEIoKoXge8Br4rIfqzQtWuwOt0tdnn/i/W048lzwBrnZLZHueewPLMmquouO63dOu25j38HlqvqR1hxrg8Af8QaznKyFnhTRPLtVUfzgZftegqx2sqT17ECF3UaVf0Q6wlojp00wS7fcIVjvMcaDFc4IvI+ME1Vzwe43PeA6bbBNFzBGENhMFzhiMjfYM01eE6Id6bMAVgruza1mdnQ5TGGwmAwGAw+MXMUBoPBYPCJMRQGg8Fg8IkxFAaDwWDwiTEUBoPBYPCJMRQGg8Fg8Mn/A00sAdkufsXrAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "---------------------Average---------------------\n",
      "AUC (Avg. +/- Std.) is  0.902 +/- 0.020\n",
      "Accuracy (Avg. +/- Std.) is  0.855 +/- 0.017\n",
      "Avg. CM is [[30, 8], [9, 78]]\n",
      "Total for all folds CM is [[150, 44], [48, 394]]\n",
      "Sensitivity (Avg. +/- Std.) is  0.757 +/- 0.059\n",
      "Specificity (Avg. +/- Std.) is  0.900 +/- 0.045\n",
      "Precision (Avg. +/- Std.) is  0.782 +/- 0.062\n",
      "FOR (Avg. +/- Std.) is  0.107 +/- 0.022\n",
      "DOR (Avg. +/- Std.) is  32.748 +/- 7.570\n"
     ]
    }
   ],
   "source": [
    "i=0                                                                          # for verification of fold number \n",
    "Accuracy = []                                                                # for store the value of accuracy \n",
    "FP = []                                                                      # for store False Positive \n",
    "TN = []                                                                      # for True Negative\n",
    "FN = []                                                                      # for False Negative\n",
    "TP = []                                                                      # for True Positive\n",
    "tprs = []                                                                    # for true positive rates\n",
    "aucs_ens = []                                                                # for store the values of auc\n",
    "sn = []                                                                      # for sensitivity \n",
    "sp = []                                                                      # for specificity\n",
    "pr = []                                                                      # for precision\n",
    "FOR = []                                                                     # for False omission rate \n",
    "DOR = [] \n",
    "iterator=0\n",
    "mean_fpr = np.linspace(0, 1, 100)\n",
    "\n",
    "\n",
    "#########################################################\n",
    "\n",
    "##      call grid  search for optimization       ##\n",
    "\n",
    "# grid_results = grid.fit(X_Data, to_categorical(Y_Lavel,2))\n",
    "   \n",
    "##            tuned/optimized parameters        ##\n",
    "\n",
    "# activation=grid_results.best_params_['activation']\n",
    "# batch_size=grid_results.best_params_['batch_size']\n",
    "# epochs=grid_results.best_params_['epochs']\n",
    "# learn_rate=grid_results.best_params_['learn_rate']\n",
    "# dropout_rate=grid_results.best_params_['dropout_rate']\n",
    "# init=grid_results.best_params_['init']\n",
    "\n",
    "##########################################################\n",
    "  \n",
    "    \n",
    "##########################################################\n",
    "##     best optimizers found  are defined here by experiment ##\n",
    "activation=\"relu\"\n",
    "batch_size=8\n",
    "epochs=200\n",
    "learn_rate=.001\n",
    "dropout_rate=0.6\n",
    "init=\"normal\"\n",
    "neuron1,neuron2,neuron3,neuron4=64,16,64,64\n",
    "############################################################\n",
    "print(activation,batch_size,epochs,learn_rate,dropout_rate,init,\n",
    "    neuron1,neuron2,neuron3,neuron4)\n",
    "\n",
    "\n",
    "\n",
    "for train_index, test_index in kf.split(X_Data,Y_Lavel):                  # for k fold experiment \n",
    "  print('------------------->>>>>>>>>>Fold no = ',i+1)\n",
    "  X_Train, X_Test = X_Data[train_index], X_Data[test_index]               # the train data and label\n",
    "  Y_Train, Y_Test = Y_Lavel[train_index], Y_Lavel[test_index]             # the  test data and label\n",
    "\n",
    "\n",
    "  Y_Train_1Hot = to_categorical(Y_Train,2)                                #convert train output to catagorical\n",
    "  Y_Test_1Hot = to_categorical(Y_Test,2)                                  #convert test output to catagorical\n",
    "\n",
    "  model =nn_opt(activation,                                               #build model using tuned parameters\n",
    "            dropout_rate,\n",
    "            init,\n",
    "            learn_rate)    \n",
    "  np.random.seed(6)\n",
    "  model.fit(x=X_Train,                                                     # fit our model \n",
    "            y=Y_Train_1Hot,\n",
    "            batch_size=batch_size,\n",
    "            epochs=epochs,\n",
    "            shuffle=False,\n",
    "            verbose=1)\n",
    "    \n",
    "  probas_ = model.predict(X_Test)                                           #predict the class probability\n",
    "\n",
    "  y_pred = np.argmax(model.predict(X_Test), axis=1)                         #find the max. probability of output class\n",
    "\n",
    "\n",
    "  tn, fp, fn, tp, roc_auc, fpr, tpr = metrics (y_true = Y_Test,           #evaluation parameters of ensembelled model \n",
    "                                              y_pred = y_pred,\n",
    "                                              probas_ = probas_)\n",
    "  tprs.append(interp(mean_fpr, fpr, tpr))\n",
    "  tprs[-1][0] = 0.0\n",
    "  aucs_ens.append(roc_auc)\n",
    "  plot_Current_ROC (fpr,tpr,iterator,roc_auc)                             #plot the ROC curve of current fold\n",
    "  iterator += 1\n",
    "  TN.append(tn)\n",
    "  FP.append(fp)\n",
    "  FN.append(fn)\n",
    "  TP.append(tp)\n",
    "  Accuracy.append(accuracy_score(Y_Test, y_pred))\n",
    "  sn.append(tp/(tp+fn))\n",
    "  sp.append(tn/(fp+tn))\n",
    "  pr.append(tp/(tp+fp))\n",
    "  FOR.append(fn/(tn+fn))\n",
    "  DOR.append((tp*tn)/(fp*fn))\n",
    "  print((tp*tn)/(fp*fn))\n",
    "  i+=1\n",
    "\n",
    "average_ROC(mean_fpr,tprs,aucs_ens,TP,TN,FP,FN)                             #plot average ROC\n",
    "average_performance(aucs_ens,Accuracy,TP,TN,FP,FN)                          #print the average performance  \n",
    "print(\"Sensitivity (Avg. +/- Std.) is  %0.3f +/- %0.3f\" %(np.mean(sn),np.std(sn)))\n",
    "print(\"Specificity (Avg. +/- Std.) is  %0.3f +/- %0.3f\" %(np.mean(sp),np.std(sp)))\n",
    "print(\"Precision (Avg. +/- Std.) is  %0.3f +/- %0.3f\" %(np.mean(pr),np.std(pr)))\n",
    "print(\"FOR (Avg. +/- Std.) is  %0.3f +/- %0.3f\" %(np.mean(FOR),np.std(FOR)))\n",
    "print(\"DOR (Avg. +/- Std.) is  %0.3f +/- %0.3f\" %(np.mean(DOR),np.std(DOR)))"
   ]
  }
 ],
 "metadata": {
  "_change_revision": 0,
  "_is_fork": false,
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Final_DNN.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}