DrivenData - Genetic Engineering Attribution (r4).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DrivenData - Genetic Engineering Attribution Challenge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 01-05-2021, additional verification\n",
    "+ tried Baysian Gaussian Mixture for categorizing and additional feature creation. (not effective. may have to create at least 1314 categories to make it work?)\n",
    "+ trained robust scaler only on training data, not for both as previously.\n",
    "+ removed header and footer features since it has not provide major contribution.\n",
    "+ model architecture:\n",
    "    + tried no hidden layers for keras. less time for each epoch but not as effective. take more than 30 epoch to reach 86% and seems it stoped at 88.8% (after 92 epoch)\n",
    "    + 3 layers structure is working for now.\n",
    "    + 4 layers takes longer to train and less effective."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 01-04-2021, additional tryings\n",
    "+ tried to split the headers into cateogorical columns, not effective and takes a great deal of time running\n",
    "+ found out missed out header and footer while running yesterday. added back in and result is as previous, between 89 ~ 91%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 01-03-2021, additional improvement\n",
    "+ Oversampling with RandomResampling and SMOTE\n",
    "+ test undersampling (not successful)\n",
    "+ implement the ability to show iterative progress of function\n",
    "+ impute N in the DNA sequence (for now, ignore N in the DNA sequence completely.)\n",
    "    - Idea 1 during the n-gram feature generation phase, add the N related feature occurance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 01-02-2021, Neuron network implemented\n",
    "+ tested saving test data. not a great idea.\n",
    "+ save for neuron network model.\n",
    "+ header feature changed\n",
    "+ added feature for sequence length logic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 12-19-2020, Group Hackthon collaborated efforts with RFC, Native Bayes,\n",
    "+ RBF and Neuron Network In progress (session 420 and 430)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 12-10-2020, Vanessa added dataset links "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 12-07-2020, Jim set up Benchmark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### future work\n",
    "+ implement the ability to auto generate and eliminate features\n",
    "+ figure out how to premutate more than 5 n-gram (maybe write one by myself?)\n",
    "+ output spec evaluation\n",
    "+ maybe generate a combined feature based from header and length of the sequence?\n",
    "\n",
    "#### drop\n",
    "\n",
    "+ implement test out RBF. (tried, not working, require way to much of memory for the data after the imputation.\n",
    "    - use guassian process for RBF?\n",
    "+ undersampling try Wilson's Editing approach (failed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Libraries Loaded\n"
     ]
    }
   ],
   "source": [
    "### Import Libraries\n",
    "\n",
    "from pathlib import Path\n",
    "\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "\n",
    "%matplotlib inline\n",
    "sns.set()\n",
    "\n",
    "print(\"Libraries Loaded\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Auxiliary libraries\n",
    "\n",
    "import winsound as alarm \n",
    "import time\n",
    "\n",
    "### Auxiliary parameter setup\n",
    "\n",
    "duration = 1500  # milliseconds\n",
    "freq = 440  # Hz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 100 Read Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Following code for reading off of JMO Harddrive\n",
    "\n",
    "#DATA_DIR = 'C:/Users/jorrison/Documents/Home/Harvard/DataMinds/Genetic Engineering/Data/'\n",
    "\n",
    "#train_values = pd.read_csv(DATA_DIR + 'train_values.csv', index_col='sequence_id')\n",
    "#train_labels = pd.read_csv(DATA_DIR + 'train_labels.csv', index_col='sequence_id')\n",
    "#test_values  = pd.read_csv(DATA_DIR + 'test_values.csv' , index_col='sequence_id')\n",
    "\n",
    "#train_values.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Read local by roland\n",
    "\n",
    "train_labels = pd.read_csv('train_labels.csv', index_col='sequence_id')\n",
    "train_values = pd.read_csv('train_values.csv', index_col='sequence_id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "### remote read\n",
    "\n",
    "#train_values = pd.read_csv('https://www.dropbox.com/s/3atnfvr65oh0y4y/train_values.csv?dl=1', index_col='sequence_id')\n",
    "#train_labels = pd.read_csv('https://www.dropbox.com/s/71yg5f3dbc4h2yd/train_labels.csv?dl=1', index_col='sequence_id')\n",
    "#test_values  = pd.read_csv('https://www.dropbox.com/s/4g1astwaccer5lz/test_values.csv?dl=1' , index_col='sequence_id')\n",
    "\n",
    "# train_values.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 110 Explore the Training Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "train_values.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 120 remove the lab has the highest frequency occurance in dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "largelab_rownum = train_labels[ train_labels['I7FXTVDP'] > 0 ].index\n",
    "\n",
    "largelab_rownum.shape\n",
    "\n",
    "largelab_rownum\n",
    "\n",
    "train_labels_dropped = train_labels.drop(largelab_rownum)\n",
    "\n",
    "train_values_dropped = train_values.drop(largelab_rownum)\n",
    "\n",
    "train_labels_dropped.shape\n",
    "\n",
    "train_values_dropped.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 130 reduce the output labels dimension to string label text.\n",
    "\n",
    "#lab_ids = pd.DataFrame(train_labels.idxmax(axis=1), columns = ['lab_id'])\n",
    "\n",
    "lab_ids = pd.DataFrame(train_labels_dropped.idxmax(axis=1), columns = ['lab_id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    63017.000000\n",
       "mean      4839.025501\n",
       "std       3883.148431\n",
       "min         20.000000\n",
       "25%        909.000000\n",
       "50%       4741.000000\n",
       "75%       7490.000000\n",
       "max      60099.000000\n",
       "Name: sequence, dtype: float64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequence_lengths = train_values.sequence.apply(len)\n",
    "\n",
    "sequence_lengths.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 140 visualize number of sequence code with length of 2500 or less.\n",
    "sequence_lengths.plot(kind='hist',\n",
    "                     title= 'Distribution of DNA Sequence Length', \n",
    "                     bins = 2500,\n",
    "                     xlim = (0, 2500))   # Suppress a bar longer than 20,000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 150 visualize number of genetic sequence code with length of 1700 or less\n",
    "sequence_lengths[sequence_lengths<1700].plot(kind='hist',\n",
    "                     title= 'Distribution of DNA Sequence Length', \n",
    "                     bins = 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 160 extrapolate data with genetic sequence length 1700 or less\n",
    "\n",
    "index_ferdous = sequence_lengths[sequence_lengths<1700].index\n",
    "\n",
    "df_ferdous_values = train_values_dropped.loc[index_ferdous]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 170 visualize the value occurance of other 39 features contained in training values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sorted_binary_features = train_values.iloc[:, 1:].mean().sort_values()\n",
    "\n",
    "ax = sorted_binary_features.plot(kind = 'barh',\n",
    "                                stacked = True,\n",
    "                                figsize = (5,12),\n",
    "                                title   = 'Prevalence of Binary Features')\n",
    "ax.set_xlabel('Proportion of sequences')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 175 inspect the output lab occurrance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "train_labels.sum().describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 180 Explore the Test Data\n",
    "\n",
    "skipped, since we can't check the test data result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 190 Explore Training Labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": true
   },
   "source": [
    "train_labels.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Get the column with the max value in each row\n",
    "\n",
    "lab_ids = pd.DataFrame(train_labels.idxmax(axis=1), columns = ['lab_id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    1314.000000\n",
       "mean       47.958143\n",
       "std       262.552258\n",
       "min         1.000000\n",
       "25%         9.000000\n",
       "50%        15.000000\n",
       "75%        34.000000\n",
       "max      8286.000000\n",
       "Name: lab_id, dtype: float64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "##### Get the distribution of lab prevalence in the training set\n",
    "lab_ids['lab_id'].value_counts().describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "I7FXTVDP    0.131488\n",
       "RKJHZGDQ    0.043353\n",
       "GTVTUGVY    0.042401\n",
       "A18S09P2    0.016884\n",
       "Q2K8NHZY    0.015440\n",
       "Name: lab_id, dtype: float64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#### Sort lab ids by prevalence\n",
    "(lab_ids['lab_id'].value_counts(normalize = True).sort_values(ascending=False)).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 200 Enrich Data Sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "from itertools import permutations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 210 Construct Features from DNA Sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 211 create bases from DNA Sequences. (got lazy so I just put bases as values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5 element bases\n",
    "bases = set(''.join(train_values.sequence.values))\n",
    "\n",
    "bases5 ={'A', 'C', 'G', 'N', 'T'}\n",
    "bases5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'A', 'C', 'G', 'T'}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 4 element bases\n",
    "\n",
    "bases4 ={'A', 'C', 'G', 'T'}\n",
    "bases4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 212 Reduced Permutation to test out the effect. seems the 2-gram approach is not as good as 4 grams\n",
    "n = 2\n",
    "subsequences = [''.join(permutation) for permutation in permutations(bases, r=n)]\n",
    "print(f\"Number of subsequences: {len(subsequences)}\")\n",
    "subsequences[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 212 Example of built-in count method on strings\n",
    "Because it's non-overlapping, \"atta\" is only counted twice\n",
    "\"gattattattaca\".count(\"atta\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 212 Counting the Subsequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1  gram variation count:  4\n",
      "2  gram variation count:  12\n",
      "3  gram variation count:  24\n",
      "4  gram variation count:  24\n",
      "n gram permutation time:  0.00022859999999980118\n",
      " |██████████████████████████████████████████████████| 100.0% \n"
     ]
    }
   ],
   "source": [
    "# Calculate 4 base features on our training set (ferdous training set)\n",
    "features_4b = get_nbase_features(train_values, bases4)\n",
    "\n",
    "alarm.Beep(freq,duration)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ Calculate 5 base features on our training set (ferdous training set)\n",
    "+ currently not as effective as bases4\n",
    "\n",
    "features_5b = get_nbase_features(train_values, bases5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 221 Header and Footer ordinate and binary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### 221 Right footer amino stop sequence extraction function\n",
    "\n",
    "rightAMINO=lambda x: x[-3:]\n",
    "\n",
    "#### 222 Left header 5 prime UTR extraction\n",
    "\n",
    "fivePUTR=lambda x: x[0:15]\n",
    "\n",
    "#### 223 extract both footer and header for value validation and evaluation\n",
    "\n",
    "footer = train_values.sequence.apply(rightAMINO)\n",
    "header = train_values.sequence.apply(fivePUTR)\n",
    "\n",
    "#### 225 use label encoder to transfer all the categorical variables\n",
    "####     into numerical variables\n",
    "\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "headerEnc = LabelEncoder()\n",
    "header_code = headerEnc.fit_transform(header)\n",
    "#all_features[\"header\"]=pd.Series(header_code,index = train_values.index)\n",
    "\n",
    "footerEnc = LabelEncoder()\n",
    "footer_code = footerEnc.fit_transform(footer)\n",
    "#all_features[\"footer\"]=pd.Series(footer_code,index = train_values.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [],
   "source": [
    "header_bin = ordin2Bin(pd.Series(header_code,index = train_values.index),\"header\")\n",
    "footer_bin = ordin2Bin(pd.Series(footer_code,index = train_values.index),\"footer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 230 add binary feature for sequence length below 1750 and above 1750"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [],
   "source": [
    "bins = [0,1750,np.inf]\n",
    "labels = [0,1]\n",
    "\n",
    "len_bin = pd.cut(sequence_lengths,bins=bins,labels=labels)\n",
    "len_bin.columns = \"length seperation\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 240 all features combined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_features=train_values.drop(\"sequence\",axis=1).join([len_bin,features_4b])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### 300 Split and scale"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 310 Break apart Training Data Set into Train-Validate-Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 311 split the data into training and validation sets. for multiple columns binary output split."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": true
   },
   "source": [
    "X,X_validate,y_lab,y_lab_validate = randomSplit_lab(all_features,lab_ids,TEST_SIZE=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [],
   "source": [
    "X,X_validate,y_cat,y_cat_validate = train_test_split(all_features,train_labels,test_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 312 save the split into files so we don't have to split the file but just load the files.\n",
    "\n",
    "start = time.perf_counter()\n",
    "\n",
    "X_bin.to_csv(\"X_bin-2021-01-02.csv\")\n",
    "X_bin_validate.to_csv(\"X_bin_validate-2021-01-02.csv\")\n",
    "y_bin.to_csv(\"y_bin-2021-01-02.csv\")\n",
    "y_bin_validate.to_csv(\"y_bin_validate-2021-01-02.csv\")\n",
    "end   = time.perf_counter()\n",
    "\n",
    "print(end-start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 320 Oversampling minorities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "from imblearn.over_sampling import SMOTE, BorderlineSMOTE, KMeansSMOTE,ADASYN\n",
    "from imblearn.over_sampling import RandomOverSampler\n",
    "import math"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 321 Random over sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "y_lab_counts = pd.DataFrame(y_lab.value_counts(),columns=[\"n_train\"])\n",
    "y_lab_counts[\"n_adjusted\"] = (y_lab_counts[\"n_train\"].apply(math.log)+1)*7\n",
    "y_lab_counts[\"n_max\"]=y_lab_counts.max(axis=1)\n",
    "\n",
    "df_sampling = y_lab_counts[\"n_max\"].astype(\"int32\")\n",
    "try:\n",
    "    index = [i[0] for i in list(df_sampling.keys())]\n",
    "    values = list(df_sampling.values)\n",
    "    \n",
    "    sampling_strategy = pd.Series(values,index = index).to_dict()\n",
    "except:\n",
    "    sampling_strategy = y_lab_counts[\"n_max\"].astype(\"int32\").to_dict()\n",
    "\n",
    "\n",
    "#sampling_strategy = sampling_strategy.to_dict()\n",
    "\n",
    "if \"df_top10_categorical\" in __name__:\n",
    "    print(\"yes\")\n",
    "else:\n",
    "    print(\"no\")\n",
    "    #sampling_strategy = (2-(df_top10_categorical+1))*sampling_stragegy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### confusion matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sampler = RandomOverSampler(sampling_strategy= sampling_strategy)\n",
    "\n",
    "X_R_res,y_R_lab_res = sampler.fit_resample(X,y_lab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 322 SMOTE over sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sm = SMOTE(sampling_strategy='minority',n_jobs = 4)\n",
    "X_res,y_lab_res=sm.fit_resample(X_R_res,y_R_lab_res)\n",
    "\n",
    "alarm.Beep(freq,duration)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 330 undersampling attempts with (apparently only if I include not minority, there will be undersampling and it resulted a few lab samples were remove from the records.)\n",
    "\n",
    "+ NeighbourhoodCleaningRule\n",
    "+ AllKNN\n",
    "+ EditedNearestNeighbours"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 331 NeighbourhoodCleaningRule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "from imblearn.under_sampling import NeighbourhoodCleaningRule\n",
    "\n",
    "ncr = NeighbourhoodCleaningRule(sampling_strategy=\"majority\",n_jobs=4)\n",
    "X_resampled, y_resampled = ncr.fit_resample(X_res.to_numpy(), y_cat_res.to_numpy())\n",
    "\n",
    "alarm.Beep(freq,duration)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 332 EditedNearestNeighbours"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "from imblearn.under_sampling import EditedNearestNeighbours\n",
    "\n",
    "ENN = EditedNearestNeighbours(n_jobs=4)\n",
    "X_resampled, y_resampled = ENN.fit_resample(X_res.to_numpy(), y_cat_res.to_numpy())\n",
    "\n",
    "alarm.Beep(freq,duration)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 340 Create Baysian Gaussian Mixture features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "X_res,BGM = BGMfeatureGen(X_res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 380 scale all the features across the columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler,RobustScaler,MinMaxScaler\n",
    "\n",
    "default_scaler = RobustScaler(with_centering=False)\n",
    "#default_scaler = MinMaxScaler()\n",
    "default_scaler.fit(X)\n",
    "X_scaled = pd.DataFrame(default_scaler.transform(X),columns = X.columns,index = X.index)\n",
    "X_scaled_validate = pd.DataFrame(default_scaler.transform(X_validate),columns = X_validate.columns,index = X_validate.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 390 create categorical output set for neuron network to train on"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "y_cat_res = pd.get_dummies(y_lab_res)\n",
    "y_cat_validate = pd.get_dummies(y_lab_validate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 391 a quick validation to check both the train and validate have covered\n",
    "the whole spectrum of the output lab labels. for categorical output only\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No of records in training set:  50413\n",
      "No of records in validate set:  12604\n",
      "Labs have 0 record in training set: Series([], dtype: float64)\n",
      "Labs have 0 record in validate set: 03GRNN7N    0.0\n",
      "0L3Y6ZB2    0.0\n",
      "13LZE1F7    0.0\n",
      "1EDZ6CA7    0.0\n",
      "1KZHNVYR    0.0\n",
      "           ... \n",
      "XU8GASLQ    0.0\n",
      "Y5YH740Y    0.0\n",
      "YY5Y32CI    0.0\n",
      "Z8BWVZZX    0.0\n",
      "ZB6DPIG5    0.0\n",
      "Length: 119, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "print('No of records in training set: ',y_cat.shape[0])\n",
    "print('No of records in validate set: ',y_cat_validate.shape[0])\n",
    "\n",
    "print('Labs have 0 record in training set: '+str(y_cat.sum()[y_cat.sum()==0]))\n",
    "print('Labs have 0 record in validate set: '+str(y_cat_validate.sum()[y_cat_validate.sum()==0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 400 ML Model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "featureLS = list(X_scaled.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 410 Random Forest (functional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "rf = RandomForestClassifier(n_estimators =400,\n",
    "                            class_weight=\"balanced\",\n",
    "                            max_depth =10,\n",
    "                            bootstrap =False,\n",
    "                            n_jobs = 5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### fit our model\n",
    "start = time.perf_counter()\n",
    "rf.fit(X_res,y_lab_res)\n",
    "end   = time.perf_counter()\n",
    "\n",
    "print(end-start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### validation evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "start = time.perf_counter()\n",
    "\n",
    "df_rf_predict= rf.predict(X_validate)\n",
    "top10_result=top10_accuracy_label(rf, X_validate, y_lab_validate.to_numpy())\n",
    "\n",
    "end   = time.perf_counter()\n",
    "\n",
    "print(\"Top 10 value accuracy:\"+str(top10_result))\n",
    "print(\"Top 10 value process time:\"+str(end-start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### just a note\n",
    "\n",
    "+ changing the estimator number to 400 improved the result significantly\n",
    "+ increase the depth to 10 improved the result.\n",
    "+ scaling the features improve the result by 4%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 420 Keras neuro-network (not functional yet, use with caution.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential,load_model\n",
    "from keras.layers import Dense\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from keras.metrics import BinaryAccuracy\n",
    "from keras.metrics import top_k_categorical_accuracy\n",
    "import functools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 421 Load previously saved Neuron Network\n",
    "\n",
    "keras_nn = load_model(\"3l_nnet_2021-01-05-1300.nn\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 422 top 10 categorical accuracy metrics function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [],
   "source": [
    "top10_acc = functools.partial(top_k_categorical_accuracy, k=10)\n",
    "top10_acc.__name__ = 'top10_acc'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'X_scaled' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-1-fd5293bdc7f6>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0minput_dim\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mX_scaled\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[0moutput_dim\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0my_cat\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[0mnnet\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mSequential\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[0mnnet\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0madd\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mDense\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minput_dim\u001b[0m\u001b[1;33m+\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mactivation\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;34m\"tanh\"\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0minput_dim\u001b[0m \u001b[1;33m=\u001b[0m\u001b[0minput_dim\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'X_scaled' is not defined"
     ]
    }
   ],
   "source": [
    "input_dim = scaled.shape[1]\n",
    "output_dim = y_cat.shape[1]\n",
    "\n",
    "nnet = Sequential()\n",
    "nnet.add(Dense(input_dim+1,activation = \"tanh\",input_dim =input_dim))\n",
    "nnet.add(Dense(output_dim+1, activation='relu'))\n",
    "nnet.add(Dense(output_dim, activation='softmax'))\n",
    "# Compile model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 422 use weighted categorical crossentropy loss function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "weights = df_top10_weight\n",
    "\n",
    "mohammed.compile(loss=weighted_categorical_crossentropy(weights), \n",
    "                 optimizer='adam',\n",
    "                 metrics=[top10_acc])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [],
   "source": [
    "nnet.compile(loss=\"categorical_crossentropy\", \n",
    "                 optimizer='adam',\n",
    "                 metrics=[top10_acc])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/7\n",
      "1576/1576 [==============================] - 30s 19ms/step - loss: 3.2245 - top10_acc: 0.6841\n",
      "Epoch 2/7\n",
      "1576/1576 [==============================] - 30s 19ms/step - loss: 1.8822 - top10_acc: 0.8541\n",
      "Epoch 3/7\n",
      "1576/1576 [==============================] - 30s 19ms/step - loss: 1.4761 - top10_acc: 0.9016 2s - loss - ETA: 0s - loss: 1.4765 - top10\n",
      "Epoch 4/7\n",
      "1576/1576 [==============================] - 31s 20ms/step - loss: 1.2122 - top10_acc: 0.9317\n",
      "Epoch 5/7\n",
      "1576/1576 [==============================] - 32s 20ms/step - loss: 1.0128 - top10_acc: 0.9532 0s - loss: 1.0131 - top10_acc: 0.953\n",
      "Epoch 6/7\n",
      "1576/1576 [==============================] - 31s 20ms/step - loss: 0.8569 - top10_acc: 0.9675\n",
      "Epoch 7/7\n",
      "1576/1576 [==============================] - 32s 20ms/step - loss: 0.7378 - top10_acc: 0.9773\n"
     ]
    }
   ],
   "source": [
    "train_history = nnet.fit(X_scaled,y_cat,epochs=7)\n",
    "alarm.Beep(freq, duration)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trainning parameters:  {'verbose': 1, 'epochs': 7, 'steps': 1576}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x26c9cdbf190>]"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD7CAYAAAB+B7/XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAApl0lEQVR4nO3de1RU57038O/eM8MMw3BnZrhfREQxKgTvJipJvES0Jmob42lp08bV9M2qq1ldqTmGJK1N2hzjSk88edPzvmmavkk9bWkTbTRWYzRqFNRIREWIoghylYsgchvmst8/BkdRFBgG9ly+n7WyZmZfht8T5MvDs595tiBJkgQiIvIaotwFEBGRazHYiYi8DIOdiMjLMNiJiLwMg52IyMsw2ImIvAyDnYjIyyjlLgAAWlo6YLMNfTp9eLgOzc3tI1DR6GNb3I+3tANgW9yVs20RRQGhoQF33e8WwW6zSU4F+41zvQXb4n68pR0A2+KuRqItHIohIvIyDHYiIi/DYCci8jIMdiIiL8NgJyLyMh4d7FxxmIjoTh4b7GcvXcWzb3yBjm6z3KUQEbkVjw32oAA/VF25js9PVMtdChGRW/HYYI8z6DBjYiQ+P1GFLpNF7nKIiNyGxwY7AHznkXHo6Lbgi5M1cpdCROQ2PDrYx8WH4r6kMOw5fhmmHqvc5RARuQWPDnYAWDo7Edc7zTh4qlbuUoiI3ILHB/u4uBCkxoVg97FKmC3stRMReXywA8CyOYlobe/B4dN1cpdCRCQ7rwj2CQmhSI4Owq6jlbBYbXKXQ0QkK68IdkEQsHR2IprbTCg4Wy93OUREsvKKYAeAycnhiDfq8GlBJaw29tqJyHd5TbALgoBlsxPR0NKFr0ob5C6HiEg2XhPsAJAxTo+YiADsLKiEjQuEEZGP8qpgFwUB2bMTUNvUga/PNcpdDhGRLLwq2AFg+ngjjKH+2JlfwWV9icgnDSrY33rrLSxZsgTZ2dl4//3379hfWlqKlStXYtGiRXjxxRdhsci3KJcoCsielYjLDe04fbFZtjqIiOQyYLAfP34cR48exSeffIKPPvoIH374IcrLy/sc8/zzz+Oll17Cnj17IEkS8vLyRqzgwZg50YjwIA12sNdORD5owGCfPn06PvjgAyiVSjQ3N8NqtUKr1Tr219TUoLu7G+np6QCAFStWYPfu3SNW8GAoFSKWzEpAeW0bSipbZK2FiGi0KQdzkEqlwpYtW/DHP/4RixcvhtFodOxraGiAXq93vNbr9bhy5cqQiggP1w3p+Fvp9YH9bn/8oRR8WlCJPV9VYf60BKfffzTdrS2eyFva4i3tANgWdzUSbRlUsAPAunXrsHbtWjzzzDPIy8vDE088AaD/+44KgjCkIpqb22GzDX3IRK8PRGPj9bvuXzQtDn/ZV4YjX1dhXFzIkN9/NA3UFk/iLW3xlnYAbIu7crYtoijcs0M84FDMxYsXUVpaCgDw9/fHwoULce7cOcd+o9GIpqYmx+vGxkYYDIYhFzoS5qZHI0irwo78CrlLISIaNQMGe3V1NXJzc9HT04Oenh7s27cPmZmZjv0xMTFQq9UoLCwEAGzfvh1z584duYqHQK1SYNH0eJy9dBXltW1yl0NENCoGDPZ58+Zh3rx5eOyxx7By5UpkZGQgOzsba9euxZkzZwAAmzdvxm9/+1s8+uij6OrqQk5OzogXPljzM2IQoFFiJ3vtROQjBMkN5gOO1Bj7DZ8cvoTthy/hl09NQ7zRPS+6cNzQ/XhLOwC2xV3JNsbuDR6ZGgt/tQI7CyrlLoWIaMT5RLBrNSo8dH8sCr9pQG1Th9zlEBGNKJ8IdgBYMC0OKpWITwsq5C6FiGhE+UywB2n9kJURg6MlV9DQ0il3OUREI8Zngh0AFk2Ph0IU8SnH2onIi/lUsIfo1Jg7JQr5xfVovtYtdzlERCPCp4IdAB6dYV835l/H2GsnIu/kc8EeHqzBnEmROHSqDq3tJrnLISJyOZ8LdgBYMjMBNpuE3ccuy10KEZHL+WSwG0K1mJFmxIGiGrR19shdDhGRS/lksANA9qwEmM027P2qSu5SiIhcymeDPToiAJnjDdhXWI2ObrPc5RARuYzPBjsALJudiO4eK/adqJa7FCIil/HpYI8z6JA+NgJ7T1Shy2SRuxwiIpfw6WAHgGVzEtHRbcEXJ2vkLoWIyCV8PtiTooJwX1IY9hy/DJPZKnc5RETD5vPBDgBLZyfieqcZB4tq5S6FiGjYGOwAxsWFIDUuBLuPVcJsYa+diDwbg73XsjmJaG3vweEz9XKXQkQ0LAz2XhMSQpEcHYRdBZWwWG1yl0NE5DQGey9BELB0diKa27pRcJa9diLyXAz2W0xODke8UYdPCyphs0lyl0NE5BQG+y0EQcCy2YloaOnC8dIrcpdDROQUBvttMsbpERMRgJ0FlbBJ7LUTkedhsN9GFARkz05AbVMHTp5vlLscIqIhY7D3Y/p4I4yh/tiRXwGJvXYi8jAM9n6IooDsWYm4fKUdpy82y10OEdGQMNjvYuZEI8KDNOy1E5HHYbDfhVIhYsmsBJTXtqGkskXucoiIBm1Qwf72228jOzsb2dnZ2LRpU7/7s7KysHz5cixfvhxbt251eaFyeGBSFEJ0fth5pELuUoiIBk050AH5+fk4fPgwtm3bBkEQ8PTTT2Pv3r1YsGCB45ji4mK8+eabyMjIGNFiR5tKKeLRGQn4y74ynK9qxbi4ELlLIiIa0IA9dr1ejxdeeAF+fn5QqVRITk5GbW3f5W2Li4vx7rvvYtmyZdi4cSNMJtOIFTza5qZHI0irwo78CrlLISIalAGDPSUlBenp6QCAiooK7Nq1C/PmzXPs7+jowIQJE7B+/Xps27YNbW1teOedd0as4NGmVimwaHo8zl66ivLaNrnLISIakCANcspHWVkZfvzjH+OnP/0pHn/88bseV1JSgg0bNmD79u2uqlF2nd1m/OjVvZg4Jhy5P5whdzlERPc04Bg7ABQWFmLdunXYsGEDsrOz++yrra1Ffn4+Vq1aBQCQJAlK5aDe1qG5ud2pRbf0+kA0Nl4f8nnOeCQzFtsPX0JhcS3ijYEuf//RbMtI85a2eEs7ALbFXTnbFlEUEB6uu/v+gd6grq4Ozz77LDZv3nxHqAOARqPBG2+8gaqqKkiShK1bt/a5sOotHpkaC3+1AjsLKuUuhYjongbsWr/33nswmUx4/fXXHdtWr16N/fv3Y926dZg0aRI2btyIn/zkJzCbzbj//vvx1FNPjWjRctBqVHjo/ljsKqhEbVMHoiMC5C6JiKhfgx5jH0meMBQDAG2dPfjF7/OROU6PtcsmuvS9+eel+/GWdgBsi7uSbSiGbgrS+iErIwZHS66goaVT7nKIiPrFYB+iRdPjoRBFfMqxdiJyUwz2IQrRqTF3ShTyi+vRfK1b7nKIiO7AYHfCozMSAAD/OsZeOxG5Hwa7E8KDNZgzKRKHTtWhtd17lk8gIu/AYHfSkpkJsNkk7D52We5SiIj6YLA7yRCqxYw0Iw4U1aCts0fucoiIHBjsw5A9KwFmsw17v6qSuxQiIgcG+zBERwQgc7wB+wqr0dFtlrscIiIADPZhWzY7Ed09Vuw7US13KUREABjswxZn0CF9bAT2nqhCl8kidzlERAx2V1g2JxEd3RZ8cbJG7lKIiBjsrpAUFYT7ksKw5/hlmMxWucshIh/HYHeRpbMTcb3TjINFtQMfTEQ0ghjsLjIuLgSpcSHYfawSZgt77UQkHwa7Cy2bk4jW9h4cPlMvdylE5MMY7C40ISEUydFB2FVQCYvVJnc5ROSjGOwuJAgCls5ORHNbNwrOstdORPJgsLvY5ORwxBt1+LSg0qnb/RERDReD3cUEQcCy2YloaOnC8dIrcpdDRD6IwT4CMsbpERMRgJ0FlbDJf69wIvIxDPYRIAoCsmcnoLapAyfPN8pdDhH5GAb7CJk+3ghjqD925FdAYq+diEYRg32EiKKA7FmJuHylHacvNstdDhH5EAb7CJo50YjwIA177UQ0qhjsI0ipELFkVgLKa9tQUtkidzlE5CMY7CPsgUlRCNH5YeeRCrlLISIfwWAfYSqliEdnJOBcVSvOV7XKXQ4R+QAG+yiYmx6NIK0KO/Ir5C6FiHwAg30UqFUKLJoej7OXrqK8tk3ucojIyw0q2N9++21kZ2cjOzsbmzZtumN/aWkpVq5ciUWLFuHFF1+ExcJ7f95ufkYMAjRK7GSvnYhG2IDBnp+fj8OHD2Pbtm3Yvn07zp49i7179/Y55vnnn8dLL72EPXv2QJIk5OXljVjBnspfrcSCqXEoutCEy1euy10OEXmxAYNdr9fjhRdegJ+fH1QqFZKTk1Fbe/P2bzU1Neju7kZ6ejoAYMWKFdi9e/eIFezJHpkaC3+1AjsLKuUuhYi82IDBnpKS4gjtiooK7Nq1C/PmzXPsb2hogF6vd7zW6/W4coWrGvZHq1HhoftjUfhNA2qbOuQuh4i8lHKwB5aVleHHP/4x1q9fj8TERMf2/j5RKQjCkIoID9cN6fhb6fWBTp8rhycXT8DnhdX4/GQNfr4ms88+T2vLvXhLW7ylHQDb4q5Goi2DCvbCwkKsW7cOGzZsQHZ2dp99RqMRTU1NjteNjY0wGAxDKqK5ud2pm1Lo9YFobPS88er56dH47KsqLJ4aC0OoFoDntqU/3tIWb2kHwLa4K2fbIorCPTvEAw7F1NXV4dlnn8XmzZvvCHUAiImJgVqtRmFhIQBg+/btmDt37pAL9SWLpsdDIYrYdZRj7UTkegP22N977z2YTCa8/vrrjm2rV6/G/v37sW7dOkyaNAmbN29Gbm4uOjo6kJaWhpycnBEt2tOF6NSYOyUKB4tqsWx2EsKDNXKXREReRJDcYNlBXxuKAYDma9144f8UYF56NL67MNWj23I7b2mLt7QDYFvclWxDMTQywoM1mDMpEodO1aG13SR3OUTkRRjsMloyMwE2m4Tdxy7LXQoReREGu4wMoVrMSDPiQFENrrHXTkQuwmCXWfasBJjNNvzPnm9gk/9yBxF5AQa7zKIjAvDglGjsyq/Ab/9ciOqGdrlLIiIPx2B3A99fnIrnnszAlatd+NWfvsLfD1yAyWyVuywi8lCDXlKARo4gCHhoajySDDrkfXEB/zp6GV+VNuC7C1MxOTlc7vKIyMOwx+5GdP4q/HDJBKxfkwGVUsR//v0U3tlejJbrvLBKRIPHYHdDqfGh+OVT0/H4g0koKmtC7h+OYl9htVMf4iIi38Ngd1MqpYhlc5Lw6x9NR1JUELbuPY/XPizkTTqIaEAMdjdnDNPi50+kY+2yNDRf68LGP53A3/aXobuHtx8kov7x4qkHEAQBsyZGYtKYcPzjwEXsOV6Fr75pwHcXpCI9JULu8ojIzbDH7kF0/ir84NHx+Pfv3g9/PyW2fHQab398BlfbuuUujYjcCIPdA6XEhuCVp6Zh5bwxKC5vxot/OIa9J6p4cZWIADDYPZZSISJ7ViI2Pj0DKbHB+MvnZfj1BydQUd8md2lEJDMGu4czhPjjuW9PwTPLJ6L1ugm//n8n8D+fn0eXiRdXiXwVL556AUEQMH2CEfclheGjQ+XYd6IahecaseaRcbh/XMSQby5ORJ6NPXYvotWo8L2FqdiQk4kAjQr/e9sZ/NdHZ9B8jRdXiXwJg90LJUcH4+UfTMV3ssaipPIqcv9wDLuPXYbVZpO7NCIaBQx2L6VUiFg8Ix6vPj0D4+NDkPfFBfz6TydQXsuLq0TejsHu5SKC/bFu1WQ8+/h9uN5lxmsfnMCfPzuHzm5eXCXyVrx46gMEQUBmqgFpiWHYdqgc+76uRuF5+8XVqal6Xlwl8jLssfsQf7USaxaMQ27OVIQEqPH77cV46x+n0dTaJXdpRORCDHYflBQVhNzvZ2L1wyk4V9WK3D8cw7+OVsJi5cVVIm/AoRgfpRBFLJwWh6mpemzdex5/P3ARBWfrkbN4PMbGBMtdHhENA3vsPi4sSIOfrpyMn66YhE6TBb/5sBAf7P4GHd1muUsjIiexx04AgIxxekxIDMX2Ly9h74kqfH2+EasfScGMCUZeXCXyMOyxk4PGT4nVD6fg5e9PQ3iwBv/3kxK8+bciNLR0yl0aEQ0Bg53ukBAZiBe/NxX/tmAcLta24aX3jmNnfgUvrhJ5CA7FUL9EUcDDmbG4f5wef9lXho8PleNoyRXkLErFuLgQucsjonsYdI+9vb0dS5cuRXV19R373n77bWRlZWH58uVYvnw5tm7d6tIiST6hgWr8r8fuw8++PRmmHite3/o13t9VivYuXlwlcleD6rGfOnUKubm5qKio6Hd/cXEx3nzzTWRkZLiyNnIjk5Mj8OrTofjkyCXsOV6FogtNeOKhsZg1MZIXV4nczKB67Hl5eXjllVdgMBj63V9cXIx3330Xy5Ytw8aNG2EymVxaJLkHtZ8C384ai1eemgZDiD/+sLMUm/9ahPqrvLhK5E4GFeyvvfYapk6d2u++jo4OTJgwAevXr8e2bdvQ1taGd955x6VFknuJM+jw79/LRM6iVFTUX8fL7x3DPw9fgtnCi6tE7kCQJGnQd0B+6KGH8MEHHyA2Nvaux5SUlGDDhg3Yvn27K+ojN9fS1o0/fFKMQydrEKPX4dlVUzBpbITcZRH5tGHPiqmtrUV+fj5WrVoFAJAkCUrl0N62ubkdNtugf7846PWBaGy8PuTz3JEnt+UHi1IxdVwEPtxzDht+fwT3jzdgTpoRk8eGQyF67oxaT/6e3I5tcU/OtkUUBYSH6+66f9jBrtFo8MYbb2DGjBmIjY3F1q1bsWDBguG+LXmY+5LC8esfzcCer6pw6FQtvv6mAaGBajw4OQpzp0QjLEgjd4lEPsPp7tTatWtx5swZhIWFYePGjfjJT36CxYsXQ5IkPPXUU66skTyEn0qBZbMT8d6LC/DTFZMQq9dhx5EKPP/7fGz5x2mcvtjs1F9mRDQ0QxpjHykcivHetjS2duHQqVp8eaoWbZ1mhAdpMC89Gg9OjkKwTi1zpffmrd8TT8e2jMJQDNG96EP8sXJeMpY/kISTZU04cLIGHx8qxz8PX0JGSgTmZ8RgfEIoRM6FJ3IZBjuNCqVCxLTxBkwbb0D91U4cLKrB4dN1OHGuEYZQf8xPj8GcSZEI1PrJXSqRx2Ow06iLDNPiiYdSsGLuGJw414gDJ2uQ98UFfHzoIqamGjA/IwYpscH8RCuRkxjsJBuVUoFZEyMxa2IkahrbcaCoFvnF9ThacgXREQGYlx6N2fdFIkCjkrtUIo/CYCe3EKPX4d8WjMOq+ck4XnoFB07W4i+fl+GjAxcxbYK9Fz8mKoi9eKJBYLCTW1GrFHhwcjQenByNyvrrOFhUg4KSKzhyph7xBh3mZcRgZpoR/mr+0yW6G/50kNtKiAxEzuLx+HbWWBwruYIvTtbgwz3nkPfFBcxMM2J+egwSIgPlLpPI7TDYye35q5WYnxGDeenRKK9rw8GTtSgorsfBolokRQVhfkY0pk8wQq1SyF0qkVtgsJPHEAQBydHBSI4OxuqHxyK/uB4Himrx/q5v8Nd9FzD7vkjMT49GjP7uH9wg8gUMdvJIWo0Kj0yNw8OZsSirvoYDJ2twsKgG+wqrkRIbjPkZMZiaqodKyV48+R4GO3k0QRAwLi4E4+JC8GRnCo6cqceBohq8u6MEf/lchTmTIjEvPQaRYVq5SyUaNQx28hqBWj8snhGPhdPj8E1lCw6crMHnJ6qx53gVJiSEYn5GDDJSIqBUeO5SwkSDwWAnryMKAtISw5CWGIZr7SZ8eboOB4tq8fvtxQjSqvDglGjMnRINfYi/3KUSjQgGO3m1YJ0aS2cnYsnMBBRfuooDJ2uw62gldhVUYuKYMGSlx3j8DUGIbsdgJ58gigImJ4djcnI4rrZ125cSPl2H//r4DG8IQl6HwU4+JyxIg8ceHINlcxJx+kIzviiqwY4jFdiRX4EpyfalhO9LCpO7TCKnMdjJZylEERnj9MgYp+9zQ5CiC00ID9Lg0TmJSIsL4Ywa8jgMdiL0f0OQP//rGwD2ZYYzUiKQnhKB5OhgiCIXIiP3xmAnusWtNwSRlArsP1aJorJGfPZVFf517DICtSpMSbaH/MTEMKj9+AEocj8MdqK7MIRq8XBmLB7OjEVntwXFl5pRVNaEr8834vCZOigVItISQ5GREoEpYyMQ4ub3cCXfwWAnGgStRonpE4yYPsEIi9WGsqpWnLzQhKKyJpy+2AzgHJKighxDNjERAVw7nmTDYCcaIqVCxITEMExIDMOTD6egprHDEfIfHyrHx4fKERGsQXpKBDJS9EiJDeanXWlUMdiJhkEQBMQadIg16LBsdiJarptw6qI95A+crMXnJ6qhVSsxOTkc6SkRuC8pHFoNf+xoZPFfGJELhQaqMT89BvPTY2DqsaL40lUUXWjEqQvNOFpyBQpRwPj4EKSn6JE+NgLhwfxAFLkeg51ohKj9FMhM1SMzVQ+bTcKFmmso6h2y2br3PLbuPY94gw7pvePyCcZAjsuTSzDYiUaBKN5cXvg7WWNR19zhCPkd+RX45EgFQgPVSB9rD/nx8aFQKTkuT85hsBPJICo8AFHhAXh0RgLaOntw5mIzTpY14UhxHb44WQO1nwKTksKQnhKByckR0Pmr5C6ZPAiDnUhmQVo/zJkUhTmTomC2WFFa2YKTZU0outCEE+caIQoCUmKDHUM2xlAucUD3xmAnciMqpQKTk+299O9JEirrr9tDvqwRf9t/AX/bfwHREQGOIZsx0UEQOS5Pt2GwE7kpURCQFBWEpKggrJg7Bo2tXY5x+d3HLmPX0UoEaVWY0hvyaYlhUKu4xAENMtjb29uxevVq/Pd//zdiY2P77CstLUVubi7a29sxdepU/OpXv4JSyd8XRK6mD/HHgqlxWDA1Dh3dZpwpty9xcOJcA748XQc/pYi0RPu4/JSxEQgO8JO7ZJLJgAl86tQp5ObmoqKiot/9zz//PF599VWkp6djw4YNyMvLw5o1a1xdJxHdIkCjwsy0SMxMi4TFasO5qlYU9Q7ZFF1oggBgTHRQ77i8HhEROrlLplE0YLDn5eXhlVdewS9+8Ys79tXU1KC7uxvp6ekAgBUrVmDLli0MdqJRpFSImJgYhomJYVjzSAqqGtpRdKEJJ8ua8NHBcnx0sByGMC1SooOQ0jvl0hjqzznzXmzAYH/ttdfuuq+hoQF6vd7xWq/X48qVK66pjIiGTBAExBsDEW8MxLfmJOFqWzdOXWxGWc01nLrYjCPF9QCAoAA/pMQG2+fWx4YgzqDjOvNeZFiD4ZIk3bHNmV5AeLjzfybq9YFOn+tu2Bb34+nt0OsDkZps73xJkoTqhnacLW/G2UvNKClvRuG5RgD21SvHJ4ZhYlI4Jo4Jx7j4EKiU7nsh1t2/L5JkA2xWSLYbj1bAZrO/luyvBdEevyPRlmEFu9FoRFNTk+N1Y2MjDAbDkN+nubkdNtudvyQGotcHorHx+pDPc0dsi/vxlnYA9rY0NbVDIwKZY8OROTYcANB8rRvnq1tRVtWK89XX8PU3DQDswztjogIdQzdjY4Lhrx7epIgbYQebFbBaINksjteSzQJYrUDvNsl6Y98tx/Vu0wWocL2t075NstkDU7IHp2ObZHOEKaSbj5JN6vPaHrY3z7lxfp9ze78GbNabbXA8SjfPv/WcQYpc8zI6dGOG/P9SFIV7doiH9Z2KiYmBWq1GYWEhMjMzsX37dsydO3c4b0lEQyRJUm/wmSFZeuyP1h7AYu59bkZnmwqWq9dvCVMLJJsVgVYLMm0WZEZZIRks6DH14Oq1TrRe68C19i50nu5G3SkbGgUbAjUigv0VCNSICFCLUAq23oC+JZBtFkcA3xHYks0l7e0e6ABBAAQFIIqOR0FUAIIIOB5FCLcdA1EBQRABQYSgVPfuG/q5EG/uE249RrjlOFEBKP3gH5+GjqsDtmjInAr2tWvXYt26dZg0aRI2b96M3NxcdHR0IC0tDTk5Oa6ukcgjSJIEWG+GKSy9AWs1Q+oNWVh6evf1Pt66z9rT+7zHcX7fsO77nn2e9zMsequuIbQjTFAgTKEAFEogWAGLJMJsE2CyAF3XgZY2EU0QoVAqoVarofFXI0AbAD+1GqKoAEQlBIX90R5yCvuwg0J587mouPO1qICgUN48T6G0h6po3yb0HhOuD0ZzS2ffkLzl0ZMuCgsKFQbxq2ro7yv1N1A+yjgUw7aMNEmyAWYTJIvJ/mju7n3eDclsAiy928y92yw90Cis6GrvuBmqdwnrG8ELq3l4RYoKQOEHQakCFCr7D71SZd+mUAFKP8c2QWHffvO5qve8m8fe3KZCaHgwWtt6AMWtQaq8M0xFxT2D0WK1oaL+OsqqWnGuqhUXqq+h02QBYF+y2H4xNhgpcSGIjggYkU/FuuO/L2c525YRHYohcjV7r7enN2BNkCzdtwWx/fmNkHY8N3f3hrOp38CGpWcIVQiASg2bnwY28daAVUFQ+kFQa+8I0xv7bg3TG6/77HMEs9+dx4gjt5qjRh8IhQvCUKkQMTYmGGNjgvHozATYJAk1jR04X9WKsupWfHO5BcdK7DPjAjRKpMTax+hT4oKRYAzknaRGCYOdhk2y9EAydUAytUPqbkdHiwBzc2ufsL0RtI6gvqPnfDOkgSH89abwg6BSAyqN/VGphqDSQPAP6n3eu693O1Rq+/hp7/H9bYPCD4IgeFXPcKSIgoA4gw5xBh0ezoyFJElobO3C+aprjouyRRfsEyz8VCKSo4OREhuM1LgQjIkJ5hIII4TBTg6SzXpLQHcAvUF947XkeN3RZ9vtveF+x3NFpSNAHWGq0kAICIXoCOXeR5UaguP5bYF9S1BDqR7RXi4NnSAIMIRqYQjV4oHJUQCAa+0mnK++Zu/VV7Vix5EKfAJAIQpIiAzEuFh7jz4lNoTLE7sIg90LSZIE9HTaA9gRzO19Xzue39yGnntcYhNECOoACOoAQKODEBAOMTwegloHQaOz79PoIKh1CDOEo6XDejPElWr7OC75pGCdGtPGGzBtvH0qdGe3GRdqrjl69Z8XVmH38csAgBh9gCPox8WGICyItw50Bn/a3JgkSfbZE3f0lPt5beoAHNs67j21zE97M4g1OojBxn4D+tZjoNLYZyAMgtpF47nknbQalWNpYgDoMVtxqa7N0avPP1uPL07WAAAigjWOO0+lxAYjMoxr0Q8Gg11GkmSDrfkyrDWlaDQ1o6u1pe+wh6ndPif4bpR+fQM5PO6OQLb3snvDubfHLYgc1yT34adSIDU+FKnxoQAAq82GqoZ2nK+6hrKqVpwpb0b+jaUQtCqkJobBGKJBnCEQcQYdDCH+XA7hNgz2USRJEqRr9bDUlMBaUwJL3TeAqQMAYAkIgeSnhaDWQQw2QFAnAf31ojUBN8NbyWVZyfsoRBGJkUFIjAzCwmlxkCQJ9Vc7cb6qFeerrqG6qQOFpQ2w9c7U9lOJiNXrEKvXOS7kxhl0w/6krCfz3ZaPElv7VVhrS2CpKYW1tgRSRwsAQNCFQ5lwP5QxE6CIngBjYjxnYBD1QxAExz1i56XHQK8PRG1dK2qaOlDV0I6qhnZUN7Sj8FwDDp2qdZwXEazpE/RxBh0iQvx94o5TDHYXk7rbYakthbW2FJaaEkjX7H9CCppAKKLtIa6MSYMQZPCoT8gRuROVUuHo1d8gSRJarpscYX/jv6ILTY4P5qr9FIjr7dnH9oZ9rD4AGj/vikLvao0MJHM3rHXnYaktgbWmFLbmywAkQKWBIioVygnzoYhJgxgWO+iLj0Q0dIIgICxIg7AgDaaMjXBsN5mtqL3Ru7/SjqqG6zhacgVdvRdoBQD6UH97r/6W4ZzwYI3Hdr4Y7EMkWS2wNlyEtaYE1tpSWK9ctK/mJiqhMI6F39THoIxOg2hIcizLSUTyUasUjnvH3iBJEprbuu/o3X99rtHx8Th/tRJx+gBHzz7OEIgYfYBHfKiKyTMAyWaD7epl+8XOmhJY68/bP5AjCBAjEuE3eREUMWlQRKbYPzRDRG5PEAREBPsjItgfGSk3bxbU3WNBTWNHn7A/UlwPU4+19zzAGKrtM5QTb9AhNFDtVr17Bvtt7jVzRQyNhir1QShi0qCMGm//sA4ReQ2NnxLJMcFIjgl2bLNJEpqudTuGcaoa2lFR34aveteuB+zr4tjH63t790YdYiICZLtZCYMdt85csQ+v9DtzJSYNojZE3kKJaNSJggBDiD8MIf7ITL3Zu+8yWVDdeHNWTlVDO748XQeT2eo4LzJc67hAe2PefYjOb8R79z4Z7I6ZKzUlsNSW3jlzJSbNPnMlUO9Wf14RkfvwV9tXr0yJDXFss0kSGlu6+gzlXKi+5ljxEgB0/irEGXRIiAzE95fdNyK1+USwDzxzJat35koMZ64QkdNEQYAxTAtjmBZTx9+8TWhnt9nes2/scAzn5J+pw8PTExAe4PqFz7wy2AeeufI4lDFpEPWJnLlCRCNOq1H1WTbhhpFaGtorUk2y9a65UnuXmStTFkMRnQZF5FjOXCEir+exwS5ZLWgr3I2ub77uZ+bK3N6ZK6mcuUJEPsdjg91SeRJNn78LQRcOVeL99rnk0RM4c4WIfJ7HBrsyaSoSnvsTmtslzlwhIrqFx04BEQQBCm0gQ52I6DYeG+xERNQ/BjsRkZdhsBMReRkGOxGRl2GwExF5GQY7EZGXcYt57KLo/JTF4ZzrbtgW9+Mt7QDYFnflTFsGOkeQpBu3eSUiIm/AoRgiIi/DYCci8jIMdiIiL8NgJyLyMgx2IiIvw2AnIvIyDHYiIi/DYCci8jIMdiIiL+Oxwb5jxw4sWbIECxYswNatW+UuZ1ja29uxdOlSVFdXy13KsLz99tvIzs5GdnY2Nm3aJHc5w/LWW29hyZIlyM7Oxvvvvy93OcP2H//xH3jhhRfkLmNYcnJykJ2djeXLl2P58uU4deqU3CU5bf/+/VixYgUWL16MV1991fVfQPJA9fX1UlZWltTS0iJ1dHRIy5Ytk8rKyuQuyylFRUXS0qVLpYkTJ0pVVVVyl+O0I0eOSE888YRkMpmknp4eKScnR/rss8/kLsspx44dk1avXi2ZzWapq6tLysrKki5evCh3WU7Lz8+XZsyYIa1fv17uUpxms9mkOXPmSGazWe5Shu3y5cvSAw88INXV1Uk9PT3Sk08+KR04cMClX8Mje+z5+fmYOXMmQkJCoNVqsWjRIuzevVvuspySl5eHV155BQaDQe5ShkWv1+OFF16An58fVCoVkpOTUVtbK3dZTpk+fTo++OADKJVKNDc3w2q1QqvVyl2WU1pbW/G73/0OzzzzjNylDEt5eTkEQcDatWvxrW99C3/+85/lLslpe/fuxZIlSxAZGQmVSoXf/e53mDJliku/hlus7jhUDQ0N0Ov1jtcGgwGnT5+WsSLnvfbaa3KX4BIpKSmO5xUVFdi1axf++te/yljR8KhUKmzZsgV//OMfsXjxYhiNRrlLcsrLL7+M5557DnV1dXKXMixtbW2YNWsWfvnLX6K7uxs5OTlISkrCnDlz5C5tyCorK6FSqfCjH/0IjY2NyMrKws9+9jOXfg2P7LFL/SxIKQjes4ynJysrK8MPf/hDrF+/HomJiXKXMyzr1q1DQUEB6urqkJeXJ3c5Q/b3v/8dUVFRmDVrltylDFtGRgY2bdoErVaLsLAwrFq1CgcPHpS7LKdYrVYUFBTgjTfeQF5eHs6cOYNt27a59Gt4ZLAbjUY0NTU5Xjc0NHj8UIY3KCwsxA9+8AP8/Oc/x+OPPy53OU67ePEiSktLAQD+/v5YuHAhzp07J3NVQ7dr1y4cOXIEy5cvx5YtW7B//3785je/kbssp5w4cQIFBQWO15IkQan0yAEHREREYNasWQgLC4NGo8HDDz/s8hEHjwz22bNno6CgAFevXkVXVxc+++wzzJ07V+6yfFpdXR2effZZbN68GdnZ2XKXMyzV1dXIzc1FT08Penp6sG/fPmRmZspd1pC9//772LlzJ/75z39i3bp1eOihh7Bhwwa5y3LK9evXsWnTJphMJrS3t2Pbtm1YsGCB3GU5JSsrC4cPH0ZbWxusViu+/PJLTJw40aVfwyN/5RmNRjz33HPIycmB2WzGqlWrMHnyZLnL8mnvvfceTCYTXn/9dce21atX48knn5SxKufMmzcPp06dwmOPPQaFQoGFCxd6/C8rT5eVleX4nthsNqxZswYZGRlyl+WUKVOm4Omnn8aaNWtgNpsxZ84crFy50qVfg3dQIiLyMh45FENERHfHYCci8jIMdiIiL8NgJyLyMgx2IiIvw2AnIvIyDHYiIi/DYCci8jL/HyPu8xDyVBKUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"trainning parameters: \",train_history.params)\n",
    "plt.plot(train_history.history[\"loss\"])\n",
    "plt.plot(train_history.history[\"top10_acc\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "X_validate_t,BGM = BGMfeatureGen(X=X_validate,BGM=BGM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 1 Validation Accuracy:  0.699857188194224\n",
      "Top 10 Validation Accuracy:  0.9064582672167566\n",
      "Evaluation process time:  3.8446800999954576\n"
     ]
    }
   ],
   "source": [
    "start = time.perf_counter()\n",
    "nnet_predict = nnet.predict(X_scaled_validate)\n",
    "df_nnet_predict = pd.DataFrame(nnet_predict,index = y_cat_validate.index,columns = y_cat.columns)\n",
    "\n",
    "df_top1_validation  = top1_accuracy_cat(df_nnet_predict,y_cat_validate)\n",
    "df_top10_validation = top10_accuracy_cat(df_nnet_predict,y_cat_validate)\n",
    "\n",
    "end   = time.perf_counter()\n",
    "\n",
    "print(\"Evaluation process time: \",end-start)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----top 1 categorical result----\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD7CAYAAACPDORaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAVqUlEQVR4nO3df0xV9/3H8dfFqyitSzt3LmkMIVnblNmsaLassi0Ql3mhwtVWzIYyWbNVMa2ausapgHGxsRLr5maqzUyabbVtVuYqRcKudjWSGLqQkVWisa2ZgPXXvYCNApUrcM/3j317Mwf13gv3cuTD85GYcM+P+3m/vZcXh8895+CybdsWAMBIKU4XAABIHkIeAAxGyAOAwQh5ADAYIQ8ABiPkAcBghDwAGMztdAH/67PP+hQOx3/q/qxZ96q7uzcJFd296HlyoOfJYbQ9p6S4dP/993zp+rsu5MNhe1Qh/8W+kw09Tw70PDkko2emawDAYIQ8ABiMkAcAgxHyAGAwQh4ADEbIA4DBCHkAMNhdd548ADhl5ldmaHqqM7F4a2AoKc9LyAPA/5ue6pbvhXcdGfvIr5ck5XmZrgEAgxHyAGAwQh4ADEbIA4DBCHkAMBghDwAGI+QBwGCEPAAYjJAHAIMR8gBgMEIeAAxGyAOAwWIK+d/97ndatGiRCgsL9Yc//EGS1NTUJJ/PJ6/Xqz179kS2PXv2rIqLi5Wfn6/KykoNDg4mp3IAQFRRQ765uVn/+Mc/VFdXp7/+9a86ePCgPvroI1VUVGj//v1qaGjQ6dOn1djYKEnauHGjtm7dqqNHj8q2bdXU1CS9CQDAyKKG/He+8x29/vrrcrvd6u7u1tDQkG7cuKHMzExlZGTI7XbL5/PJ7/fr0qVL6u/v19y5cyVJS5culd/vT3YPAIAvEdN0zdSpU7V3714VFhYqJydHwWBQlmVF1ns8HgUCgWHLLctSIBBIfNUAgJjE/EdD1q9fr1WrVmnNmjVqb28ftt7lcsm27RGXx2PWrHvj2v6/WdbMUe87UdHz5EDPk0Myeo4a8v/+979169YtfeMb39CMGTPk9Xrl9/s1ZcqUyDbBYFAej0fp6enq6uqKLO/s7JTH44mroO7uXoXDw39YRGNZM9XZ2RP3fhMZPU8O9Dy+4zppND2npLjueHAcdbrm4sWLqqqq0q1bt3Tr1i29//77KikpUVtbmzo6OjQ0NKT6+nrl5uZq9uzZSk1NVUtLiySptrZWubm5cRcNAEiMqEfyeXl5OnXqlJ588klNmTJFXq9XhYWF+upXv6p169YpFAopLy9PBQUFkqTdu3erqqpKfX19mjNnjsrKypLeBABgZC57pIl0BzFdEzt6nhzoeXzHdfIPeTsyXQMAmLgIeQAwGCEPAAaL+Tz5u92tgSHHTn/qDw2q58ZNR8YGgDsxJuSnTZ3i6Acmk+tjMQATBdM1AGAwQh4ADEbIA4DBCHkAMBghDwAGI+QBwGCEPAAYjJAHAIMR8gBgMEIeAAxGyAOAwQh5ADAYIQ8ABiPkAcBghDwAGIyQBwCDEfIAYDBCHgAMFtOf/3vllVf0t7/9TZKUl5enX/7yl9qyZYtaWlo0Y8YMSdLatWu1cOFCNTU1aefOnQqFQnriiSe0YcOG5FUPALijqCHf1NSkkydP6vDhw3K5XHrmmWf03nvv6fTp03rjjTfk8Xgi2/b396uiokIHDx7UAw88oPLycjU2NiovLy+pTQAARhZ1usayLG3evFnTpk3T1KlT9eCDD+ry5cu6fPmytm7dKp/Pp7179yocDqu1tVWZmZnKyMiQ2+2Wz+eT3+8fjz4AACOIeiT/8MMPR75ub29XQ0OD3nrrLTU3N2v79u1KS0tTeXm5Dh06pLS0NFmWFdne4/EoEAjEVdCsWffGtf3dwrJmTqpxnUTPkwM9J0ZMc/KSdO7cOZWXl2vTpk36+te/rn379kXWrVy5UrW1tSooKBi2n8vliqug7u5ehcN2XPtIzr8hOjt7xn1My5rpyLhOoufJwameJ2KOpKS47nhwHNPZNS0tLXr66af1wgsv6KmnntLHH3+so0ePRtbbti2326309HR1dXVFlgeDwdvm7AEA4ytqyF+5ckXPPfecdu/ercLCQkn/CfWXXnpJ169f18DAgN5++20tXLhQ2dnZamtrU0dHh4aGhlRfX6/c3NykNwEAGFnU6ZrXXntNoVBI1dXVkWUlJSVavXq1li9frsHBQXm9XhUVFUmSqqurtW7dOoVCIeXl5Y04hQMAGB9RQ76qqkpVVVUjristLR22LCcnR3V1dWOvDAAwZlzxCgAGI+QBwGCEPAAYjJAHAIMR8gBgMEIeAAxGyAOAwQh5ADAYIQ8ABiPkAcBghDwAGIyQBwCDEfIAYDBCHgAMRsgDgMEIeQAwGCEPAAYj5AHAYIQ8ABiMkAcAgxHyAGAwQh4ADBZTyL/yyisqLCxUYWGhdu3aJUlqamqSz+eT1+vVnj17ItuePXtWxcXFys/PV2VlpQYHB5NTOQAgqqgh39TUpJMnT+rw4cOqra3VmTNnVF9fr4qKCu3fv18NDQ06ffq0GhsbJUkbN27U1q1bdfToUdm2rZqamqQ3AQAYWdSQtyxLmzdv1rRp0zR16lQ9+OCDam9vV2ZmpjIyMuR2u+Xz+eT3+3Xp0iX19/dr7ty5kqSlS5fK7/cnuwcAwJeIGvIPP/xwJLTb29vV0NAgl8sly7Ii23g8HgUCAQWDwduWW5alQCCQ+KoBADFxx7rhuXPnVF5erk2bNsntdqutre229S6XS7ZtD9vP5XLFVdCsWffGtf3dwrJmTqpxnUTPkwM9J0ZMId/S0qL169eroqJChYWFam5uVldXV2R9MBiUx+NRenr6bcs7Ozvl8XjiKqi7u1fh8PAfFtE4/Ybo7OwZ9zEta6Yj4zqJnicHp3qeiDmSkuK648Fx1OmaK1eu6LnnntPu3btVWFgoScrOzlZbW5s6Ojo0NDSk+vp65ebmavbs2UpNTVVLS4skqba2Vrm5uXEXDQBIjKhH8q+99ppCoZCqq6sjy0pKSlRdXa1169YpFAopLy9PBQUFkqTdu3erqqpKfX19mjNnjsrKypJXPQDgjqKGfFVVlaqqqkZcV1dXN2xZVlaWDh06NPbKAABjxhWvAGAwQh4ADEbIA4DBCHkAMBghDwAGI+QBwGCEPAAYjJAHAIMR8gBgMEIeAAxGyAOAwQh5ADAYIQ8ABiPkAcBghDwAGIyQBwCDEfIAYDBCHgAMRsgDgMEIeQAwWNQ/5A1AmvmVGZqe6sy3y62BIUfGhRkIeSAG01Pd8r3wriNjH/n1EkfGhRmYrgEAg8Uc8r29vSoqKtLFixclSVu2bJHX69WSJUu0ZMkSvffee5KkpqYm+Xw+eb1e7dmzJzlVAwBiEtN0zalTp1RVVaX29vbIstOnT+uNN96Qx+OJLOvv71dFRYUOHjyoBx54QOXl5WpsbFReXl7CCwcARBfTkXxNTY22bdsWCfTPP/9cly9f1tatW+Xz+bR3716Fw2G1trYqMzNTGRkZcrvd8vl88vv9SW0AAPDlYjqS37Fjx22Pu7u7NX/+fG3fvl1paWkqLy/XoUOHlJaWJsuyItt5PB4FAoHEVgwAiNmozq7JyMjQvn37Io9Xrlyp2tpaFRQUDNvW5XLF9dyzZt07mpIcZ1kzJ9W4TqLnyYGeE2NUIf/xxx+rvb1d+fn5kiTbtuV2u5Wenq6urq7IdsFg8LY5+1h0d/cqHLbjrsnpN0RnZ8+4j2lZMx0Z10lO9TwZ319O4nWOXUqK644Hx6M6hdK2bb300ku6fv26BgYG9Pbbb2vhwoXKzs5WW1ubOjo6NDQ0pPr6euXm5o5mCABAAozqSD4rK0urV6/W8uXLNTg4KK/Xq6KiIklSdXW11q1bp1AopLy8vBGncAAA4yOukD9+/Hjk69LSUpWWlg7bJicnR3V1dWOvDAAwZlzxCgAGI+QBwGCEPAAYjJAHAIMR8gBgMEIeAAxGyAOAwQh5ADAYIQ8ABiPkAcBghDwAGIyQBwCDEfIAYDBCHgAMRsgDgMEIeQAwGCEPAAYj5AHAYIQ8ABiMkAcAgxHyAGAwQh4ADEbIA4DBYgr53t5eFRUV6eLFi5KkpqYm+Xw+eb1e7dmzJ7Ld2bNnVVxcrPz8fFVWVmpwcDA5VQMAYhI15E+dOqXly5ervb1dktTf36+Kigrt379fDQ0NOn36tBobGyVJGzdu1NatW3X06FHZtq2ampqkFg8AuLOoIV9TU6Nt27bJ4/FIklpbW5WZmamMjAy53W75fD75/X5dunRJ/f39mjt3riRp6dKl8vv9SS0eAHBn7mgb7Nix47bHwWBQlmVFHns8HgUCgWHLLctSIBCIu6BZs+6Ne5+7gWXNnFTjOomeJwd6ToyoIf+/bNsetszlcn3p8nh1d/cqHB7+XNE4/Ybo7OwZ9zEta6Yj4zrJqZ4n4/vLSbzOsUtJcd3x4Djus2vS09PV1dUVeRwMBuXxeIYt7+zsjEzxAACcEXfIZ2dnq62tTR0dHRoaGlJ9fb1yc3M1e/ZspaamqqWlRZJUW1ur3NzchBcMAIhd3NM1qampqq6u1rp16xQKhZSXl6eCggJJ0u7du1VVVaW+vj7NmTNHZWVlCS8YABC7mEP++PHjka9zcnJUV1c3bJusrCwdOnQoMZUBAMaMK14BwGCEPAAYjJAHAIPF/cErACTbrYEhx89ZNwUhD+CuM23qFPleeHfcxz3y6yXjPmayMV0DAAbjSB6jMvMrMzQ9dfzfPrcGhsZ9TGAiI+QxKtNT3fw6DUwATNcAgMEIeQAwGCEPAAYj5AHAYIQ8ABiMkAcAgxHyAGAwQh4ADMbFUABG5NRVzUgsXkEAI3LqqmaJK5sTiZAH7nLcdhdjQcgDdzluu4ux4INXADAYR/ITGL/GA4hmTCFfVlam7u5uud3/eZrt27frwoULevXVVzUwMKCnn35apaWlCSkUwzn1a7zEr/LARDHqkLdtW+fPn9eJEyciIR8IBLRhwwa98847mjZtmkpKSvT444/roYceSljBAIDYjTrkz58/L5fLpVWrVqm7u1s/+tGPdM8992j+/Pm67777JEn5+fny+/1au3ZtouoFAMRh1B+83rhxQzk5Odq3b5/++Mc/6s9//rMuX74sy7Ii23g8HgUCgYQUCgCI36iP5OfNm6d58+ZJktLS0rRs2TLt3LlTa9asuW07l8sV1/POmnXvaEtyFB+Ajh/+r2GqZLy3Rx3y//znPzUwMKCcnBxJ/5mjnz17trq6uiLbBINBeTyeuJ63u7tX4bAddz1Of+N3dvaM+5hO9+wU/q9hqtG8t1NSXHc8OB71dE1PT4927dqlUCik3t5eHT58WC+//LI++OADXbt2TTdv3tSxY8eUm5s72iEAAGM06iP5BQsW6NSpU3ryyScVDoe1YsUKfetb39KGDRtUVlamgYEBLVu2TI899lgi6wUAxGFM58k///zzev75529b5vP55PP5xvK0AIAE4YpXTChc5QvEh5DHhMLNuoD4cIMyADAYIQ8ABiPkAcBghDwAGIyQBwCDEfIAYDBCHgAMRsgDgMEIeQAwGFe8JgCX2gO4WxHyCcCl9gDuVkzXAIDBCHkAMBghDwAGI+QBwGCEPAAYjJAHAIMR8gBgMEIeAAxGyAOAwQh5ADBYUkL+yJEjWrRokRYuXKg333wzGUMAAGKQ8HvXBAIB7dmzR++8846mTZumkpISPf7443rooYcSPRQAIIqEh3xTU5Pmz5+v++67T5KUn58vv9+vtWvXxrR/Sopr1GN77p8x6n3Hyqmx6dn8cZ0cm57H12jyL9o+Ltu27dEWNJLf//73+vzzz7VhwwZJ0l/+8he1trbqxRdfTOQwAIAYJHxOfqSfGS7X6I/OAQCjl/CQT09PV1dXV+RxMBiUx+NJ9DAAgBgkPOS/+93v6oMPPtC1a9d08+ZNHTt2TLm5uYkeBgAQg4R/8Jqenq4NGzaorKxMAwMDWrZsmR577LFEDwMAiEHCP3gFANw9uOIVAAxGyAOAwQh5ADAYIQ8ABptwIR/t5mdnz55VcXGx8vPzVVlZqcHBQQeqTKxoPf/973/XkiVLtHjxYj377LO6fv26A1UmVqw3uTtx4oR+8IMfjGNlyROt5/Pnz2vlypVavHixfv7zn0+K1/nMmTMqLi7W4sWLVV5erhs3bjhQZWL19vaqqKhIFy9eHLYuKfllTyBXr161FyxYYH/22Wd2X1+f7fP57HPnzt22TWFhof2vf/3Ltm3b3rJli/3mm286UGniROu5p6fH/t73vmdfvXrVtm3b/u1vf2u/+OKLTpWbELG8zrZt252dnXZBQYG9YMECB6pMrGg9h8Nh2+v12o2NjbZt2/bLL79s79q1y6lyEyKW13n58uX2iRMnbNu27Z07d9q/+c1vnCg1YT788EO7qKjIfvTRR+1PP/102Ppk5NeEOpL/75ufpaWlRW5+9oVLly6pv79fc+fOlSQtXbr0tvUTUbSeBwYG9Ktf/Urp6emSpEceeURXrlxxqtyEiNbzF6qqqmK+8d3dLlrPZ86cUVpaWuTCwjVr1qi0tNSpchMiltc5HA6rr69PknTz5k1Nnz7diVITpqamRtu2bRvxLgDJyq8JFfLBYFCWZUUeezweBQKBL11vWdZt6yeiaD3ff//9+uEPfyhJ6u/v14EDByKPJ6poPUvS66+/rjlz5ig7O3u8y0uKaD1fuHBBX/va17Rp0yb5fD5t27ZNaWlpTpSaMLG8zps3b1ZlZaW+//3vq6mpSSUlJeNdZkLt2LFD3/72t0dcl6z8mlAhb0e5+Vm09RNRrD319PRo1apVysrK0lNPPTUepSVNtJ4/+eQTHTt2TM8+++x4lpVU0XoeHBxUc3OzfvKTn+jIkSPKyMhQdXX1eJaYcNF67u/vV2Vlpf70pz/p5MmTWrFihTZt2jSeJY6rZOXXhAr5aDc/+9/1nZ2dE/7maLHc8C0YDGrFihXKysrSjh07xrvEhIvWs9/vV2dnp4qLi7V69epI/xNZtJ4ty1JmZqa++c1vSpKKiorU2to67nUmUrSeP/nkE6WmpkZui/LjH/9Yzc3N417neElWfk2okI9287PZs2crNTVVLS0tkqTa2toJf3O0aD0PDQ1pzZo1euKJJ1RZWTnhf3ORove8fv16HT16VO+++64OHDggj8ejt956y8GKxy5az/PmzdO1a9f00UcfSZKOHz+uRx991KlyEyJaz5mZmbp69arOnz8vSXr//fcjP+RMlLT8GvNHt+Osrq7OLiwstL1er33gwAHbtm37mWeesVtbW23btu2zZ8/axcXFdkFBgf2LX/zCDoVCTpabEHfq+dixY/YjjzxiL168OPKvoqLC4YrHLtrr/IVPP/3UiLNrbDt6zx9++KFdXFxsL1q0yP7Zz35md3V1OVluQkTr+cSJE7bP57OLiorsn/70p/aFCxecLDdhFixYEDm7Jtn5xQ3KAMBgE2q6BgAQH0IeAAxGyAOAwQh5ADAYIQ8ABiPkAcBghDwAGIyQBwCD/R9SufbD5cLlMAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "categorical accuracy:  count    1195.000000\n",
      "mean        0.531983\n",
      "std         0.383486\n",
      "min         0.000000\n",
      "25%         0.142857\n",
      "50%         0.500000\n",
      "75%         0.953835\n",
      "max         1.000000\n",
      "dtype: float64\n",
      "----top 10 categorical result----\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD7CAYAAACPDORaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAWJ0lEQVR4nO3dcWzUd/3H8deVgwKC2WDfq0vXNFGXEaajRLKtzlzTqe1BOZgt0QJaFzcEUYjVELAt4pykyDqrRjsl0enIjOkQCm2aKzpCIymKNA4CdtkipbAO7q5lDtrR0l6/vz/8rb/xa8f3Wu56u889HwkJ9/1+vv2837nyui+fu37qsm3bFgDASGmJLgAAED+EPAAYjJAHAIMR8gBgMEIeAAxGyAOAwQh5ADCYO9EF/H9vvdWvkZGJf3R//vw56u3ti0NFH1z0nBroOTVMtue0NJfuvPND73v+AxfyIyP2pEL+3WtTDT2nBnpODfHomeUaADAYIQ8ABiPkAcBghDwAGIyQBwCDEfIAYDBCHgAM9oH7nDwAJMrcD8/SzPTExOKNoUhcvi4hDwD/a2a6W/7vHkzI3I3ProzL12W5BgAMRsgDgMEIeQAwGCEPAAYj5AHAYIQ8ABiMkAcAgxHyAGAwQh4ADEbIA4DBCHkAMBghDwAGI+QBwGCEPAAYLKqQP3LkiIqLi+Xz+fSjH/1IktTW1ia/36+CggLV1taOju3o6FBJSYkKCwtVWVmp4eHh+FQOAHDkGPIXL17Ujh07VFdXp8bGRv3rX/9Sa2urKioqVFdXp+bmZp05c0atra2SpC1btmj79u1qaWmRbduqr6+PexMAgPE5hvyf//xnLVu2TB/5yEc0ffp01dbWatasWcrOzlZWVpbcbrf8fr8CgYC6u7s1MDCgnJwcSVJxcbECgUC8ewAAvA/H3wzV1dWl6dOn64knnlA4HFZ+fr7uvfdeWZY1Osbj8SgYDCoUCt103LIsBYPB+FQOAHDkGPKRSEQnT57U3r17NXv2bG3cuFGzZs0aM87lcsm27XGPT8T8+XMmNP69LGvupK9NVvScGug5NcSjZ8eQv+uuu5Sbm6t58+ZJkj772c8qEAho2rRpo2NCoZA8Ho8yMjLU09MzejwcDsvj8UyooN7ePo2MjH2xcGJZcxUOX5vwdcmMnlMDPU/tvIk0mZ7T0ly3vDl2XJPPz8/XsWPHdPXqVUUiEf31r3+Vz+dTZ2enurq6FIlE1NTUJK/Xq8zMTKWnp6u9vV2S1NDQIK/XO+GiAQCx4Xgnv2jRIj355JNas2aNhoaG9Mgjj2j16tX66Ec/qk2bNmlwcFB5eXny+XySpJqaGlVVVam/v18LFy5UWVlZ3JsAAIzPMeQladWqVVq1atVNx3Jzc3Xo0KExYxcsWKB9+/bFpjoAwG3hJ14BwGCEPAAYjJAHAIMR8gBgMEIeAAxGyAOAwQh5ADAYIQ8ABiPkAcBghDwAGIyQBwCDEfIAYDBCHgAMRsgDgMEIeQAwGCEPAAYj5AHAYIQ8ABiMkAcAgxHyAGAwQh4ADEbIA4DB3NEMKisrU29vr9zu/w7/4Q9/qAsXLui5557T0NCQHn/8ca1du1aS1NbWpurqag0ODmrp0qUqLy+PX/UAgFtyDHnbtnXu3DkdPXp0NOSDwaDKy8u1f/9+zZgxQ6WlpXrooYd0zz33qKKiQnv37tXdd9+t9evXq7W1VXl5eXFvBAAwlmPInzt3Ti6XS+vWrVNvb6+++MUv6kMf+pAefvhh3XHHHZKkwsJCBQIBPfjgg8rOzlZWVpYkye/3KxAIEPIAkCCOa/JXr15Vbm6ufvnLX+p3v/ud/vjHP+rNN9+UZVmjYzwej4LBoEKh0LjHAQCJ4Xgnv3jxYi1evFiSNHv2bK1atUrV1dXasGHDTeNcLpds2x5zvcvlmlBB8+fPmdD497KsuZO+NlnRc2qg59QQj54dQ/7kyZMaGhpSbm6upP+u0WdmZqqnp2d0TCgUksfjUUZGxrjHJ6K3t08jI2NfLJxY1lyFw9cmfF0yo+fUQM9TO28iTabntDTXLW+OHZdrrl27pt27d2twcFB9fX06cOCAnnnmGR0/flxXrlzR9evXdfjwYXm9Xi1atEidnZ3q6upSJBJRU1OTvF7vhIsGAMSG4518fn6+Tp06pccee0wjIyNas2aNPvWpT6m8vFxlZWUaGhrSqlWr9MADD0iSdu3apU2bNmlwcFB5eXny+XxxbwIAMD6XPd5CegKxXBM9ek4N9Dy18/q/e3DK55WkxmdXJma5BgCQvAh5ADAYIQ8ABiPkAcBghDwAGIyQBwCDEfIAYDBCHgAMRsgDgMEIeQAwGCEPAAYj5AHAYIQ8ABiMkAcAgxHyAGAwQh4ADEbIA4DBCHkAMBghDwAGI+QBwGCEPAAYjJAHAIMR8gBgsKhD/sc//rG2bdsmSero6FBJSYkKCwtVWVmp4eFhSdKbb76ptWvXyufz6Rvf+Ib6+/vjUzUAICpRhfzx48d14MCB0cdbtmzR9u3b1dLSItu2VV9fL0l66qmntGbNGgUCAX3iE59QXV1dfKoGAETFMeT/85//qLa2Vhs2bJAkdXd3a2BgQDk5OZKk4uJiBQIBDQ0N6R//+IcKCwtvOg4ASBy304Dvf//7Ki8v16VLlyRJoVBIlmWNnrcsS8FgUG+99ZbmzJkjt9t90/GJmj9/zoSv+b9a5k762mRFz6mBnlNDPHq+Zci/9NJLuvvuu5Wbm6v9+/dLkmzbHjPO5XK97/GJ6u3t08jI2K/lxLLmKhy+NuHrkhk9pwZ6ntp5E2kyPaeluW55c3zLkG9ublY4HNbKlSv19ttv65133pHL5VJPT897igrL4/Fo3rx56uvrUyQS0bRp00aPAwAS55Zr8s8//7yampp08OBBbd68WY8++qiqq6uVnp6u9vZ2SVJDQ4O8Xq+mT5+uJUuWqLm5+abjAIDEmdTn5GtqalRdXa2lS5fq+vXrKisrkyTt2LFD9fX1WrZsmU6ePKlvf/vbsawVADBBjm+8vqu4uFjFxcWSpAULFmjfvn1jxmRmZmrv3r2xqw4AcFv4iVcAMBghDwAGI+QBwGCEPAAYjJAHAIMR8gBgMEIeAAxGyAOAwQh5ADAYIQ8ABiPkAcBghDwAGIyQBwCDEfIAYDBCHgAMRsgDgMEIeQAwGCEPAAYj5AHAYIQ8ABiMkAcAgxHyAGCwqEL+Zz/7mZYtW6aioiI9//zzkqS2tjb5/X4VFBSotrZ2dGxHR4dKSkpUWFioyspKDQ8Px6dyAIAjx5A/ceKE/va3v+nQoUP605/+pL179+rVV19VRUWF6urq1NzcrDNnzqi1tVWStGXLFm3fvl0tLS2ybVv19fVxbwIAMD7HkH/wwQf1wgsvyO12q7e3V5FIRFevXlV2draysrLkdrvl9/sVCATU3d2tgYEB5eTkSJKKi4sVCATi3QMA4H24oxk0ffp0/fznP9dvf/tb+Xw+hUIhWZY1et7j8SgYDI45blmWgsHghAqaP3/OhMa/l2XNnfS1yYqeUwM9p4Z49BxVyEvS5s2btW7dOm3YsEHnz58fc97lcsm27XGPT0Rvb59GRsZ+HSeWNVfh8LUJX5fM6Dk10PPUzptIk+k5Lc11y5tjx+Waf//73+ro6JAkzZo1SwUFBfr73/+unp6e0TGhUEgej0cZGRk3HQ+Hw/J4PBMuGgAQG44h/8Ybb6iqqko3btzQjRs39PLLL6u0tFSdnZ3q6upSJBJRU1OTvF6vMjMzlZ6ervb2dklSQ0ODvF5v3JsAAIzPcbkmLy9Pp06d0mOPPaZp06apoKBARUVFmjdvnjZt2qTBwUHl5eXJ5/NJkmpqalRVVaX+/n4tXLhQZWVlcW8CADA+lz3eQnoCsSYfPXpODfQ8tfP6v3twyueVpMZnVyZmTR4AkLwIeQAwGCEPAAYj5AHAYIQ8ABiMkAcAgxHyAGAwQh4ADEbIA4DBCHkAMBghDwAGI+QBwGCEPAAYjJAHAIMR8gBgMEIeAAxGyAOAwQh5ADAYIQ8ABiPkAcBghDwAGIyQBwCDRRXyv/jFL1RUVKSioiLt3r1bktTW1ia/36+CggLV1taOju3o6FBJSYkKCwtVWVmp4eHh+FQOAHDkGPJtbW06duyYDhw4oIaGBp09e1ZNTU2qqKhQXV2dmpubdebMGbW2tkqStmzZou3bt6ulpUW2bau+vj7uTQAAxucY8pZladu2bZoxY4amT5+uj33sYzp//ryys7OVlZUlt9stv9+vQCCg7u5uDQwMKCcnR5JUXFysQCAQ7x4AAO/DMeTvvffe0dA+f/68mpub5XK5ZFnW6BiPx6NgMKhQKHTTccuyFAwGY181ACAq7mgHvv7661q/fr22bt0qt9utzs7Om867XC7Ztj3mOpfLNaGC5s+fM6Hx72VZcyd9bbKi59RAz6khHj1HFfLt7e3avHmzKioqVFRUpBMnTqinp2f0fCgUksfjUUZGxk3Hw+GwPB7PhArq7e3TyMjYFwsnljVX4fC1CV+XzOg5NdDz1M6bSJPpOS3NdcubY8flmkuXLumb3/ymampqVFRUJElatGiROjs71dXVpUgkoqamJnm9XmVmZio9PV3t7e2SpIaGBnm93gkXDQCIDcc7+d/85jcaHBzUrl27Ro+VlpZq165d2rRpkwYHB5WXlyefzydJqqmpUVVVlfr7+7Vw4UKVlZXFr3oAwC257PEW0hOI5Zro0XNqoOepndf/3YNTPq8kNT67MjHLNQCA5EXIA4DBCHkAMBghDwAGI+QBwGCEPAAYjJAHAIMR8gBgMEIeAAxGyAOAwQh5ADAYIQ8ABiPkAcBghDwAGIyQBwCDEfIAYDBCHgAMRsgDgMEIeQAwmOMv8gbwX3M/PEsz06f+n8yNociUzwlzEPJAlGamuxPyS54bn1055XPCHMaE/I2hiCxrbkLmHhgc1rWr1xMyNwDcStQh39fXp9LSUv3qV7/SPffco7a2NlVXV2twcFBLly5VeXm5JKmjo0NVVVXq6+vTkiVL9NRTT8ntjv9ryYzp0xJylyX9907rWkJmBoBbi+qN11OnTmn16tU6f/68JGlgYEAVFRWqq6tTc3Ozzpw5o9bWVknSli1btH37drW0tMi2bdXX18eteADArUUV8vX19dqxY4c8Ho8k6fTp08rOzlZWVpbcbrf8fr8CgYC6u7s1MDCgnJwcSVJxcbECgUDcigcA3FpU6yg7d+686XEoFJJlWaOPPR6PgsHgmOOWZSkYDMaoVADARE1qsdy27THHXC7X+x6fiPnz50ympIRL1Ju+iZo3kejZfIn8IEUixaPnSYV8RkaGenp6Rh+HQiF5PJ4xx8Ph8OgST7R6e/s0MjL2xcJJor8hwuGpf+vVsuYmZN5ESmTPifweS8XnORU/rjqZ5zktzXXLm+NJ/cTrokWL1NnZqa6uLkUiETU1Ncnr9SozM1Pp6elqb2+XJDU0NMjr9U5mCgBADEzqTj49PV27du3Spk2bNDg4qLy8PPl8PklSTU2Nqqqq1N/fr4ULF6qsrCymBQMAojehkD9y5Mjo33Nzc3Xo0KExYxYsWKB9+/bdfmUAgNvGBmUAYDBCHgAMRsgDgMGM2aAMMFWiPjPOxntmIOSBD7hEbb7HxntmYLkGAAxGyAOAwQh5ADAYa/IAxpWqm4SZhpAHMK5E/7Y1xAbLNQBgMEIeAAxGyAOAwQh5ADAYIQ8ABiPkAcBghDwAGIzPyWNS5n54lmamT/23z42hyJTPCSQzQh6TMjPdnbCdEQFEj+UaADAYIQ8ABiPkAcBgrMknsVTcJTAVewZuR1xCvrGxUc8995yGhob0+OOPa+3atfGYJuWl4i6BqdgzcDtiHvLBYFC1tbXav3+/ZsyYodLSUj300EP6+Mc/HuupAAAOYh7ybW1tevjhh3XHHXdIkgoLCxUIBPStb30rquvT0lyTnttz56xJX3u7bqfu25HInhM1Nz2bP28i5062HHG6xmXbtj3Zgsbz61//Wu+8847Ky8slSS+99JJOnz6tp59+OpbTAACiEPNP14z3muFyJeYuFwBSXcxDPiMjQz09PaOPQ6GQPB5PrKcBAEQh5iH/6U9/WsePH9eVK1d0/fp1HT58WF6vN9bTAACiEPM3XjMyMlReXq6ysjINDQ1p1apVeuCBB2I9DQAgCjF/4xUA8MHBtgYAYDBCHgAMRsgDgMEIeQAwWNKFfGNjo5YtW6bPf/7zevHFF8ec7+joUElJiQoLC1VZWanh4eEEVBlbTj3/5S9/0cqVK7VixQpt3LhRb7/9dgKqjC2nnt919OhRPfroo1NYWfw49Xzu3Dl95Stf0YoVK/TEE0+kxPN89uxZlZSUaMWKFVq/fr2uXr2agCpjq6+vT8uXL9cbb7wx5lxc8stOIpcvX7bz8/Ptt956y+7v77f9fr/9+uuv3zSmqKjI/uc//2nbtm1/73vfs1988cUEVBo7Tj1fu3bNfuSRR+zLly/btm3bP/3pT+2nn346UeXGRDTPs23bdjgctn0+n52fn5+AKmPLqeeRkRG7oKDAbm1ttW3btp955hl79+7diSo3JqJ5nlevXm0fPXrUtm3brq6utn/yk58kotSYeeWVV+zly5fb999/v33x4sUx5+ORX0l1J//ezc9mz549uvnZu7q7uzUwMKCcnBxJUnFx8U3nk5FTz0NDQ/rBD36gjIwMSdJ9992nS5cuJarcmHDq+V1VVVVRb3z3QefU89mzZzV79uzRHyzcsGFD0m/hHc3zPDIyov7+fknS9evXNXPmzESUGjP19fXasWPHuLsAxCu/kirkQ6GQLMsafezxeBQMBt/3vGVZN51PRk4933nnnfrc5z4nSRoYGNCePXtGHycrp54l6YUXXtDChQu1aNGiqS4vLpx6vnDhgu666y5t3bpVfr9fO3bs0OzZsxNRasxE8zxv27ZNlZWV+sxnPqO2tjaVlpZOdZkxtXPnTi1ZsmTcc/HKr6QKedth8zOn88ko2p6uXbumdevWacGCBfrCF74wFaXFjVPPr732mg4fPqyNGzdOZVlx5dTz8PCwTpw4oS9/+ctqbGxUVlaWdu3aNZUlxpxTzwMDA6qsrNTvf/97HTt2TGvWrNHWrVunssQpFa/8SqqQd9r87P+fD4fDSb85WjQbvoVCIa1Zs0YLFizQzp07p7rEmHPqORAIKBwOq6SkRF//+tdH+09mTj1blqXs7Gx98pOflCQtX75cp0+fnvI6Y8mp59dee03p6emj26J86Utf0okTJ6a8zqkSr/xKqpB32vwsMzNT6enpam9vlyQ1NDQk/eZoTj1HIhFt2LBBS5cuVWVlZdL/z0Vy7nnz5s1qaWnRwYMHtWfPHnk8Hv3hD39IYMW3z6nnxYsX68qVK3r11VclSUeOHNH999+fqHJjwqnn7OxsXb58WefOnZMkvfzyy6MvciaKW37d9lu3U+zQoUN2UVGRXVBQYO/Zs8e2bdt+8skn7dOnT9u2bdsdHR12SUmJ7fP57O985zv24OBgIsuNiVv1fPjwYfu+++6zV6xYMfqnoqIiwRXfPqfn+V0XL1404tM1tu3c8yuvvGKXlJTYy5Yts7/2ta/ZPT09iSw3Jpx6Pnr0qO33++3ly5fbX/3qV+0LFy4kstyYyc/PH/10Tbzziw3KAMBgSbVcAwCYGEIeAAxGyAOAwQh5ADAYIQ8ABiPkAcBghDwAGIyQBwCD/Q8X+YFZ9mptcAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "categorical accuracy:  count    1195.000000\n",
      "mean        0.768064\n",
      "std         0.322588\n",
      "min         0.000000\n",
      "25%         0.600000\n",
      "50%         0.981132\n",
      "75%         1.000000\n",
      "max         1.000000\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "print(\"----top 1 categorical result----\")\n",
    "df_top1_categorical,df_top1_weight= categoricalAccuracy(df_top1_validation,y_cat_validate)\n",
    "\n",
    "print(\"----top 10 categorical result----\")\n",
    "df_top10_categorical,df_top10_weight = categoricalAccuracy(df_top10_validation,y_cat_validate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 448,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: 3l_nnet_2021-01-05-1800\\assets\n"
     ]
    }
   ],
   "source": [
    "#### 429 Save neuron networks\n",
    "\n",
    "nnet.save(\"3l_nnet_2021-01-05-1800\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 449,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 449,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD7CAYAAAB+B7/XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAT0ElEQVR4nO3df2yVZ/3/8VfL4RT40E2s5zTGNI3MZcRFGcmi1CkNRg6V09NudpkFBI2OH9lGs0ki0HZhcSk2uImSbYkk0ziZyeomUAhW5ghNSFUCcRAIbsRSNgT6a4S2jJ729FzfPxbOFwS8z7nPfXra6zwfyZLe527v+/3uufvi3nXu+7rzjDFGAABr5Ge7AACAtwh2ALAMwQ4AliHYAcAyBDsAWIZgBwDLEOwAYBlftnZ8+fJVxeMT9xL6oqKZ6u8fynYZGUefdqFPu9zYZ35+nmbN+r+kfi5rwR6Pmwkd7JImfH1eoU+70Kdd3PTJUAwAWIZgBwDLEOwAYBmCHQAsk1Sw/+pXv9KSJUsUDof129/+VpLU0dGhSCSiUCikbdu2ZbRIAEDyHK+KOXLkiP7+97+rtbVVsVhMS5YsUVlZmerr6/X73/9en/3sZ7VmzRq1t7ervLx8PGoGAPwPjmfsX/nKV/Taa6/J5/Opv79fY2NjGhgYUGlpqUpKSuTz+RSJRNTW1jYe9QIAHCR1HfvUqVO1fft2/eY3v1FFRYV6enoUCAQS64PBoLq7u1PacVHRzNQqHSfx2IjyfX5JUiBQeNOyrQKBwmyXMC7o0y70eWdJ36BUV1enVatWae3aterq6rplfV5eXko77u8fmpA3GAQChepsqkksz254S729g1msKLMCgUKr+7uOPu2Si33m5+clfULsOBTz73//W6dPn5YkTZ8+XaFQSP/4xz/U19eX+J6enh4Fg0E3dQMAPOYY7OfPn1djY6NGRkY0MjKid955R7W1tTp79qzOnTunsbEx7du3TwsWLBiPegEADhyHYsrLy3X8+HE9/PDDmjJlikKhkMLhsD796U9r3bp1ikajKi8vV0VFxXjUCwBwkNQYe11dnerq6m56raysTK2trRkpCgDgHneeAoBlCHYAsAzBDgCWIdgBwDIEOwBYhmAHAMsQ7ABgGYIdACyT9CRgALxVeNd0TStI/09wOBrT4MA1DyqCLQh2IEumFfgUWb8n7e3sfbFa9s9ziFQwFAMAliHYAcAyBDsAWIZgBwDLEOwAYBmCHQAsQ7ADgGUIdgCwDMEOAJYh2AHAMgQ7AFiGYAcAyxDsAGAZgh0ALEOwA4BlkpqP/aWXXtKf//xnSVJ5ebl+8pOfaNOmTTp27JimT58uSXrqqae0aNGizFUKAEiKY7B3dHTo8OHD2rVrl/Ly8vT444/r7bff1smTJ7Vz504Fg8HxqBMAkCTHoZhAIKCNGzfK7/dr6tSpuueee3ThwgVduHBBzz77rCKRiLZv3654PD4e9QIAHDgG+7333qsHHnhAktTV1aX9+/frG9/4hubPn68tW7aopaVFR48e1ZtvvpnpWgEASUj6madnzpzRmjVrtGHDBs2ePVsvv/xyYt2KFSu0e/duPfbYY0nvuKhoZmqVZlEgUJjtEjLK9v6us7nPG3uzuc8b0eedJRXsx44dU11dnerr6xUOh/Xee++pq6tLixcvliQZY+TzpfZc7P7+IcXjJuWCM+12v8TeXnsfFRwIFFrd33UTsU8vg+l6bxOxz0zIxT7z8/OSPiF2HIq5ePGinnzySb3wwgsKh8OSPgnyLVu26MqVKxodHdUbb7zBFTEAMEE4nma/+uqrikajam5uTrxWW1ur1atXa+nSpYrFYgqFQqqsrMxooQCA5DgGe2NjoxobG2+7bvny5Z4XBABID3eeAoBlCHYAsAzBDgCWIdgBwDIEOwBYhmAHAMsQ7ABgGYIdACxDsAOAZVKbuQuACu+armkF/Olg4uLoBFI0rcCnyPo9aW9n74vVHlQD3IqhGACwDMEOAJYh2AHAMgQ7AFgm5z88nXW3Xz5/QbbLAADP5Hyw+/wF6myqSSzPbngri9UAQPoYigEAyxDsAGAZgh0ALEOwA4BlCHYAsAzBDgCWIdgBwDIEOwBYhmAHAMskFewvvfSSwuGwwuGwtm7dKknq6OhQJBJRKBTStm3bMlokACB5jsHe0dGhw4cPa9euXdq9e7dOnTqlffv2qb6+Xq+88or279+vkydPqr29fTzqBQA4cAz2QCCgjRs3yu/3a+rUqbrnnnvU1dWl0tJSlZSUyOfzKRKJqK2tbTzqBQA4cJwE7N5770183dXVpf3792vFihUKBAKJ14PBoLq7u1PacVHRzJS+P1visREFAoU3Lef7/FmsyHs39mczm/u8sTeb+7wRfd5Z0rM7njlzRmvWrNGGDRvk8/l09uzZm9bn5eWltOP+/iHF4yaln8kEp19avs9/y+yPvb2DmS5r3AQChVb1cyde9jkRA+V6b7yfdrmxz/z8vKRPiJP68PTYsWP6wQ9+oPXr1+uRRx5RcXGx+vr6Eut7enoUDAZdlA0A8JpjsF+8eFFPPvmkXnjhBYXDYUnS3LlzdfbsWZ07d05jY2Pat2+fFixYkPFiAQDOHIdiXn31VUWjUTU3Nydeq62tVXNzs9atW6doNKry8nJVVFRktFAAQHIcg72xsVGNjY23Xdfa2up5QQCA9HDnKQBYhmAHAMsQ7ABgGYIdACxDsAOAZQh2ALAMwQ4AliHYAcAyBDsAWIZgBwDLEOwAYBmCHQAsQ7ADgGUIdgCwDMEOAJZJ+pmnACamkdExTx5mPRyNaXDgmldlIYsIdmCS80+dosj6PWlvZ++L1bL/8dC5gaEYALAMwQ4AliHYAcAyBDsAWIZgBwDLEOwAYBmCHQAsQ7ADgGWSDvahoSFVVlbq/PnzkqRNmzYpFAqpurpa1dXVevvttzNWJAAgeUndeXr8+HE1Njaqq6sr8drJkye1c+dOBYPBTNUGAHAhqTP2lpYWbd68ORHiH3/8sS5cuKBnn31WkUhE27dvVzwez2ihAIDkJBXsTU1NevDBBxPL/f39mj9/vrZs2aKWlhYdPXpUb775ZsaKBAAkz9UkYCUlJXr55ZcTyytWrNDu3bv12GOPJb2NoqKZbnY9IbidPW+isq2fO8mVPtMxmX5Hk6nWdLjp01Wwv/fee+rq6tLixYslScYY+Xypbaq/f0jxuHGze0+5+aX19tozB14gUGhVP3fiZZ82B8pkORZy8bjNz89L+oTY1eWOxhht2bJFV65c0ejoqN544w0tWrTIzaYAAB5zdcY+Z84crV69WkuXLlUsFlMoFFJlZaXXtQEAXEgp2A8ePJj4evny5Vq+fLnnBQEA0sOdpwBgGYIdACxDsAOAZQh2ALAMwQ4AliHYAcAyBDsAWIZgBwDLEOwAYBlXUwoAsM/I6JgnE5wNR2MaHLjmQUVwi2AHIEnyT52iyPo9aW9n74vVsn/exYmNoRgAsAzBDgCWIdgBwDIEOwBYhg9PUxSPjdxy5UBsJKrLV0ayVBEA3IxgT1G+z6/OppqbXpvd8JYkgh3AxMBQDABYhmAHAMsQ7ABgGYIdACxDsAOAZXLuqphZd/vl8xdkuwxkQeFd0zWtIOcOeeSgnDvKff6Cmy5X/ORSReSCaQU+zya5AiYyhmIAwDIEOwBYJqlgHxoaUmVlpc6fPy9J6ujoUCQSUSgU0rZt2zJaIAAgNY7Bfvz4cS1dulRdXV2SpOHhYdXX1+uVV17R/v37dfLkSbW3t2e6TgBAkhyDvaWlRZs3b1YwGJQknThxQqWlpSopKZHP51MkElFbW1vGCwUAJMfxqpimpqablnt6ehQIBBLLwWBQ3d3dKe+4qGhmyj8zkXnxrMhsmcy1Y2Iaj2MqV45bN32mfLmjMeaW1/Ly8lLecX//kOLxW7eVaZk6GHp7J+dTHgOBwklbeypyJQQmikwfU7l03F7vMz8/L+kT4pSviikuLlZfX19iuaenJzFMAwDIvpSDfe7cuTp79qzOnTunsbEx7du3TwsWLMhEbQAAF1IeiikoKFBzc7PWrVunaDSq8vJyVVRUZKI2AIALSQf7wYMHE1+XlZWptbU1IwUBANLDnacAYBmCHQAsQ7ADgGUIdgCwDMEOAJYh2AHAMgQ7AFiGYAcAyxDsAGAZgh0ALEOwA4BlCHYAsAzBDgCWIdgBwDIEOwBYhmAHAMsQ7ABgGYIdACxDsAOAZQh2ALAMwQ4AliHYAcAyvmwXAMAuI6NjCgQK097OcDSmwYFrHlSUewh2AJ7yT52iyPo9aW9n74vVGvSgnlzEUAwAWCatM/aVK1eqv79fPt8nm/npT3+quXPnelIYAMAd18FujFFnZ6cOHTqUCHYAQPa5Horp7OxUXl6eVq1apaqqKu3cudPLugAALrk+1R4YGFBZWZmee+45DQ8Pa+XKlfr85z+vhx56yMv6AAApch3s8+bN07x58yRJM2bM0KOPPqr29vakg72oaKbbXU848djITZd3xWMjyvf5s1hRary4NA3IhP91bObKceumT9fBfvToUY2OjqqsrEzSJ2PuqYy19/cPKR43bnfvWiYOhnyfX51NNYnl2Q1vqbd3clyoFQgUTppa05ErIWCbOx2buXTcXu8zPz8v6RNi12Psg4OD2rp1q6LRqIaGhrRr1y4tWrTI7eYAAB5xfca+cOFCHT9+XA8//LDi8biWLVuWGJoBAGRPWtcpPv3003r66ac9KgUA4AXuPAUAyxDsAGAZ628ZnXW3Xz5/QbbLQBoK75quaQXWH6qAZ6z/a/H5C265FBGTy7QCn2ezBQK5gKEYALAMwQ4AliHYAcAyBDsAWIZgBwDLEOwAYBmCHQAsQ7ADgGUIdgCwDMEOAJYh2AHAMtbPFQNgchoZHfPkmafD0ZgGB655VdakQLADmJD8U6d4Nvmb/U9HvRlDMQBgGYIdACxDsAOAZQh2ALAMwQ4Alpl0V8X89zNMYyNRXb4yksWKAOQCr569Ox6XX066YL/9M0wJdgCZ5eWzdzN9+SVDMQBgGYIdACyTVrDv3btXS5Ys0aJFi/T66697VRMAIA2ux9i7u7u1bds2/elPf5Lf71dtba2++tWv6gtf+IKX9QEAUuQ62Ds6OjR//nx96lOfkiQtXrxYbW1teuqpp5L6+fz8PLe7lu/uQErb+u/v93rZTU0TyWSoNThrOtthO655dYxno57r35vKz+QZY0zKVUn69a9/rY8//ljPPPOMJOmPf/yjTpw4oeeff97N5gAAHnE9xn67fw/y8ib+mR8A2M51sBcXF6uvry+x3NPTo2Aw6ElRAAD3XAf71772Nf3tb3/TRx99pGvXrunAgQNasGCBl7UBAFxw/eFpcXGxnnnmGa1cuVKjo6N69NFH9eUvf9nL2gAALrj+8BQAMDFx5ykAWIZgBwDLEOwAYBmCHQAsk/PB7jSR2enTp1VTU6PFixeroaFBsVgsC1Wmz6nPv/71r6qurlZVVZWeeOIJXblyJQtVpi/ZiekOHTqkb37zm+NYmbec+uzs7NSKFStUVVWlH/3oR9a+n6dOnVJNTY2qqqq0Zs0aDQwMZKHK9A0NDamyslLnz5+/ZZ2rDDI57NKlS2bhwoXm8uXL5urVqyYSiZgzZ87c9D3hcNj885//NMYYs2nTJvP6669nodL0OPU5ODhoHnroIXPp0iVjjDG//OUvzfPPP5+tcl1L5v00xpje3l5TUVFhFi5cmIUq0+fUZzweN6FQyLS3txtjjPn5z39utm7dmq1yXUvm/Vy6dKk5dOiQMcaYn/3sZ+YXv/hFNkpNy7vvvmsqKyvN/fffbz788MNb1rvJoJw+Y79xIrMZM2YkJjK77j//+Y+Gh4f1wAMPSJK+853v3LR+snDqc3R0VM8995yKi4slSffdd58uXryYrXJdc+rzusbGxqQnq5uInPo8deqUZsyYkbhhcO3atVq+fHm2ynUtmfczHo/r6tWrkqRr165p2rRp2Sg1LS0tLdq8efNt79x3m0E5Hew9PT0KBP7/rIzBYFDd3d13XB8IBG5aP1k49Tlr1ix961vfkiQNDw9rx44dieXJxKlPSXrttdf0xS9+UXPnzh3v8jzj1OcHH3ygz3zmM9qwYYMikYg2b96sGTNmZKPUtCTzfm7cuFENDQ36+te/ro6ODtXW1o53mWlramrSgw8+eNt1bjMop4PdOExk5rR+ski2j8HBQa1atUpz5szRI488Mh6lecqpz/fff18HDhzQE088MZ5lec6pz1gspiNHjuh73/ue9u7dq5KSEjU3N49niZ5w6nN4eFgNDQ363e9+p8OHD2vZsmXasGHDeJaYcW4zKKeD3Wkis/9e39vbOyknOktmwraenh4tW7ZMc+bMUVNT03iX6AmnPtva2tTb26uamhqtXr060fNk49RnIBBQaWmpvvSlL0mSKisrdeLEiXGvM11Ofb7//vsqKChITGXy3e9+V0eOHBn3OjPJbQbldLA7TWT2uc99TgUFBTp27Jgkaffu3ZNyojOnPsfGxrR27Vp9+9vfVkNDw6T8vxLJuc+6ujr95S9/0Z49e7Rjxw4Fg0H94Q9/yGLF7jj1OW/ePH300Uf617/+JUk6ePCg7r///myV65pTn6Wlpbp06ZI6OzslSe+8807iHzNbuM4gzz7anaRaW1tNOBw2oVDI7NixwxhjzOOPP25OnDhhjDHm9OnTpqamxlRUVJgf//jHJhqNZrNc1/5XnwcOHDD33XefqaqqSvxXX1+f5YrdcXo/r/vwww8n7VUxxjj3+e6775qamhqzZMkS88Mf/tD09fVls1zXnPo8dOiQiUQiprKy0nz/+983H3zwQTbLTcvChQsTV8Wkm0FMAgYAlsnpoRgAsBHBDgCWIdgBwDIEOwBYhmAHAMsQ7ABgGYIdACxDsAOAZf4fJMXMVskfxJoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#### 428 first layer neurons weight result check.\n",
    "df_feature_weight = pd.DataFrame(nnet.get_weights()[0],index = X.columns)\n",
    "\n",
    "df_feature_weight.max().hist()\n",
    "df_feature_weight.mean().hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "nnet_predict_cm = pd.DataFrame(df_nnet_predict.idxmax(axis=1), columns = ['lab_id'])\n",
    "y_cat_cm = pd.DataFrame(y_cat_validate.idxmax(axis=1), columns = ['lab_id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "df_new = confusion_matrix(y_cat_cm,nnet_predict_cm, normalize=\"true\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 430 X boost (functional but took a long time.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "from xgboost.sklearn import XGBClassifier\n",
    "\n",
    "xgb = XGBClassifier(n_estimators=400,learning_rate=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": true
   },
   "source": [
    "xgb.fit(X_res,y_lab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "start = time.perf_counter()\n",
    "result= xgb.score(X_validate, y_lab_validate)\n",
    "end   = time.perf_counter()\n",
    "\n",
    "print(result)\n",
    "print(end-start)\n",
    "\n",
    "start = time.perf_counter()\n",
    "top10_result=top10_accuracy_label(xgb, X_validate, y_lab_validate.to_numpy())\n",
    "end   = time.perf_counter()\n",
    "\n",
    "print(top10_result)\n",
    "print(end-start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 440 MultinomialNB (functional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.inspection import permutation_importance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MNB = MultinomialNB(alpha=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "MNB.fit(X_scaled,y_lab_res)\n",
    "\n",
    "alarm.Beep(freq,duration)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.perf_counter()\n",
    "result= MNB.score(X_v, y_lab_validate)\n",
    "\n",
    "print(\"Single value accuracy rating:\",result)\n",
    "\n",
    "top10_result=top10_accuracy_label(MNB, X_v, y_lab_validate.to_numpy())\n",
    "end   = time.perf_counter()\n",
    "\n",
    "print(\"Top 10 value accuracy:\",top10_result)\n",
    "print(\"Process time:\",end-start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "param_metric = {\"alpha\":[0.005,0.001,0.0005],\n",
    "               \"fit_prior\":[True]}\n",
    "\n",
    "\n",
    "GS_MNB = GridSearchCV(MNB, \n",
    "                      param_grid=param_metric,\n",
    "                      scoring=\"accuracy\",\n",
    "                      n_jobs= 4,\n",
    "                      verbose = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": true
   },
   "source": [
    "MNB_best = GS_MNB.best_estimator_\n",
    "\n",
    "start = time.perf_counter()\n",
    "result= MNB_best.score(X_validate, y_lab_validate)\n",
    "end   = time.perf_counter()\n",
    "\n",
    "print(\"Single value accuracy rating:\" + str(result))\n",
    "print(\"Single value process time:\"+ str(end-start))\n",
    "\n",
    "start = time.perf_counter()\n",
    "top10_result=top10_accuracy_label(MNB_best, X_validate, y_lab_validate.to_numpy())\n",
    "end   = time.perf_counter()\n",
    "\n",
    "print(\"Top 10 value accuracy:\"+str(top10_result))\n",
    "print(\"Top 10 value process time:\"+str(end-start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 441 get premutation importance for each feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.perf_counter()\n",
    "r = permutation_importance(MNB,\n",
    "                           X_v,\n",
    "                           y_lab_validate,\n",
    "                           n_repeats=30,\n",
    "                           n_jobs = 4,\n",
    "                           random_state=42)\n",
    "end   = time.perf_counter()\n",
    "\n",
    "print(\"Premutation importance process time:\"+ str(end-start))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "effectives = getEffective(r,featureLS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alarm.Beep(freq,duration)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 442 refit the Input matrix with effective feature for MNB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_effectives = X_scaled[effectives]\n",
    "X_effectives.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 443 try it again with the MNB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MNB.fit(X_effectives,y_lab_res)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.perf_counter()\n",
    "result= MNB.score(X_v[effectives], y_lab_validate)\n",
    "end   = time.perf_counter()\n",
    "\n",
    "print(\"Single value accuracy rating:\" + str(result))\n",
    "print(\"Single value process time:\"+ str(end-start))\n",
    "\n",
    "start = time.perf_counter()\n",
    "top10_result=top10_accuracy_label(MNB, X_v[effectives], y_lab_validate.to_numpy())\n",
    "end   = time.perf_counter()\n",
    "\n",
    "print(\"Top 10 value accuracy:\"+str(top10_result))\n",
    "print(\"Top 10 value process time:\"+str(end-start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 445 Complement Naive Bayes classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "from sklearn.naive_bayes import ComplementNB\n",
    "from sklearn.metrics import fbeta_score, make_scorer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CNB_params = {\"alpha\":[.01,.001,0.001,.0001],\n",
    "               \"fit_prior\":[True]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CNB = ComplementNB()\n",
    "\n",
    "GS_CNB = GridSearchCV(CNB, \n",
    "                      param_grid=CNB_params,\n",
    "                      scoring=\"accuracy\",\n",
    "                      n_jobs= 4,\n",
    "                      verbose = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": true
   },
   "source": [
    "GS_CNB.fit(X_res,y_lab_res)\n",
    "\n",
    "print(\"The best parameters searched: \",GS_CNB.best_params_)\n",
    "print(\"The best model scores: \",GS_CNB.best_score_)\n",
    "\n",
    "alarm.Beep(freq,duration)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CNB_best = GS_CNB.best_estimator_\n",
    "\n",
    "start = time.perf_counter()\n",
    "result= CNB_best.score(X_validate, y_lab_validate)\n",
    "end   = time.perf_counter()\n",
    "\n",
    "print(\"Single value accuracy rating:\" + str(result))\n",
    "print(\"Single value process time:\"+ str(end-start))\n",
    "\n",
    "start = time.perf_counter()\n",
    "top10_result=top10_accuracy_label(CNB_best, X_validate, y_lab_validate.to_numpy())\n",
    "end   = time.perf_counter()\n",
    "\n",
    "print(\"Top 10 value accuracy:\"+str(top10_result))\n",
    "print(\"Top 10 value process time:\"+str(end-start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 450 RBF SVM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "from sklearn.gaussian_process import GaussianProcessClassifier\n",
    "from sklearn.gaussian_process.kernels import RBF\n",
    "\n",
    "kernel = 1.0 * RBF(1)\n",
    "\n",
    "gpc = GaussianProcessClassifier(kernel = kernel, n_jobs = 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "gpc.fit(X_res,y_lab_res)\n",
    "\n",
    "##### ops, oversized. may wanna try cross validation later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 500 Make Predictions (untouched, only useful if we are submitting to DataDriven site.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "test_values.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "test_values_filled = test_values.fillna(0)\n",
    "#test_values_filled = test_values.dropna(0)\n",
    "test_values_filled.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#test_ngram_features = get_ngram_features(test_values, subsequences)\n",
    "test_ngram_features = get_ngram_features(test_values_filled, subsequences)\n",
    "all_test_features   = test_ngram_features.join(test_values_filled.drop('sequence', axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Visualize the building blocks to \"all_test_features\" and \"all_test_features\"\n",
    "\n",
    "#test_values.describe()\n",
    "\n",
    "#test_ngram_features.describe()\n",
    "\n",
    "all_test_features.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "probas = rf.predict_proba(all_test_features)\n",
    "\n",
    "probas[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 520 Save Submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "submission_format = pd.read_csv(DATA_DIR + 'submission_format_3TFRxH6.csv', index_col='sequence_id')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "submission_format.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "test_values.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "probas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "probas.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "submission_format.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "test_values_filled.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "assert submission_format.shape == probas.shape\n",
    "#assert (rf.classes_ == submission_format.columns).all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 900 Utility functions below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 901 print progress bar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print iterations progress\n",
    "def printProgressBar (iteration, total, prefix = '', suffix = '', decimals = 1, length = 100, fill = '█', printEnd = \"\\r\"):\n",
    "    \"\"\"\n",
    "    Call in a loop to create terminal progress bar\n",
    "    @params:\n",
    "        iteration   - Required  : current iteration (Int)\n",
    "        total       - Required  : total iterations (Int)\n",
    "        prefix      - Optional  : prefix string (Str)\n",
    "        suffix      - Optional  : suffix string (Str)\n",
    "        decimals    - Optional  : positive number of decimals in percent complete (Int)\n",
    "        length      - Optional  : character length of bar (Int)\n",
    "        fill        - Optional  : bar fill character (Str)\n",
    "        printEnd    - Optional  : end character (e.g. \"\\r\", \"\\r\\n\") (Str)\n",
    "    \"\"\"\n",
    "    percent = (\"{0:.\" + str(decimals) + \"f}\").format(100 * (iteration / float(total)))\n",
    "    filledLength = int(length * iteration // total)\n",
    "    bar = fill * filledLength + '-' * (length - filledLength)\n",
    "    print(f'\\r{prefix} |{bar}| {percent}% {suffix}', end = printEnd)\n",
    "    # Print New Line on Complete\n",
    "    if iteration == total: \n",
    "        print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 902 quickSplit will function as train_test_split except when there is only one record\n",
    "\n",
    "Train_test_split will generally split the data set as per describe the test_size.\n",
    "\n",
    "However under the condition when there is only one set of data available, train_test_split will return an error.\n",
    "\n",
    "For quickSplit, it will solve the issue as per general train_test_split except for the single record, which quickSplit will return the exact value back as train and test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def quickSplit(X,y, TEST_SIZE=0.2):\n",
    "    if X.shape[0] ==1:\n",
    "        X_train = X\n",
    "        X_test  = X\n",
    "        y_train = y\n",
    "        y_test  = y\n",
    "    else:\n",
    "        X_train,X_test,y_train,y_test = train_test_split(X,y,test_size = TEST_SIZE)\n",
    "    return X_train,X_test,y_train,y_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 903 randomSplit is used for one column label output split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def randomSplit_lab(df_X,df_y,col_label = \"lab_id\",TEST_SIZE=0.2):\n",
    "    start = time.perf_counter()\n",
    "    \n",
    "    ls_y_labels = list(df_y[col_label].unique())\n",
    "\n",
    "    X_train = pd.DataFrame()\n",
    "    X_test  = pd.DataFrame()\n",
    "    y_train = pd.DataFrame()\n",
    "    y_test  = pd.DataFrame()\n",
    "    \n",
    "    i = 1\n",
    "    total = len(ls_y_labels)\n",
    "    \n",
    "    for label in ls_y_labels:\n",
    "        y = df_y[df_y[col_label] == label]\n",
    "        X = df_X.loc[y.index]\n",
    "        \n",
    "        Xa,Xb,ya,yb  =  quickSplit(X,y,TEST_SIZE)\n",
    "\n",
    "        X_train = X_train.append(Xa)\n",
    "        X_test  = X_test.append(Xb)\n",
    "        y_train = y_train.append(ya)\n",
    "        y_test  = y_test.append(yb)\n",
    "        \n",
    "        printProgressBar(i,total)\n",
    "        i+=1\n",
    "    \n",
    "    end   = time.perf_counter()\n",
    "    print(\"training, holdout split complete\",end-start)\n",
    "    alarm.Beep(freq, duration)\n",
    "    \n",
    "    return X_train,X_test,y_train,y_test\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 904 randomSplit_cat is used for multiple columns categorical output split."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "def randomSplit_cat(df_X,df_y,TEST_SIZE=0.2):\n",
    "    start = time.perf_counter()\n",
    "\n",
    "    ls_y_labels = list(df_y.columns)\n",
    "\n",
    "    X_train = pd.DataFrame()\n",
    "    X_test  = pd.DataFrame()\n",
    "    y_train = pd.DataFrame()\n",
    "    y_test  = pd.DataFrame()\n",
    "    \n",
    "    i = 1\n",
    "    total = len(ls_y_labels)\n",
    "    \n",
    "    for label in ls_y_labels:\n",
    "        y = df_y[df_y[label]==1]\n",
    "        X = df_X.loc[y.index]\n",
    "        \n",
    "        Xa,Xb,ya,yb  =  quickSplit(X,y,TEST_SIZE)\n",
    "\n",
    "        X_train = X_train.append(Xa)\n",
    "        X_test  = X_test.append(Xb)\n",
    "        y_train = y_train.append(ya)\n",
    "        y_test  = y_test.append(yb)\n",
    "        \n",
    "        printProgressBar(iteration = i, total = total, length = 50)\n",
    "        i+=1\n",
    "    \n",
    "    end   = time.perf_counter()\n",
    "    print(\"training, holdout split complete\",end-start)\n",
    "    alarm.Beep(freq, duration)\n",
    "    \n",
    "    return X_train,X_test,y_train,y_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 905 get n gram features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_ngram_features(data, subsequences):\n",
    "    \"\"\"Generates counts for each subsequence.\n",
    "    \n",
    "    Args:\n",
    "        data (DataFrame): The data you want to create features from. Must subsequences (list): A list of subsequences to count\n",
    "        \n",
    "    Returns:\n",
    "        DataFrame: A DataFrame with one column for each subsequence\n",
    "    \"\"\"\n",
    "    start = time.perf_counter()\n",
    "    i = 1\n",
    "    total = len(subsequences)\n",
    "    \n",
    "    features = pd.DataFrame(index=data.index)\n",
    "    \n",
    "    for subseq in subsequences:\n",
    "        features[subseq] = data.sequence.str.count(subseq)\n",
    "        \n",
    "        printProgressBar(iteration = i, total = total, length = 50)\n",
    "        i+=1\n",
    "    \n",
    "    end = time.perf_counter()\n",
    "    \n",
    "    print(\"process time: \",end-start)\n",
    "    return features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 906 get n base features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_nbase_features(data, bases):\n",
    "    \"\"\"Generates counts for each subsequence of n gram of the bases.\n",
    "    \n",
    "    Args:\n",
    "        data (DataFrame): The data you want to create features from. Must \n",
    "        bases (list): A list of base elements to construct sequences\n",
    "        \n",
    "    Returns:\n",
    "        DataFrame: A DataFrame with one column for each subsequence\n",
    "    \"\"\"\n",
    "    start = time.perf_counter()\n",
    "    \n",
    "    ngrams = [x+1 for x in range(len(bases))]\n",
    "    subseqs_ng = []\n",
    "    \n",
    "    for n in ngrams:\n",
    "        subseqs_ng.append([''.join(permutation) for permutation in permutations(bases, r=n)])\n",
    "        print(n,\" gram variation count: \",len(subseqs_ng[n-1]))\n",
    "    \n",
    "    end = time.perf_counter()\n",
    "    \n",
    "    print(\"n gram permutation time: \",end-start)\n",
    "    \n",
    "    features = pd.DataFrame(index=data.index)\n",
    "    i = 1\n",
    "    total = len(subseqs_ng)\n",
    "    \n",
    "    for subseqs in subseqs_ng:\n",
    "        printProgressBar(iteration = i, total = total, length = 50)\n",
    "        \n",
    "        for subseq in subseqs:\n",
    "            features[subseq] = data.sequence.str.count(subseq)\n",
    "        \n",
    "        #print(i, \" gram features generation time: \",end-start)\n",
    "        i+=1\n",
    "    \n",
    "    return features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 907 weighted categorical crossentropy loss function with weight value as input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "A weighted version of categorical_crossentropy for keras (2.0.6). This lets you apply a weight to unbalanced classes.\n",
    "@url: https://gist.github.com/wassname/ce364fddfc8a025bfab4348cf5de852d\n",
    "@author: wassname\n",
    "\"\"\"\n",
    "from keras import backend as K\n",
    "def weighted_categorical_crossentropy(weights):\n",
    "    \"\"\"\n",
    "    A weighted version of keras.objectives.categorical_crossentropy\n",
    "    \n",
    "    Variables:\n",
    "        weights: numpy array of shape (C,) where C is the number of classes\n",
    "    \n",
    "    Usage:\n",
    "        weights = np.array([0.5,2,10]) # Class one at 0.5, class 2 twice the normal weights, class 3 10x.\n",
    "        loss = weighted_categorical_crossentropy(weights)\n",
    "        model.compile(loss=loss,optimizer='adam')\n",
    "    \"\"\"\n",
    "    \n",
    "    weights = K.variable(weights)\n",
    "        \n",
    "    def loss(y_true, y_pred):\n",
    "        # scale predictions so that the class probas of each sample sum to 1\n",
    "        y_pred /= K.sum(y_pred, axis=-1, keepdims=True)\n",
    "        # clip to prevent NaN's and Inf's\n",
    "        y_pred = K.clip(y_pred, K.epsilon(), 1 - K.epsilon())\n",
    "        # calc\n",
    "        loss = y_true * K.log(y_pred) * weights\n",
    "        loss = -K.sum(loss, -1)\n",
    "        return loss\n",
    "    \n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 908 top 10 finder which take a binary dummy output matrix and "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "####     compare with another binary dummy validation matrix and return\n",
    "####     validation result and print top 10 validation accuracy\n",
    "\n",
    "def top10_accuracy_cat(y_predict, y_validate):\n",
    "    top10_idx = np.argpartition(y_predict.to_numpy(), -10, axis=1)[:, -10:]\n",
    "    \n",
    "    validate_idx = np.argpartition(y_validate.to_numpy(),-1,axis=1)[:,-1:]\n",
    "    df_top10 = pd.DataFrame(top10_idx)\n",
    "    df_validate = pd.DataFrame(validate_idx)\n",
    "    \n",
    "    df_top10 = df_top10.sub(df_validate[0],axis = 0)\n",
    "    df_top10[\"validation\"] = df_top10.eq(0).any(axis=1).astype(\"int32\")\n",
    "    \n",
    "    df_result = pd.DataFrame(df_top10[\"validation\"],columns = [\"validation\"])\n",
    "    df_result[\"index\"]=y_validate.index\n",
    "    df_result.set_index(\"index\",drop=True,inplace=True)\n",
    "    \n",
    "    print(\"Top 10 Validation Accuracy: \",df_top10[\"validation\"].mean())\n",
    "    \n",
    "    return df_result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 909 top 1 finder which take a binary dummy output matrix and "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "####     compare with another binary dummy validation matrix and return\n",
    "####     validation result and print top 1 validation accuracy\n",
    "\n",
    "def top1_accuracy_cat(y_predict, y_validate):\n",
    "    top1_idx = np.argpartition(y_predict.to_numpy(), -1, axis=1)[:, -1:]\n",
    "    \n",
    "    validate_idx = np.argpartition(y_validate.to_numpy(),-1,axis=1)[:,-1:]\n",
    "    df_top1 = pd.DataFrame(top1_idx)\n",
    "    df_validate = pd.DataFrame(validate_idx)\n",
    "    \n",
    "    df_top1 = df_top1.sub(df_validate[0],axis = 0)\n",
    "    df_top1[\"validation\"] = df_top1.eq(0).any(axis=1).astype(\"int32\")\n",
    "    \n",
    "    df_result = pd.DataFrame(df_top1[\"validation\"],columns = [\"validation\"])\n",
    "    df_result[\"index\"]=y_validate.index\n",
    "    df_result.set_index(\"index\",drop=True,inplace=True)\n",
    "\n",
    "    \n",
    "    print(\"Top 1 Validation Accuracy: \",df_top1[\"validation\"].mean())\n",
    "    \n",
    "    return df_result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 909 top 10 finder for normal estimators which accepts label output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def top10_accuracy_label(estimator, X, y):\n",
    "    \"\"\"A custom scorer that evaluates a model on whether the correct label the top 10 most probably predictions.\n",
    "    \n",
    "    Args:\n",
    "        estimatory (sklearn estimator): The skLearn model that should be evaluatioed \n",
    "        X (numpy array): The validation data.\n",
    "        y (numpy array): The ground truth lables.\n",
    "        \n",
    "    Returns:\n",
    "        float: Accuracy of the model as defined by the proportion of prediction in which the correct label was in Top 10\n",
    "    \"\"\"\n",
    "    # predict the probabilities across all possible labels for rows in our training set\n",
    "    probas = estimator.predict_proba(X)\n",
    "    \n",
    "    # get the indices for top 10 predictions for each row; these are these are the last ten in each row\n",
    "    # Note: We use argpartition, which is O(n), vs argsort, which uses the quicksort algorithm \n",
    "    # by default and is O(n^2) in the worst case. We can do this because we only need the top ten\n",
    "    # partitioned, not in sorted order.\n",
    "    \n",
    "    # Documentation: https://numpy.org/doc/1.18/reference/generated/numpy.argpartition.html\n",
    "    top10_idx = np.argpartition(probas, -10, axis=1)[:, -10:]\n",
    "    \n",
    "    # index into the classes list using the top ten indices to get the class names\n",
    "    top10_preds = estimator.classes_[top10_idx]\n",
    "\n",
    "    # check if y-true is in top 10 for each set of predictions\n",
    "    mask = top10_preds == y.reshape((y.size, 1))\n",
    "    \n",
    "    # take the mean\n",
    "    top_10_accuracy = mask.any(axis=1).mean()\n",
    " \n",
    "    return top_10_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 910 per category result top 10 accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "####     input: dataframe result from top 1 validation or top 10 validation and validation output.\n",
    "####     output:print out the categorical accuracy\n",
    "\n",
    "import math\n",
    "\n",
    "def categoricalAccuracy(df_validation,y_validate):\n",
    "    \n",
    "    df_filtered       = y_validate*df_validation.to_numpy()\n",
    "    df_result         = df_filtered.sum()/y_validate.sum()\n",
    "    categoricalWeight = ((y_validate.sum()/df_filtered.sum()+1)/2).apply(math.tanh).to_numpy()\n",
    "    \n",
    "    df_result.hist()\n",
    "    plt.show()\n",
    "    print(\"categorical accuracy: \",df_result.describe())\n",
    "    \n",
    "    return df_result,categoricalWeight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 911 Bayesian Gaussian Mixture feature generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def BGMfeatureGen(X,y=None, BGM = None):\n",
    "    start = time.perf_counter()\n",
    "    \n",
    "    if BGM is None:\n",
    "        from sklearn.mixture import BayesianGaussianMixture\n",
    "\n",
    "        BGM = BayesianGaussianMixture()\n",
    "        BGM = BayesianGaussianMixture(n_components=100,max_iter=100)\n",
    "        \n",
    "        if y is None:\n",
    "            BGM.fit(X)\n",
    "        else:\n",
    "            BGM.fit(X,y)\n",
    "    else:\n",
    "        pass\n",
    "    \n",
    "    BGM_cluster   = pd.DataFrame(BGM.predict(X),columns = [\"BGM\"])\n",
    "    X_transformed = X.join(BGM_cluster)\n",
    "    \n",
    "    alarm.Beep(freq,duration)\n",
    "\n",
    "    end = time.perf_counter()\n",
    "    print(\"Bayesian Gaussian Mixture transform Time:\"+ str(end-start))\n",
    "        \n",
    "    return X_transformed,BGM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 912 get effective feature list from the number of features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getEffective(r,featureLS):\n",
    "    effectives = []\n",
    "    for i in r.importances_mean.argsort()[::-1]:\n",
    "        if r.importances_mean[i] - 2 * r.importances_std[i] > 0:\n",
    "            #print(f\"{featureLS[i]:<8}\"f\"{r.importances_mean[i]:.3f}\"f\" +/- {r.importances_std[i]:.3f}\")\n",
    "            effectives.append(featureLS[i])\n",
    "    return effectives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 913 ordinal data to binary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ordin2Bin(s_ord,label = \"new\"):\n",
    "    digits = math.floor(math.log(s_ord.max(),2))\n",
    "    df_bin = pd.DataFrame()\n",
    "    \n",
    "    for digit in range(digits,-1,-1):\n",
    "        getDigit = lambda x: math.floor(x/pow(2,digit))\n",
    "        getRemainder = lambda x: x%pow(2,digit)\n",
    "        \n",
    "        df_bin[label+str(digit)]=s_ord.apply(getDigit)\n",
    "    \n",
    "        s_ord = s_ord.apply(getRemainder)\n",
    "        \n",
    "    return df_bin"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}