kaggle_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preparing the Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- A base de dados em questão foi retirada do site kaggle.\n",
    "\n",
    "- Fonte: [Software Defect Prediction](https://www.kaggle.com/datasets/semustafacevik/software-defect-prediction/data)\n",
    "\n",
    "\"This is a PROMISE data set made publicly available in order to encourage repeatable, verifiable, refutable, and/or improvable predictive models of software engineering.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.utils import shuffle\n",
    "from sklearn import preprocessing\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import torch\n",
    "from torch.utils.data import TensorDataset, Dataset, DataLoader\n",
    "from torchvision import transforms, utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aqui usamos a função `read_csv` do pandas para ler os arquivos jm1.csv, train.csv e test.csv. Esse último, em primeiro momento, não será efetivamente utilizado com o modelo, mas está aí caso necessário"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lendo os arquivos de base de dados\n",
    "original_database = pd.read_csv('data/jm1.csv')\n",
    "train_data = pd.read_csv('data/train.csv')\n",
    "test_data = pd.read_csv('data/test.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Duplicated Lines:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0, 0)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Verificando duplicatas\n",
    "print('Duplicated Lines:')\n",
    "train_data.duplicated().sum(), test_data.duplicated().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "O arquivo jm1.csv e train.csv são diferentes em quantidade, sendo train.csv em torno de 10 vezes maior, mas usando concatenação, podemos juntá-los e adicionar mais dados ao treino."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.concat([train_data, original_database], axis=0, ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data\n",
    "data_id = train_data.pop('id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>loc</th>\n",
       "      <th>v(g)</th>\n",
       "      <th>ev(g)</th>\n",
       "      <th>iv(g)</th>\n",
       "      <th>n</th>\n",
       "      <th>v</th>\n",
       "      <th>l</th>\n",
       "      <th>d</th>\n",
       "      <th>i</th>\n",
       "      <th>e</th>\n",
       "      <th>...</th>\n",
       "      <th>lOCode</th>\n",
       "      <th>lOComment</th>\n",
       "      <th>lOBlank</th>\n",
       "      <th>locCodeAndComment</th>\n",
       "      <th>uniq_Op</th>\n",
       "      <th>uniq_Opnd</th>\n",
       "      <th>total_Op</th>\n",
       "      <th>total_Opnd</th>\n",
       "      <th>branchCount</th>\n",
       "      <th>defects</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.1</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1.3</td>\n",
       "      <td>1.30</td>\n",
       "      <td>1.30</td>\n",
       "      <td>1.30</td>\n",
       "      <td>1.30</td>\n",
       "      <td>1.30</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1.2</td>\n",
       "      <td>1.2</td>\n",
       "      <td>1.2</td>\n",
       "      <td>1.2</td>\n",
       "      <td>1.4</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>72.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>1134.13</td>\n",
       "      <td>0.05</td>\n",
       "      <td>20.31</td>\n",
       "      <td>55.85</td>\n",
       "      <td>23029.10</td>\n",
       "      <td>...</td>\n",
       "      <td>51</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>17</td>\n",
       "      <td>36</td>\n",
       "      <td>112</td>\n",
       "      <td>86</td>\n",
       "      <td>13</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>190.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>600.0</td>\n",
       "      <td>4348.76</td>\n",
       "      <td>0.06</td>\n",
       "      <td>17.06</td>\n",
       "      <td>254.87</td>\n",
       "      <td>74202.67</td>\n",
       "      <td>...</td>\n",
       "      <td>129</td>\n",
       "      <td>29</td>\n",
       "      <td>28</td>\n",
       "      <td>2</td>\n",
       "      <td>17</td>\n",
       "      <td>135</td>\n",
       "      <td>329</td>\n",
       "      <td>271</td>\n",
       "      <td>5</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>37.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>599.12</td>\n",
       "      <td>0.06</td>\n",
       "      <td>17.19</td>\n",
       "      <td>34.86</td>\n",
       "      <td>10297.30</td>\n",
       "      <td>...</td>\n",
       "      <td>28</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>16</td>\n",
       "      <td>76</td>\n",
       "      <td>50</td>\n",
       "      <td>7</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10880</th>\n",
       "      <td>18.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>241.48</td>\n",
       "      <td>0.14</td>\n",
       "      <td>7.33</td>\n",
       "      <td>32.93</td>\n",
       "      <td>1770.86</td>\n",
       "      <td>...</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>15</td>\n",
       "      <td>30</td>\n",
       "      <td>22</td>\n",
       "      <td>7</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10881</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>129.66</td>\n",
       "      <td>0.12</td>\n",
       "      <td>8.25</td>\n",
       "      <td>15.72</td>\n",
       "      <td>1069.68</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>8</td>\n",
       "      <td>19</td>\n",
       "      <td>11</td>\n",
       "      <td>3</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10882</th>\n",
       "      <td>42.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>103.0</td>\n",
       "      <td>519.57</td>\n",
       "      <td>0.04</td>\n",
       "      <td>26.40</td>\n",
       "      <td>19.68</td>\n",
       "      <td>13716.72</td>\n",
       "      <td>...</td>\n",
       "      <td>29</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "      <td>15</td>\n",
       "      <td>59</td>\n",
       "      <td>44</td>\n",
       "      <td>7</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10883</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>147.15</td>\n",
       "      <td>0.12</td>\n",
       "      <td>8.44</td>\n",
       "      <td>17.44</td>\n",
       "      <td>1241.57</td>\n",
       "      <td>...</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "      <td>21</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10884</th>\n",
       "      <td>19.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>272.63</td>\n",
       "      <td>0.09</td>\n",
       "      <td>11.57</td>\n",
       "      <td>23.56</td>\n",
       "      <td>3154.67</td>\n",
       "      <td>...</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>14</td>\n",
       "      <td>31</td>\n",
       "      <td>27</td>\n",
       "      <td>5</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10885 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         loc  v(g)  ev(g)  iv(g)      n        v     l      d       i  \\\n",
       "0        1.1   1.4    1.4    1.4    1.3     1.30  1.30   1.30    1.30   \n",
       "1        1.0   1.0    1.0    1.0    1.0     1.00  1.00   1.00    1.00   \n",
       "2       72.0   7.0    1.0    6.0  198.0  1134.13  0.05  20.31   55.85   \n",
       "3      190.0   3.0    1.0    3.0  600.0  4348.76  0.06  17.06  254.87   \n",
       "4       37.0   4.0    1.0    4.0  126.0   599.12  0.06  17.19   34.86   \n",
       "...      ...   ...    ...    ...    ...      ...   ...    ...     ...   \n",
       "10880   18.0   4.0    1.0    4.0   52.0   241.48  0.14   7.33   32.93   \n",
       "10881    9.0   2.0    1.0    2.0   30.0   129.66  0.12   8.25   15.72   \n",
       "10882   42.0   4.0    1.0    2.0  103.0   519.57  0.04  26.40   19.68   \n",
       "10883   10.0   1.0    1.0    1.0   36.0   147.15  0.12   8.44   17.44   \n",
       "10884   19.0   3.0    1.0    1.0   58.0   272.63  0.09  11.57   23.56   \n",
       "\n",
       "              e  ...  lOCode  lOComment  lOBlank  locCodeAndComment  uniq_Op  \\\n",
       "0          1.30  ...       2          2        2                  2      1.2   \n",
       "1          1.00  ...       1          1        1                  1        1   \n",
       "2      23029.10  ...      51         10        8                  1       17   \n",
       "3      74202.67  ...     129         29       28                  2       17   \n",
       "4      10297.30  ...      28          1        6                  0       11   \n",
       "...         ...  ...     ...        ...      ...                ...      ...   \n",
       "10880   1770.86  ...      13          0        2                  0       10   \n",
       "10881   1069.68  ...       5          0        2                  0       12   \n",
       "10882  13716.72  ...      29          1       10                  0       18   \n",
       "10883   1241.57  ...       6          0        2                  0        9   \n",
       "10884   3154.67  ...      13          0        2                  1       12   \n",
       "\n",
       "       uniq_Opnd total_Op total_Opnd branchCount defects  \n",
       "0            1.2      1.2        1.2         1.4   False  \n",
       "1              1        1          1           1    True  \n",
       "2             36      112         86          13    True  \n",
       "3            135      329        271           5    True  \n",
       "4             16       76         50           7    True  \n",
       "...          ...      ...        ...         ...     ...  \n",
       "10880         15       30         22           7   False  \n",
       "10881          8       19         11           3   False  \n",
       "10882         15       59         44           7   False  \n",
       "10883          8       21         15           1   False  \n",
       "10884         14       31         27           5   False  \n",
       "\n",
       "[10885 rows x 22 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "original_database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>loc</th>\n",
       "      <th>v(g)</th>\n",
       "      <th>ev(g)</th>\n",
       "      <th>iv(g)</th>\n",
       "      <th>n</th>\n",
       "      <th>v</th>\n",
       "      <th>l</th>\n",
       "      <th>d</th>\n",
       "      <th>i</th>\n",
       "      <th>...</th>\n",
       "      <th>lOCode</th>\n",
       "      <th>lOComment</th>\n",
       "      <th>lOBlank</th>\n",
       "      <th>locCodeAndComment</th>\n",
       "      <th>uniq_Op</th>\n",
       "      <th>uniq_Opnd</th>\n",
       "      <th>total_Op</th>\n",
       "      <th>total_Opnd</th>\n",
       "      <th>branchCount</th>\n",
       "      <th>defects</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>278.63</td>\n",
       "      <td>0.06</td>\n",
       "      <td>19.56</td>\n",
       "      <td>14.25</td>\n",
       "      <td>...</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>151.27</td>\n",
       "      <td>0.14</td>\n",
       "      <td>7.00</td>\n",
       "      <td>21.11</td>\n",
       "      <td>...</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>197.65</td>\n",
       "      <td>0.11</td>\n",
       "      <td>8.05</td>\n",
       "      <td>22.76</td>\n",
       "      <td>...</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>94.01</td>\n",
       "      <td>0.19</td>\n",
       "      <td>5.25</td>\n",
       "      <td>17.86</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>60.94</td>\n",
       "      <td>0.18</td>\n",
       "      <td>5.63</td>\n",
       "      <td>12.44</td>\n",
       "      <td>...</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112643</th>\n",
       "      <td>NaN</td>\n",
       "      <td>18.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>241.48</td>\n",
       "      <td>0.14</td>\n",
       "      <td>7.33</td>\n",
       "      <td>32.93</td>\n",
       "      <td>...</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>15</td>\n",
       "      <td>30</td>\n",
       "      <td>22</td>\n",
       "      <td>7</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112644</th>\n",
       "      <td>NaN</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>129.66</td>\n",
       "      <td>0.12</td>\n",
       "      <td>8.25</td>\n",
       "      <td>15.72</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>8</td>\n",
       "      <td>19</td>\n",
       "      <td>11</td>\n",
       "      <td>3</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112645</th>\n",
       "      <td>NaN</td>\n",
       "      <td>42.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>103.0</td>\n",
       "      <td>519.57</td>\n",
       "      <td>0.04</td>\n",
       "      <td>26.40</td>\n",
       "      <td>19.68</td>\n",
       "      <td>...</td>\n",
       "      <td>29</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "      <td>15</td>\n",
       "      <td>59</td>\n",
       "      <td>44</td>\n",
       "      <td>7</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112646</th>\n",
       "      <td>NaN</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>147.15</td>\n",
       "      <td>0.12</td>\n",
       "      <td>8.44</td>\n",
       "      <td>17.44</td>\n",
       "      <td>...</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "      <td>21</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112647</th>\n",
       "      <td>NaN</td>\n",
       "      <td>19.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>272.63</td>\n",
       "      <td>0.09</td>\n",
       "      <td>11.57</td>\n",
       "      <td>23.56</td>\n",
       "      <td>...</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>14</td>\n",
       "      <td>31</td>\n",
       "      <td>27</td>\n",
       "      <td>5</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>112648 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         id   loc  v(g)  ev(g)  iv(g)      n       v     l      d      i  ...  \\\n",
       "0       0.0  22.0   3.0    1.0    2.0   60.0  278.63  0.06  19.56  14.25  ...   \n",
       "1       1.0  14.0   2.0    1.0    2.0   32.0  151.27  0.14   7.00  21.11  ...   \n",
       "2       2.0  11.0   2.0    1.0    2.0   45.0  197.65  0.11   8.05  22.76  ...   \n",
       "3       3.0   8.0   1.0    1.0    1.0   23.0   94.01  0.19   5.25  17.86  ...   \n",
       "4       4.0  11.0   2.0    1.0    2.0   17.0   60.94  0.18   5.63  12.44  ...   \n",
       "...     ...   ...   ...    ...    ...    ...     ...   ...    ...    ...  ...   \n",
       "112643  NaN  18.0   4.0    1.0    4.0   52.0  241.48  0.14   7.33  32.93  ...   \n",
       "112644  NaN   9.0   2.0    1.0    2.0   30.0  129.66  0.12   8.25  15.72  ...   \n",
       "112645  NaN  42.0   4.0    1.0    2.0  103.0  519.57  0.04  26.40  19.68  ...   \n",
       "112646  NaN  10.0   1.0    1.0    1.0   36.0  147.15  0.12   8.44  17.44  ...   \n",
       "112647  NaN  19.0   3.0    1.0    1.0   58.0  272.63  0.09  11.57  23.56  ...   \n",
       "\n",
       "        lOCode  lOComment  lOBlank  locCodeAndComment  uniq_Op  uniq_Opnd  \\\n",
       "0           17          1        1                  0     16.0        9.0   \n",
       "1           11          0        1                  0     11.0       11.0   \n",
       "2            8          0        1                  0     12.0       11.0   \n",
       "3            4          0        2                  0      8.0        6.0   \n",
       "4            7          0        2                  0      7.0        6.0   \n",
       "...        ...        ...      ...                ...      ...        ...   \n",
       "112643      13          0        2                  0       10         15   \n",
       "112644       5          0        2                  0       12          8   \n",
       "112645      29          1       10                  0       18         15   \n",
       "112646       6          0        2                  0        9          8   \n",
       "112647      13          0        2                  1       12         14   \n",
       "\n",
       "        total_Op total_Opnd branchCount defects  \n",
       "0           38.0       22.0         5.0   False  \n",
       "1           18.0       14.0         3.0   False  \n",
       "2           28.0       17.0         3.0   False  \n",
       "3           16.0        7.0         1.0    True  \n",
       "4           10.0       10.0         3.0   False  \n",
       "...          ...        ...         ...     ...  \n",
       "112643        30         22           7   False  \n",
       "112644        19         11           3   False  \n",
       "112645        59         44           7   False  \n",
       "112646        21         15           1   False  \n",
       "112647        31         27           5   False  \n",
       "\n",
       "[112648 rows x 23 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A coluna de rótulo é denominada \"defects\". Ela originalmente é discreta e binária.\n",
    "\n",
    "Utilizamos a função `map()` para atribui `0` ao rótulo `False` e `1` ao rótulo `True`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Coluna de rótulos\n",
    "label_name = 'defects'\n",
    "# Transformação do rótulo para 0 e 1\n",
    "data[label_name] = data[label_name].map({False: 0, True: 1})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Algumas colunas podem apresentar valores nulos ou discretos, desinteressantes. Utilizamos a função `replace_with_nan` para substituir esses valores por 'NaN', que, futuramente, será utilizado como parâmetro de drop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Função para substituir valores \"descartáveis\" por 'NaN'\n",
    "def replace_with_nan(element):\n",
    "    if type(element) == str and not element.isalnum():\n",
    "        element = \"NaN\"\n",
    "    return element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>loc</th>\n",
       "      <th>v(g)</th>\n",
       "      <th>ev(g)</th>\n",
       "      <th>iv(g)</th>\n",
       "      <th>n</th>\n",
       "      <th>v</th>\n",
       "      <th>l</th>\n",
       "      <th>d</th>\n",
       "      <th>i</th>\n",
       "      <th>...</th>\n",
       "      <th>lOCode</th>\n",
       "      <th>lOComment</th>\n",
       "      <th>lOBlank</th>\n",
       "      <th>locCodeAndComment</th>\n",
       "      <th>uniq_Op</th>\n",
       "      <th>uniq_Opnd</th>\n",
       "      <th>total_Op</th>\n",
       "      <th>total_Opnd</th>\n",
       "      <th>branchCount</th>\n",
       "      <th>defects</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>278.63</td>\n",
       "      <td>0.06</td>\n",
       "      <td>19.56</td>\n",
       "      <td>14.25</td>\n",
       "      <td>...</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>151.27</td>\n",
       "      <td>0.14</td>\n",
       "      <td>7.00</td>\n",
       "      <td>21.11</td>\n",
       "      <td>...</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>197.65</td>\n",
       "      <td>0.11</td>\n",
       "      <td>8.05</td>\n",
       "      <td>22.76</td>\n",
       "      <td>...</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>94.01</td>\n",
       "      <td>0.19</td>\n",
       "      <td>5.25</td>\n",
       "      <td>17.86</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>60.94</td>\n",
       "      <td>0.18</td>\n",
       "      <td>5.63</td>\n",
       "      <td>12.44</td>\n",
       "      <td>...</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112643</th>\n",
       "      <td>NaN</td>\n",
       "      <td>18.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>241.48</td>\n",
       "      <td>0.14</td>\n",
       "      <td>7.33</td>\n",
       "      <td>32.93</td>\n",
       "      <td>...</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112644</th>\n",
       "      <td>NaN</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>129.66</td>\n",
       "      <td>0.12</td>\n",
       "      <td>8.25</td>\n",
       "      <td>15.72</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112645</th>\n",
       "      <td>NaN</td>\n",
       "      <td>42.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>103.0</td>\n",
       "      <td>519.57</td>\n",
       "      <td>0.04</td>\n",
       "      <td>26.40</td>\n",
       "      <td>19.68</td>\n",
       "      <td>...</td>\n",
       "      <td>29</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>59.0</td>\n",
       "      <td>44.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112646</th>\n",
       "      <td>NaN</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>147.15</td>\n",
       "      <td>0.12</td>\n",
       "      <td>8.44</td>\n",
       "      <td>17.44</td>\n",
       "      <td>...</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112647</th>\n",
       "      <td>NaN</td>\n",
       "      <td>19.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>272.63</td>\n",
       "      <td>0.09</td>\n",
       "      <td>11.57</td>\n",
       "      <td>23.56</td>\n",
       "      <td>...</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>12.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>112648 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         id   loc  v(g)  ev(g)  iv(g)      n       v     l      d      i  ...  \\\n",
       "0       0.0  22.0   3.0    1.0    2.0   60.0  278.63  0.06  19.56  14.25  ...   \n",
       "1       1.0  14.0   2.0    1.0    2.0   32.0  151.27  0.14   7.00  21.11  ...   \n",
       "2       2.0  11.0   2.0    1.0    2.0   45.0  197.65  0.11   8.05  22.76  ...   \n",
       "3       3.0   8.0   1.0    1.0    1.0   23.0   94.01  0.19   5.25  17.86  ...   \n",
       "4       4.0  11.0   2.0    1.0    2.0   17.0   60.94  0.18   5.63  12.44  ...   \n",
       "...     ...   ...   ...    ...    ...    ...     ...   ...    ...    ...  ...   \n",
       "112643  NaN  18.0   4.0    1.0    4.0   52.0  241.48  0.14   7.33  32.93  ...   \n",
       "112644  NaN   9.0   2.0    1.0    2.0   30.0  129.66  0.12   8.25  15.72  ...   \n",
       "112645  NaN  42.0   4.0    1.0    2.0  103.0  519.57  0.04  26.40  19.68  ...   \n",
       "112646  NaN  10.0   1.0    1.0    1.0   36.0  147.15  0.12   8.44  17.44  ...   \n",
       "112647  NaN  19.0   3.0    1.0    1.0   58.0  272.63  0.09  11.57  23.56  ...   \n",
       "\n",
       "        lOCode  lOComment  lOBlank  locCodeAndComment  uniq_Op  uniq_Opnd  \\\n",
       "0           17          1        1                  0     16.0        9.0   \n",
       "1           11          0        1                  0     11.0       11.0   \n",
       "2            8          0        1                  0     12.0       11.0   \n",
       "3            4          0        2                  0      8.0        6.0   \n",
       "4            7          0        2                  0      7.0        6.0   \n",
       "...        ...        ...      ...                ...      ...        ...   \n",
       "112643      13          0        2                  0     10.0       15.0   \n",
       "112644       5          0        2                  0     12.0        8.0   \n",
       "112645      29          1       10                  0     18.0       15.0   \n",
       "112646       6          0        2                  0      9.0        8.0   \n",
       "112647      13          0        2                  1     12.0       14.0   \n",
       "\n",
       "        total_Op  total_Opnd  branchCount  defects  \n",
       "0           38.0        22.0          5.0        0  \n",
       "1           18.0        14.0          3.0        0  \n",
       "2           28.0        17.0          3.0        0  \n",
       "3           16.0         7.0          1.0        1  \n",
       "4           10.0        10.0          3.0        0  \n",
       "...          ...         ...          ...      ...  \n",
       "112643      30.0        22.0          7.0        0  \n",
       "112644      19.0        11.0          3.0        0  \n",
       "112645      59.0        44.0          7.0        0  \n",
       "112646      21.0        15.0          1.0        0  \n",
       "112647      31.0        27.0          5.0        0  \n",
       "\n",
       "[112648 rows x 23 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Colunas que podem ter valores faltantes\n",
    "fix_cols = [\"uniq_Op\", \"uniq_Opnd\", \"total_Op\", \"total_Opnd\", \"branchCount\"]\n",
    "\n",
    "def fix_columns(df, cols):\n",
    "    # Cópia do dataframe\n",
    "    df_new = df.copy(deep = True)\n",
    "    # Aplicar a função 'replace_with_nan' para cara elemento de uma coluna\n",
    "    for col in cols:\n",
    "        df_new[col] = df_new[col].apply(replace_with_nan).astype(\"float\")\n",
    "    return df_new\n",
    "\n",
    "# Dataframe ajustado por 'replace_with_nan'\n",
    "data_fixed = fix_columns(data, fix_cols)\n",
    "data_fixed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[pandas.Dataframe.apply](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html): Aplica uma função ao longo de um eixo do dataframe.\n",
    "\n",
    "`df.apply(function, axis=0)`\n",
    "\n",
    "`axis`: {0 or 'index', 1 or 'column'}, default 0\n",
    "\n",
    "- 0: Aplica a função para cara coluna\n",
    "- 1: Aplica a função para cara \n",
    "\n",
    "`df = pd.DataFrame([[4, 9]] * 3, columns=['A', 'B'])`\n",
    "\n",
    "``df.apply(np.sqrt)``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Utilizamos a função `dropna()` e `drop()` descartar os nulos \"NaN\" e a coluna 'id'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loc                  0\n",
      "v(g)                 0\n",
      "ev(g)                0\n",
      "iv(g)                0\n",
      "n                    0\n",
      "v                    0\n",
      "l                    0\n",
      "d                    0\n",
      "i                    0\n",
      "e                    0\n",
      "b                    0\n",
      "t                    0\n",
      "lOCode               0\n",
      "lOComment            0\n",
      "lOBlank              0\n",
      "locCodeAndComment    0\n",
      "uniq_Op              0\n",
      "uniq_Opnd            0\n",
      "total_Op             0\n",
      "total_Opnd           0\n",
      "branchCount          0\n",
      "defects              0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "data_fixed_drop = data_fixed.dropna() # Remover valores nulos\n",
    "data_fixed_drop = data_fixed_drop.drop(labels='id',axis=1) # Remover coluna 'id'\n",
    "print(data_fixed_drop.isna().sum()) # Verificar os nulos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alguns atributos, como \"v(g)\", \"ev(g)\", \"l\", \"d\", \"i\", \"e\" e \"t\", podem ser descartados. A função `drop_columns` foi posta para descartar essas colunas de forma conjunta."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 101763 entries, 0 to 101762\n",
      "Data columns (total 15 columns):\n",
      " #   Column             Non-Null Count   Dtype  \n",
      "---  ------             --------------   -----  \n",
      " 0   loc                101763 non-null  float64\n",
      " 1   iv(g)              101763 non-null  float64\n",
      " 2   n                  101763 non-null  float64\n",
      " 3   v                  101763 non-null  float64\n",
      " 4   b                  101763 non-null  float64\n",
      " 5   lOCode             101763 non-null  int64  \n",
      " 6   lOComment          101763 non-null  int64  \n",
      " 7   lOBlank            101763 non-null  int64  \n",
      " 8   locCodeAndComment  101763 non-null  int64  \n",
      " 9   uniq_Op            101763 non-null  float64\n",
      " 10  uniq_Opnd          101763 non-null  float64\n",
      " 11  total_Op           101763 non-null  float64\n",
      " 12  total_Opnd         101763 non-null  float64\n",
      " 13  branchCount        101763 non-null  float64\n",
      " 14  defects            101763 non-null  int64  \n",
      "dtypes: float64(10), int64(5)\n",
      "memory usage: 12.4 MB\n"
     ]
    }
   ],
   "source": [
    "# Remover alguns atributos\n",
    "drop_cols = [\"v(g)\", \"ev(g)\", \"l\", \"d\", \"i\", \"e\", \"t\"] \n",
    "def drop_columns(df, cols):\n",
    "    df_new = df.copy(deep=True)\n",
    "    df_new = df_new.drop(labels=cols, axis=1)\n",
    "    return df_new\n",
    "# Dataframe ajustado e com menos atributos\n",
    "data_fixed_drop = drop_columns(data_fixed_drop, drop_cols)\n",
    "data_fixed_drop.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Retirada de valores nulos e da coluna 'id'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 101763 entries, 0 to 101762\n",
      "Data columns (total 15 columns):\n",
      " #   Column             Non-Null Count   Dtype  \n",
      "---  ------             --------------   -----  \n",
      " 0   loc                101763 non-null  float64\n",
      " 1   iv(g)              101763 non-null  float64\n",
      " 2   n                  101763 non-null  float64\n",
      " 3   v                  101763 non-null  float64\n",
      " 4   b                  101763 non-null  float64\n",
      " 5   lOCode             101763 non-null  int64  \n",
      " 6   lOComment          101763 non-null  int64  \n",
      " 7   lOBlank            101763 non-null  int64  \n",
      " 8   locCodeAndComment  101763 non-null  int64  \n",
      " 9   uniq_Op            101763 non-null  float64\n",
      " 10  uniq_Opnd          101763 non-null  float64\n",
      " 11  total_Op           101763 non-null  float64\n",
      " 12  total_Opnd         101763 non-null  float64\n",
      " 13  branchCount        101763 non-null  float64\n",
      " 14  defects            101763 non-null  int64  \n",
      "dtypes: float64(10), int64(5)\n",
      "memory usage: 12.4 MB\n"
     ]
    }
   ],
   "source": [
    "data_fixed_drop.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Como estamos utilizando dataframes panda, podemos aproveitar o sckit learn, pela função `train_test_split()` para dividir a base de dados para treino e teste"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "targets = 'defects'\n",
    "train_set, test_set = train_test_split(data_fixed_drop,  \n",
    "                                                    test_size=0.3,\n",
    "                                                    random_state=30)\n",
    "train_y = train_set['defects'].copy()\n",
    "test_y = test_set['defects'].copy()\n",
    "train_X = train_set.drop(labels='defects', axis=1)\n",
    "test_X = test_set.drop(['defects'], axis=1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pré-processamento de escala dos atributos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adicionando features de média para alguns atributos\n",
    "def add_feat(X):\n",
    "    df=X.copy()\n",
    "    df['mean_bnv']         = (df['n'] + df['v'] + df['b']) /3;\n",
    "    df['mean_uniqOpOpend'] = (df['uniq_Op'] + df['uniq_Opnd']) /2;\n",
    "    df['mean_totOpOpend']  = (df['total_Op'] + df['total_Opnd']) /2;\n",
    "    return df\n",
    "train_X = add_feat(train_X)\n",
    "test_data = add_feat(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Escala dos atributos para normalizar a variação entre valores discretos e números, além de diferentes dimensionalidades.\n",
    "\n",
    "Até então tínhamos alguns atributos int64 e object. Com a função `scale()`, podemos normalizar a escala e convertê-los em float64, que posteriormente será interessante para transformamos em dataset do pytorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 30529 entries, 0 to 30528\n",
      "Data columns (total 14 columns):\n",
      " #   Column             Non-Null Count  Dtype  \n",
      "---  ------             --------------  -----  \n",
      " 0   loc                30529 non-null  float64\n",
      " 1   iv(g)              30529 non-null  float64\n",
      " 2   n                  30529 non-null  float64\n",
      " 3   v                  30529 non-null  float64\n",
      " 4   b                  30529 non-null  float64\n",
      " 5   lOCode             30529 non-null  float64\n",
      " 6   lOComment          30529 non-null  float64\n",
      " 7   lOBlank            30529 non-null  float64\n",
      " 8   locCodeAndComment  30529 non-null  float64\n",
      " 9   uniq_Op            30529 non-null  float64\n",
      " 10  uniq_Opnd          30529 non-null  float64\n",
      " 11  total_Op           30529 non-null  float64\n",
      " 12  total_Opnd         30529 non-null  float64\n",
      " 13  branchCount        30529 non-null  float64\n",
      "dtypes: float64(14)\n",
      "memory usage: 3.3 MB\n"
     ]
    }
   ],
   "source": [
    "# Faz o ajuste de escala e transforma todos os atributos para float\n",
    "def scale(df):\n",
    "    scaler = preprocessing.RobustScaler()\n",
    "    robust_df = scaler.fit_transform(df)\n",
    "    robust_df = pd.DataFrame(robust_df, columns =df.columns)\n",
    "    return robust_df\n",
    "train_X = scale(train_X)\n",
    "test_data =  scale(test_data)\n",
    "test_X = scale(test_X)\n",
    "test_X.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 30529 entries, 0 to 30528\n",
      "Data columns (total 14 columns):\n",
      " #   Column             Non-Null Count  Dtype  \n",
      "---  ------             --------------  -----  \n",
      " 0   loc                30529 non-null  float64\n",
      " 1   iv(g)              30529 non-null  float64\n",
      " 2   n                  30529 non-null  float64\n",
      " 3   v                  30529 non-null  float64\n",
      " 4   b                  30529 non-null  float64\n",
      " 5   lOCode             30529 non-null  float64\n",
      " 6   lOComment          30529 non-null  float64\n",
      " 7   lOBlank            30529 non-null  float64\n",
      " 8   locCodeAndComment  30529 non-null  float64\n",
      " 9   uniq_Op            30529 non-null  float64\n",
      " 10  uniq_Opnd          30529 non-null  float64\n",
      " 11  total_Op           30529 non-null  float64\n",
      " 12  total_Opnd         30529 non-null  float64\n",
      " 13  branchCount        30529 non-null  float64\n",
      "dtypes: float64(14)\n",
      "memory usage: 3.3 MB\n"
     ]
    }
   ],
   "source": [
    "test_X.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tentando transformar em tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transformando a base de treino em tensor float\n",
    "train_X_tensor = torch.FloatTensor(train_X.values)\n",
    "train_y_tensor = torch.FloatTensor(train_y.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transformando a base de treino em tensor float\n",
    "test_X_tensor = torch.FloatTensor(test_X.values)\n",
    "test_y_tensor = torch.FloatTensor(test_y.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Datasets\n",
    "train_dataset = TensorDataset(train_X_tensor, train_y_tensor)\n",
    "val_dataset = TensorDataset(test_X_tensor, test_y_tensor)\n",
    "# DataLoaders\n",
    "train_dataloader = DataLoader(train_dataset, batch_size=128)\n",
    "val_dataloader = DataLoader(val_dataset, batch_size=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Referências\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Kaggle: Software Defect Prediction](https://www.kaggle.com/datasets/semustafacevik/software-defect-prediction/code)\n",
    "\n",
    "- [PS S3 E23 Feature Selection and Ensemble](https://www.kaggle.com/code/liudacheldieva/ps-s3-e23-feature-selection-and-ensemble)\n",
    "\n",
    "- [Solving sklearn datasets with PyTorch](https://www.kaggle.com/code/glebbuzin/solving-sklearn-datasets-with-pytorch)\n",
    "\n",
    "- [Binary Classification with a Software Defects Data](https://www.kaggle.com/code/imessam/binary-classification-with-a-software-defects-data)\n",
    "\n",
    "- [Playground Series - Season 3, Episode 23](https://www.kaggle.com/code/jpedrou/playground-series-season-3-episode-23/notebook)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}