01_TestTrainSplit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, let's check capabilities of SkLearn befere using NNs\n",
    "\n",
    "Which method to use? Of course ones that enlisted [here](https://github.com/lemikhovalex/DA_CV) (repo for ML classes at MIPT)\n",
    "## Preprocess data\n",
    "But before using data let's create features and labels\n",
    "\n",
    "The following features are to be created, accoarding to papers about LSTM:\n",
    "- $t_{w}$ - time passed between Sunday midnight and the event\n",
    "- $t_{e}$ - time passed between the completion of the given event and the completion of the previous one\n",
    "- $t_{t}$ - time passed between the start of the trace and the given event\n",
    "- one-hot-encoded labels\n",
    "- make smth like n-gramm, but for process and use it as features\n",
    "\n",
    "### Test train split\n",
    "Ok it seems to be rational not to take random transitions, but whole traces, cause `n-gramm` preprocessing will be conducted natively, if data is provided as a sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: python-dateutil in c:\\users\\alemikhov\\appdata\\local\\programs\\python\\python36\\lib\\site-packages (2.8.1)\n",
      "Requirement already satisfied: six>=1.5 in c:\\users\\alemikhov\\appdata\\local\\programs\\python\\python36\\lib\\site-packages (from python-dateutil) (1.15.0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: You are using pip version 21.0.1; however, version 21.1.2 is available.\n",
      "You should consider upgrading via the 'c:\\users\\alemikhov\\appdata\\local\\programs\\python\\python36\\python.exe -m pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "! pip install python-dateutil"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import datetime\n",
    "import random\n",
    "from dateutil.parser import parse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('datasets/bpi_12.csv')\n",
    "df['timestamp'] = df['timestamp'].apply(lambda x: parse(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>trace_id</th>\n",
       "      <th>activity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-10-01 11:45:13.917000+02:00</td>\n",
       "      <td>173688</td>\n",
       "      <td>W_Completeren aanvraag</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-10-01 12:17:08.924000+02:00</td>\n",
       "      <td>173688</td>\n",
       "      <td>W_Nabellen offertes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-10-08 16:32:00.886000+02:00</td>\n",
       "      <td>173688</td>\n",
       "      <td>W_Nabellen offertes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-10-10 11:33:05.791000+02:00</td>\n",
       "      <td>173688</td>\n",
       "      <td>W_Nabellen offertes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-10-13 10:37:37.026000+02:00</td>\n",
       "      <td>173688</td>\n",
       "      <td>W_Valideren aanvraag</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72408</th>\n",
       "      <td>2012-03-01 09:28:21.228000+01:00</td>\n",
       "      <td>214370</td>\n",
       "      <td>W_Afhandelen leads</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72409</th>\n",
       "      <td>2012-03-01 09:27:52.388000+01:00</td>\n",
       "      <td>214373</td>\n",
       "      <td>W_Afhandelen leads</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72410</th>\n",
       "      <td>2012-03-01 20:22:41.157000+01:00</td>\n",
       "      <td>214373</td>\n",
       "      <td>W_Completeren aanvraag</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72411</th>\n",
       "      <td>2012-03-10 12:50:54.881000+01:00</td>\n",
       "      <td>214373</td>\n",
       "      <td>W_Nabellen offertes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72412</th>\n",
       "      <td>2012-03-01 09:27:41.325000+01:00</td>\n",
       "      <td>214376</td>\n",
       "      <td>W_Afhandelen leads</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>72413 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                              timestamp  trace_id                activity\n",
       "0      2011-10-01 11:45:13.917000+02:00    173688  W_Completeren aanvraag\n",
       "1      2011-10-01 12:17:08.924000+02:00    173688     W_Nabellen offertes\n",
       "2      2011-10-08 16:32:00.886000+02:00    173688     W_Nabellen offertes\n",
       "3      2011-10-10 11:33:05.791000+02:00    173688     W_Nabellen offertes\n",
       "4      2011-10-13 10:37:37.026000+02:00    173688    W_Valideren aanvraag\n",
       "...                                 ...       ...                     ...\n",
       "72408  2012-03-01 09:28:21.228000+01:00    214370      W_Afhandelen leads\n",
       "72409  2012-03-01 09:27:52.388000+01:00    214373      W_Afhandelen leads\n",
       "72410  2012-03-01 20:22:41.157000+01:00    214373  W_Completeren aanvraag\n",
       "72411  2012-03-10 12:50:54.881000+01:00    214373     W_Nabellen offertes\n",
       "72412  2012-03-01 09:27:41.325000+01:00    214376      W_Afhandelen leads\n",
       "\n",
       "[72413 rows x 3 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also dont want to deal with short traces - composing features as window will fail it("
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 9658 traces\n"
     ]
    }
   ],
   "source": [
    "traces = list(set(df['trace_id']))\n",
    "print(f'There are {len(traces)} traces')\n",
    "traces_len = {trace_id: df[df['trace_id'] == trace_id].shape[0] for trace_id in traces}\n",
    "traces_time = {}\n",
    "for trace_id in traces:\n",
    "    _df = df[df['trace_id'] == trace_id]\n",
    "    if _df.shape[0] > 7:\n",
    "        traces_time[trace_id] = (_df['timestamp'].max() - _df['timestamp'].min()).total_seconds()\n",
    "traces_time = pd.DataFrame({'trace_id': list(traces_time.keys()), 'tt': list(traces_time.values())})\n",
    "traces_time.sort_values(by=['tt'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>trace_id</th>\n",
       "      <th>tt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1142</th>\n",
       "      <td>174421</td>\n",
       "      <td>1175.742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1084</th>\n",
       "      <td>174120</td>\n",
       "      <td>4601.686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1133</th>\n",
       "      <td>174376</td>\n",
       "      <td>5961.455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2700</th>\n",
       "      <td>184607</td>\n",
       "      <td>8422.561</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2484</th>\n",
       "      <td>182194</td>\n",
       "      <td>8647.784</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      trace_id        tt\n",
       "1142    174421  1175.742\n",
       "1084    174120  4601.686\n",
       "1133    174376  5961.455\n",
       "2700    184607  8422.561\n",
       "2484    182194  8647.784"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "traces_time.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's calculate 4 equivalent quantiles 4 trace length distribution and sample 30% from each"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_qs = 4\n",
    "sh = traces_time.shape[0]\n",
    "qs = [0.]\n",
    "qs.extend([traces_time[_i * sh // n_qs: (_i + 1) * sh // n_qs]['tt'].max() for _i in range(n_qs)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_ids = []\n",
    "test_relative = 0.25\n",
    "for _i in range(n_qs):\n",
    "    _lb_cond = (traces_time['tt'] > qs[_i])\n",
    "    _rb_cond = (traces_time['tt'] <= qs[_i+1])\n",
    "    _cond = _lb_cond & _rb_cond\n",
    "    _pop = traces_time[_cond]['trace_id'].values.tolist()\n",
    "    test_ids.extend(random.sample(population=_pop,\n",
    "                                  k=int(len(_pop) * test_relative)\n",
    "                                 )\n",
    "                   )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 948 traces for test\n"
     ]
    }
   ],
   "source": [
    "print(f'There are {len(test_ids)} traces for test')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ids = traces\n",
    "for _test_id in test_ids:\n",
    "    train_ids.remove(_test_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 8710 traces for train\n"
     ]
    }
   ],
   "source": [
    "print(f'There are {len(train_ids)} traces for train')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>trace_id</th>\n",
       "      <th>activity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-10-01 11:45:13.917000+02:00</td>\n",
       "      <td>173688</td>\n",
       "      <td>W_Completeren aanvraag</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-10-01 12:17:08.924000+02:00</td>\n",
       "      <td>173688</td>\n",
       "      <td>W_Nabellen offertes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-10-08 16:32:00.886000+02:00</td>\n",
       "      <td>173688</td>\n",
       "      <td>W_Nabellen offertes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-10-10 11:33:05.791000+02:00</td>\n",
       "      <td>173688</td>\n",
       "      <td>W_Nabellen offertes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-10-13 10:37:37.026000+02:00</td>\n",
       "      <td>173688</td>\n",
       "      <td>W_Valideren aanvraag</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          timestamp  trace_id                activity\n",
       "0  2011-10-01 11:45:13.917000+02:00    173688  W_Completeren aanvraag\n",
       "1  2011-10-01 12:17:08.924000+02:00    173688     W_Nabellen offertes\n",
       "2  2011-10-08 16:32:00.886000+02:00    173688     W_Nabellen offertes\n",
       "3  2011-10-10 11:33:05.791000+02:00    173688     W_Nabellen offertes\n",
       "4  2011-10-13 10:37:37.026000+02:00    173688    W_Valideren aanvraag"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df = df[df['trace_id'].isin(test_ids)]\n",
    "train_df = df[df['trace_id'].isin(train_ids)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train_df.to_csv('datasets/train_df.csv', index=False)\n",
    "# test_df.to_csv('datasets/test_df.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets use test like a holdout dataframe\n",
    "\n",
    "# Custom split for GridSearchCV\n",
    "\n",
    "## Split strategy\n",
    "\n",
    "- Split for 2 pieces that are prepared for test and train - time relations\n",
    "- Random split each piece to `n_splits`, for folds\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df_s = train_df.sort_values(by=['timestamp'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>trace_id</th>\n",
       "      <th>activity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>2011-10-01 10:10:25.759000+02:00</td>\n",
       "      <td>173712</td>\n",
       "      <td>W_Afhandelen leads</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>2011-10-01 10:16:49.843000+02:00</td>\n",
       "      <td>173706</td>\n",
       "      <td>W_Afhandelen leads</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>2011-10-01 10:27:07.853000+02:00</td>\n",
       "      <td>173709</td>\n",
       "      <td>W_Completeren aanvraag</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>2011-10-01 11:40:24.141000+02:00</td>\n",
       "      <td>173703</td>\n",
       "      <td>W_Completeren aanvraag</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>2011-10-01 11:43:02.623000+02:00</td>\n",
       "      <td>173709</td>\n",
       "      <td>W_Completeren aanvraag</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69379</th>\n",
       "      <td>2012-03-14 15:54:04.151000+01:00</td>\n",
       "      <td>212238</td>\n",
       "      <td>W_Valideren aanvraag</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68517</th>\n",
       "      <td>2012-03-14 15:56:07.999000+01:00</td>\n",
       "      <td>211624</td>\n",
       "      <td>W_Nabellen incomplete dossiers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71747</th>\n",
       "      <td>2012-03-14 15:57:49.624000+01:00</td>\n",
       "      <td>213855</td>\n",
       "      <td>W_Nabellen offertes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70876</th>\n",
       "      <td>2012-03-14 16:00:09.680000+01:00</td>\n",
       "      <td>213276</td>\n",
       "      <td>W_Nabellen incomplete dossiers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68518</th>\n",
       "      <td>2012-03-14 16:04:54.681000+01:00</td>\n",
       "      <td>211624</td>\n",
       "      <td>W_Nabellen incomplete dossiers</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>58738 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                              timestamp  trace_id  \\\n",
       "40     2011-10-01 10:10:25.759000+02:00    173712   \n",
       "32     2011-10-01 10:16:49.843000+02:00    173706   \n",
       "36     2011-10-01 10:27:07.853000+02:00    173709   \n",
       "30     2011-10-01 11:40:24.141000+02:00    173703   \n",
       "37     2011-10-01 11:43:02.623000+02:00    173709   \n",
       "...                                 ...       ...   \n",
       "69379  2012-03-14 15:54:04.151000+01:00    212238   \n",
       "68517  2012-03-14 15:56:07.999000+01:00    211624   \n",
       "71747  2012-03-14 15:57:49.624000+01:00    213855   \n",
       "70876  2012-03-14 16:00:09.680000+01:00    213276   \n",
       "68518  2012-03-14 16:04:54.681000+01:00    211624   \n",
       "\n",
       "                             activity  \n",
       "40                 W_Afhandelen leads  \n",
       "32                 W_Afhandelen leads  \n",
       "36             W_Completeren aanvraag  \n",
       "30             W_Completeren aanvraag  \n",
       "37             W_Completeren aanvraag  \n",
       "...                               ...  \n",
       "69379            W_Valideren aanvraag  \n",
       "68517  W_Nabellen incomplete dossiers  \n",
       "71747             W_Nabellen offertes  \n",
       "70876  W_Nabellen incomplete dossiers  \n",
       "68518  W_Nabellen incomplete dossiers  \n",
       "\n",
       "[58738 rows x 3 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df_s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}