TLRClassification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_4NzfRZGk-Y-"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tobhey/NoRBERT_for_TLR/blob/master/TLRClassification.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0MiVHuYUwZSt"
   },
   "source": [
    "# Binary Classification of Functional Aspects in Requirments Elements of Traceability Link Recovery Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "56AfAWCbwf07"
   },
   "source": [
    "This notebook includes all code needed to train and evaluate binary classifiers for predicting the functional aspects present in a requirement element."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WaiYTEPHwtEV"
   },
   "source": [
    "Note: some cells are hidden and only the title is shown. To display the code, double-click the cell to switch the display mode."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-djxLyuc-Opk"
   },
   "source": [
    "## Prepare\n",
    "Install required libraries and import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Epk5taxa99eI",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "#@title Install needed libraries {display-mode: \"form\"}\n",
    "!pip install fastai==1.0.61 fastcore==1.3.29 fastprogress==1.0.3 pytorch-transformers==1.2.0 sklearn==0.0 spacy==3.6.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "Fr6bTWdl-XzF"
   },
   "outputs": [],
   "source": [
    "#@title Import python packages\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import os\n",
    "\n",
    "from fastai import *\n",
    "from fastai.text import *\n",
    "from fastai.callbacks import *\n",
    "from sklearn.model_selection import train_test_split, StratifiedKFold\n",
    "from sklearn.metrics import classification_report, precision_recall_fscore_support\n",
    "from sklearn.utils.multiclass import unique_labels\n",
    "\n",
    "from pytorch_transformers import BertTokenizer, BertPreTrainedModel, BertModel, BertConfig\n",
    "from pytorch_transformers import AdamW\n",
    "\n",
    "from fastprogress import master_bar, progress_bar\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "Wtzha3q7QjjU"
   },
   "outputs": [],
   "source": [
    "#@title Check, if and what kind of GPU is used\n",
    "def get_memory_usage():\n",
    "    return torch.cuda.memory_allocated(device)/1000000\n",
    "\n",
    "def get_memory_usage_str():\n",
    "    return 'Memory usage: {:.2f} MB'.format(get_memory_usage())\n",
    "\n",
    "cuda_available = torch.cuda.is_available()\n",
    "if cuda_available:\n",
    "    curr_device = torch.cuda.current_device()\n",
    "    print(torch.cuda.get_device_name(curr_device))\n",
    "device = torch.device(\"cuda\" if cuda_available else \"cpu\")\n",
    "device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vyi4b_Acw9Gg"
   },
   "source": [
    "### Define configuration used in this experiment run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mOKXVgJgGtYV"
   },
   "source": [
    "Create config and set hyperparameters.\n",
    "One can configure:\n",
    "\n",
    "\n",
    "*   BERT model to use (model_name)\n",
    "*   Learning Rate to use (max_lr)\n",
    "*   Momentum (moms)\n",
    "*   Epoch number for training (epochs)\n",
    "*   Badge size for training (bs)\n",
    "*   Weight decay for training (weight_decay)\n",
    "*   Maximal sequence length (max_seq_len)\n",
    "*   Train size used for both test/train and train/validation split (train_size)\n",
    "*   Loss function used for training (loss_func)\n",
    "*   The random seed used for shuffling, sampling and splitting (seed)\n",
    "*   Whether, or not to use early stopping (es)\n",
    "*   The minimal delta used to indicate early stopping (min_delta)\n",
    "*   The number of epochs that need to undergo this delta to early stop training (patience)\n",
    "*   The way of folding used for this experiment (either test/train split (No), ten-fold cross validation (TenFold), or project specific folding (ProjFold))\n",
    "*   Which kind of sampling to use (either OverSampling minority class, UnderSampling majority class, or NoSampling at all)\n",
    "\n",
    "*   Which class to predict (clazz)\n",
    "\n",
    "Further one can configure, where to get the dataset from and where to save log, result and model files.\n",
    "By setting the classes Array one can decide which binary classifiers to train and evaluate in one experiment run.\n",
    "One boolean is provided to decide whether to save the model file.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "i0lgLyC6Gsnf"
   },
   "outputs": [],
   "source": [
    "class Config(dict):\n",
    "    def __init__(self, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        for k, v in kwargs.items():\n",
    "            setattr(self, k, v)\n",
    "    \n",
    "    def set(self, key, val):\n",
    "        self[key] = val\n",
    "        setattr(self, key, val)\n",
    "\n",
    "class Fold(Enum):\n",
    "  No = 1\n",
    "  TenFold = 2\n",
    "  ProjFold = 3\n",
    "\n",
    "class Sampling(Enum):\n",
    "  NoSampling = 1\n",
    "  UnderSampling = 2\n",
    "  OverSampling = 3\n",
    "\n",
    "config = Config(\n",
    "    num_labels = 2, # will be set automatically afterwards\n",
    "    model_name=\"bert-large-cased\", # bert_base_uncased, bert_large_cased, bert_large_uncased\n",
    "    max_lr=2e-5, # default: 2e-5\n",
    "    moms=(0.8, 0.7), # default: (0.8, 0.7); alt.(0.95, 0.85)\n",
    "    epochs=16, # 10, 16, 32, 50\n",
    "    bs=16, # default: 16\n",
    "    weight_decay = 0.01,\n",
    "    max_seq_len=128, # 50, 128\n",
    "    train_size=0.75, # 0.8\n",
    "    loss_func=nn.CrossEntropyLoss(),\n",
    "    seed=904727489, #default: 904727489, 42 (as in Dalpiaz) or None\n",
    "    es = False, # True\n",
    "    min_delta = 0.01,\n",
    "    patience = 3,\n",
    "    fold = Fold.ProjFold, # Fold.No, Fold.TenFold, Fold.ProjFold\n",
    "    sampling = Sampling.OverSampling, #Sampling.UnderSampling, Sampling.NoSampling, Sampling.OverSampling\n",
    ")\n",
    "\n",
    "clazz = 'all' # 'F', 'Q', 'OnlyF', 'OnlyQ' class to train classification on\n",
    "\n",
    "config_data = Config(\n",
    "    root_folder = '.', # where is the root folder? Keep it that way if you want to load from Google Drive\n",
    "    data_folder = '/', # where is the folder containing the datasets; relative to root\n",
    "    train_data = ['classification_dataset.csv'],\n",
    "    label_column = clazz,\n",
    "    log_folder_name = '/log/',\n",
    "    log_file = clazz + '_' + Fold(config.fold).name + '_' + Sampling(config.sampling).name + '_classifierPredictions_' + datetime.now().strftime('%Y%m%d-%H%M') + '.txt', # log-file name (make sure log folder exists)\n",
    "    result_file = clazz + '_' + Fold(config.fold).name + '_' + Sampling(config.sampling).name + '_classifierResults_' + datetime.now().strftime('%Y%m%d-%H%M') + '.txt', # result-file name (make sure log folder exists)\n",
    "    model_path = '/models/', # where is the folder for the model(s); relative to the root\n",
    "    model_name = 'NoRBERT.pkl', # what is the model name? \n",
    "   \n",
    "    # Project split to use, either p-fold (as in Dalpiaz) or loPo\n",
    "    #project_fold = [[3, 9, 11], [1, 5, 12], [6, 10, 13], [1, 8, 14], [3, 12, 15], [2, 5, 11], [6, 9, 14], [7, 8, 13], [2, 4, 15], [4, 7, 10] ], # p-fold\n",
    "    project_fold = [['eTour'], ['iTrust'], ['SMOS'], ['eAnci'], ['LibEST'] ], # loPo\n",
    "    classes= ['F', 'Function', 'Behavior', 'Data' , 'UserRelated'], # this array defines which classes are trained and evaluated in one run\n",
    "   \n",
    ")\n",
    "\n",
    "save_model = False # True, if you want to use save the model file (make sure model folder exists)\n",
    "input_col = 'text'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OmGISBrhW-VJ"
   },
   "outputs": [],
   "source": [
    "#@title Prepare data loading:  {display-mode: \"form\"}\n",
    "data_path = config_data.train_data[0]\n",
    "data_file = Path(data_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "er1yzLHFQq1U"
   },
   "outputs": [],
   "source": [
    "#@title Define logging functions and seed generation {display-mode: \"form\"}\n",
    "def initLog(firstRun):\n",
    "    logfolder = config_data.root_folder + config_data.log_folder_name\n",
    "   \n",
    "    if not os.path.isdir(logfolder):\n",
    "      print(\"Log folder does not exist, trying to create folder.\")\n",
    "      try:\n",
    "        os.mkdir(logfolder)\n",
    "      except OSError:\n",
    "        print (\"Creation of the directory %s failed\" % logfolder)\n",
    "      else:\n",
    "        print (\"Successfully created the directory %s\" % logfolder)\n",
    "    logfile = logfolder + config_data.log_file\n",
    "    log_txt = datetime.now().strftime('%Y-%m-%d %H:%M') + ' ' + get_info()\n",
    "    if firstRun:\n",
    "        with open(logfile, 'w') as log:\n",
    "            log.write(log_txt + '\\n')\n",
    "    else:\n",
    "        with open(logfile, 'a') as log:\n",
    "            log.write(log_txt + '\\n')\n",
    "    \n",
    "\n",
    "def logLine(line):\n",
    "    logfile = config_data.root_folder + config_data.log_folder_name  + config_data.log_file\n",
    "    with open(logfile, 'a') as log:\n",
    "        log.write(line + '\\n')\n",
    "\n",
    "def logResult(result):\n",
    "    logfile = config_data.root_folder + config_data.log_folder_name + config_data.result_file\n",
    "    with open(logfile, 'a') as log:\n",
    "        log.write(get_info() + '\\n')\n",
    "        log.write(result + '\\n')\n",
    "\n",
    "def get_info():\n",
    "     model_config = 'model: {}, max_lr: {}, epochs: {}, bs: {}, train_size: {}, weight decay: {},  Seed: {}, Data: {}, Column: {}, EarlyStopping: {}:{};pat:{}'.format(config.model_name, config.max_lr, config.epochs, config.bs, config.train_size, config.weight_decay, config.seed, config_data.train_data, config_data.label_column, config.es, config.min_delta, config.patience)\n",
    "     return model_config\n",
    "\n",
    "def set_seed(seed):\n",
    "    if seed is None:\n",
    "        seed = random.randint(0, 2**31)\n",
    "    random.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    torch.backends.cudnn.deterministic = True\n",
    "    torch.backends.cudnn.benchmark = False\n",
    "    if torch.cuda.is_available(): torch.cuda.manual_seed_all(seed)\n",
    "    return seed\n",
    "\n",
    "set_seed(config.seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ptp6NhIC_FQb"
   },
   "source": [
    "## Learner\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6anB63ppBAtB"
   },
   "outputs": [],
   "source": [
    "#@title Create proper tokenizer for our data (adapting FastAiTokenizer to use BertTokenizer) {display-mode: \"form\"}\n",
    "class FastAiBertTokenizer(BaseTokenizer):\n",
    "    \"\"\"Wrapper around BertTokenizer to be compatible with fast.ai\"\"\"\n",
    "    def __init__(self, tokenizer: BertTokenizer, max_seq_len: int=512, **kwargs):\n",
    "        self._pretrained_tokenizer = tokenizer\n",
    "        self.max_seq_len = max_seq_len\n",
    "\n",
    "    def __call__(self, *args, **kwargs):\n",
    "        return self\n",
    "\n",
    "    def tokenizer(self, t:str):\n",
    "        \"\"\"Limits the maximum sequence length. Prepend with [CLS] and append [SEP]\"\"\"\n",
    "        return [\"[CLS]\"] + self._pretrained_tokenizer.tokenize(t)[:self.max_seq_len - 2] + [\"[SEP]\"]\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1G8rFbEEJWyu"
   },
   "source": [
    "Now, we can create our own databunch using the tokenizer above. Notice we're passing the include_bos=False and include_eos=False options. This is to prevent fastai from adding its own SOS/EOS tokens that will interfere with BERT's SOS/EOS tokens.\n",
    "\n",
    "We can pass our own list of Preprocessors to the databunch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TNRRj6jIJrp2"
   },
   "outputs": [],
   "source": [
    "#@title Define Processors and Databunch {display-mode: \"form\"}\n",
    "class BertTokenizeProcessor(TokenizeProcessor):\n",
    "    \"\"\"Special Tokenizer, where we remove sos/eos tokens since we add that ourselves in the tokenizer.\"\"\"\n",
    "    def __init__(self, tokenizer):\n",
    "        super().__init__(tokenizer=tokenizer, include_bos=False, include_eos=False)\n",
    "\n",
    "class BertNumericalizeProcessor(NumericalizeProcessor):\n",
    "    \"\"\"Use a custom vocabulary to match the original BERT model.\"\"\"\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super().__init__(*args, vocab=Vocab(list(bert_tok.vocab.keys())), **kwargs)\n",
    "\n",
    "def get_bert_processor(tokenizer:Tokenizer=None, vocab:Vocab=None):\n",
    "    return [BertTokenizeProcessor(tokenizer=tokenizer),\n",
    "            NumericalizeProcessor(vocab=vocab)]\n",
    "\n",
    "class BertDataBunch(TextDataBunch):\n",
    "    @classmethod\n",
    "    def from_df(cls, path:PathOrStr, train_df:DataFrame, valid_df:DataFrame, test_df:Optional[DataFrame]=None,\n",
    "              tokenizer:Tokenizer=None, vocab:Vocab=None, classes:Collection[str]=None, text_cols:IntsOrStrs=1,\n",
    "              label_cols:IntsOrStrs=0, **kwargs) -> DataBunch:\n",
    "        \"Create a `TextDataBunch` from DataFrames.\"\n",
    "        p_kwargs, kwargs = split_kwargs_by_func(kwargs, get_bert_processor)\n",
    "        # use our custom processors while taking tokenizer and vocab as kwargs\n",
    "        processor = get_bert_processor(tokenizer=tokenizer, vocab=vocab, **p_kwargs)\n",
    "        if classes is None and is_listy(label_cols) and len(label_cols) > 1: classes = label_cols\n",
    "        src = ItemLists(path, TextList.from_df(train_df, path, cols=text_cols, processor=processor),\n",
    "                      TextList.from_df(valid_df, path, cols=text_cols, processor=processor))\n",
    "        src = src.label_for_lm() if cls==TextLMDataBunch else src.label_from_df(cols=label_cols, classes=classes)\n",
    "        if test_df is not None: src.add_test(TextList.from_df(test_df, path, cols=text_cols))\n",
    "        return src.databunch(**kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "he_PRt9Q3eUB"
   },
   "outputs": [],
   "source": [
    "#@title Define own BertTextClassifier class{display-mode: \"form\"}\n",
    "class BertTextClassifier(BertPreTrainedModel):\n",
    "    def __init__(self, model_name, num_labels):\n",
    "        config = BertConfig.from_pretrained(model_name)\n",
    "        super(BertTextClassifier, self).__init__(config)\n",
    "        self.num_labels = num_labels\n",
    "        \n",
    "        self.bert = BertModel.from_pretrained(model_name, config=config)\n",
    "        \n",
    "        self.dropout = nn.Dropout(self.config.hidden_dropout_prob)\n",
    "        self.classifier = nn.Linear(self.config.hidden_size, num_labels)\n",
    "\n",
    "    \n",
    "    def forward(self, tokens, labels=None, position_ids=None, token_type_ids=None, attention_mask=None, head_mask=None):\n",
    "        outputs = self.bert(tokens, position_ids=position_ids, token_type_ids=token_type_ids, attention_mask=attention_mask, head_mask=head_mask)\n",
    "        \n",
    "        pooled_output = outputs[1]\n",
    "\n",
    "        dropout_output = self.dropout(pooled_output)\n",
    "        logits = self.classifier(dropout_output)\n",
    "\n",
    "        activation = nn.Softmax(dim=1)\n",
    "        probs = activation(logits)   \n",
    "\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sMK89CEyx2Kc"
   },
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IwxQFKykzpQq"
   },
   "source": [
    "Load the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oeaTvNRTypP0"
   },
   "outputs": [],
   "source": [
    "#@title Define functions to load data {display-mode: \"form\"}\n",
    "def load_data(filename):\n",
    "    fpath = config_data.root_folder + config_data.data_folder + filename\n",
    "    print(fpath)\n",
    "    df = pd.read_csv(fpath, delimiter=',', header=0, encoding='utf8', names=['ProjectID', 'file', 'ID', 'text', 'functional', 'Function', 'Behavior', 'Data', 'OnlyF', 'F', 'OnlyQ', 'Q', 'UserRelated'], dtype= {'UserRelated':int})\n",
    "    df = df.dropna()  \n",
    "    df = df.reset_index()\n",
    "    return df\n",
    "\n",
    "def load_all_data(filenames, label_column):\n",
    "    df = load_data(filenames[0])\n",
    "    for i in range(1, len(filenames)):\n",
    "        df = df.append(load_data(filenames[i]))\n",
    "\n",
    "    # shuffle the dataset a bit and get the amount of classes\n",
    "    df = df.sample(frac=1, axis=0, random_state = config.seed)\n",
    "    config.num_labels = df[label_column].nunique()\n",
    "\n",
    "    print(df.shape)\n",
    "    print(df[label_column].value_counts())\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-6o6UU0tUYck"
   },
   "outputs": [],
   "source": [
    "#@title Actually load the dataset{display-mode: \"form\"}\n",
    "# load the train datasets\n",
    "df = load_data(config_data.train_data[0])\n",
    "# shuffle the dataset a bit\n",
    "df = df.sample(frac=1, axis=0, random_state = config.seed)\n",
    "\n",
    "print(df.shape)\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#@title Prepare dataframe to put results in {display-mode: \"form\"}\n",
    "df_result = df.copy()\n",
    "cols = ['functional', 'Function', 'Behavior', 'Data', 'OnlyF', 'F', 'OnlyQ', 'Q', 'UserRelated']\n",
    "df_result = df_result.assign(**{c:0 for c in cols})\n",
    "display(df_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TWP1X17N5tJx"
   },
   "outputs": [],
   "source": [
    "#@title Create the dictionary that contains the labels along with their indices. This is useful for evaluation and similar. {display-mode: \"form\"}\n",
    "def create_label_indices(df):\n",
    "    #prepare label\n",
    "    labels = ['not_' + config_data.label_column, config_data.label_column]\n",
    "  \n",
    "    #create dict\n",
    "    labelDict = dict()\n",
    "    for i in range (0, len(labels)):\n",
    "        labelDict[i] = labels[i]\n",
    "    return labelDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Co7QpGwgyKEp"
   },
   "outputs": [],
   "source": [
    "#@title Define functions for under-/oversample dataset {display-mode: \"form\"}\n",
    "def undersample(df_trn, major_label, minor_label):\n",
    "  sample_size = sum(df_trn[config_data.label_column] == minor_label)\n",
    "  majority_indices = df_trn[df_trn[config_data.label_column] == major_label].index\n",
    "  random_indices = np.random.choice(majority_indices, sample_size, replace=False)\n",
    "  sample = df_trn.loc[random_indices]\n",
    "  sample = sample.append(df_trn[df_trn[config_data.label_column] == minor_label])\n",
    "  df_trn = sample\n",
    "  df_trn = df_trn.sample(frac=1, axis=0, random_state = config.seed)\n",
    "  print(df_trn[config_data.label_column].value_counts())\n",
    "  return df_trn\n",
    "\n",
    "def oversample(df_trn, major_label, minor_label):\n",
    "  minor_size = sum(df_trn[config_data.label_column] == minor_label)\n",
    "  major_size = sum(df_trn[config_data.label_column] == major_label)\n",
    "  multiplier = major_size//minor_size\n",
    "  sample = df_trn\n",
    "  minority_indices = df_trn[df_trn[config_data.label_column] == minor_label].index\n",
    "  diff = major_size - (multiplier * minor_size)     \n",
    "  random_indices = np.random.choice(minority_indices, diff, replace=False)\n",
    "  sample = pd.concat([df_trn.loc[random_indices], sample], ignore_index=True)\n",
    "  for i in range(multiplier - 1):\n",
    "    sample = pd.concat([sample, df_trn[df_trn[config_data.label_column] == minor_label]], ignore_index=True)\n",
    "  df_trn = sample\n",
    "  df_trn = df_trn.sample(frac=1, axis=0, random_state = config.seed)\n",
    "  print(df_trn[config_data.label_column].value_counts())\n",
    "  return df_trn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yq7yLhQ8yNKf"
   },
   "outputs": [],
   "source": [
    "#@title Function to split dataframe according to Sampling strategy and train size {display-mode: \"form\"}\n",
    "def split_dataframe(df, train_size = 0.8, random_state = None):\n",
    "    # split data into training and validation set\n",
    "    df_trn, df_valid = train_test_split(df, stratify = df[config_data.label_column], train_size = train_size, random_state = random_state)\n",
    "    # apply sample strategy\n",
    "    sizeOne = sum(df_trn[config_data.label_column] == 1)\n",
    "    sizeZero = sum(df_trn[config_data.label_column] == 0)\n",
    "    major_label = 0\n",
    "    minor_label = 1\n",
    "    if sizeOne > sizeZero:\n",
    "      major_label = 1\n",
    "      minor_label = 0\n",
    "    if config.sampling == Sampling.UnderSampling:\n",
    "      df_trn = undersample(df_trn, major_label, minor_label)\n",
    "    elif config.sampling == Sampling.OverSampling:\n",
    "      df_trn = oversample(df_trn, major_label, minor_label)\n",
    "    return df_trn, df_valid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SClv488eQC8B"
   },
   "source": [
    "## Predictor\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qubb_Ka-C78O"
   },
   "outputs": [],
   "source": [
    "#@title Create a predictor class{display-mode: \"form\"}\n",
    "class Predictor:\n",
    "    def __init__(self, classifier):\n",
    "        self.classifier = classifier\n",
    "        self.classes = self.classifier.data.classes\n",
    "\n",
    "    def predict(self, text):\n",
    "        prediction = self.classifier.predict(text)\n",
    "        prediction_class = prediction[1]\n",
    "        return self.classes[prediction_class]  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zyVQS13d5Sft"
   },
   "source": [
    "## Create and train the learner/classifier\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "T83UogVz5XJJ"
   },
   "outputs": [],
   "source": [
    "#@title Define functions to create databunch, learner and actual classifier{display-mode: \"form\"}\n",
    "def create_databunch(config, df_trn, df_valid):\n",
    "    bert_tok = BertTokenizer.from_pretrained(config.model_name,)\n",
    "    fastai_tokenizer = Tokenizer(tok_func=FastAiBertTokenizer(bert_tok, max_seq_len=config.max_seq_len), pre_rules=[], post_rules=[])\n",
    "    fastai_bert_vocab = Vocab(list(bert_tok.vocab.keys()))\n",
    "    return BertDataBunch.from_df(\".\", \n",
    "                   train_df=df_trn,\n",
    "                   valid_df=df_valid,\n",
    "                   tokenizer=fastai_tokenizer,\n",
    "                   vocab=fastai_bert_vocab,\n",
    "                   bs=config.bs,\n",
    "                   text_cols=input_col,\n",
    "                   label_cols=config_data.label_column,\n",
    "                   collate_fn=partial(pad_collate, pad_first=False, pad_idx=0),\n",
    "              )\n",
    "\n",
    "\n",
    "def create_learner(config, databunch):\n",
    "    model = BertTextClassifier(config.model_name, config.num_labels)\n",
    "\n",
    "    optimizer = partial(AdamW)\n",
    "    if config.es:\n",
    "      learner = Learner(\n",
    "        databunch, model,\n",
    "        optimizer,\n",
    "        wd = config.weight_decay,\n",
    "        metrics=FBeta(beta=1), #accuracy, (metric to optimize on)\n",
    "        loss_func=config.loss_func, callback_fns=[partial(EarlyStoppingCallback, monitor='f_beta', min_delta=config.min_delta, patience=config.patience)]\n",
    "      )\n",
    "    else:\n",
    "      learner = Learner(\n",
    "        databunch, model,\n",
    "        optimizer,\n",
    "        wd = config.weight_decay,\n",
    "        metrics=FBeta(beta=1), #accuracy, (metric to optimize on)\n",
    "        loss_func=config.loss_func,\n",
    "      )\n",
    "    \n",
    "    return learner\n",
    "\n",
    "# Create the classifier\n",
    "def create_classifier(config, df):\n",
    "  df_trn, df_valid = split_dataframe(df, train_size = config.train_size, random_state = config.seed)\n",
    "  databunch = create_databunch(config, df_trn, df_valid)\n",
    "\n",
    "  return create_learner(config, databunch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "da-sBmbOCLYc"
   },
   "outputs": [],
   "source": [
    "#@title Define predict loop {display-mode: \"form\"}\n",
    "def predict_and_log_result(classifier, df_eval):\n",
    "  predictor = Predictor(classifier)\n",
    "  flat_predictions, flat_true_labels = [], []\n",
    "  column_index = df_eval.columns.get_loc(config_data.label_column)\n",
    "  for row in progress_bar(df_eval.itertuples(), total=len(df_eval)):\n",
    "      class_text = row.text\n",
    "      class_label = row[column_index+1]\n",
    "      flat_true_labels.append(class_label)\n",
    "      prediction = predictor.predict(class_text)\n",
    "      flat_predictions.append(prediction)\n",
    "      df_result.loc[df_result['index'] == row.index, config_data.label_column] = prediction\n",
    "\n",
    "      log_text = 'PID: {}, {}, {} -> {}'.format(row.ProjectID, class_text, label_indices.get(class_label), label_indices.get(prediction))\n",
    "      logLine(log_text)\n",
    "  \n",
    "  # get labels in correct order\n",
    "  target_names = []\n",
    "  test_labels = unique_labels(flat_true_labels, flat_predictions) \n",
    "  test_labels = np.sort(test_labels)\n",
    "  for x in test_labels:\n",
    "    target_names.append(label_indices.get(x))\n",
    "\n",
    "  result = classification_report(flat_true_labels, flat_predictions, target_names=target_names, digits = 5)\n",
    "  logResult(result)\n",
    "  print(result)\n",
    "  return flat_predictions, flat_true_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PHRYZ-rX-860"
   },
   "outputs": [],
   "source": [
    "#@title Define train and test loop{display-mode: \"form\"}\n",
    "def train_and_predict(df_train, df_eval, overall_flat_predictions, overall_flat_true_labels, results):\n",
    "  classifier = create_classifier(config, df_train)\n",
    "  # Train the classifier on train set\n",
    "  print(classifier.fit_one_cycle(config.epochs, max_lr=config.max_lr, moms=config.moms, wd=config.weight_decay))\n",
    "  #Predict on test set\n",
    "  flat_predictions, flat_true_labels = predict_and_log_result(classifier, df_eval)\n",
    "  overall_flat_predictions.extend(flat_predictions)\n",
    "  overall_flat_true_labels.extend(flat_true_labels)\n",
    "  test_labels = df_eval[config_data.label_column].unique()\n",
    "  test_labels = np.sort(test_labels)\n",
    "  results.extend(precision_recall_fscore_support(flat_true_labels, flat_predictions, labels = test_labels))\n",
    "  return classifier, overall_flat_predictions, overall_flat_true_labels, results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#@title Define function to calculate averaged metric results {display-mode: \"form\"}\n",
    "def calcAverageMetrics(results):\n",
    "  precisions, recalls, fscores = [], [], []\n",
    "  for i in range(int(len(results)/4)):\n",
    "    precisions.append(results[i*4])\n",
    "    recalls.append(results[i*4+1])\n",
    "    fscores.append(results[i*4+2])\n",
    "  precision = [0]*len(precisions[0])\n",
    "  recall = [0]*len(recalls[0])\n",
    "  fscore = [0]*len(fscores[0])\n",
    "  for i in range(len(precisions)):\n",
    "    precision = precision + precisions[i]\n",
    "    recall = recall + recalls[i]\n",
    "    fscore = fscore + fscores[i]\n",
    "  precision = precision / int(len(results)/4)\n",
    "  recall = recall / int(len(results)/4)\n",
    "  fscore = fscore / int(len(results)/4)\n",
    "  return precision, recall, fscore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sqGyLEubBw60",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#@title Decide how to fold and train the classifier {display-mode: \"form\"}\n",
    "# run train/eval loop for each in classes defined class subsequently\n",
    "firstRun = True\n",
    "for cl in config_data.classes:\n",
    "    set_seed(config.seed)\n",
    "    config_data.label_column = cl\n",
    "    # load the train dataset\n",
    "    label_indices = create_label_indices(df)\n",
    "    print(label_indices)\n",
    "\n",
    "    overall_flat_predictions, overall_flat_true_labels, results = [], [], []\n",
    "    initLog(firstRun)\n",
    "    if config.fold == Fold.TenFold:\n",
    "        skf = StratifiedKFold(n_splits=10)\n",
    "        fold_number = 1\n",
    "        for train, test in skf.split(df, df[config_data.label_column]):\n",
    "            df_train = df.iloc[train]\n",
    "            df_eval = df.iloc[test]\n",
    "            log_text = '/////////////////////// Fold: {} of {} /////////////////////////////'.format(fold_number,10)\n",
    "            logLine(log_text)\n",
    "            classifier, overall_flat_predictions, overall_flat_true_labels, results = train_and_predict(df_train, df_eval, overall_flat_predictions, overall_flat_true_labels, results)\n",
    "            fold_number = fold_number + 1\n",
    "    elif config.fold == Fold.ProjFold:     \n",
    "        for k in config_data.project_fold:\n",
    "            train = df.loc[~df['ProjectID'].isin(k)].index\n",
    "            test = df.loc[df['ProjectID'].isin(k)].index\n",
    "            if k == \"UserRelated\":\n",
    "                k = [\"UserRelated\", 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]\n",
    "                train = df.loc[~df['ProjectID'].isin(k)].index\n",
    "            df_train = df.loc[train]\n",
    "            df_eval = df.loc[test]\n",
    "            log_text = '/////////////////////// Test-Projects: {} /////////////////////////////'.format(k)\n",
    "            logLine(log_text)\n",
    "            classifier, overall_flat_predictions, overall_flat_true_labels, results = train_and_predict(df_train, df_eval, overall_flat_predictions, overall_flat_true_labels, results)\n",
    "    else:\n",
    "        df_train, df_eval = train_test_split(df,stratify=df[config_data.label_column], train_size=config.train_size, random_state= config.seed)\n",
    "        classifier, overall_flat_predictions, overall_flat_true_labels, results = train_and_predict(df_train, df_eval, overall_flat_predictions, overall_flat_true_labels, results)\n",
    "\n",
    "    target_names = []\n",
    "    test_labels = df[config_data.label_column].unique()\n",
    "\n",
    "    test_labels = np.sort(test_labels)\n",
    "    for x in test_labels:\n",
    "        target_names.append(label_indices.get(x))\n",
    "\n",
    "    print('/////////////////////// Aggregated Predictions Result /////////////////////////////')\n",
    "    resultText = '/////////////////////// Aggregated Predictions Result /////////////////////////////\\n'\n",
    "    result = classification_report(overall_flat_true_labels, overall_flat_predictions, target_names=target_names, digits = 5)\n",
    "    resultText += result\n",
    "    print(result)\n",
    "    print('/////////////////////// Averaged Metrics Result /////////////////////////////')\n",
    "    resultText += '/////////////////////// Averaged Metrics Result /////////////////////////////\\n'\n",
    "    precision, recall, fscore = calcAverageMetrics(results)\n",
    "    print(\"              precision    recall  f1-score\")\n",
    "    resultText += \"              precision    recall  f1-score\\n\"\n",
    "    for i in range(len(precision)):\n",
    "        print('{:<14}'.format(target_names[i]) + '  {:.5f}'.format(precision[i]) + '   {:.5f}'.format(recall[i]) + '   {:.5f}'.format(fscore[i]))\n",
    "        resultText += '{:<14}'.format(target_names[i]) + '  {:.5f}'.format(precision[i]) + '   {:.5f}'.format(recall[i]) + '   {:.5f}'.format(fscore[i]) + '\\n'\n",
    "    logResult(resultText)\n",
    "    firstRun = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cwAQt_Zhy6Ar"
   },
   "outputs": [],
   "source": [
    "#@title Display and log overall evaluation results {display-mode: \"form\"}\n",
    "target_names = []\n",
    "test_labels = df_eval[config_data.label_column].unique()\n",
    "\n",
    "test_labels = np.sort(test_labels)\n",
    "for x in test_labels:\n",
    "  target_names.append(label_indices.get(x))\n",
    "\n",
    "print('/////////////////////// Aggregated Predictions Result /////////////////////////////')\n",
    "resultText = '/////////////////////// Aggregated Predictions Result /////////////////////////////\\n'\n",
    "result = classification_report(overall_flat_true_labels, overall_flat_predictions, target_names=target_names, digits = 5)\n",
    "resultText += result\n",
    "print(result)\n",
    "print('/////////////////////// Averaged Metrics Result /////////////////////////////')\n",
    "resultText += '/////////////////////// Averaged Metrics Result /////////////////////////////\\n'\n",
    "precision, recall, fscore = calcAverageMetrics(results)\n",
    "print(\"              precision    recall  f1-score\")\n",
    "resultText += \"              precision    recall  f1-score\\n\"\n",
    "for i in range(len(precision)):\n",
    "    print('{:<14}'.format(target_names[i]) + '  {:.5f}'.format(precision[i]) + '   {:.5f}'.format(recall[i]) + '   {:.5f}'.format(fscore[i]))\n",
    "    resultText += '{:<14}'.format(target_names[i]) + '  {:.5f}'.format(precision[i]) + '   {:.5f}'.format(recall[i]) + '   {:.5f}'.format(fscore[i]) + '\\n'\n",
    "logResult(resultText)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Save Result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#@title Safe result dataframe {display-mode: \"form\"}\n",
    "def create_result_csv_name():\n",
    "    name = 'classification_result_{fold}_{model}_e{epochs}_{sampling}_{clasz}.csv'.format(fold=Fold(config.fold).name,model=config.model_name,epochs=str(config.epochs),sampling=Sampling(config.sampling).name,clasz=clazz)\n",
    "    return name\n",
    "\n",
    "df_result.sort_index(inplace=True)\n",
    "if 'index' in df_result:\n",
    "    df_result.drop(['index'],axis=1,inplace=True)\n",
    "df_result.to_csv(config_data.root_folder + config_data.data_folder + create_result_csv_name(), index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UeoCn0Bs-Wds"
   },
   "source": [
    "## Save Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DXTWGILJ4kJx"
   },
   "outputs": [],
   "source": [
    "#@title Save the model along with its config\n",
    "def create_model_name():\n",
    "    name = 'NoRBERT_{clasz}_e{epochs}_{sampling}'.format(clasz=clazz, epochs=str(config.epochs),sampling=Sampling(config.sampling).name)\n",
    "    return name\n",
    "\n",
    "def save_config(model_save_path, model_name):\n",
    "    settings = ''\n",
    "    for item in config.__dict__:\n",
    "        value = config[item]\n",
    "        setting = '{item}={value},\\n'.format(item=item, value=value)\n",
    "        settings += setting\n",
    "    save_path = model_save_path + model_name + '.config'\n",
    "    with open(save_path, 'w', encoding='utf-8') as out:\n",
    "        out.write(settings)\n",
    "\n",
    "if save_model:\n",
    "  model_name = create_model_name()\n",
    "  model_save_path = config_data.root_folder + config_data.model_path\n",
    "  if not os.path.isdir(model_save_path):\n",
    "    print(\"Models folder does not exist, trying to create folder.\")\n",
    "    try:\n",
    "      os.mkdir(model_save_path)\n",
    "    except OSError:\n",
    "      print (\"Creation of the directory %s failed\" % model_save_path)\n",
    "    else:\n",
    "      print (\"Successfully created the directory %s\" % model_save_path)\n",
    "  save_config(model_save_path, model_name)\n",
    "  model_save_file = model_save_path + model_name + '.pkl'\n",
    "  classifier.export(file = model_save_file)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "machine_shape": "hm",
   "name": "TLRClassification.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}