Reinforcement-learning-fine-tuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d7f10b19-c061-42b2-8b42-e3cbafa3b1da",
   "metadata": {},
   "source": [
    "# Fine-Tune FLAN-T5 with Reinforcement Learning (PPO) and PEFT to Generate Less-Toxic Summaries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36ef668a-9c51-489b-be47-a07a09ef2289",
   "metadata": {},
   "source": [
    "This notebook focuses on fine-tuning a FLAN-T5 model to generate less toxic content using Meta AI's hate speech reward model, a binary classifier that predicts \"not hate\" or \"hate.\" Proximal Policy Optimization (PPO) will be used for this fine-tuning to reduce the model's toxicity."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89f973f1-f095-4915-86d0-bc16380da22d",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a name='1'></a>\n",
    "## 1 - Import the necessary components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d8c20bed-6a30-4847-a507-02969ecb4465",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from transformers import pipeline, AutoTokenizer, AutoModelForSequenceClassification, AutoModelForSeq2SeqLM, GenerationConfig\n",
    "from datasets import load_dataset\n",
    "from peft import PeftModel, PeftConfig, LoraConfig, TaskType\n",
    "\n",
    "# trl: Transformer Reinforcement Learning library\n",
    "from trl import PPOTrainer, PPOConfig, AutoModelForSeq2SeqLMWithValueHead\n",
    "from trl import create_reference_model\n",
    "from trl.core import LengthSampler\n",
    "\n",
    "import torch\n",
    "import evaluate\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# tqdm library makes the loops show a smart progress meter.\n",
    "from tqdm import tqdm\n",
    "tqdm.pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b76eea84-8e3a-4487-9692-613977e6c8e3",
   "metadata": {},
   "source": [
    "<a name='2'></a>\n",
    "## 2 - Load FLAN-T5 Model, Prepare Reward Model and Toxicity Evaluator"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a5f97d4-ea5f-4072-b5d6-785d1d833ed4",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a name='2.1'></a>\n",
    "### 2.1 - Load Data and FLAN-T5 Model Fine-Tuned with Summarization Instruction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90dc0211-4032-4967-946d-3a538829d5c9",
   "metadata": {
    "tags": []
   },
   "source": [
    "Continue using the Hugging Face dataset DialogSum and the pre-trained model FLAN-T5 for the fine-tuning process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b058b52b-ec4d-4426-8d71-91e898f727f6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['id', 'dialogue', 'summary', 'topic'],\n",
       "        num_rows: 12460\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['id', 'dialogue', 'summary', 'topic'],\n",
       "        num_rows: 500\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['id', 'dialogue', 'summary', 'topic'],\n",
       "        num_rows: 1500\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_name=\"google/flan-t5-base\"\n",
    "huggingface_dataset_name = \"knkarthick/dialogsum\"\n",
    "\n",
    "dataset_original = load_dataset(huggingface_dataset_name)\n",
    "\n",
    "dataset_original"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "668d30d6-6f81-4e52-a81a-3057163ddb0e",
   "metadata": {},
   "source": [
    "The next step involves preprocessing the dataset by selecting a portion of it, filtering dialogues by length for readability, wrapping each dialogue with instructions, and tokenizing the prompts. This can be organized into a function called `build_dataset` that saves the token IDs in the `input_ids` field and the decoded prompts in the `query` field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "51469abe-4d72-4093-a6c6-8e04e19f09eb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DatasetDict({\n",
      "    train: Dataset({\n",
      "        features: ['id', 'dialogue', 'summary', 'topic', 'input_ids', 'query'],\n",
      "        num_rows: 8017\n",
      "    })\n",
      "    test: Dataset({\n",
      "        features: ['id', 'dialogue', 'summary', 'topic', 'input_ids', 'query'],\n",
      "        num_rows: 2005\n",
      "    })\n",
      "})\n"
     ]
    }
   ],
   "source": [
    "def build_dataset(model_name,\n",
    "                  dataset_name,\n",
    "                  input_min_text_length, \n",
    "                  input_max_text_length):\n",
    "\n",
    "    \"\"\"\n",
    "    Preprocess the dataset and split it into train and test parts.\n",
    "\n",
    "    Parameters:\n",
    "    - model_name (str): Tokenizer model name.\n",
    "    - dataset_name (str): Name of the dataset to load.\n",
    "    - input_min_text_length (int): Minimum length of the dialogues.\n",
    "    - input_max_text_length (int): Maximum length of the dialogues.\n",
    "        \n",
    "    Returns:\n",
    "    - dataset_splits (datasets.dataset_dict.DatasetDict): Preprocessed dataset containing train and test parts.\n",
    "    \"\"\"\n",
    "    \n",
    "    # load dataset (only \"train\" part will be enough for this lab).\n",
    "    dataset = load_dataset(dataset_name, split=\"train\")\n",
    "    \n",
    "    # Filter the dialogues of length between input_min_text_length and input_max_text_length characters.\n",
    "    dataset = dataset.filter(lambda x: len(x[\"dialogue\"]) > input_min_text_length and len(x[\"dialogue\"]) <= input_max_text_length, batched=False)\n",
    "\n",
    "    # Prepare tokenizer. Setting device_map=\"auto\" allows to switch between GPU and CPU automatically.\n",
    "    tokenizer = AutoTokenizer.from_pretrained(model_name, device_map=\"auto\")\n",
    "    \n",
    "    def tokenize(sample):\n",
    "        \n",
    "        # Wrap each dialogue with the instruction.\n",
    "        prompt = f\"\"\"\n",
    "Summarize the following conversation.\n",
    "\n",
    "{sample[\"dialogue\"]}\n",
    "\n",
    "Summary:\n",
    "\"\"\"\n",
    "        sample[\"input_ids\"] = tokenizer.encode(prompt)\n",
    "        \n",
    "        # This must be called \"query\", which is a requirement of our PPO library.\n",
    "        sample[\"query\"] = tokenizer.decode(sample[\"input_ids\"])\n",
    "        return sample\n",
    "\n",
    "    # Tokenize each dialogue.\n",
    "    dataset = dataset.map(tokenize, batched=False)\n",
    "    dataset.set_format(type=\"torch\")\n",
    "    \n",
    "    # Split the dataset into train and test parts.\n",
    "    dataset_splits = dataset.train_test_split(test_size=0.2, shuffle=False, seed=42)\n",
    "\n",
    "    return dataset_splits\n",
    "\n",
    "dataset = build_dataset(model_name=model_name,\n",
    "                        dataset_name=huggingface_dataset_name,\n",
    "                        input_min_text_length=200, \n",
    "                        input_max_text_length=1000)\n",
    "\n",
    "print(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d03155e-649b-45bb-a5a0-94edd682c069",
   "metadata": {
    "tags": []
   },
   "source": [
    "Load the fully trained PEFT model checkpoint in this section to proceed with inference or further tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dec8bea4-addd-4b29-b3af-6db6ea2baeb7",
   "metadata": {
    "tags": []
   },
   "source": [
    "List the model item and check its size (it's less than 15 Mb):"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4226923-67c0-4ea6-8e47-030136b2f191",
   "metadata": {},
   "source": [
    "Define a function to retrieve the number of model parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a1f06806-a194-4c14-b64d-e31afd7b658c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def print_number_of_trainable_model_parameters(model):\n",
    "    trainable_model_params = 0\n",
    "    all_model_params = 0\n",
    "    for _, param in model.named_parameters():\n",
    "        all_model_params += param.numel()\n",
    "        if param.requires_grad:\n",
    "            trainable_model_params += param.numel()\n",
    "    return f\"\\ntrainable model parameters: {trainable_model_params}\\nall model parameters: {all_model_params}\\npercentage of trainable model parameters: {100 * trainable_model_params / all_model_params:.2f}%\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21e06a57-fb80-4f8c-a967-4c7c42a7bfda",
   "metadata": {
    "tags": []
   },
   "source": [
    "Add the adapter to the original FLAN-T5 model by passing the LoRA configurations, setting `is_trainable=True` for training purposes, unlike in the previous lab where only inference was performed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a1a94b14-b375-45e7-9e49-a7f2c341b4ff",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PEFT model parameters to be updated:\n",
      "\n",
      "trainable model parameters: 3538944\n",
      "all model parameters: 251116800\n",
      "percentage of trainable model parameters: 1.41%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "lora_config = LoraConfig(\n",
    "    r=32, # Rank\n",
    "    lora_alpha=32,\n",
    "    target_modules=[\"q\", \"v\"],\n",
    "    lora_dropout=0.05,\n",
    "    bias=\"none\",\n",
    "    task_type=TaskType.SEQ_2_SEQ_LM # FLAN-T5\n",
    ")\n",
    "\n",
    "model = AutoModelForSeq2SeqLM.from_pretrained(model_name, \n",
    "                                              torch_dtype=torch.bfloat16)\n",
    "\n",
    "peft_model = PeftModel.from_pretrained(model, \n",
    "                                       './peft-dialogue-summary-checkpoint/', \n",
    "                                       lora_config=lora_config,\n",
    "                                       torch_dtype=torch.bfloat16, \n",
    "                                       device_map=\"auto\",                                       \n",
    "                                       is_trainable=True)\n",
    "\n",
    "print(f'PEFT model parameters to be updated:\\n{print_number_of_trainable_model_parameters(peft_model)}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a950ae8a-76b9-4951-9c78-9ac7a6349e17",
   "metadata": {},
   "source": [
    "Prepare to fine-tune the LLM using Reinforcement Learning (RL) by setting up the Proximal Policy Optimization (PPO) model with the instruct-fine-tuned PEFT model. PPO will optimize the RL policy based on the reward model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1e86bab0-6dee-4dff-a754-b584ed962723",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PPO model parameters to be updated (ValueHead + 769 params):\n",
      "\n",
      "trainable model parameters: 3539713\n",
      "all model parameters: 251117569\n",
      "percentage of trainable model parameters: 1.41%\n",
      "\n",
      "ValueHead(\n",
      "  (dropout): Dropout(p=0.1, inplace=False)\n",
      "  (summary): Linear(in_features=768, out_features=1, bias=True)\n",
      "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "ppo_model = AutoModelForSeq2SeqLMWithValueHead.from_pretrained(peft_model,                                                               \n",
    "                                                               torch_dtype=torch.bfloat16,\n",
    "                                                               is_trainable=True)\n",
    "\n",
    "print(f'PPO model parameters to be updated (ValueHead + 769 params):\\n{print_number_of_trainable_model_parameters(ppo_model)}\\n')\n",
    "print(ppo_model.v_head)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2913ef05-737e-4cdf-9bac-467ee6cf9f76",
   "metadata": {},
   "source": [
    "During PPO, only a few parameters will be updated. Specifically, the parameters of the `ValueHead`. The number of trainable parameters can be computed as $(n+1)*m$, where $n$ is the number of input units (here $n=768$) and $m$ is the number of output units (you have $m=1$). The $+1$ term in the equation takes into account the bias term."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76c7e2df-0c75-4bd0-bf8d-1f1545ef864e",
   "metadata": {},
   "source": [
    "Create a frozen copy of the PPO model to serve as a reference model, representing the LLM before detoxification. This reference model will have its parameters unchanged during PPO training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "18a9b30a-ad14-4189-8088-d4de447fe247",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reference model parameters to be updated:\n",
      "\n",
      "trainable model parameters: 0\n",
      "all model parameters: 251117569\n",
      "percentage of trainable model parameters: 0.00%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ref_model = create_reference_model(ppo_model)\n",
    "\n",
    "print(f'Reference model parameters to be updated:\\n{print_number_of_trainable_model_parameters(ref_model)}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3a14848-e83d-4fdd-bc68-eb770c5951d7",
   "metadata": {
    "tags": []
   },
   "source": [
    "Everything is set. It is time to prepare the reward model!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bfdf1f7-3509-4adc-812a-2b22bd330137",
   "metadata": {},
   "source": [
    "<a name='2.2'></a>\n",
    "### 2.2 - Prepare Reward Model\n",
    "Reinforcement Learning (RL) involves agents taking actions in an environment to maximize cumulative rewards, with the agent's behavior defined by a policy. In this context, the original policy is based on the instruct PEFT model, representing the LLM before detoxification. Human labeler feedback on toxicity can be costly, so a reward model is used instead to encourage detoxification of dialogue summaries.\n",
    "\n",
    "The reward model will use Meta AI's RoBERTa-based hate speech model, which outputs logits predicting probabilities for the classes **nothate** and **hate**. The logits for **nothate** will serve as positive rewards for training.\n",
    "\n",
    "To implement this, create an instance of the required model class for the RoBERTa model and load the corresponding tokenizer for testing. Remember that label **0** corresponds to **nothate** and label **1** to **hate**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7f038a9f-e04b-49bc-8923-8ef3816919ee",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0: 'nothate', 1: 'hate'}\n"
     ]
    }
   ],
   "source": [
    "toxicity_model_name = \"facebook/roberta-hate-speech-dynabench-r4-target\"\n",
    "toxicity_tokenizer = AutoTokenizer.from_pretrained(toxicity_model_name, device_map=\"auto\")\n",
    "toxicity_model = AutoModelForSequenceClassification.from_pretrained(toxicity_model_name, device_map=\"auto\")\n",
    "print(toxicity_model.config.id2label)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79d68799-a6e8-42d7-8d61-002e47210c18",
   "metadata": {
    "tags": []
   },
   "source": [
    "Take some non-toxic text, tokenize it, and pass it to the model. Print the output logits, probabilities, and the corresponding reward that will be used for fine-tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8f4e6a05-2398-4ca7-a176-d5a1ff27fe39",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logits [not hate, hate]: [3.114102363586426, -2.489619016647339]\n",
      "probabilities [not hate, hate]: [0.9963293671607971, 0.0036706042010337114]\n",
      "reward (high): [3.114102363586426]\n"
     ]
    }
   ],
   "source": [
    "non_toxic_text = \"#Person 1# tells Tommy that he didn't like the movie.\"\n",
    "\n",
    "toxicity_input_ids = toxicity_tokenizer(non_toxic_text, return_tensors=\"pt\").input_ids\n",
    "\n",
    "logits = toxicity_model(input_ids=toxicity_input_ids).logits\n",
    "print(f'logits [not hate, hate]: {logits.tolist()[0]}')\n",
    "\n",
    "# Print the probabilities for [not hate, hate]\n",
    "probabilities = logits.softmax(dim=-1).tolist()[0]\n",
    "print(f'probabilities [not hate, hate]: {probabilities}')\n",
    "\n",
    "# get the logits for \"not hate\" - this is the reward!\n",
    "not_hate_index = 0\n",
    "nothate_reward = (logits[:, not_hate_index]).tolist()\n",
    "print(f'reward (high): {nothate_reward}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63f729c5-98c3-4745-96e8-3484670215db",
   "metadata": {},
   "source": [
    "Let's show a toxic comment.  This will have a low reward because it is more toxic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0ffc81e4-b220-4f4e-95ec-98ac418612d1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logits [not hate, hate]: [-0.6921197175979614, 0.37227365374565125]\n",
      "probabilities [not hate, hate]: [0.2564707398414612, 0.743529200553894]\n",
      "reward (low): [-0.6921197175979614]\n"
     ]
    }
   ],
   "source": [
    "toxic_text = \"#Person 1# tells Tommy that the movie was terrible, dumb and stupid.\"\n",
    "\n",
    "toxicity_input_ids = toxicity_tokenizer(toxic_text, return_tensors=\"pt\").input_ids\n",
    "\n",
    "logits = toxicity_model(toxicity_input_ids).logits\n",
    "print(f'logits [not hate, hate]: {logits.tolist()[0]}')\n",
    "\n",
    "# Print the probabilities for [not hate, hate]\n",
    "probabilities = logits.softmax(dim=-1).tolist()[0]\n",
    "print(f'probabilities [not hate, hate]: {probabilities}')\n",
    "\n",
    "# Get the logits for \"not hate\" - this is the reward!\n",
    "nothate_reward = (logits[:, not_hate_index]).tolist() \n",
    "print(f'reward (low): {nothate_reward}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc6e656e-fefe-4623-8bbb-472a8cf1c3c5",
   "metadata": {},
   "source": [
    "Setup Hugging Face inference pipeline to simplify the code for the toxicity reward model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "73aab8c6-33eb-4bf4-a816-3866fc3460af",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reward model output:\n",
      "For non-toxic text\n",
      "[{'label': 'nothate', 'score': 3.114102363586426}, {'label': 'hate', 'score': -2.489619016647339}]\n",
      "[{'label': 'nothate', 'score': 0.9963293671607971}, {'label': 'hate', 'score': 0.0036706042010337114}]\n",
      "For toxic text\n",
      "[{'label': 'hate', 'score': 0.37227365374565125}, {'label': 'nothate', 'score': -0.6921197175979614}]\n",
      "[{'label': 'hate', 'score': 0.743529200553894}, {'label': 'nothate', 'score': 0.2564707398414612}]\n"
     ]
    }
   ],
   "source": [
    "device = 0 if torch.cuda.is_available() else \"cpu\"\n",
    "\n",
    "sentiment_pipe = pipeline(\"sentiment-analysis\", \n",
    "                          model=toxicity_model_name, \n",
    "                          device=device)\n",
    "reward_logits_kwargs = {\n",
    "    \"top_k\": None, # Return all scores.\n",
    "    \"function_to_apply\": \"none\", # Set to \"none\" to retrieve raw logits.\n",
    "    \"batch_size\": 16\n",
    "}\n",
    "\n",
    "reward_probabilities_kwargs = {\n",
    "    \"top_k\": None, # Return all scores.\n",
    "    \"function_to_apply\": \"softmax\", # Set to \"softmax\" to apply softmax and retrieve probabilities.\n",
    "    \"batch_size\": 16\n",
    "}\n",
    "\n",
    "print(\"Reward model output:\")\n",
    "print(\"For non-toxic text\")\n",
    "print(sentiment_pipe(non_toxic_text, **reward_logits_kwargs))\n",
    "print(sentiment_pipe(non_toxic_text, **reward_probabilities_kwargs))\n",
    "print(\"For toxic text\")\n",
    "print(sentiment_pipe(toxic_text, **reward_logits_kwargs))\n",
    "print(sentiment_pipe(toxic_text, **reward_probabilities_kwargs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21302d74-59d8-451f-b287-e86245bf3324",
   "metadata": {},
   "source": [
    "The outputs are the logits for both `nothate` (positive) and `hate` (negative) classes. But PPO will be using logits only of the `nothate` class as the positive reward signal used to help detoxify the LLM outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "36ff3925-70c4-495c-ae26-68e2fc36296b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'label': 'nothate', 'score': 3.114102363586426}, {'label': 'hate', 'score': -2.489619016647339}]\n",
      "[{'label': 'nothate', 'score': 0.9963293671607971}, {'label': 'hate', 'score': 0.0036706042010337114}]\n"
     ]
    }
   ],
   "source": [
    "print(sentiment_pipe(non_toxic_text, **reward_logits_kwargs))\n",
    "print(sentiment_pipe(non_toxic_text, **reward_probabilities_kwargs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8d11618b-5887-489a-b390-2139e364987f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'label': 'hate', 'score': 0.37227365374565125}, {'label': 'nothate', 'score': -0.6921197175979614}]\n",
      "[{'label': 'hate', 'score': 0.743529200553894}, {'label': 'nothate', 'score': 0.2564707398414612}]\n"
     ]
    }
   ],
   "source": [
    "print(sentiment_pipe(toxic_text, **reward_logits_kwargs))\n",
    "print(sentiment_pipe(toxic_text, **reward_probabilities_kwargs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56513033-9bb1-41d5-81e2-54d1249c5c89",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a name='2.3'></a>\n",
    "### 2.3 - Evaluate Toxicity\n",
    "To evaluate the model before and after fine-tuning for detoxification, set up the toxicity evaluation metric. The **toxicity score** ranges from 0 to 1, with 1 indicating the highest level of toxicity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "7de8e99d-60ea-48a2-bdf5-817f80b48979",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "toxicity_evaluator = evaluate.load(\"toxicity\", \n",
    "                                    toxicity_model_name,\n",
    "                                    module_type=\"measurement\",\n",
    "                                    toxic_label=\"hate\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "840fbc47-c5c2-469a-b5f2-6407e8f0bfde",
   "metadata": {
    "tags": []
   },
   "source": [
    "The toxicity scores will be the probabilities of the **hate** class returned directly from the reward model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "5298f91c-30d1-4d17-95a7-553952ac97b5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Toxicity score for non-toxic text:\n",
      "[0.0036706042010337114]\n",
      "\n",
      "Toxicity score for toxic text:\n",
      "[0.743529200553894]\n"
     ]
    }
   ],
   "source": [
    "toxicity_score = toxicity_evaluator.compute(predictions=[\n",
    "    non_toxic_text\n",
    "])\n",
    "\n",
    "print(\"Toxicity score for non-toxic text:\")\n",
    "print(toxicity_score[\"toxicity\"])\n",
    "\n",
    "toxicity_score = toxicity_evaluator.compute(predictions=[\n",
    "    toxic_text\n",
    "])\n",
    "\n",
    "print(\"\\nToxicity score for toxic text:\")\n",
    "print(toxicity_score[\"toxicity\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d3e835b-14b9-4646-b1c9-c975ef3ea944",
   "metadata": {
    "tags": []
   },
   "source": [
    "Use the evaluator to compute the toxicity of the dialogues by passing the test dataset (dataset[\"test\"]), tokenizer, frozen PEFT model and toxicity evaluator. It’s helpful to encapsulate these steps in a function called `evaluate_toxicity`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "316ab128-33ff-4a1e-8936-47bfa29d48a3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def evaluate_toxicity(model, \n",
    "                      toxicity_evaluator, \n",
    "                      tokenizer, \n",
    "                      dataset, \n",
    "                      num_samples):\n",
    "    \n",
    "    \"\"\"\n",
    "    Preprocess the dataset and split it into train and test parts.\n",
    "\n",
    "    Parameters:\n",
    "    - model (trl model): Model to be evaluated.\n",
    "    - toxicity_evaluator (evaluate_modules toxicity metrics): Toxicity evaluator.\n",
    "    - tokenizer (transformers tokenizer): Tokenizer to be used.\n",
    "    - dataset (dataset): Input dataset for the evaluation.\n",
    "    - num_samples (int): Maximum number of samples for the evaluation.\n",
    "        \n",
    "    Returns:\n",
    "    tuple: A tuple containing two numpy.float64 values:\n",
    "    - mean (numpy.float64): Mean of the samples toxicity.\n",
    "    - std (numpy.float64): Standard deviation of the samples toxicity.\n",
    "    \"\"\"\n",
    "\n",
    "    max_new_tokens=100\n",
    "\n",
    "    toxicities = []\n",
    "    input_texts = []\n",
    "    for i, sample in tqdm(enumerate(dataset)):\n",
    "        input_text = sample[\"query\"]\n",
    "\n",
    "        if i > num_samples:\n",
    "            break\n",
    "            \n",
    "        input_ids = tokenizer(input_text, return_tensors=\"pt\", padding=True).input_ids\n",
    "        \n",
    "        generation_config = GenerationConfig(max_new_tokens=max_new_tokens,\n",
    "                                             top_k=0.0,\n",
    "                                             top_p=1.0,\n",
    "                                             do_sample=True)\n",
    "\n",
    "        response_token_ids = model.generate(input_ids=input_ids,\n",
    "                                            generation_config=generation_config)\n",
    "        \n",
    "        generated_text = tokenizer.decode(response_token_ids[0], skip_special_tokens=True)\n",
    "        \n",
    "        toxicity_score = toxicity_evaluator.compute(predictions=[(input_text + \" \" + generated_text)])\n",
    "\n",
    "        toxicities.extend(toxicity_score[\"toxicity\"])\n",
    "\n",
    "    # Compute mean & std using np.\n",
    "    mean = np.mean(toxicities)\n",
    "    std = np.std(toxicities)\n",
    "        \n",
    "    return mean, std"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aed269c3-dbd7-4d45-bc44-c6ab6d4ae141",
   "metadata": {
    "tags": []
   },
   "source": [
    "And now perform the calculation of the model toxicity before fine-tuning/detoxification:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c11ede15-dc1a-4a7e-a60d-b9cadfc7d876",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "51it [14:25, 16.97s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "toxicity [mean, std] before detox: [0.03329359488898441, 0.04593935212341302]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(model_name, device_map=\"auto\")\n",
    "\n",
    "mean_before_detoxification, std_before_detoxification = evaluate_toxicity(model=ref_model, \n",
    "                                                                          toxicity_evaluator=toxicity_evaluator, \n",
    "                                                                          tokenizer=tokenizer, \n",
    "                                                                          dataset=dataset[\"test\"], \n",
    "                                                                          num_samples=50)\n",
    "\n",
    "print(f'toxicity [mean, std] before detox: [{mean_before_detoxification}, {std_before_detoxification}]')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ba81c90-1ac8-4403-ac1a-d4c75c6df4f0",
   "metadata": {},
   "source": [
    "<a name='3'></a>\n",
    "## 3 - Perform Fine-Tuning to Detoxify the Summaries\n",
    "Optimize a RL policy against the reward model using Proximal Policy Optimization (PPO)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5516e318-8fce-4ca7-bf19-b7baf5255480",
   "metadata": {},
   "source": [
    "<a name='3.1'></a>\n",
    "### 3.1 - Initialize `PPOTrainer`\n",
    "For initializing the `PPOTrainer`, define a collator function that transforms the input dictionaries in the required format. Once defined, test the collator to ensure it functions correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8b7be1c0-382a-4fe2-8174-470f3e333e84",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collator input: [{'key1': 'value1', 'key2': 'value2', 'key3': 'value3'}]\n",
      "Collator output: {'key1': ['value1'], 'key2': ['value2'], 'key3': ['value3']}\n"
     ]
    }
   ],
   "source": [
    "def collator(data):\n",
    "    return dict((key, [d[key] for d in data]) for key in data[0])\n",
    "\n",
    "test_data = [{\"key1\": \"value1\", \"key2\": \"value2\", \"key3\": \"value3\"}]\n",
    "print(f'Collator input: {test_data}')\n",
    "print(f'Collator output: {collator(test_data)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "080c2e92-4988-4944-8353-0e1bb2048072",
   "metadata": {},
   "source": [
    "Set up the configuration parameters. Load the `ppo_model` and the tokenizer. You will also load a frozen version of the model `ref_model`. The first model is optimized while the second model serves as a reference to calculate the KL-divergence from the starting point. This works as an additional reward signal in the PPO training to make sure the optimized model does not deviate too much from the original LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "55e3afa3-39b0-41f8-9769-82019b13a73c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# learning_rate=1.41e-5\n",
    "# max_ppo_epochs=1\n",
    "# mini_batch_size=4\n",
    "# batch_size=16\n",
    "\n",
    "# config = PPOConfig(\n",
    "#     model_name=model_name,    \n",
    "#     learning_rate=learning_rate,\n",
    "#     ppo_epochs=max_ppo_epochs,\n",
    "#     mini_batch_size=mini_batch_size,\n",
    "#     batch_size=batch_size\n",
    "# )\n",
    "\n",
    "# ppo_trainer = PPOTrainer(config=config, \n",
    "#                          model=ppo_model, \n",
    "#                          ref_model=ref_model, \n",
    "#                          tokenizer=tokenizer, \n",
    "#                          dataset=dataset[\"train\"], \n",
    "#                          data_collator=collator)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ad77d2c-3800-4e15-bb38-3851d94ad374",
   "metadata": {},
   "source": [
    "<a name='3.2'></a>\n",
    "### 3.2 - Fine-Tune the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cac21fb-fea5-4e80-a741-87f35ae72c62",
   "metadata": {},
   "source": [
    "The fine-tuning loop consists of the following main steps:\n",
    "1. Get the query responses from the policy LLM (PEFT model).\n",
    "2. Get sentiments for query/responses from hate speech RoBERTa model.\n",
    "3. Optimize policy with PPO using the (query, response, reward) triplet.\n",
    "\n",
    "The operation is running if you see the following metrics appearing:\n",
    "* `objective/kl`: minimize kl divergence,\n",
    "* `ppo/returns/mean`: maximize mean returns,\n",
    "* `ppo/policy/advantages_mean`: maximize advantages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b4304a2b-7f9b-4950-8338-943b65eff21b",
   "metadata": {},
   "outputs": [],
   "source": [
    "output_min_length = 100\n",
    "output_max_length = 400\n",
    "output_length_sampler = LengthSampler(output_min_length, output_max_length)\n",
    "\n",
    "generation_kwargs = {\n",
    "    \"min_length\": 5,\n",
    "    \"top_k\": 0.0,\n",
    "    \"top_p\": 1.0,\n",
    "    \"do_sample\": True\n",
    "}\n",
    "\n",
    "reward_kwargs = {\n",
    "    \"top_k\": None, # Return all scores.\n",
    "    \"function_to_apply\": \"none\", # You want the raw logits without softmax.\n",
    "    \"batch_size\": 16\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "922324d2-cd28-41ea-afaf-0337f8f9886a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# max_ppo_steps = 30\n",
    "\n",
    "# for step, batch in tqdm(enumerate(ppo_trainer.dataloader)):\n",
    "#     # Break when you reach max_steps.\n",
    "#     if step >= max_ppo_steps:\n",
    "#         break   \n",
    "\n",
    "#     prompt_tensors = batch[\"input_ids\"]\n",
    "\n",
    "#     # Get response from FLAN-T5/PEFT LLM.\n",
    "#     summary_tensors = []\n",
    "\n",
    "#     for prompt_tensor in prompt_tensors:\n",
    "#         max_new_tokens = output_length_sampler()        \n",
    "            \n",
    "#         generation_kwargs[\"max_new_tokens\"] = max_new_tokens\n",
    "#         summary = ppo_trainer.generate(prompt_tensor, **generation_kwargs)\n",
    "        \n",
    "#         summary_tensors.append(summary.squeeze()[-max_new_tokens:])\n",
    "        \n",
    "#     # This needs to be called \"response\".\n",
    "#     batch[\"response\"] = [tokenizer.decode(r.squeeze()) for r in summary_tensors]\n",
    "\n",
    "#     # Compute reward outputs.\n",
    "#     query_response_pairs = [q + r for q, r in zip(batch[\"query\"], batch[\"response\"])]    \n",
    "#     rewards = sentiment_pipe(query_response_pairs, **reward_kwargs)\n",
    "\n",
    "#     # You use the `nothate` item because this is the score for the positive `nothate` class.\n",
    "#     reward_tensors = [torch.tensor(reward[not_hate_index][\"score\"]) for reward in rewards]    \n",
    "\n",
    "#     # Run PPO step.\n",
    "#     stats = ppo_trainer.step(prompt_tensors, summary_tensors, reward_tensors)\n",
    "#     ppo_trainer.log_stats(stats, batch, reward_tensors)\n",
    "    \n",
    "#     print(f'objective/kl: {stats[\"objective/kl\"]}')\n",
    "#     print(f'ppo/returns/mean: {stats[\"ppo/returns/mean\"]}')\n",
    "#     print(f'ppo/policy/advantages_mean: {stats[\"ppo/policy/advantages_mean\"]}')\n",
    "#     print('-'.join('' for x in range(100)))\n",
    "\n",
    "# # Save Final Checkpoint\n",
    "# !mkdir -p ./ppo-peft-dialogue-summary-checkpoint\n",
    "# ppo_model.save_pretrained(f'./ppo-peft-dialogue-summary-checkpoint')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d4dfb134-43c0-4db2-a844-6aafeb063928",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AutoModelForSeq2SeqLMWithValueHead(\n",
       "  (pretrained_model): PeftModelForSeq2SeqLM(\n",
       "    (base_model): LoraModel(\n",
       "      (model): T5ForConditionalGeneration(\n",
       "        (shared): Embedding(32128, 768)\n",
       "        (encoder): T5Stack(\n",
       "          (embed_tokens): Embedding(32128, 768)\n",
       "          (block): ModuleList(\n",
       "            (0): T5Block(\n",
       "              (layer): ModuleList(\n",
       "                (0): T5LayerSelfAttention(\n",
       "                  (SelfAttention): T5Attention(\n",
       "                    (q): lora.Linear(\n",
       "                      (base_layer): Linear(in_features=768, out_features=768, bias=False)\n",
       "                      (lora_dropout): ModuleDict(\n",
       "                        (default): Dropout(p=0.05, inplace=False)\n",
       "                      )\n",
       "                      (lora_A): ModuleDict(\n",
       "                        (default): Linear(in_features=768, out_features=32, bias=False)\n",
       "                      )\n",
       "                      (lora_B): ModuleDict(\n",
       "                        (default): Linear(in_features=32, out_features=768, bias=False)\n",
       "                      )\n",
       "                      (lora_embedding_A): ParameterDict()\n",
       "                      (lora_embedding_B): ParameterDict()\n",
       "                      (lora_magnitude_vector): ModuleDict()\n",
       "                    )\n",
       "                    (k): Linear(in_features=768, out_features=768, bias=False)\n",
       "                    (v): lora.Linear(\n",
       "                      (base_layer): Linear(in_features=768, out_features=768, bias=False)\n",
       "                      (lora_dropout): ModuleDict(\n",
       "                        (default): Dropout(p=0.05, inplace=False)\n",
       "                      )\n",
       "                      (lora_A): ModuleDict(\n",
       "                        (default): Linear(in_features=768, out_features=32, bias=False)\n",
       "                      )\n",
       "                      (lora_B): ModuleDict(\n",
       "                        (default): Linear(in_features=32, out_features=768, bias=False)\n",
       "                      )\n",
       "                      (lora_embedding_A): ParameterDict()\n",
       "                      (lora_embedding_B): ParameterDict()\n",
       "                      (lora_magnitude_vector): ModuleDict()\n",
       "                    )\n",
       "                    (o): Linear(in_features=768, out_features=768, bias=False)\n",
       "                    (relative_attention_bias): Embedding(32, 12)\n",
       "                  )\n",
       "                  (layer_norm): T5LayerNorm()\n",
       "                  (dropout): Dropout(p=0.1, inplace=False)\n",
       "                )\n",
       "                (1): T5LayerFF(\n",
       "                  (DenseReluDense): T5DenseGatedActDense(\n",
       "                    (wi_0): Linear(in_features=768, out_features=2048, bias=False)\n",
       "                    (wi_1): Linear(in_features=768, out_features=2048, bias=False)\n",
       "                    (wo): Linear(in_features=2048, out_features=768, bias=False)\n",
       "                    (dropout): Dropout(p=0.1, inplace=False)\n",
       "                    (act): NewGELUActivation()\n",
       "                  )\n",
       "                  (layer_norm): T5LayerNorm()\n",
       "                  (dropout): Dropout(p=0.1, inplace=False)\n",
       "                )\n",
       "              )\n",
       "            )\n",
       "            (1-11): 11 x T5Block(\n",
       "              (layer): ModuleList(\n",
       "                (0): T5LayerSelfAttention(\n",
       "                  (SelfAttention): T5Attention(\n",
       "                    (q): lora.Linear(\n",
       "                      (base_layer): Linear(in_features=768, out_features=768, bias=False)\n",
       "                      (lora_dropout): ModuleDict(\n",
       "                        (default): Dropout(p=0.05, inplace=False)\n",
       "                      )\n",
       "                      (lora_A): ModuleDict(\n",
       "                        (default): Linear(in_features=768, out_features=32, bias=False)\n",
       "                      )\n",
       "                      (lora_B): ModuleDict(\n",
       "                        (default): Linear(in_features=32, out_features=768, bias=False)\n",
       "                      )\n",
       "                      (lora_embedding_A): ParameterDict()\n",
       "                      (lora_embedding_B): ParameterDict()\n",
       "                      (lora_magnitude_vector): ModuleDict()\n",
       "                    )\n",
       "                    (k): Linear(in_features=768, out_features=768, bias=False)\n",
       "                    (v): lora.Linear(\n",
       "                      (base_layer): Linear(in_features=768, out_features=768, bias=False)\n",
       "                      (lora_dropout): ModuleDict(\n",
       "                        (default): Dropout(p=0.05, inplace=False)\n",
       "                      )\n",
       "                      (lora_A): ModuleDict(\n",
       "                        (default): Linear(in_features=768, out_features=32, bias=False)\n",
       "                      )\n",
       "                      (lora_B): ModuleDict(\n",
       "                        (default): Linear(in_features=32, out_features=768, bias=False)\n",
       "                      )\n",
       "                      (lora_embedding_A): ParameterDict()\n",
       "                      (lora_embedding_B): ParameterDict()\n",
       "                      (lora_magnitude_vector): ModuleDict()\n",
       "                    )\n",
       "                    (o): Linear(in_features=768, out_features=768, bias=False)\n",
       "                  )\n",
       "                  (layer_norm): T5LayerNorm()\n",
       "                  (dropout): Dropout(p=0.1, inplace=False)\n",
       "                )\n",
       "                (1): T5LayerFF(\n",
       "                  (DenseReluDense): T5DenseGatedActDense(\n",
       "                    (wi_0): Linear(in_features=768, out_features=2048, bias=False)\n",
       "                    (wi_1): Linear(in_features=768, out_features=2048, bias=False)\n",
       "                    (wo): Linear(in_features=2048, out_features=768, bias=False)\n",
       "                    (dropout): Dropout(p=0.1, inplace=False)\n",
       "                    (act): NewGELUActivation()\n",
       "                  )\n",
       "                  (layer_norm): T5LayerNorm()\n",
       "                  (dropout): Dropout(p=0.1, inplace=False)\n",
       "                )\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "          (final_layer_norm): T5LayerNorm()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (decoder): T5Stack(\n",
       "          (embed_tokens): Embedding(32128, 768)\n",
       "          (block): ModuleList(\n",
       "            (0): T5Block(\n",
       "              (layer): ModuleList(\n",
       "                (0): T5LayerSelfAttention(\n",
       "                  (SelfAttention): T5Attention(\n",
       "                    (q): lora.Linear(\n",
       "                      (base_layer): Linear(in_features=768, out_features=768, bias=False)\n",
       "                      (lora_dropout): ModuleDict(\n",
       "                        (default): Dropout(p=0.05, inplace=False)\n",
       "                      )\n",
       "                      (lora_A): ModuleDict(\n",
       "                        (default): Linear(in_features=768, out_features=32, bias=False)\n",
       "                      )\n",
       "                      (lora_B): ModuleDict(\n",
       "                        (default): Linear(in_features=32, out_features=768, bias=False)\n",
       "                      )\n",
       "                      (lora_embedding_A): ParameterDict()\n",
       "                      (lora_embedding_B): ParameterDict()\n",
       "                      (lora_magnitude_vector): ModuleDict()\n",
       "                    )\n",
       "                    (k): Linear(in_features=768, out_features=768, bias=False)\n",
       "                    (v): lora.Linear(\n",
       "                      (base_layer): Linear(in_features=768, out_features=768, bias=False)\n",
       "                      (lora_dropout): ModuleDict(\n",
       "                        (default): Dropout(p=0.05, inplace=False)\n",
       "                      )\n",
       "                      (lora_A): ModuleDict(\n",
       "                        (default): Linear(in_features=768, out_features=32, bias=False)\n",
       "                      )\n",
       "                      (lora_B): ModuleDict(\n",
       "                        (default): Linear(in_features=32, out_features=768, bias=False)\n",
       "                      )\n",
       "                      (lora_embedding_A): ParameterDict()\n",
       "                      (lora_embedding_B): ParameterDict()\n",
       "                      (lora_magnitude_vector): ModuleDict()\n",
       "                    )\n",
       "                    (o): Linear(in_features=768, out_features=768, bias=False)\n",
       "                    (relative_attention_bias): Embedding(32, 12)\n",
       "                  )\n",
       "                  (layer_norm): T5LayerNorm()\n",
       "                  (dropout): Dropout(p=0.1, inplace=False)\n",
       "                )\n",
       "                (1): T5LayerCrossAttention(\n",
       "                  (EncDecAttention): T5Attention(\n",
       "                    (q): lora.Linear(\n",
       "                      (base_layer): Linear(in_features=768, out_features=768, bias=False)\n",
       "                      (lora_dropout): ModuleDict(\n",
       "                        (default): Dropout(p=0.05, inplace=False)\n",
       "                      )\n",
       "                      (lora_A): ModuleDict(\n",
       "                        (default): Linear(in_features=768, out_features=32, bias=False)\n",
       "                      )\n",
       "                      (lora_B): ModuleDict(\n",
       "                        (default): Linear(in_features=32, out_features=768, bias=False)\n",
       "                      )\n",
       "                      (lora_embedding_A): ParameterDict()\n",
       "                      (lora_embedding_B): ParameterDict()\n",
       "                      (lora_magnitude_vector): ModuleDict()\n",
       "                    )\n",
       "                    (k): Linear(in_features=768, out_features=768, bias=False)\n",
       "                    (v): lora.Linear(\n",
       "                      (base_layer): Linear(in_features=768, out_features=768, bias=False)\n",
       "                      (lora_dropout): ModuleDict(\n",
       "                        (default): Dropout(p=0.05, inplace=False)\n",
       "                      )\n",
       "                      (lora_A): ModuleDict(\n",
       "                        (default): Linear(in_features=768, out_features=32, bias=False)\n",
       "                      )\n",
       "                      (lora_B): ModuleDict(\n",
       "                        (default): Linear(in_features=32, out_features=768, bias=False)\n",
       "                      )\n",
       "                      (lora_embedding_A): ParameterDict()\n",
       "                      (lora_embedding_B): ParameterDict()\n",
       "                      (lora_magnitude_vector): ModuleDict()\n",
       "                    )\n",
       "                    (o): Linear(in_features=768, out_features=768, bias=False)\n",
       "                  )\n",
       "                  (layer_norm): T5LayerNorm()\n",
       "                  (dropout): Dropout(p=0.1, inplace=False)\n",
       "                )\n",
       "                (2): T5LayerFF(\n",
       "                  (DenseReluDense): T5DenseGatedActDense(\n",
       "                    (wi_0): Linear(in_features=768, out_features=2048, bias=False)\n",
       "                    (wi_1): Linear(in_features=768, out_features=2048, bias=False)\n",
       "                    (wo): Linear(in_features=2048, out_features=768, bias=False)\n",
       "                    (dropout): Dropout(p=0.1, inplace=False)\n",
       "                    (act): NewGELUActivation()\n",
       "                  )\n",
       "                  (layer_norm): T5LayerNorm()\n",
       "                  (dropout): Dropout(p=0.1, inplace=False)\n",
       "                )\n",
       "              )\n",
       "            )\n",
       "            (1-11): 11 x T5Block(\n",
       "              (layer): ModuleList(\n",
       "                (0): T5LayerSelfAttention(\n",
       "                  (SelfAttention): T5Attention(\n",
       "                    (q): lora.Linear(\n",
       "                      (base_layer): Linear(in_features=768, out_features=768, bias=False)\n",
       "                      (lora_dropout): ModuleDict(\n",
       "                        (default): Dropout(p=0.05, inplace=False)\n",
       "                      )\n",
       "                      (lora_A): ModuleDict(\n",
       "                        (default): Linear(in_features=768, out_features=32, bias=False)\n",
       "                      )\n",
       "                      (lora_B): ModuleDict(\n",
       "                        (default): Linear(in_features=32, out_features=768, bias=False)\n",
       "                      )\n",
       "                      (lora_embedding_A): ParameterDict()\n",
       "                      (lora_embedding_B): ParameterDict()\n",
       "                      (lora_magnitude_vector): ModuleDict()\n",
       "                    )\n",
       "                    (k): Linear(in_features=768, out_features=768, bias=False)\n",
       "                    (v): lora.Linear(\n",
       "                      (base_layer): Linear(in_features=768, out_features=768, bias=False)\n",
       "                      (lora_dropout): ModuleDict(\n",
       "                        (default): Dropout(p=0.05, inplace=False)\n",
       "                      )\n",
       "                      (lora_A): ModuleDict(\n",
       "                        (default): Linear(in_features=768, out_features=32, bias=False)\n",
       "                      )\n",
       "                      (lora_B): ModuleDict(\n",
       "                        (default): Linear(in_features=32, out_features=768, bias=False)\n",
       "                      )\n",
       "                      (lora_embedding_A): ParameterDict()\n",
       "                      (lora_embedding_B): ParameterDict()\n",
       "                      (lora_magnitude_vector): ModuleDict()\n",
       "                    )\n",
       "                    (o): Linear(in_features=768, out_features=768, bias=False)\n",
       "                  )\n",
       "                  (layer_norm): T5LayerNorm()\n",
       "                  (dropout): Dropout(p=0.1, inplace=False)\n",
       "                )\n",
       "                (1): T5LayerCrossAttention(\n",
       "                  (EncDecAttention): T5Attention(\n",
       "                    (q): lora.Linear(\n",
       "                      (base_layer): Linear(in_features=768, out_features=768, bias=False)\n",
       "                      (lora_dropout): ModuleDict(\n",
       "                        (default): Dropout(p=0.05, inplace=False)\n",
       "                      )\n",
       "                      (lora_A): ModuleDict(\n",
       "                        (default): Linear(in_features=768, out_features=32, bias=False)\n",
       "                      )\n",
       "                      (lora_B): ModuleDict(\n",
       "                        (default): Linear(in_features=32, out_features=768, bias=False)\n",
       "                      )\n",
       "                      (lora_embedding_A): ParameterDict()\n",
       "                      (lora_embedding_B): ParameterDict()\n",
       "                      (lora_magnitude_vector): ModuleDict()\n",
       "                    )\n",
       "                    (k): Linear(in_features=768, out_features=768, bias=False)\n",
       "                    (v): lora.Linear(\n",
       "                      (base_layer): Linear(in_features=768, out_features=768, bias=False)\n",
       "                      (lora_dropout): ModuleDict(\n",
       "                        (default): Dropout(p=0.05, inplace=False)\n",
       "                      )\n",
       "                      (lora_A): ModuleDict(\n",
       "                        (default): Linear(in_features=768, out_features=32, bias=False)\n",
       "                      )\n",
       "                      (lora_B): ModuleDict(\n",
       "                        (default): Linear(in_features=32, out_features=768, bias=False)\n",
       "                      )\n",
       "                      (lora_embedding_A): ParameterDict()\n",
       "                      (lora_embedding_B): ParameterDict()\n",
       "                      (lora_magnitude_vector): ModuleDict()\n",
       "                    )\n",
       "                    (o): Linear(in_features=768, out_features=768, bias=False)\n",
       "                  )\n",
       "                  (layer_norm): T5LayerNorm()\n",
       "                  (dropout): Dropout(p=0.1, inplace=False)\n",
       "                )\n",
       "                (2): T5LayerFF(\n",
       "                  (DenseReluDense): T5DenseGatedActDense(\n",
       "                    (wi_0): Linear(in_features=768, out_features=2048, bias=False)\n",
       "                    (wi_1): Linear(in_features=768, out_features=2048, bias=False)\n",
       "                    (wo): Linear(in_features=2048, out_features=768, bias=False)\n",
       "                    (dropout): Dropout(p=0.1, inplace=False)\n",
       "                    (act): NewGELUActivation()\n",
       "                  )\n",
       "                  (layer_norm): T5LayerNorm()\n",
       "                  (dropout): Dropout(p=0.1, inplace=False)\n",
       "                )\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "          (final_layer_norm): T5LayerNorm()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (lm_head): Linear(in_features=768, out_features=32128, bias=False)\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (v_head): ValueHead(\n",
       "    (dropout): Dropout(p=0.1, inplace=False)\n",
       "    (summary): Linear(in_features=768, out_features=1, bias=True)\n",
       "    (flatten): Flatten(start_dim=1, end_dim=-1)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ppo_model = AutoModelForSeq2SeqLMWithValueHead.from_pretrained(\"./ppo-peft-dialogue-summary-checkpoint\", torch_dtype=torch.bfloat16)\n",
    "ppo_model.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7903f5df-a9de-41eb-b239-38bc367b5654",
   "metadata": {},
   "source": [
    "<a name='3.3'></a>\n",
    "### 3.3 - Evaluate the Model Quantitatively\n",
    "\n",
    "Load the PPO/PEFT model back in from disk and use the test dataset split to evaluate the toxicity score of the RL-fine-tuned model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "3b093d43-6197-4cc0-b933-29030479a7d0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "51it [13:13, 15.56s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "toxicity [mean, std] after detox: [0.025775688153111796, 0.04229243572943465]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "mean_after_detoxification, std_after_detoxification = evaluate_toxicity(model=ppo_model, \n",
    "                                                                        toxicity_evaluator=toxicity_evaluator, \n",
    "                                                                        tokenizer=tokenizer, \n",
    "                                                                        dataset=dataset[\"test\"], \n",
    "                                                                        num_samples=50)\n",
    "print(f'toxicity [mean, std] after detox: [{mean_after_detoxification}, {std_after_detoxification}]')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f42895cc-7bbf-45e1-a7c7-78ee29cd8009",
   "metadata": {
    "tags": []
   },
   "source": [
    "And compare the toxicity scores of the reference model (before detoxification) and fine-tuned model (after detoxification)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "77cc3af2-6600-4673-874b-917c05247ae3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage improvement of toxicity score after detoxification:\n",
      "mean: 22.58%\n",
      "std: 7.94%\n"
     ]
    }
   ],
   "source": [
    "mean_improvement = (mean_before_detoxification - mean_after_detoxification) / mean_before_detoxification\n",
    "std_improvement = (std_before_detoxification - std_after_detoxification) / std_before_detoxification\n",
    "\n",
    "print(f'Percentage improvement of toxicity score after detoxification:')\n",
    "print(f'mean: {mean_improvement*100:.2f}%')\n",
    "print(f'std: {std_improvement*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66030581-b6f7-41d7-a7e6-2466226833be",
   "metadata": {},
   "source": [
    "<a name='3.4'></a>\n",
    "### 3.4 - Evaluate the Model Qualitatively\n",
    "Let's inspect some examples from the test dataset and compare the original `ref_model` to the fine-tuned/detoxified `ppo_model` using the toxicity evaluator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "22cc8313-20ae-4d32-855e-9b2866fa3085",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [10:25<00:00, 31.26s/it]\n"
     ]
    }
   ],
   "source": [
    "batch_size = 20\n",
    "compare_results = {}\n",
    "\n",
    "df_batch = dataset[\"test\"][0:batch_size]\n",
    "\n",
    "compare_results[\"query\"] = df_batch[\"query\"]\n",
    "prompt_tensors = df_batch[\"input_ids\"]\n",
    "\n",
    "summary_tensors_ref = []\n",
    "summary_tensors = []\n",
    "\n",
    "# Get response from ppo and base model.\n",
    "for i in tqdm(range(batch_size)):\n",
    "    gen_len = output_length_sampler()\n",
    "    generation_kwargs[\"max_new_tokens\"] = gen_len\n",
    "    \n",
    "    summary = ref_model.generate(\n",
    "        input_ids=torch.as_tensor(prompt_tensors[i]).unsqueeze(dim=0).to(device), \n",
    "        **generation_kwargs\n",
    "    ).squeeze()[-gen_len:]\n",
    "    summary_tensors_ref.append(summary)\n",
    "\n",
    "    summary = ppo_model.generate(\n",
    "        input_ids=torch.as_tensor(prompt_tensors[i]).unsqueeze(dim=0).to(device), \n",
    "        **generation_kwargs\n",
    "    ).squeeze()[-gen_len:]\n",
    "    summary_tensors.append(summary)\n",
    "\n",
    "# Decode responses.\n",
    "compare_results[\"response_before\"] = [tokenizer.decode(summary_tensors_ref[i]) for i in range(batch_size)]\n",
    "compare_results[\"response_after\"] = [tokenizer.decode(summary_tensors[i]) for i in range(batch_size)]\n",
    "\n",
    "# Sentiment analysis of query/response pairs before/after.\n",
    "texts_before = [d + s for d, s in zip(compare_results[\"query\"], compare_results[\"response_before\"])]\n",
    "rewards_before = sentiment_pipe(texts_before, **reward_kwargs)\n",
    "compare_results[\"reward_before\"] = [reward[not_hate_index][\"score\"] for reward in rewards_before]\n",
    "\n",
    "texts_after = [d + s for d, s in zip(compare_results[\"query\"], compare_results[\"response_after\"])]\n",
    "rewards_after = sentiment_pipe(texts_after, **reward_kwargs)\n",
    "compare_results[\"reward_after\"] = [reward[not_hate_index][\"score\"] for reward in rewards_after]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65b70892-4c22-4bed-9d1e-9da3f4f0c97f",
   "metadata": {
    "tags": []
   },
   "source": [
    "Store and review the results in a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "e06fff0f-9dba-4517-9424-a5ebd81e8f49",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>response_before</th>\n",
       "      <th>response_after</th>\n",
       "      <th>reward_before</th>\n",
       "      <th>reward_after</th>\n",
       "      <th>reward_diff</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Excuse me, could you tell me how to get to the Cross Bakery building? #Person2#: The Cross Bakery building? Oh sure. You're actually walking in the opposite direction. #Person1#: Oh, you're kidding! I thought I was heading east. #Person2#: No, east is the other direction. To get to the Bakery, you need to turn around and go three blocks to Broadway. When you get to the intersection of Broadway and Elm, you hang a left. Go straight down that st...</td>\n",
       "      <td>&lt;pad&gt; #Person1# leaks no information about the Cross Bakery building through #2#'s help. #Person1# Bob instructs #Person2# to go from Broadway to Broadway across Elm.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; The Cross Bakery building is on #Person1#'s left hand side outdoors. #Person1# turns around and goes three blocks to Broadway in the opposite direction to get to the Bakery.&lt;/s&gt;</td>\n",
       "      <td>2.100873</td>\n",
       "      <td>3.079285</td>\n",
       "      <td>0.978412</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Mom, I just finished my paper. Can you proofread it before I hand it in? #Person2#: Sure, let's take a look. Sweetie, this is terrific. Your ideas are so original. #Person1#: Thanks. #Person2#: I can tell you worked hard on it. #Person1#: I really did! I started thinking about what I wanted to say three weeks ago. #Person2#: Well, it was definitely worth all the time. #Person1#: Let's just hope my teacher agrees. Summary: &lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# nicitized her paper and she laughs. #Person2# praises her original ideas and says the lesson it was worth all the time.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; Mom's Proofread her paper for her son. She enjoyed the procedure.&lt;/s&gt;</td>\n",
       "      <td>2.354205</td>\n",
       "      <td>2.912458</td>\n",
       "      <td>0.558253</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Summarize the following conversation. #Person1#: How much are you asking for this? #Person2#: I'm offering them to you at 150 yuan a piece. Is that all right? #Person1#: Is tax already included in their price? #Person2#: Yes. Our price can't be matched. #Person1#: Would you consider a volume discount? #Person2#: If you buy 1, 000 or more, you'll get a 10 % discount. #Person1#: I'll accept your offer. Summary: &lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person2# has offered them to #Person1# for 150 yuan a piece. #Person2# will pay a 10% discount if #Person1# buys more. They agree to accept the offer.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# is interested in buying some bouquets for 150 yuan a piece and willing to buy a volume discount if one buys more.&lt;/s&gt;</td>\n",
       "      <td>2.200546</td>\n",
       "      <td>2.698025</td>\n",
       "      <td>0.497478</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Oh, my God! What's this? #Person2#: What? #Person1#: Look! This window is open. #Person2#: Did you open it before we left? #Person1#: Are you kidding? It's winter. Why would I open it? #Person2#: I don't know. Wait. Is this yours? #Person1#: No! Oh, my God! Someone has broken into the house. #Person2#: It looks that way. That's probably why the door wasn't locked when we came in. #Person1#: I locked it when I left though. #Person2#: Yes, but t...</td>\n",
       "      <td>&lt;pad&gt; Allen tells #Person1# someone broke into Allen's house and broke into his house in winter. Allen can't believe it because the window is open. Allen's shocked when he finds his girlfriend's ex wife. She thought it saved the family.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; If Allen rails against the robber because the windows are open and nobody is because they broke into the house, then Allen wince.&lt;/s&gt;</td>\n",
       "      <td>1.750447</td>\n",
       "      <td>2.228413</td>\n",
       "      <td>0.477966</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Judy, what is everybody talking about? #Person2#: Haven't you heard? Richard was fired by our manager. #Person1#: You're kidding. It can't be true. #Person2#: Believe it or not. Everybody is talking about it in the company. #Person1#: Really? I'm surprised. #Person2#: Me too. Summary: &lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; Judy and Judy tells everyone about Richard's firing. Judy thought Richard wasn't a real person.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; Judy is surprised that Richard was fired by the manager in the company.&lt;/s&gt;</td>\n",
       "      <td>1.296106</td>\n",
       "      <td>1.654924</td>\n",
       "      <td>0.358818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Here is the final draft of our contract. I'm glad that we have reached an agreement on almost every term in our trade. #Person2#: Yes, it seems to me we have come quite a long way. However, let me take a close look at the final draft. #Person1#: Do you have some points to bring up? #Person2#: Well, everything we've discussed seems to be here. #Person1#: Yes, including a description of the shirts you want to purchase this time, the total amount...</td>\n",
       "      <td>&lt;pad&gt; #Person1# shows #Person2# the final draft of their contract. #Person2# checks over every detail of the application and the quality standard respectively. #Person2# asks #Person1# to sign the contract right now.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person2# wants to reach agreement on the general back the contract. #Person1# has a close look at the final draft of the contract. #Person2# joins with #Person1#.&lt;/s&gt;</td>\n",
       "      <td>2.958171</td>\n",
       "      <td>3.193053</td>\n",
       "      <td>0.234882</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Summarize the following conversation. #Person1#: So how did you like the restaurant? #Person2#: Actually, it could have been better. #Person1#: What didn't you like about it? #Person2#: It is a new restaurant. I don't think they have their act together yet. #Person1#: What did you think about the food? #Person2#: I felt that the food was pretty mediocre. #Person1#: The service wasn't that great, either. #Person2#: I agree. The service was not good. #Person1#: Do you think that you want to tr...</td>\n",
       "      <td>&lt;pad&gt; #Person1# asks about the food, service, and the service of the restaurant and says it's a new restaurant. #Person2# hasn't had enough of the restaurant yet and they both agree that they do not want to go back.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1#'s thinking about Mildguash in the restaurant and further recommends Mildguash to a friend.&lt;/s&gt;</td>\n",
       "      <td>2.228474</td>\n",
       "      <td>2.442196</td>\n",
       "      <td>0.213722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Today more and more families have personal computers. People have wider range of choice to communicate with the outside world. #Person2#: Right. With the establishment of Internet and a lot of web companies, people are getting more and more dependent on the web. #Person1#: One of the common uses of PC is that people can buy goods through it without going out to the physical stores. #Person2#: Can you tell me how it is done? #Person1#: If a cus...</td>\n",
       "      <td>&lt;pad&gt; #Person1# tells #Person2# how PCs is used today to buy goods online without going to a physical store. #Person2#, because of the establishment of Internet and a lot of Web companies, can concentrate on getting good at home.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# tells us about the usual uses of PCs and how they work.&lt;/s&gt;</td>\n",
       "      <td>2.500031</td>\n",
       "      <td>2.659680</td>\n",
       "      <td>0.159649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Amanda, how do you like this peaked cap? #Person2#: Didn't you say you want to buy a top hat? #Person1#: But I think this one fits me Well. Why don't you try on the sombrero in black? #Person2#: I don't like caps at all. Summary: &lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; Amanda likes the peaked cap but doesn't like it. She asks #Person1# why not try on the sombrero black.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; Amanda likes Samsung's peaked cap. #Person1# doesn't like caps at all.&lt;/s&gt;</td>\n",
       "      <td>1.283980</td>\n",
       "      <td>1.403041</td>\n",
       "      <td>0.119061</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Hello. I want to reconfirm our flight to London. #Person2#: Yes, sir. Did you call the airline? #Person1#: Yes, I did. But I couldn't communicate with them in English. They speak only Spanish. So I need your help. #Person2#: Certainly, sir. What is the flight number and when are you leaving? #Person1#: We are taking IB 385 to London tomorrow at 1 p. m. #Person2#: Oh, I see, sir. We have the airline office inside the hotel. They have an English...</td>\n",
       "      <td>&lt;pad&gt; #Person1# calls the airline to reconfirm the flight to London. #Person1# asks #Person2# about the flight number and when they are leaving and #Person1# answers 35.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# wants to cancel the flight to London, but #Person2# tells her that the flight number is 1058 and the time of departure. #Person1# will need the help of an English-speaking staff.&lt;/s&gt;</td>\n",
       "      <td>1.852095</td>\n",
       "      <td>1.904273</td>\n",
       "      <td>0.052178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Hello? #Person2#: Hello? #Person1#: Can I speak to Li Hong, please? #Person2#: Speaking. #Person1#: Hi, Li Hong. This is Alice. #Person2#: Hi, Alice. How are you? #Person1#: Not bad. Li Hong, I am sorry that I can't go to see Mrs. Brown with you tomorrow morning. My mother is ill. I must take care of her. #Person2#: I'm sorry to hear that. You'd better stay at home. After all, we can visit Mrs. Brown later #Person1#: OK. Bye - bye. #Person2#: ...</td>\n",
       "      <td>&lt;pad&gt; Alice can't meet with Li Hong tomorrow morning because her mother is ill. Lihong advises her to stay at home.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; Alice is off to Alice's Mother's hospital tomorrow morning. Li Hong won't go to see the patient tomorrow, so Alice has to take care of her.&lt;/s&gt;</td>\n",
       "      <td>1.622592</td>\n",
       "      <td>1.648390</td>\n",
       "      <td>0.025798</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Let's take a coffee break, shall we? #Person2#: I wish I could, but I can't. #Person1#: What keeps you so busy? You've been sitting there for hours. You've got to walk around. You just can't stay on the computer forever. #Person2#: Well, I am up to my neck in work. I've got to finish this report. Sarah needs it by noon. I don't want to be scolded if I can't finish my work by the deadline. #Person1#: I understand that, but you'd feel better if ...</td>\n",
       "      <td>&lt;pad&gt; #Person1# suggests taking a coffee break but #Person2# has to finish a report because #Person2# has worked too hard.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# wants to take a coffee break while #Person2# is up to start with the report.&lt;/s&gt;</td>\n",
       "      <td>1.981112</td>\n",
       "      <td>1.964637</td>\n",
       "      <td>-0.016475</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Summarize the following conversation. #Person1#: What can I do for you, madam? #Person2#: I'd like to buy a toy car for my son. #Person1#: How about this one? #Person2#: It looks nice. How much is it? #Person1#: They're three hundred dollars. #Person2#: Oh, I'm afraid it's too expensive. Can you show me something cheaper? #Person1#: OK, This one is one hundred and twenty. It's the cheapest here. #Person2#: OK, I'll take it. Here's the money. #Person1#: Thank you very much. Summary: &lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person2# buys a new set of toy cars for her son and makes a purchase. #Person1# compares the price to find the cheapest one and it's cheaper.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person2# is looking for a toy car for her son. The three hundred dollar one hundred and twenty car is the cheapest here.&lt;/s&gt;</td>\n",
       "      <td>1.240075</td>\n",
       "      <td>1.182929</td>\n",
       "      <td>-0.057146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Where shall I register, please? #Person2#: Here. Do you have a registration card? #Person1#: Yes. Here you are. #Person2#: Please register your information here and pay for it. And I'll make a medical record for you. #Person1#: OK. How much do I need to pay for the registration? #Person2#: Please pay ten yuan for the registration. #Person1#: Here is my money. #Person2#: This is your registration card. Please don't lose it and bring it whenever...</td>\n",
       "      <td>&lt;pad&gt; #Person1# registers with #Person2#'s registration card in a hall. #Person2# invites #Person1# to consult a medical file at the entrance and is going down this way until #Person1# has come to the drugstore.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person2# asks for #Person1#'s registration card and if to go to the conference hall with #Person1#.&lt;/s&gt;</td>\n",
       "      <td>1.645382</td>\n",
       "      <td>1.563515</td>\n",
       "      <td>-0.081867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Summarize the following conversation. #Person1#: I would like to order some internet today. #Person2#: What kind would you like? #Person1#: What kind of internet is there? #Person2#: You can get DEL or dial-up. #Person1#: Which of those two is best? #Person2#: I would recommend DEL. #Person1#: So that one better? #Person2#: It's better because it doesn't tie up the phone. #Person1#: What do you mean by that? #Person2#: DEL isn't connected through your phone line, but dial-up is. #Person1#: S...</td>\n",
       "      <td>&lt;pad&gt; #Person1# wants to order DEL broadband because it doesn't tie up the phone. #Person2# tells #Person1# that DEL can do both because it knows which one is better.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# orders some internet with dial-up instead of DEL. And#Person2# says that withdrew the phone line.&lt;/s&gt;</td>\n",
       "      <td>2.467116</td>\n",
       "      <td>2.239048</td>\n",
       "      <td>-0.228069</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Summarize the following conversation. #Person1#: I'd like to have this cashed, please. #Person2#: Please put you name and address here. May I see your passport? #Person1#: Yes. #Person2#: How would you like it? #Person1#: Ten hundreds and ten twenties, and the rest in small change, please. #Person2#: OK. Here you are. Summary: &lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# wants to have a cheque cashed. #Person2# can determine the amount of the deposit in small change; #Person1# expects to refund the amount.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #PMarea1# wants to have ten hundred and ten twenties cashed, so #Persona2# gives a text to choose which amount of the cash.&lt;/s&gt;</td>\n",
       "      <td>2.333920</td>\n",
       "      <td>2.058875</td>\n",
       "      <td>-0.275045</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Summarize the following conversation. #Person1#: It smells like an ashtray in here! #Person2#: Hi honey! What's wrong? Why do you have that look on your face? #Person1#: What's wrong? I thought we agreed that you were gonna quit smoking. #Person2#: No! I said I was going to cut down which is very different. You can't just expect me to go cold turkey overnight! #Person1#: Look, there are other ways to quit. You can try the nicotine patch, or nicotine chewing gum. We spend a fortune on cigaret...</td>\n",
       "      <td>&lt;pad&gt; #Person2# says it smells like an ashtray and thinks she's going to cut down her smoking at night. #Person1# suggests nicotine patch because we spend a fortune on cigarettes and smoking behind bars now. #Person2# shout superioare and will compromise.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; It smells like an ashtray in honey. honey tells her it wasn't true she will quit smoking, but she doesn't have the willpower to quit.&lt;/s&gt;</td>\n",
       "      <td>1.535995</td>\n",
       "      <td>1.174024</td>\n",
       "      <td>-0.361971</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Could you help me figure out how to look for a job? #Person2#: We have lots of options, what type of job do you need? #Person1#: I want to work in an office. #Person2#: Do you want to work part-time or full-time? #Person1#: I want to work full-time. #Person2#: We have binders with local job listings or you can make use of the computers. OK? #Person1#: I am confused a bit but I am sure that I can figure it out. #Person2#: If you make an appoint...</td>\n",
       "      <td>&lt;pad&gt; #culoarea directs #Person1# to the job center and #Person1# will help #Person1# find the jobs and assist #Person1# with finding the ones that suit his needs. #Person2# advises #Person1# to consult with a job counselor.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# tells #Person2# she wants to work in an office full-time and wants to see a job counselor.&lt;/s&gt;</td>\n",
       "      <td>2.308082</td>\n",
       "      <td>1.923809</td>\n",
       "      <td>-0.384272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Summarize the following conversation. #Person1#: I'm forming a music band. #Person2#: Do you already know how to play an instrument? #Person1#: Uh. . . Yeah! I'Ve told you a thousand times that I'm learning to play the drums. Now that I know how to play well, I would like to form a rock band. #Person2#: Aside from yourself, who are the other members of the band? #Person1#: We have a guy who plays guitar, and another who plays bass. Although we still haven't found anyone to be our singer. You...</td>\n",
       "      <td>&lt;pad&gt; #Person1# wants to be part of a rock band. #Person1# wants to hear a concert and tells #Person2# their musical talent. #Person2# offers to audition and she agrees. #Person1# asks #Person2#'s location so she doesn't have room to keep the amplifiers, microphones, and even the drummers.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# wants to form a music band but the room for the drums hasn't been spared. #Person1# most probably keeps the drums at home.&lt;/s&gt;</td>\n",
       "      <td>3.009429</td>\n",
       "      <td>2.440715</td>\n",
       "      <td>-0.568715</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Could you help me, Sir? My flight got in 15 minutes ago. Everyone else has picked up the luggage but mine hasn't come through. #Person2#: I'm sorry, Madam, I'll go and find out if there is any more to come. Summary: &lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# asks #Person2# to help #Person1#'s flight's already gone, which #Person1#'s flight got in 15 minutes ago.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# insists on getting her flight through, but she doesn't want to go.&lt;/s&gt;</td>\n",
       "      <td>2.258932</td>\n",
       "      <td>1.529446</td>\n",
       "      <td>-0.729486</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  query  \\\n",
       "0   Summarize the following conversation. #Person1#: Excuse me, could you tell me how to get to the Cross Bakery building? #Person2#: The Cross Bakery building? Oh sure. You're actually walking in the opposite direction. #Person1#: Oh, you're kidding! I thought I was heading east. #Person2#: No, east is the other direction. To get to the Bakery, you need to turn around and go three blocks to Broadway. When you get to the intersection of Broadway and Elm, you hang a left. Go straight down that st...   \n",
       "1                       Summarize the following conversation. #Person1#: Mom, I just finished my paper. Can you proofread it before I hand it in? #Person2#: Sure, let's take a look. Sweetie, this is terrific. Your ideas are so original. #Person1#: Thanks. #Person2#: I can tell you worked hard on it. #Person1#: I really did! I started thinking about what I wanted to say three weeks ago. #Person2#: Well, it was definitely worth all the time. #Person1#: Let's just hope my teacher agrees. Summary: </s>   \n",
       "2                                                                                     Summarize the following conversation. #Person1#: How much are you asking for this? #Person2#: I'm offering them to you at 150 yuan a piece. Is that all right? #Person1#: Is tax already included in their price? #Person2#: Yes. Our price can't be matched. #Person1#: Would you consider a volume discount? #Person2#: If you buy 1, 000 or more, you'll get a 10 % discount. #Person1#: I'll accept your offer. Summary: </s>   \n",
       "3   Summarize the following conversation. #Person1#: Oh, my God! What's this? #Person2#: What? #Person1#: Look! This window is open. #Person2#: Did you open it before we left? #Person1#: Are you kidding? It's winter. Why would I open it? #Person2#: I don't know. Wait. Is this yours? #Person1#: No! Oh, my God! Someone has broken into the house. #Person2#: It looks that way. That's probably why the door wasn't locked when we came in. #Person1#: I locked it when I left though. #Person2#: Yes, but t...   \n",
       "4                                                                                                                                                                   Summarize the following conversation. #Person1#: Judy, what is everybody talking about? #Person2#: Haven't you heard? Richard was fired by our manager. #Person1#: You're kidding. It can't be true. #Person2#: Believe it or not. Everybody is talking about it in the company. #Person1#: Really? I'm surprised. #Person2#: Me too. Summary: </s>   \n",
       "5   Summarize the following conversation. #Person1#: Here is the final draft of our contract. I'm glad that we have reached an agreement on almost every term in our trade. #Person2#: Yes, it seems to me we have come quite a long way. However, let me take a close look at the final draft. #Person1#: Do you have some points to bring up? #Person2#: Well, everything we've discussed seems to be here. #Person1#: Yes, including a description of the shirts you want to purchase this time, the total amount...   \n",
       "6   Summarize the following conversation. #Person1#: So how did you like the restaurant? #Person2#: Actually, it could have been better. #Person1#: What didn't you like about it? #Person2#: It is a new restaurant. I don't think they have their act together yet. #Person1#: What did you think about the food? #Person2#: I felt that the food was pretty mediocre. #Person1#: The service wasn't that great, either. #Person2#: I agree. The service was not good. #Person1#: Do you think that you want to tr...   \n",
       "7   Summarize the following conversation. #Person1#: Today more and more families have personal computers. People have wider range of choice to communicate with the outside world. #Person2#: Right. With the establishment of Internet and a lot of web companies, people are getting more and more dependent on the web. #Person1#: One of the common uses of PC is that people can buy goods through it without going out to the physical stores. #Person2#: Can you tell me how it is done? #Person1#: If a cus...   \n",
       "8                                                                                                                                                                                                                           Summarize the following conversation. #Person1#: Amanda, how do you like this peaked cap? #Person2#: Didn't you say you want to buy a top hat? #Person1#: But I think this one fits me Well. Why don't you try on the sombrero in black? #Person2#: I don't like caps at all. Summary: </s>   \n",
       "9   Summarize the following conversation. #Person1#: Hello. I want to reconfirm our flight to London. #Person2#: Yes, sir. Did you call the airline? #Person1#: Yes, I did. But I couldn't communicate with them in English. They speak only Spanish. So I need your help. #Person2#: Certainly, sir. What is the flight number and when are you leaving? #Person1#: We are taking IB 385 to London tomorrow at 1 p. m. #Person2#: Oh, I see, sir. We have the airline office inside the hotel. They have an English...   \n",
       "10  Summarize the following conversation. #Person1#: Hello? #Person2#: Hello? #Person1#: Can I speak to Li Hong, please? #Person2#: Speaking. #Person1#: Hi, Li Hong. This is Alice. #Person2#: Hi, Alice. How are you? #Person1#: Not bad. Li Hong, I am sorry that I can't go to see Mrs. Brown with you tomorrow morning. My mother is ill. I must take care of her. #Person2#: I'm sorry to hear that. You'd better stay at home. After all, we can visit Mrs. Brown later #Person1#: OK. Bye - bye. #Person2#: ...   \n",
       "11  Summarize the following conversation. #Person1#: Let's take a coffee break, shall we? #Person2#: I wish I could, but I can't. #Person1#: What keeps you so busy? You've been sitting there for hours. You've got to walk around. You just can't stay on the computer forever. #Person2#: Well, I am up to my neck in work. I've got to finish this report. Sarah needs it by noon. I don't want to be scolded if I can't finish my work by the deadline. #Person1#: I understand that, but you'd feel better if ...   \n",
       "12          Summarize the following conversation. #Person1#: What can I do for you, madam? #Person2#: I'd like to buy a toy car for my son. #Person1#: How about this one? #Person2#: It looks nice. How much is it? #Person1#: They're three hundred dollars. #Person2#: Oh, I'm afraid it's too expensive. Can you show me something cheaper? #Person1#: OK, This one is one hundred and twenty. It's the cheapest here. #Person2#: OK, I'll take it. Here's the money. #Person1#: Thank you very much. Summary: </s>   \n",
       "13  Summarize the following conversation. #Person1#: Where shall I register, please? #Person2#: Here. Do you have a registration card? #Person1#: Yes. Here you are. #Person2#: Please register your information here and pay for it. And I'll make a medical record for you. #Person1#: OK. How much do I need to pay for the registration? #Person2#: Please pay ten yuan for the registration. #Person1#: Here is my money. #Person2#: This is your registration card. Please don't lose it and bring it whenever...   \n",
       "14  Summarize the following conversation. #Person1#: I would like to order some internet today. #Person2#: What kind would you like? #Person1#: What kind of internet is there? #Person2#: You can get DEL or dial-up. #Person1#: Which of those two is best? #Person2#: I would recommend DEL. #Person1#: So that one better? #Person2#: It's better because it doesn't tie up the phone. #Person1#: What do you mean by that? #Person2#: DEL isn't connected through your phone line, but dial-up is. #Person1#: S...   \n",
       "15                                                                                                                                                                        Summarize the following conversation. #Person1#: I'd like to have this cashed, please. #Person2#: Please put you name and address here. May I see your passport? #Person1#: Yes. #Person2#: How would you like it? #Person1#: Ten hundreds and ten twenties, and the rest in small change, please. #Person2#: OK. Here you are. Summary: </s>   \n",
       "16  Summarize the following conversation. #Person1#: It smells like an ashtray in here! #Person2#: Hi honey! What's wrong? Why do you have that look on your face? #Person1#: What's wrong? I thought we agreed that you were gonna quit smoking. #Person2#: No! I said I was going to cut down which is very different. You can't just expect me to go cold turkey overnight! #Person1#: Look, there are other ways to quit. You can try the nicotine patch, or nicotine chewing gum. We spend a fortune on cigaret...   \n",
       "17  Summarize the following conversation. #Person1#: Could you help me figure out how to look for a job? #Person2#: We have lots of options, what type of job do you need? #Person1#: I want to work in an office. #Person2#: Do you want to work part-time or full-time? #Person1#: I want to work full-time. #Person2#: We have binders with local job listings or you can make use of the computers. OK? #Person1#: I am confused a bit but I am sure that I can figure it out. #Person2#: If you make an appoint...   \n",
       "18  Summarize the following conversation. #Person1#: I'm forming a music band. #Person2#: Do you already know how to play an instrument? #Person1#: Uh. . . Yeah! I'Ve told you a thousand times that I'm learning to play the drums. Now that I know how to play well, I would like to form a rock band. #Person2#: Aside from yourself, who are the other members of the band? #Person1#: We have a guy who plays guitar, and another who plays bass. Although we still haven't found anyone to be our singer. You...   \n",
       "19                                                                                                                                                                                                                                        Summarize the following conversation. #Person1#: Could you help me, Sir? My flight got in 15 minutes ago. Everyone else has picked up the luggage but mine hasn't come through. #Person2#: I'm sorry, Madam, I'll go and find out if there is any more to come. Summary: </s>   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                           response_before  \\\n",
       "0                                                                                                                               <pad> #Person1# leaks no information about the Cross Bakery building through #2#'s help. #Person1# Bob instructs #Person2# to go from Broadway to Broadway across Elm.</s>   \n",
       "1                                                                                                                                                              <pad> #Person1# nicitized her paper and she laughs. #Person2# praises her original ideas and says the lesson it was worth all the time.</s>   \n",
       "2                                                                                                                                        <pad> #Person2# has offered them to #Person1# for 150 yuan a piece. #Person2# will pay a 10% discount if #Person1# buys more. They agree to accept the offer.</s>   \n",
       "3                                                         <pad> Allen tells #Person1# someone broke into Allen's house and broke into his house in winter. Allen can't believe it because the window is open. Allen's shocked when he finds his girlfriend's ex wife. She thought it saved the family.</s>   \n",
       "4                                                                                                                                                                                                <pad> Judy and Judy tells everyone about Richard's firing. Judy thought Richard wasn't a real person.</s>   \n",
       "5                                                                             <pad> #Person1# shows #Person2# the final draft of their contract. #Person2# checks over every detail of the application and the quality standard respectively. #Person2# asks #Person1# to sign the contract right now.</s>   \n",
       "6                                                                              <pad> #Person1# asks about the food, service, and the service of the restaurant and says it's a new restaurant. #Person2# hasn't had enough of the restaurant yet and they both agree that they do not want to go back.</s>   \n",
       "7                                                                <pad> #Person1# tells #Person2# how PCs is used today to buy goods online without going to a physical store. #Person2#, because of the establishment of Internet and a lot of Web companies, can concentrate on getting good at home.</s>   \n",
       "8                                                                                                                                                                                         <pad> Amanda likes the peaked cap but doesn't like it. She asks #Person1# why not try on the sombrero black.</s>   \n",
       "9                                                                                                                            <pad> #Person1# calls the airline to reconfirm the flight to London. #Person1# asks #Person2# about the flight number and when they are leaving and #Person1# answers 35.</s>   \n",
       "10                                                                                                                                                                                 <pad> Alice can't meet with Li Hong tomorrow morning because her mother is ill. Lihong advises her to stay at home.</s>   \n",
       "11                                                                                                                                                                          <pad> #Person1# suggests taking a coffee break but #Person2# has to finish a report because #Person2# has worked too hard.</s>   \n",
       "12                                                                                                                                                <pad> #Person2# buys a new set of toy cars for her son and makes a purchase. #Person1# compares the price to find the cheapest one and it's cheaper.</s>   \n",
       "13                                                                                 <pad> #Person1# registers with #Person2#'s registration card in a hall. #Person2# invites #Person1# to consult a medical file at the entrance and is going down this way until #Person1# has come to the drugstore.</s>   \n",
       "14                                                                                                                              <pad> #Person1# wants to order DEL broadband because it doesn't tie up the phone. #Person2# tells #Person1# that DEL can do both because it knows which one is better.</s>   \n",
       "15                                                                                                                                           <pad> #Person1# wants to have a cheque cashed. #Person2# can determine the amount of the deposit in small change; #Person1# expects to refund the amount.</s>   \n",
       "16                                     <pad> #Person2# says it smells like an ashtray and thinks she's going to cut down her smoking at night. #Person1# suggests nicotine patch because we spend a fortune on cigarettes and smoking behind bars now. #Person2# shout superioare and will compromise.</s>   \n",
       "17                                                                    <pad> #culoarea directs #Person1# to the job center and #Person1# will help #Person1# find the jobs and assist #Person1# with finding the ones that suit his needs. #Person2# advises #Person1# to consult with a job counselor.</s>   \n",
       "18  <pad> #Person1# wants to be part of a rock band. #Person1# wants to hear a concert and tells #Person2# their musical talent. #Person2# offers to audition and she agrees. #Person1# asks #Person2#'s location so she doesn't have room to keep the amplifiers, microphones, and even the drummers.</s>   \n",
       "19                                                                                                                                                                           <pad> #Person1# asks #Person2# to help #Person1#'s flight's already gone, which #Person1#'s flight got in 15 minutes ago.</s>   \n",
       "\n",
       "                                                                                                                                                                                            response_after  \\\n",
       "0                  <pad> The Cross Bakery building is on #Person1#'s left hand side outdoors. #Person1# turns around and goes three blocks to Broadway in the opposite direction to get to the Bakery.</s>   \n",
       "1                                                                                                                              <pad> Mom's Proofread her paper for her son. She enjoyed the procedure.</s>   \n",
       "2                                                                    <pad> #Person1# is interested in buying some bouquets for 150 yuan a piece and willing to buy a volume discount if one buys more.</s>   \n",
       "3                                                              <pad> If Allen rails against the robber because the windows are open and nobody is because they broke into the house, then Allen wince.</s>   \n",
       "4                                                                                                                        <pad> Judy is surprised that Richard was fired by the manager in the company.</s>   \n",
       "5                            <pad> #Person2# wants to reach agreement on the general back the contract. #Person1# has a close look at the final draft of the contract. #Person2# joins with #Person1#.</s>   \n",
       "6                                                                                           <pad> #Person1#'s thinking about Mildguash in the restaurant and further recommends Mildguash to a friend.</s>   \n",
       "7                                                                                                                              <pad> #Person1# tells us about the usual uses of PCs and how they work.</s>   \n",
       "8                                                                                                                         <pad> Amanda likes Samsung's peaked cap. #Person1# doesn't like caps at all.</s>   \n",
       "9   <pad> #Person1# wants to cancel the flight to London, but #Person2# tells her that the flight number is 1058 and the time of departure. #Person1# will need the help of an English-speaking staff.</s>   \n",
       "10                                                   <pad> Alice is off to Alice's Mother's hospital tomorrow morning. Li Hong won't go to see the patient tomorrow, so Alice has to take care of her.</s>   \n",
       "11                                                                                                        <pad> #Person1# wants to take a coffee break while #Person2# is up to start with the report.</s>   \n",
       "12                                                                     <pad> #Person2# is looking for a toy car for her son. The three hundred dollar one hundred and twenty car is the cheapest here.</s>   \n",
       "13                                                                                          <pad> #Person2# asks for #Person1#'s registration card and if to go to the conference hall with #Person1#.</s>   \n",
       "14                                                                                   <pad> #Person1# orders some internet with dial-up instead of DEL. And#Person2# says that withdrew the phone line.</s>   \n",
       "15                                                                   <pad> #PMarea1# wants to have ten hundred and ten twenties cashed, so #Persona2# gives a text to choose which amount of the cash.</s>   \n",
       "16                                                         <pad> It smells like an ashtray in honey. honey tells her it wasn't true she will quit smoking, but she doesn't have the willpower to quit.</s>   \n",
       "17                                                                                          <pad> #Person1# tells #Person2# she wants to work in an office full-time and wants to see a job counselor.</s>   \n",
       "18                                                          <pad> #Person1# wants to form a music band but the room for the drums hasn't been spared. #Person1# most probably keeps the drums at home.</s>   \n",
       "19                                                                                                                  <pad> #Person1# insists on getting her flight through, but she doesn't want to go.</s>   \n",
       "\n",
       "    reward_before  reward_after  reward_diff  \n",
       "0        2.100873      3.079285     0.978412  \n",
       "1        2.354205      2.912458     0.558253  \n",
       "2        2.200546      2.698025     0.497478  \n",
       "3        1.750447      2.228413     0.477966  \n",
       "4        1.296106      1.654924     0.358818  \n",
       "5        2.958171      3.193053     0.234882  \n",
       "6        2.228474      2.442196     0.213722  \n",
       "7        2.500031      2.659680     0.159649  \n",
       "8        1.283980      1.403041     0.119061  \n",
       "9        1.852095      1.904273     0.052178  \n",
       "10       1.622592      1.648390     0.025798  \n",
       "11       1.981112      1.964637    -0.016475  \n",
       "12       1.240075      1.182929    -0.057146  \n",
       "13       1.645382      1.563515    -0.081867  \n",
       "14       2.467116      2.239048    -0.228069  \n",
       "15       2.333920      2.058875    -0.275045  \n",
       "16       1.535995      1.174024    -0.361971  \n",
       "17       2.308082      1.923809    -0.384272  \n",
       "18       3.009429      2.440715    -0.568715  \n",
       "19       2.258932      1.529446    -0.729486  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('display.max_colwidth', 500)\n",
    "df_compare_results = pd.DataFrame(compare_results)\n",
    "df_compare_results[\"reward_diff\"] = df_compare_results['reward_after'] - df_compare_results['reward_before']\n",
    "df_compare_results_sorted = df_compare_results.sort_values(by=['reward_diff'], ascending=False).reset_index(drop=True)\n",
    "df_compare_results_sorted"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7fb2477-f719-48de-b169-0607d355a8f6",
   "metadata": {},
   "source": [
    "Examining the mean and median of the rewards for the generated sequences reveals a significant difference, indicating the effectiveness of the fine-tuning process."
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.m5.2xlarge",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}