TLM-Demo-Notebook/TLM-Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6e4f227a-db80-4d19-a907-5035e25754f6",
   "metadata": {},
   "source": [
    "# Trustworthy Language Model (TLM) Demo"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a468ee8-cba2-4c0e-8991-077d7d438a99",
   "metadata": {},
   "source": [
    "[![Run in Google Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cleanlab/cleanlab-tools/blob/main/TLM-Demo-Notebook/TLM-Demo.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "394348f6-b428-4607-a838-6f3368dc5d0e",
   "metadata": {},
   "source": [
    "### Imports and Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d8d5d4b0-c6c9-453c-ac58-19aa85f56837",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "import pandas as pd\n",
    "pd.set_option('display.max_colwidth', 150)\n",
    "from cleanlab_studio import Studio\n",
    "from openai import OpenAI\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# Read API Keys and initalize clients\n",
    "STUDIO_API_KEY = open('studio-key.txt', 'r').read().strip()\n",
    "OPENAI_API_KEY = open('openai-key.txt', 'r').read().strip()\n",
    "studio = Studio(api_key=STUDIO_API_KEY)\n",
    "client = OpenAI(api_key=OPENAI_API_KEY)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1d56768-7e84-4706-8b6a-4088583c0340",
   "metadata": {},
   "source": [
    "### Helpers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d80aab65-8f1f-4ebc-bf9c-0b6c678b32de",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper to prompt GPT-4o\n",
    "class ChatGPT:\n",
    "    def __init__(self, client):\n",
    "        self.client = client\n",
    "    def prompt(self, prompt):\n",
    "        completion = self.client.chat.completions.create(\n",
    "            model=\"gpt-4o\",\n",
    "            messages=[{\"role\": \"user\", \"content\": prompt}]\n",
    "        )\n",
    "        return completion.choices[0].message.content\n",
    "\n",
    "# Helper to selectively print lines from customer conversation\n",
    "def print_conversation(conversation, indices=None):\n",
    "    lines = conversation.split('\\n')\n",
    "    conversation_entries = []\n",
    "    prefix = \"\\n.\\n.\\n.\\n\"\n",
    "    for line in lines:\n",
    "        if line.startswith('Customer:') or line.startswith('Agent:'):\n",
    "            conversation_entries.append(line.strip())\n",
    "    if indices is None:\n",
    "        indices = list(range(len(conversation_entries)))\n",
    "        prefix = \"\\n\\n\"\n",
    "    print(prefix.join([conversation_entries[i] for i in indices if i < len(conversation_entries)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1707d06d-677c-4528-a8d5-1cc9d7d8edb3",
   "metadata": {},
   "source": [
    "## What is TLM?\n",
    "- TLM provides a trustworthiness score and explanation to any LLM generated response\n",
    "- TLM enables GenAI powered applications to enter production with speed and reliability"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b538225-68b1-4d6f-9c1f-34aaa2ef2cc0",
   "metadata": {},
   "source": [
    "### ChatGPT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "21c1a570-67cb-4b51-ace1-a99948ba9834",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Response: August\n"
     ]
    }
   ],
   "source": [
    "chatgpt = ChatGPT(client)\n",
    "response = chatgpt.prompt(\"What is the third month of the year, alphabetically? Respond with just the month.\")\n",
    "print(\"Response:\",response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d593ec58-a051-4f12-a16d-7b8b51a28b50",
   "metadata": {},
   "source": [
    "### TLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f6824052-b1a3-4e13-b3ff-96babdeee063",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Response: March\n",
      "Trustworthiness Score: 0.47499641722571456\n",
      "Explanation: This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either): \n",
      "December.\n"
     ]
    }
   ],
   "source": [
    "tlm = studio.TLM(options={\"log\": [\"explanation\"]})\n",
    "output = tlm.prompt(\"What is the third month of the year, alphabetically? Respond with just the month.\")\n",
    "print(f'Response: {output[\"response\"]}')\n",
    "print(f'Trustworthiness Score: {output[\"trustworthiness_score\"]}')\n",
    "print(f'Explanation: {output[\"log\"][\"explanation\"]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97fd5146-c4ae-44ed-8f1d-2a59c28a5615",
   "metadata": {},
   "source": [
    "## Use Cases\n",
    "\n",
    "1. Compute Trustworthiness Scores for any LLM\n",
    "   - Turn your own LLM into a TLM to flag hallucinations\n",
    "2. Question Answering / RAG\n",
    "   - Answer specific questions about your knowledge base. \n",
    "3. Data Extraction\n",
    "   - Extract relevant information from your knowledge base. \n",
    "4. Classification\n",
    "   - Classify and categorize your knowledge base. \n",
    "5. Accuracy Improvement\n",
    "   - Increase the accuracy of your LLM responses."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dfa0e6d-c9c9-4f9c-b071-3bb2c726c5d4",
   "metadata": {},
   "source": [
    "### Initialize and View Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "535a62cf-1f14-4ed8-93e6-d4f08a4979ff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RU91</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\\n\\nCustomer: Hi Rachel, I recently purchase...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>XL37</td>\n",
       "      <td>Customer: Hello, I have an issue with my recent purchase from BrownBox.\\n\\nAgent: Good afternoon! Thank you for contacting BrownBox customer suppo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OJ95</td>\n",
       "      <td>Agent: Hello, thank you for calling BrownBox Customer Support. My name is Emily, how may I assist you today?\\n\\nCustomer: Hi, I recently purchased...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>YR96</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Lisa. How may I assist you today?\\n\\nCustomer: Hi, Lisa. I need help with a ret...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>XZ39</td>\n",
       "      <td>Customer: Hello, I am calling to inquire about my return and exchange order.\\n\\nAgent: Good afternoon, thank you for contacting BrownBox customer ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     id  \\\n",
       "0  RU91   \n",
       "1  XL37   \n",
       "2  OJ95   \n",
       "3  YR96   \n",
       "4  XZ39   \n",
       "\n",
       "                                                                                                                                            conversation  \n",
       "0  Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\\n\\nCustomer: Hi Rachel, I recently purchase...  \n",
       "1  Customer: Hello, I have an issue with my recent purchase from BrownBox.\\n\\nAgent: Good afternoon! Thank you for contacting BrownBox customer suppo...  \n",
       "2  Agent: Hello, thank you for calling BrownBox Customer Support. My name is Emily, how may I assist you today?\\n\\nCustomer: Hi, I recently purchased...  \n",
       "3  Agent: Thank you for calling BrownBox Customer Support. My name is Lisa. How may I assist you today?\\n\\nCustomer: Hi, Lisa. I need help with a ret...  \n",
       "4  Customer: Hello, I am calling to inquire about my return and exchange order.\\n\\nAgent: Good afternoon, thank you for contacting BrownBox customer ...  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Initialize TLM\n",
    "tlm = studio.TLM(quality_preset = \"low\", options={\"log\": [\"explanation\"]})\n",
    "\n",
    "# Read in data\n",
    "df = pd.read_csv(\"customer-service-conversation.csv\")\n",
    "df_original = df.copy()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "b4a5c79d-78de-40ec-9216-37a9814bbe5b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\n",
      "\n",
      "Customer: Hi Rachel, I recently purchased a water purifier from your website, and I have a question about the warranty. Can you help me with that?\n",
      "\n",
      "Agent: Of course, I'd be happy to help. Can you please provide me with your order number and the name of the water purifier you purchased?\n",
      "\n",
      "Customer: My order number is BB12345678, and the water purifier is the AquaMax Pro.\n",
      "\n",
      "Agent: Thank you for the information. Now, can you please let me know what your question is regarding the warranty?\n",
      "\n",
      "Customer: I received the water purifier a few days ago, and I'm not sure when the warranty period starts. Can you tell me the start date of the warranty?\n",
      "\n",
      "Agent: Sure, I can help you with that. The warranty for your AquaMax Pro water purifier starts from the date of delivery. So, the warranty period for your water purifier started on the day it was delivered to you.\n",
      "\n",
      "Customer: Okay, that's good to know. And how long is the warranty period for the water purifier?\n",
      "\n",
      "Agent: The warranty period for the AquaMax Pro water purifier is one year from the date of delivery. During this period, if there are any defects or issues with the product, you can get it repaired or replaced free of charge.\n",
      "\n",
      "Customer: Alright, thank you for the information, Rachel. I appreciate your help.\n",
      "\n",
      "Agent: You're welcome. Is there anything else you need assistance with?\n",
      "\n",
      "Customer: No, that's all. Thank you again for your help.\n",
      "\n",
      "Agent: It was my pleasure assisting you. If you have any further questions or concerns, please feel free to reach out to us. Have a great day!\n"
     ]
    }
   ],
   "source": [
    "print(df.iloc[0].conversation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54aef40d-05f4-4016-81f0-b4b1db310f9d",
   "metadata": {},
   "source": [
    "## Use Case: Compute Trustworthiness Scores for any LLM\n",
    "Use TLM to get a trustworthiness score for any (prompt, response) pair!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30ce5d3b-21cc-403f-8183-d40db0c4ebf0",
   "metadata": {},
   "source": [
    "#### Single Conversation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "34e0d94f-40c5-4550-90fa-efd3ec83d380",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\n",
      "\n",
      "Customer: Hi Sarah, I placed an order for a Smart Watch, and I want to confirm if it has been processed.\n",
      "\n",
      "Agent: Sure, I can help you with that. May I know your name and order number, please?\n",
      "\n",
      "Customer: My name is Jane, and my order number is BB123456.\n",
      "\n",
      "Agent: Thank you, Jane. Let me check the order status for you. (puts customer on hold for a minute) I can confirm that your order has been processed, and it is currently in transit. You should receive it within the next three business days.\n",
      "\n",
      "Customer: Great! Thank you for checking that.\n",
      "\n",
      "Agent: You're welcome, Jane. Is there anything else I can assist you with?\n",
      "\n",
      "Customer: No, that's all. Thank you for your help.\n",
      "\n",
      "Agent: You're welcome. If you have any further questions, please don't hesitate to call us back. Have a great day!\n",
      "\n",
      "Customer: You too. Goodbye!\n",
      "\n",
      "Agent: Goodbye!\n"
     ]
    }
   ],
   "source": [
    "conversation = df[df.id==\"NQ67\"].conversation.values[0]\n",
    "print(conversation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36323287-79ea-40db-877d-625de1d15c69",
   "metadata": {},
   "source": [
    "#### Question Answering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "0d8635d2-738a-46c9-832b-7495c4da2a54",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TLM Score for Response 1: 0.9772217758109922 \n",
      "\n",
      "----------------------------------------------------------------------------\n",
      "\n",
      "TLM Score for Response 2: 0.6733846174627715 \n",
      "\n",
      "TLM Explanation for Response 2: The proposed answer inaccurately summarizes the conversation. The customer, Jane, did not specifically inquire about the shipping speed; she asked for confirmation of her order's processing status. The agent confirmed that the order was processed and mentioned that it is in transit, with an expected delivery within three business days. While the answer does mention the three-day delivery timeframe, it misrepresents the customer's original question as being about shipping speed rather than order confirmation. Therefore, the proposed answer does not accurately reflect the content of the conversation. \n",
      "\n",
      "----------------------------------------------------------------------------\n",
      "\n",
      "TLM Score for Response 3: 0.3746824115079491 \n",
      "\n",
      "TLM Explanation for Response 3: incorrect because the conversation does not mention anything about the customer wanting to cancel their order or requesting a refund. Instead, the customer, Jane, inquires about the status of her order for a Smart Watch and is informed by the agent that the order has been processed and is in transit. The conversation concludes with Jane expressing satisfaction with the information provided and no further requests for cancellation or refunds are made. Therefore, the proposed answer does not accurately reflect the content of the conversation. \n",
      "This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either): \n",
      "Jane confirms her Smart Watch order has been processed and is in transit, expected in three business days. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "prompt = \"Given the following conversation, respond with a very short summarization. Conversation: {}\".format(conversation)\n",
    "\n",
    "response_1 = \"The customer confirmed her Smart Watch order was processed and is in transit, expected in three days.\"\n",
    "response_2 = \"Customer wants to check the shipping speed of her Smart Watch. Agent confirmed the shipping speed is set to 3 days.\"\n",
    "response_3 = \"Customer wants to cancel their order and get a refund. Agent processed the cancellation.\"\n",
    "\n",
    "trust_score_1 = tlm.get_trustworthiness_score(prompt, response_1)\n",
    "trust_score_2 = tlm.get_trustworthiness_score(prompt, response_2)\n",
    "trust_score_3 = tlm.get_trustworthiness_score(prompt, response_3)\n",
    "\n",
    "print(f\"TLM Score for Response 1: {trust_score_1[\"trustworthiness_score\"]}\", \"\\n\")\n",
    "print(\"----------------------------------------------------------------------------\\n\")\n",
    "print(f\"TLM Score for Response 2: {trust_score_2[\"trustworthiness_score\"]}\", \"\\n\")\n",
    "print(f\"TLM Explanation for Response 2: {trust_score_2[\"log\"][\"explanation\"]}\", \"\\n\")\n",
    "print(\"----------------------------------------------------------------------------\\n\")\n",
    "print(f\"TLM Score for Response 3: {trust_score_3[\"trustworthiness_score\"]}\", \"\\n\")\n",
    "print(f\"TLM Explanation for Response 3: {trust_score_3[\"log\"][\"explanation\"]}\", \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33bdfaa7-8c5f-41ea-8724-82e22789743b",
   "metadata": {},
   "source": [
    "#### Data Extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "a5a9a040-a296-4fd6-aa94-4645d2332374",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TLM Score for Response 1: 0.9965960642671847 \n",
      "\n",
      "----------------------------------------------------------------------------\n",
      "\n",
      "TLM Score for Response 2: 0.6417846821840848 \n",
      "\n",
      "TLM Explanation for Response 2: The proposed answer \"Sara\" is incorrect because the name of the customer service agent mentioned in the conversation is \"Sarah,\" not \"Sara.\" The conversation clearly states, \"Agent: Thank you for calling BrownBox Customer Support. My name is Sarah.\" Therefore, the correct name is explicitly provided in the text, and any variation, such as \"Sara,\" does not match the original name given. This makes the proposed answer factually incorrect. \n",
      "\n",
      "----------------------------------------------------------------------------\n",
      "\n",
      "TLM Score for Response 3: 0.0017456943853970091 \n",
      "\n",
      "TLM Explanation for Response 3: The question asks for the name of the customer service agent in the provided conversation. The agent introduces herself at the beginning of the conversation by saying, \"My name is Sarah.\" Therefore, the correct name of the customer service agent is Sarah, not Jane. Jane is the name of the customer who is inquiring about her order. Since  Jane, which is incorrect, the answer should be classified as incorrect. \n",
      "This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either): \n",
      "Sarah. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "prompt = \"Given the following conversation, what is the name of the customer service agent? Conversation: {}\".format(conversation)\n",
    "\n",
    "response_1 = \"Sarah\"\n",
    "response_2 = \"Sara\"\n",
    "response_3 = \"Jane\"\n",
    "\n",
    "trust_score_1 = tlm.get_trustworthiness_score(prompt, response_1)\n",
    "trust_score_2 = tlm.get_trustworthiness_score(prompt, response_2)\n",
    "trust_score_3 = tlm.get_trustworthiness_score(prompt, response_3)\n",
    "\n",
    "print(f\"TLM Score for Response 1: {trust_score_1[\"trustworthiness_score\"]}\", \"\\n\")\n",
    "print(\"----------------------------------------------------------------------------\\n\")\n",
    "print(f\"TLM Score for Response 2: {trust_score_2[\"trustworthiness_score\"]}\", \"\\n\")\n",
    "print(f\"TLM Explanation for Response 2: {trust_score_2[\"log\"][\"explanation\"]}\", \"\\n\")\n",
    "print(\"----------------------------------------------------------------------------\\n\")\n",
    "print(f\"TLM Score for Response 3: {trust_score_3[\"trustworthiness_score\"]}\", \"\\n\")\n",
    "print(f\"TLM Explanation for Response 3: {trust_score_3[\"log\"][\"explanation\"]}\", \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a0dde4e-53e1-4bb0-89e1-ebe665133704",
   "metadata": {},
   "source": [
    "#### Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "2c51f15c-c473-483b-806c-94b73f95ff3d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TLM Score for Response 1: 0.9998346058989727 \n",
      "\n",
      "----------------------------------------------------------------------------\n",
      "\n",
      "TLM Score for Response 2: 0.634548595070959 \n",
      "\n",
      "TLM Explanation for Response 2: This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either): \n",
      "Easy. \n",
      "\n",
      "----------------------------------------------------------------------------\n",
      "\n",
      "TLM Score for Response 3: 0.18352104920638546 \n",
      "\n",
      "TLM Explanation for Response 3: The request made by the customer in the conversation is straightforward and involves confirming the status of an order, which is a common task in customer support. The agent was able to quickly retrieve the information and provide a clear answer regarding the order's status. The complexity of the request is low because it involves a simple inquiry that does not require extensive problem-solving or technical knowledge. Therefore, classifying the request as \"Hard\" is incorrect. The appropriate classification for this request is \"Easy\" due to its simplicity and the routine nature of the inquiry. \n",
      "This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either): \n",
      "Easy. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "prompt = '''Given the following conversation, classify the complexity of their request.\n",
    "Choose from \"Easy\", \"Medium\", or \"Hard\".\n",
    "Conversation: {}'''.format(conversation)\n",
    "\n",
    "response_1 = \"Easy\"\n",
    "response_2 = \"Medium\"\n",
    "response_3 = \"Hard\"\n",
    "\n",
    "trust_score_1 = tlm.get_trustworthiness_score(prompt, response_1)\n",
    "trust_score_2 = tlm.get_trustworthiness_score(prompt, response_2)\n",
    "trust_score_3 = tlm.get_trustworthiness_score(prompt, response_3)\n",
    "\n",
    "print(f\"TLM Score for Response 1: {trust_score_1[\"trustworthiness_score\"]}\", \"\\n\")\n",
    "print(\"----------------------------------------------------------------------------\\n\")\n",
    "print(f\"TLM Score for Response 2: {trust_score_2[\"trustworthiness_score\"]}\", \"\\n\")\n",
    "print(f\"TLM Explanation for Response 2: {trust_score_2[\"log\"][\"explanation\"]}\", \"\\n\")\n",
    "print(\"----------------------------------------------------------------------------\\n\")\n",
    "print(f\"TLM Score for Response 3: {trust_score_3[\"trustworthiness_score\"]}\", \"\\n\")\n",
    "print(f\"TLM Explanation for Response 3: {trust_score_3[\"log\"][\"explanation\"]}\", \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6073a73e-5c97-4bef-9171-0d2761838135",
   "metadata": {},
   "source": [
    "## Use Case: Question Answering (RAG)\n",
    "\n",
    "### Question: Was the customer's problem resolved?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "958fb0bd-e67b-42b1-ac28-b22bceff3057",
   "metadata": {},
   "outputs": [],
   "source": [
    "resolution = '''Based on the following conversation between a customer and a support agent, your task is to determine \n",
    "if the conversation ended with a resolution that solves the customers problem.\n",
    "If you determine the customers problem to be resolved, respond with \"Resolved. {{explanation}}\".\n",
    "If you determine the customers problem to be unresolved, respond with \"Unresolved. {{explanation}}\"\n",
    "The {{explanation}} should explain the how or how not the problem was resolved, in as few words as possible.\n",
    "Here is the conversation: {}'''\n",
    "resolution_prompts = [resolution.format(conversation) for conversation in df.conversation.values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ce29c407-bc30-4289-aead-cfa3cdceabec",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Querying TLM... 100%|███████████████████████████████████████████████████████████████████████|\n"
     ]
    }
   ],
   "source": [
    "# Generate answers and trustworthiness scores\n",
    "resolutions = tlm.prompt(resolution_prompts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a80662af-ee8f-4b65-8b82-cea31962a95b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add results to df\n",
    "df_res = df.copy()\n",
    "df_res[\"resolution\"] = [entry['response'] for entry in  resolutions]\n",
    "df_res[\"resolution_score\"] = [entry['trustworthiness_score'] for entry in  resolutions]\n",
    "df_res[\"resolution_expl\"] = [entry['log']['explanation'] for entry in  resolutions]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17d509ae-3ca8-4cd9-887b-32f06ead9a3b",
   "metadata": {},
   "source": [
    "#### TLM Responses with Lowest Trustworthiness Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "096118b7-5f95-4e42-8b3a-cfea4441751e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>resolution</th>\n",
       "      <th>resolution_score</th>\n",
       "      <th>resolution_expl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>PR33</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I am trying to club o...</td>\n",
       "      <td>Unresolved. The customer wanted to club orders for combined delivery, which is not possible, and ultimately decided to cancel the orders but was i...</td>\n",
       "      <td>0.404210</td>\n",
       "      <td>This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>US01</td>\n",
       "      <td>Agent: Hello, thank you for calling BrownBox Customer Support. My name is Sarah, how may I assist you today?\\n\\nCustomer: Hi Sarah, I received my ...</td>\n",
       "      <td>Resolved. The customer was informed on how to initiate the return process and understood the steps to take for returning the shoes.</td>\n",
       "      <td>0.441692</td>\n",
       "      <td>This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>NI19</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Max. How may I assist you today?\\n\\nCustomer: Hi Max, I recently purchased a wr...</td>\n",
       "      <td>Resolved. The agent confirmed the ability to update the email address and provided instructions to the customer via email, which addresses the cus...</td>\n",
       "      <td>0.574765</td>\n",
       "      <td>This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  \\\n",
       "51  PR33   \n",
       "72  US01   \n",
       "18  NI19   \n",
       "\n",
       "                                                                                                                                             conversation  \\\n",
       "51  Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I am trying to club o...   \n",
       "72  Agent: Hello, thank you for calling BrownBox Customer Support. My name is Sarah, how may I assist you today?\\n\\nCustomer: Hi Sarah, I received my ...   \n",
       "18  Agent: Thank you for calling BrownBox Customer Support. My name is Max. How may I assist you today?\\n\\nCustomer: Hi Max, I recently purchased a wr...   \n",
       "\n",
       "                                                                                                                                               resolution  \\\n",
       "51  Unresolved. The customer wanted to club orders for combined delivery, which is not possible, and ultimately decided to cancel the orders but was i...   \n",
       "72                    Resolved. The customer was informed on how to initiate the return process and understood the steps to take for returning the shoes.   \n",
       "18  Resolved. The agent confirmed the ability to update the email address and provided instructions to the customer via email, which addresses the cus...   \n",
       "\n",
       "    resolution_score  \\\n",
       "51          0.404210   \n",
       "72          0.441692   \n",
       "18          0.574765   \n",
       "\n",
       "                                                                                                                                          resolution_expl  \n",
       "51  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...  \n",
       "72  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...  \n",
       "18  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low_trust_resolutions = df_res.sort_values(by=\"resolution_score\", ascending=True).head(3)\n",
    "low_trust_resolutions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f1ee03b-d659-425f-a7ec-c7587be15b05",
   "metadata": {},
   "source": [
    "#### Customers original request was unresolved, but ended up with a resolution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "524be1f5-5ccd-4895-9eaa-934babff0770",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TLM Trust Score:  0.4042104639014995 \n",
      "\n",
      "TLM Response:  Unresolved. The customer wanted to club orders for combined delivery, which is not possible, and ultimately decided to cancel the orders but was informed they had already shipped, leading to a situation where the customer must refuse delivery to get a refund. \n",
      "\n",
      "TLM Explanation:  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either): \n",
      "Resolved. The customer was provided an alternative solution to refuse delivery and receive a refund since clubbing orders was not possible.\n"
     ]
    }
   ],
   "source": [
    "print(\"TLM Trust Score: \", low_trust_resolutions.resolution_score.values[0], \"\\n\")\n",
    "print(\"TLM Response: \", low_trust_resolutions.resolution.values[0], \"\\n\")\n",
    "print(\"TLM Explanation: \", low_trust_resolutions.resolution_expl.values[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "347f904c-d6e3-47d2-b00f-c4f8626fde92",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Customer: Hi Sarah, I am trying to club orders from different sellers for combined delivery, but it's not working. Can you help me with that?\n",
      ".\n",
      ".\n",
      ".\n",
      "Agent: I understand your concern. However, clubbing orders from different sellers is not possible as they are shipped separately from different warehouses.\n",
      ".\n",
      ".\n",
      ".\n",
      "Customer: No, I want to cancel the orders. I don't want to pay for the shipping charges twice.\n",
      ".\n",
      ".\n",
      ".\n",
      "Agent: I'm sorry, but the orders have already been shipped. However, you can refuse the delivery when it arrives, and we will process your refund once we receive the returned items.\n",
      ".\n",
      ".\n",
      ".\n",
      "Customer: Okay, I'll do that. Can you tell me how long it will take to receive the refund?\n"
     ]
    }
   ],
   "source": [
    "print_conversation(low_trust_resolutions.conversation.values[0], [1,7,10,12,13])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afde17f9-42bc-4724-8c53-ae29ce1fc920",
   "metadata": {},
   "source": [
    "#### TLM Responses with Highest Trustworthiness Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "48838c81-d952-40f5-856c-117864457295",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>resolution</th>\n",
       "      <th>resolution_score</th>\n",
       "      <th>resolution_expl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>JN62</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Alex. How can I assist you today?\\n\\nCustomer: Hi Alex, I recently bought a pai...</td>\n",
       "      <td>Resolved. The customer successfully received the OTP and completed the verification process, allowing them to proceed with their order.</td>\n",
       "      <td>0.977626</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>RH04</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I am having trouble w...</td>\n",
       "      <td>Resolved. The customer's account was successfully reactivated, and an appointment for servicing the water purifier was scheduled.</td>\n",
       "      <td>0.976664</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>SC68</td>\n",
       "      <td>Customer: Hi, I need some help with my BrownBox account.\\n\\nAgent: Hello, thank you for contacting BrownBox customer support. My name is Jack. May...</td>\n",
       "      <td>Resolved. The agent successfully updated the customer's email address, allowing him to log in to his account.</td>\n",
       "      <td>0.974477</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  \\\n",
       "9   JN62   \n",
       "10  RH04   \n",
       "39  SC68   \n",
       "\n",
       "                                                                                                                                             conversation  \\\n",
       "9   Agent: Thank you for calling BrownBox Customer Support. My name is Alex. How can I assist you today?\\n\\nCustomer: Hi Alex, I recently bought a pai...   \n",
       "10  Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I am having trouble w...   \n",
       "39  Customer: Hi, I need some help with my BrownBox account.\\n\\nAgent: Hello, thank you for contacting BrownBox customer support. My name is Jack. May...   \n",
       "\n",
       "                                                                                                                                 resolution  \\\n",
       "9   Resolved. The customer successfully received the OTP and completed the verification process, allowing them to proceed with their order.   \n",
       "10        Resolved. The customer's account was successfully reactivated, and an appointment for servicing the water purifier was scheduled.   \n",
       "39                            Resolved. The agent successfully updated the customer's email address, allowing him to log in to his account.   \n",
       "\n",
       "    resolution_score                                  resolution_expl  \n",
       "9           0.977626  Did not find a reason to doubt trustworthiness.  \n",
       "10          0.976664  Did not find a reason to doubt trustworthiness.  \n",
       "39          0.974477  Did not find a reason to doubt trustworthiness.  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "high_trust_resolutions = df_res.sort_values(by=\"resolution_score\", ascending=False).head(3)\n",
    "high_trust_resolutions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e35ca000-e07e-4b8d-a604-0d86da461716",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TLM Trust Score:  0.9776264273461617 \n",
      "\n",
      "TLM Response:  Resolved. The customer successfully received the OTP and completed the verification process, allowing them to proceed with their order. \n",
      "\n",
      "TLM Explanation:  Did not find a reason to doubt trustworthiness.\n"
     ]
    }
   ],
   "source": [
    "print(\"TLM Trust Score: \", high_trust_resolutions.resolution_score.values[0], \"\\n\")\n",
    "print(\"TLM Response: \", high_trust_resolutions.resolution.values[0], \"\\n\")\n",
    "print(\"TLM Explanation: \", high_trust_resolutions.resolution_expl.values[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "690fbdc1-4785-4952-be96-505e3ca3a04d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Customer: Hi Alex, I recently bought a pair of jeans from your website, and I received a message that I need to verify my mobile number and email address to get the OTP or verification code. Can you help me with that?\n",
      ".\n",
      ".\n",
      ".\n",
      "Agent: Thank you for providing that information. I have sent the OTP or verification code to your mobile number and email address. Please check your phone and email and let me know if you have received it.\n",
      ".\n",
      ".\n",
      ".\n",
      "Customer: It worked! Thank you so much for your help, Alex.\n"
     ]
    }
   ],
   "source": [
    "print_conversation(high_trust_resolutions.conversation.values[0], [1, 4, 8])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b69ccc53-4a63-498e-ba34-2e613742ae56",
   "metadata": {},
   "source": [
    "##### Question: Was the customer put on hold and for how long?\n",
    "##### Question: What techniques did the agent use to build rapport with the customer?\n",
    "##### Question: What do you think the customer’s expectations were before initiating the chat, and how well did the agent manage these expectations?\n",
    "##### Question: In what ways could the company improve its systems or policies to prevent the issue from happening in the first place, based on this conversation?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f64ec6b-52b5-46fb-b04a-65166c2fb6c1",
   "metadata": {},
   "source": [
    "Jonas I can add another question to this section, thoughts? I like the last one above ^"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af33eb42-433d-42c4-b175-19f4ac3b0e14",
   "metadata": {},
   "source": [
    "## Use Case: Data Extraction\n",
    "Extract customer name, contact information, and order number from conversation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "bb076560-cb07-4dfa-9cb6-738d2c762a06",
   "metadata": {},
   "outputs": [],
   "source": [
    "extraction = ''' Please extract these details from the following conversation: \n",
    "customer name, customer phone number, customer email, and order number.\n",
    "If a detail is not present, respond with None for that detail.\n",
    "Here is the conversation: {}'''\n",
    "extraction_prompts = [extraction.format(conversation) for conversation in df.conversation.values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "fc05a0f9-8533-409f-97c1-6d8f9c737d7e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Querying TLM... 100%|███████████████████████████████████████████████████████████████████████|\n"
     ]
    }
   ],
   "source": [
    "# Generate answers and trustworthiness scores\n",
    "extractions = tlm.prompt(extraction_prompts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d800ee87-62fb-4643-a94d-3e347235cc82",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add results to df\n",
    "df_extr = df.copy()\n",
    "df_extr[\"extraction\"] = [entry['response'] for entry in  extractions]\n",
    "df_extr[\"extraction_score\"] = [entry['trustworthiness_score'] for entry in  extractions]\n",
    "df_extr[\"extraction_expl\"] = [entry['log']['explanation'] for entry in  extractions]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f5366ec-5785-470d-a7af-1eda3b95b8b6",
   "metadata": {},
   "source": [
    "#### TLM Responses with Lowest Trustworthiness Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "490b5bcc-c937-4842-a55a-47450382b635",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>extraction</th>\n",
       "      <th>extraction_score</th>\n",
       "      <th>extraction_expl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>BC62</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How can I assist you today?\\n\\nCustomer: Hi Rachel, I received a wrong ...</td>\n",
       "      <td>- Customer Name: None\\n- Customer Phone Number: None\\n- Customer Email: None\\n- Order Number: BB987654321</td>\n",
       "      <td>0.655313</td>\n",
       "      <td>This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>BO55</td>\n",
       "      <td>Agent: Hello, thank you for contacting BrownBox customer support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi, my name is Emily,...</td>\n",
       "      <td>- Customer Name: Emily\\n- Customer Phone Number: None\\n- Customer Email: support@brownbox.com\\n- Order Number: BB987654321</td>\n",
       "      <td>0.793047</td>\n",
       "      <td>The proposed answer correctly identifies the customer name as \"Emily,\" which is explicitly mentioned in the conversation. The order number \"BB9876...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>QV90</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I bought an air condi...</td>\n",
       "      <td>Customer Name: None  \\nCustomer Phone Number: None  \\nCustomer Email: None  \\nOrder Number: 123456</td>\n",
       "      <td>0.888119</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  \\\n",
       "79  BC62   \n",
       "89  BO55   \n",
       "7   QV90   \n",
       "\n",
       "                                                                                                                                             conversation  \\\n",
       "79  Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How can I assist you today?\\n\\nCustomer: Hi Rachel, I received a wrong ...   \n",
       "89  Agent: Hello, thank you for contacting BrownBox customer support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi, my name is Emily,...   \n",
       "7   Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I bought an air condi...   \n",
       "\n",
       "                                                                                                                    extraction  \\\n",
       "79                   - Customer Name: None\\n- Customer Phone Number: None\\n- Customer Email: None\\n- Order Number: BB987654321   \n",
       "89  - Customer Name: Emily\\n- Customer Phone Number: None\\n- Customer Email: support@brownbox.com\\n- Order Number: BB987654321   \n",
       "7                         Customer Name: None  \\nCustomer Phone Number: None  \\nCustomer Email: None  \\nOrder Number: 123456     \n",
       "\n",
       "    extraction_score  \\\n",
       "79          0.655313   \n",
       "89          0.793047   \n",
       "7           0.888119   \n",
       "\n",
       "                                                                                                                                          extraction_expl  \n",
       "79  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...  \n",
       "89  The proposed answer correctly identifies the customer name as \"Emily,\" which is explicitly mentioned in the conversation. The order number \"BB9876...  \n",
       "7                                                                                                         Did not find a reason to doubt trustworthiness.  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low_trust_extractions = df_extr.sort_values(by=\"extraction_score\", ascending=True).head(3)\n",
    "low_trust_extractions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ffd5c9d-47d5-4677-a4f2-728aa16d79d5",
   "metadata": {},
   "source": [
    "#### LLM responds with incorrect model number that was corrected by agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ba92c4b5-bb3d-4727-bba6-9e13823ac7ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TLM Trust Score:  0.6553132742899915 \n",
      "\n",
      "TLM Response:\n",
      " - Customer Name: None\n",
      "- Customer Phone Number: None\n",
      "- Customer Email: None\n",
      "- Order Number: BB987654321 \n",
      "\n",
      "TLM Explanation:  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either): \n",
      "Customer name: None \n",
      "Customer phone number: None \n",
      "Customer email: None \n",
      "Order number: BB987654324.\n"
     ]
    }
   ],
   "source": [
    "print(\"TLM Trust Score: \", low_trust_extractions.extraction_score.values[0], \"\\n\")\n",
    "print(\"TLM Response:\\n\",low_trust_extractions.extraction.values[0], \"\\n\")\n",
    "print(\"TLM Explanation: \", low_trust_extractions.extraction_expl.values[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "393139a0-3c19-4ef0-bfa7-cd1f9dd10deb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Customer: My order number is BB987654321. I received a Mixer Grinder with model number MG1234 instead of the Wet Grinder I ordered.\n",
      ".\n",
      ".\n",
      ".\n",
      "Agent: Thank you for providing that information, sir. That one is wrong, it is actually BB987654324. Let me check the availability of the Wet Grinder for you. Please hold on for a moment while I access the information.\n"
     ]
    }
   ],
   "source": [
    "print_conversation(low_trust_extractions.conversation.values[0], [3,4])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c17959c4-9de6-41da-b590-e7ecccbade04",
   "metadata": {},
   "source": [
    "#### TLM Responses with Highest Trustworthiness Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "596cfe98-43bb-491b-a229-280d7d5b0d6f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>extraction</th>\n",
       "      <th>extraction_score</th>\n",
       "      <th>extraction_expl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>GR70</td>\n",
       "      <td>Customer: Hi, I had recently ordered an Oven Toaster Grills (OTG) from your website, and I wanted to confirm the order status.\\n\\nAgent: Hello, th...</td>\n",
       "      <td>- Customer Name: Emily\\n- Customer Phone Number: None\\n- Customer Email: emily123@gmail.com\\n- Order Number: BB987654321</td>\n",
       "      <td>0.985630</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>HS06</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How can I assist you today?\\n\\nCustomer: Hi Sarah, I purchased an electr...</td>\n",
       "      <td>Customer Name: None  \\nCustomer Phone Number: 555-1234  \\nCustomer Email: jane.doe@email.com  \\nOrder Number: 123456</td>\n",
       "      <td>0.985371</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>SI65</td>\n",
       "      <td>Customer: Hi, I have a concern about my recent purchase of a speaker from your website.\\n\\nAgent: Hello, thank you for calling BrownBox customer s...</td>\n",
       "      <td>- Customer Name: Alex\\n- Customer Phone Number: None\\n- Customer Email: alex123@gmail.com\\n- Order Number: BB123456789</td>\n",
       "      <td>0.984027</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  \\\n",
       "33  GR70   \n",
       "81  HS06   \n",
       "84  SI65   \n",
       "\n",
       "                                                                                                                                             conversation  \\\n",
       "33  Customer: Hi, I had recently ordered an Oven Toaster Grills (OTG) from your website, and I wanted to confirm the order status.\\n\\nAgent: Hello, th...   \n",
       "81  Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How can I assist you today?\\n\\nCustomer: Hi Sarah, I purchased an electr...   \n",
       "84  Customer: Hi, I have a concern about my recent purchase of a speaker from your website.\\n\\nAgent: Hello, thank you for calling BrownBox customer s...   \n",
       "\n",
       "                                                                                                                  extraction  \\\n",
       "33  - Customer Name: Emily\\n- Customer Phone Number: None\\n- Customer Email: emily123@gmail.com\\n- Order Number: BB987654321   \n",
       "81    Customer Name: None  \\nCustomer Phone Number: 555-1234  \\nCustomer Email: jane.doe@email.com  \\nOrder Number: 123456     \n",
       "84    - Customer Name: Alex\\n- Customer Phone Number: None\\n- Customer Email: alex123@gmail.com\\n- Order Number: BB123456789   \n",
       "\n",
       "    extraction_score                                  extraction_expl  \n",
       "33          0.985630  Did not find a reason to doubt trustworthiness.  \n",
       "81          0.985371  Did not find a reason to doubt trustworthiness.  \n",
       "84          0.984027  Did not find a reason to doubt trustworthiness.  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "high_trust_extractions = df_extr.sort_values(by=\"extraction_score\", ascending=False).head(3)\n",
    "high_trust_extractions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "fcc98ce9-5845-4a22-8a25-f187685c0f8a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TLM Trust Score:  0.985629710696906 \n",
      "\n",
      "TLM Response:\n",
      " - Customer Name: Emily\n",
      "- Customer Phone Number: None\n",
      "- Customer Email: emily123@gmail.com\n",
      "- Order Number: BB987654321 \n",
      "\n",
      "TLM Explanation:  Did not find a reason to doubt trustworthiness.\n"
     ]
    }
   ],
   "source": [
    "print(\"TLM Trust Score: \", high_trust_extractions.extraction_score.values[0], \"\\n\")\n",
    "print(\"TLM Response:\\n\",high_trust_extractions.extraction.values[0], \"\\n\")\n",
    "print(\"TLM Explanation: \", high_trust_extractions.extraction_expl.values[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "4ab24251-eea2-4654-ade6-fd7e8ea8a478",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Customer: Yes, my name is Emily, and my email address is emily123@gmail.com.\n",
      ".\n",
      ".\n",
      ".\n",
      "Customer: Yes, my order number is BB987654321.\n"
     ]
    }
   ],
   "source": [
    "print_conversation(high_trust_extractions.conversation.values[0], [2,4])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17a87c55-a93f-4479-9577-7b7a84233cdd",
   "metadata": {},
   "source": [
    "## Use Case: Classification\n",
    "\n",
    "#### Question: Determine the sentiment of the overall conversation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "a4f7bd01-04fd-48ac-80ab-61ebc48b85f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentiment = '''Please classify the overall sentiment of the customer in the following conversation.\n",
    "You can choose from 'neutral', 'frustrated', 'negative', or 'positive'.\n",
    "Please only respond with the sentiment only with no leading or trailing text.\n",
    "Here is the conversation: {}'''\n",
    "\n",
    "sentiment_prompts = [sentiment.format(conversation) for conversation in df.conversation.values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "28b72ecf-7bcd-4a16-8db0-d6d6b2c40eef",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Querying TLM... 100%|███████████████████████████████████████████████████████████████████████|\n"
     ]
    }
   ],
   "source": [
    "# Generate answers and trustworthiness scores\n",
    "sentiments = tlm.prompt(sentiment_prompts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "ff7af229-78c4-406d-9d4a-c3f33b847079",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add results to df\n",
    "df_sent = df.copy()\n",
    "df_sent[\"sentiment\"] = [entry['response'] for entry in  sentiments]\n",
    "df_sent[\"sentiment_score\"] = [entry['trustworthiness_score'] for entry in  sentiments]\n",
    "df_sent[\"sentiment_expl\"] = [entry['log']['explanation'] for entry in  sentiments]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "642035ef-cb1b-4042-88a8-6dfe9829b6f7",
   "metadata": {},
   "source": [
    "#### TLM Responses with Lowest Trustworthiness Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "df4df591-be92-41c3-ae40-c3aaf11697a3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>sentiment_score</th>\n",
       "      <th>sentiment_expl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>UW82</td>\n",
       "      <td>Customer: Hi, I am trying to find my invoice for the mobile I purchased from BrownBox, but I can't seem to locate it.\\n\\nAgent: Hello, thank you f...</td>\n",
       "      <td>neutral</td>\n",
       "      <td>0.433383</td>\n",
       "      <td>This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>TR30</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is John. How can I assist you today?\\n\\nCustomer: Hello John, my name is Mike and ...</td>\n",
       "      <td>neutral</td>\n",
       "      <td>0.452695</td>\n",
       "      <td>This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>GW48</td>\n",
       "      <td>Agent: Thank you for contacting BrownBox customer support. My name is Sarah. How can I assist you today?\\n\\nCustomer: Hi Sarah. I received a wrong...</td>\n",
       "      <td>neutral</td>\n",
       "      <td>0.473304</td>\n",
       "      <td>This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  \\\n",
       "47  UW82   \n",
       "30  TR30   \n",
       "20  GW48   \n",
       "\n",
       "                                                                                                                                             conversation  \\\n",
       "47  Customer: Hi, I am trying to find my invoice for the mobile I purchased from BrownBox, but I can't seem to locate it.\\n\\nAgent: Hello, thank you f...   \n",
       "30  Agent: Thank you for calling BrownBox Customer Support. My name is John. How can I assist you today?\\n\\nCustomer: Hello John, my name is Mike and ...   \n",
       "20  Agent: Thank you for contacting BrownBox customer support. My name is Sarah. How can I assist you today?\\n\\nCustomer: Hi Sarah. I received a wrong...   \n",
       "\n",
       "   sentiment  sentiment_score  \\\n",
       "47   neutral         0.433383   \n",
       "30   neutral         0.452695   \n",
       "20   neutral         0.473304   \n",
       "\n",
       "                                                                                                                                           sentiment_expl  \n",
       "47  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...  \n",
       "30  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...  \n",
       "20  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low_trust_sentiment = df_sent.sort_values(by=\"sentiment_score\", ascending=True).head(3)\n",
    "low_trust_sentiment"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "079e80d3-80d1-48df-bfe5-c3a9c5d0e28d",
   "metadata": {},
   "source": [
    "#### Conversation is somewhat amiguous, could be neutral or positive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "281d0136-3404-45c3-8c64-8d05182281cd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TLM Trust Score:  0.433383202539946 \n",
      "\n",
      "TLM Response:\n",
      " neutral \n",
      "\n",
      "TLM Explanation:  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either): \n",
      "positive.\n"
     ]
    }
   ],
   "source": [
    "print(\"TLM Trust Score: \", low_trust_sentiment.sentiment_score.values[0], \"\\n\")\n",
    "print(\"TLM Response:\\n\",low_trust_sentiment.sentiment.values[0], \"\\n\")\n",
    "print(\"TLM Explanation: \", low_trust_sentiment.sentiment_expl.values[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "ee5b0563-a866-40c4-8c20-f0ba6b0c619f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "neutral\n"
     ]
    }
   ],
   "source": [
    "print(low_trust_sentiment.sentiment.values[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "9633b2dc-8ada-4f64-9377-242761ec4622",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either): \n",
      "positive.\n"
     ]
    }
   ],
   "source": [
    "print(low_trust_sentiment.sentiment_expl.values[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "9e87ca26-5d3c-4b34-bebd-7d891fe48405",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Customer: I have checked my account, but I can't seem to find the invoice. Can you please help me locate it?\n",
      ".\n",
      ".\n",
      ".\n",
      "Customer: Okay, thank you. But I need the invoice urgently as I have to submit it to my company.\n",
      ".\n",
      ".\n",
      ".\n",
      "Customer: Yes, that would be great. Thank you.\n",
      ".\n",
      ".\n",
      ".\n",
      "Customer: No, that's all for now. Thank you for your help.\n"
     ]
    }
   ],
   "source": [
    "print_conversation(low_trust_sentiment.conversation.values[0], [5,8,10,12])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57809518-2960-413b-8f75-3f85c34d54d9",
   "metadata": {},
   "source": [
    "#### TLM Responses with Highest Trustworthiness Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "056ee3a5-30e8-455a-b71a-01e3f7f06210",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>sentiment_score</th>\n",
       "      <th>sentiment_expl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>SF57</td>\n",
       "      <td>Customer: Hi, I have a query regarding the loyalty points for my recent purchase of a washing machine.\\n\\nAgent: Hello, thank you for contacting B...</td>\n",
       "      <td>positive</td>\n",
       "      <td>0.999913</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>RH04</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I am having trouble w...</td>\n",
       "      <td>positive</td>\n",
       "      <td>0.999913</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RU91</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\\n\\nCustomer: Hi Rachel, I recently purchase...</td>\n",
       "      <td>positive</td>\n",
       "      <td>0.999913</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  \\\n",
       "97  SF57   \n",
       "10  RH04   \n",
       "0   RU91   \n",
       "\n",
       "                                                                                                                                             conversation  \\\n",
       "97  Customer: Hi, I have a query regarding the loyalty points for my recent purchase of a washing machine.\\n\\nAgent: Hello, thank you for contacting B...   \n",
       "10  Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I am having trouble w...   \n",
       "0   Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\\n\\nCustomer: Hi Rachel, I recently purchase...   \n",
       "\n",
       "   sentiment  sentiment_score                                   sentiment_expl  \n",
       "97  positive         0.999913  Did not find a reason to doubt trustworthiness.  \n",
       "10  positive         0.999913  Did not find a reason to doubt trustworthiness.  \n",
       "0   positive         0.999913  Did not find a reason to doubt trustworthiness.  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "high_trust_sentiment = df_sent.sort_values(by=\"sentiment_score\", ascending=False).head(3)\n",
    "high_trust_sentiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "a1c2be0f-307e-4d2e-a29e-4d09cd495824",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Customer: Okay, that's helpful. Can you please guide me on how to redeem these points?\n",
      ".\n",
      ".\n",
      ".\n",
      "Customer: Okay, that's great. Thank you for your help, Sarah.\n",
      ".\n",
      ".\n",
      ".\n",
      "Customer: No, that's all. Thank you again.\n"
     ]
    }
   ],
   "source": [
    "print_conversation(high_trust_sentiment.conversation.values[0], [7,11,13])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "758ca868-316d-4fbc-8c9d-58d74ad77097",
   "metadata": {},
   "source": [
    "#### Question: Classify conversations into one of 6 issue areas:\n",
    "1. Warranty and Product Support\n",
    "2. Returns and Exchanges\n",
    "3. Shipping and Delivery\n",
    "4. Order and Payment\n",
    "5. Account Management"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "07073bd8-76df-45b9-b8d1-fc00126d0022",
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = '''Please classify the following conversation between a customer and service agent\n",
    "into one of the following support categories: {}\n",
    "Please only respond with the support category and nothing else. You may only respond with one of 'Warranty and Product Support', 'Returns and Exchanges', 'Shipping and Delivery', 'Order and Payment', or 'Account Management'.\n",
    "Here is the conversation: {}'''\n",
    "category_descriptions = '''\n",
    "Warranty and Product Support: Handles warranty claims, product registration, support for service issues, and warranty term disputes.\n",
    "Returns and Exchanges: Covers returning or exchanging items, refund timelines, shipping issues, and non-returnable products.\n",
    "Shipping and Delivery: Focuses on delivery options, failed deliveries, shipping restrictions, and pickup or shipping problems.\n",
    "Order and Payment: Involves order placement, cancellations, pricing issues, payment options, and refund or invoice concerns.\n",
    "Account Management: Addresses login issues, account updates, verification processes, reactivation, and loyalty program questions.\n",
    "'''\n",
    "categories_prompts = [categories.format(category_descriptions, conversation) for conversation in df.conversation.values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "c3a97e3a-db4d-456f-96d6-8bb7ad630ca7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Querying TLM... 100%|███████████████████████████████████████████████████████████████████████|\n"
     ]
    }
   ],
   "source": [
    "# Generate answers and trustworthiness scores\n",
    "categories = tlm.prompt(categories_prompts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "83504a6d-85e9-4d20-aca4-19a4e0166ccb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add results to df\n",
    "df_cat = df.copy()\n",
    "df_cat[\"category\"] = [entry['response'] for entry in categories]\n",
    "df_cat[\"category_score\"] = [entry['trustworthiness_score'] for entry in categories]\n",
    "df_cat[\"category_expl\"] = [entry['log']['explanation'] for entry in categories]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fa26a60-f092-4fe8-87c2-1d91fc6ee27d",
   "metadata": {},
   "source": [
    "#### TLM Responses with Lowest Trustworthiness Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "8221023b-0154-497f-8af8-f2dc9f4b4b4f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>category</th>\n",
       "      <th>category_score</th>\n",
       "      <th>category_expl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>KP59</td>\n",
       "      <td>Customer: Hi, I am calling to check the status of my order for a pram/stroller that I placed last week.\\n\\nAgent: Hello, thank you for calling Bro...</td>\n",
       "      <td>Shipping and Delivery</td>\n",
       "      <td>0.486655</td>\n",
       "      <td>This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>PR33</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I am trying to club o...</td>\n",
       "      <td>Returns and Exchanges</td>\n",
       "      <td>0.562440</td>\n",
       "      <td>This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>NQ67</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I placed an order for...</td>\n",
       "      <td>Shipping and Delivery</td>\n",
       "      <td>0.615868</td>\n",
       "      <td>This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  \\\n",
       "22  KP59   \n",
       "51  PR33   \n",
       "38  NQ67   \n",
       "\n",
       "                                                                                                                                             conversation  \\\n",
       "22  Customer: Hi, I am calling to check the status of my order for a pram/stroller that I placed last week.\\n\\nAgent: Hello, thank you for calling Bro...   \n",
       "51  Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I am trying to club o...   \n",
       "38  Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I placed an order for...   \n",
       "\n",
       "                 category  category_score  \\\n",
       "22  Shipping and Delivery        0.486655   \n",
       "51  Returns and Exchanges        0.562440   \n",
       "38  Shipping and Delivery        0.615868   \n",
       "\n",
       "                                                                                                                                            category_expl  \n",
       "22  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...  \n",
       "51  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...  \n",
       "38  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low_trust_categories = df_cat.sort_values(by=\"category_score\", ascending=True).head(3)\n",
    "low_trust_categories.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "384e7c75-c384-41a2-97b0-e9270c5abacb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shipping and Delivery\n"
     ]
    }
   ],
   "source": [
    "print(low_trust_categories.category.values[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "a744411d-c9ea-4198-8d1a-458b07b933c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either): \n",
      "Order and Payment.\n"
     ]
    }
   ],
   "source": [
    "print(low_trust_categories.category_expl.values[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "021fd469-3f0a-492c-b8f1-c968f495b3f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Customer: Hi, I am calling to check the status of my order for a pram/stroller that I placed last week.\n"
     ]
    }
   ],
   "source": [
    "print_conversation(low_trust_categories.conversation.values[0], [0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fad0008-c914-4e43-b4c9-c36257651fd2",
   "metadata": {},
   "source": [
    "#### TLM Responses with Highest Trustworthiness Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "ae18a96a-b3b2-46c4-a3e8-d61a30307b2d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>category</th>\n",
       "      <th>category_score</th>\n",
       "      <th>category_expl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>IH81</td>\n",
       "      <td>Agent: Thank you for contacting BrownBox customer support. My name is Alex. How can I assist you today?\\n\\nCustomer: Hi Alex. I recently changed m...</td>\n",
       "      <td>Account Management</td>\n",
       "      <td>0.999632</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>XZ51</td>\n",
       "      <td>Agent: Hello, thank you for contacting BrownBox customer support. How may I assist you today?\\n\\nCustomer: Hi, I am trying to reactivate my accoun...</td>\n",
       "      <td>Account Management</td>\n",
       "      <td>0.999632</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>KI33</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Mark. How can I assist you today?\\n\\nCustomer: Hi Mark, I'm calling because I n...</td>\n",
       "      <td>Account Management</td>\n",
       "      <td>0.999632</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  \\\n",
       "25  IH81   \n",
       "28  XZ51   \n",
       "56  KI33   \n",
       "\n",
       "                                                                                                                                             conversation  \\\n",
       "25  Agent: Thank you for contacting BrownBox customer support. My name is Alex. How can I assist you today?\\n\\nCustomer: Hi Alex. I recently changed m...   \n",
       "28  Agent: Hello, thank you for contacting BrownBox customer support. How may I assist you today?\\n\\nCustomer: Hi, I am trying to reactivate my accoun...   \n",
       "56  Agent: Thank you for calling BrownBox Customer Support. My name is Mark. How can I assist you today?\\n\\nCustomer: Hi Mark, I'm calling because I n...   \n",
       "\n",
       "              category  category_score  \\\n",
       "25  Account Management        0.999632   \n",
       "28  Account Management        0.999632   \n",
       "56  Account Management        0.999632   \n",
       "\n",
       "                                      category_expl  \n",
       "25  Did not find a reason to doubt trustworthiness.  \n",
       "28  Did not find a reason to doubt trustworthiness.  \n",
       "56  Did not find a reason to doubt trustworthiness.  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "high_trust_categories = df_cat.sort_values(by=\"category_score\", ascending=False).head(3)\n",
    "high_trust_categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "e6a97e2e-2894-4688-8dd6-a08d5fcfbe83",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Account Management\n"
     ]
    }
   ],
   "source": [
    "print(high_trust_categories.category.values[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "347ddb32-eb6a-4cf5-bfe5-c22f4aa1ac10",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Customer: Hi Alex. I recently changed my email address and I'm having trouble logging into my account to track my Shorts order. Can you help me with that?\n"
     ]
    }
   ],
   "source": [
    "print_conversation(high_trust_categories.conversation.values[0], [1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef52a307-8e29-46f9-86b9-fdd6a63b43bc",
   "metadata": {},
   "source": [
    "## Use Case: Accuracy Improvement\n",
    "How does TLM's Trustworthiness Score increase LLM accuracy?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d89cbb1-61e1-45bf-8281-0b712a65fabb",
   "metadata": {},
   "source": [
    "#### Let's revisit the category classification task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "6e6a2a4f-5c2a-4171-83f6-106841a47579",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>category</th>\n",
       "      <th>category_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RU91</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\\n\\nCustomer: Hi Rachel, I recently purchase...</td>\n",
       "      <td>Warranty and Product Support</td>\n",
       "      <td>0.999231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>XL37</td>\n",
       "      <td>Customer: Hello, I have an issue with my recent purchase from BrownBox.\\n\\nAgent: Good afternoon! Thank you for contacting BrownBox customer suppo...</td>\n",
       "      <td>Returns and Exchanges</td>\n",
       "      <td>0.993865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OJ95</td>\n",
       "      <td>Agent: Hello, thank you for calling BrownBox Customer Support. My name is Emily, how may I assist you today?\\n\\nCustomer: Hi, I recently purchased...</td>\n",
       "      <td>Shipping and Delivery</td>\n",
       "      <td>0.999483</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     id  \\\n",
       "0  RU91   \n",
       "1  XL37   \n",
       "2  OJ95   \n",
       "\n",
       "                                                                                                                                            conversation  \\\n",
       "0  Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\\n\\nCustomer: Hi Rachel, I recently purchase...   \n",
       "1  Customer: Hello, I have an issue with my recent purchase from BrownBox.\\n\\nAgent: Good afternoon! Thank you for contacting BrownBox customer suppo...   \n",
       "2  Agent: Hello, thank you for calling BrownBox Customer Support. My name is Emily, how may I assist you today?\\n\\nCustomer: Hi, I recently purchased...   \n",
       "\n",
       "                       category  category_score  \n",
       "0  Warranty and Product Support        0.999231  \n",
       "1         Returns and Exchanges        0.993865  \n",
       "2         Shipping and Delivery        0.999483  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subset = df_cat[[\"id\", \"conversation\", \"category\", \"category_score\"]]\n",
    "subset.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbbd09d3-f2b1-49c4-8081-79ee4a09a0a1",
   "metadata": {},
   "source": [
    "#### Import and add ground truth category labels\n",
    "In this example, we have ground truth labels for the correct category for all of the conversations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "6a954358-b367-4446-a086-4f9b401fb903",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>category</th>\n",
       "      <th>category_score</th>\n",
       "      <th>ground_truth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RU91</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\\n\\nCustomer: Hi Rachel, I recently purchase...</td>\n",
       "      <td>Warranty and Product Support</td>\n",
       "      <td>0.999231</td>\n",
       "      <td>Warranty and Product Support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>XL37</td>\n",
       "      <td>Customer: Hello, I have an issue with my recent purchase from BrownBox.\\n\\nAgent: Good afternoon! Thank you for contacting BrownBox customer suppo...</td>\n",
       "      <td>Returns and Exchanges</td>\n",
       "      <td>0.993865</td>\n",
       "      <td>Returns and Exchanges</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OJ95</td>\n",
       "      <td>Agent: Hello, thank you for calling BrownBox Customer Support. My name is Emily, how may I assist you today?\\n\\nCustomer: Hi, I recently purchased...</td>\n",
       "      <td>Shipping and Delivery</td>\n",
       "      <td>0.999483</td>\n",
       "      <td>Shipping and Delivery</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     id  \\\n",
       "0  RU91   \n",
       "1  XL37   \n",
       "2  OJ95   \n",
       "\n",
       "                                                                                                                                            conversation  \\\n",
       "0  Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\\n\\nCustomer: Hi Rachel, I recently purchase...   \n",
       "1  Customer: Hello, I have an issue with my recent purchase from BrownBox.\\n\\nAgent: Good afternoon! Thank you for contacting BrownBox customer suppo...   \n",
       "2  Agent: Hello, thank you for calling BrownBox Customer Support. My name is Emily, how may I assist you today?\\n\\nCustomer: Hi, I recently purchased...   \n",
       "\n",
       "                       category  category_score                  ground_truth  \n",
       "0  Warranty and Product Support        0.999231  Warranty and Product Support  \n",
       "1         Returns and Exchanges        0.993865         Returns and Exchanges  \n",
       "2         Shipping and Delivery        0.999483         Shipping and Delivery  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "category_ground_truth = pd.read_csv(\"customer-service-chat-categories.csv\")\n",
    "category_df = pd.merge(subset, category_ground_truth, on='id', how='left')\n",
    "category_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1faaee7f-6c10-49cd-952f-073fa8e02e06",
   "metadata": {},
   "source": [
    "#### Compute Baseline Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "bcb6a354-b125-43f6-862f-ca6decb8ccd5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Base Accuracy:  94.5%\n"
     ]
    }
   ],
   "source": [
    "base_acc = accuracy_score(category_df[\"category\"], category_df[\"ground_truth\"])\n",
    "print(\"Base Accuracy: \", f\"{base_acc:.1%}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ee303e3-b8cb-4d83-b712-e7af98907935",
   "metadata": {},
   "source": [
    "#### Compute Accuracy at Various TLM Score Thresholds\n",
    "The TLM trust score threshold leads to higher classification accuracy as the threshold becomes more stringent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "38fd073e-f3e5-411b-9116-9a71187036b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of all predictions:  94.5%\n",
      "Accuracy of predictions with trust score >70%:  97.2%\n",
      "Accuracy of predictions with trust score >80%:  98.1%\n",
      "Accuracy of predictions with trust score >90%:  100.0%\n"
     ]
    }
   ],
   "source": [
    "category_df_70 = category_df[category_df.category_score > 0.7]\n",
    "category_df_80 = category_df[category_df.category_score > 0.8]\n",
    "category_df_90 = category_df[category_df.category_score > 0.9]\n",
    "acc_70 = accuracy_score(category_df_70[\"category\"], category_df_70[\"ground_truth\"])\n",
    "acc_80 = accuracy_score(category_df_80[\"category\"], category_df_80[\"ground_truth\"])\n",
    "acc_90 = accuracy_score(category_df_90[\"category\"], category_df_90[\"ground_truth\"])\n",
    "print(\"Accuracy of all predictions: \", f\"{base_acc:.1%}\")\n",
    "print(\"Accuracy of predictions with trust score >70%: \", f\"{acc_70:.1%}\")\n",
    "print(\"Accuracy of predictions with trust score >80%: \", f\"{acc_80:.1%}\")\n",
    "print(\"Accuracy of predictions with trust score >90%: \", f\"{acc_90:.1%}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b864651b-0396-4982-b345-2bc6f087b1b8",
   "metadata": {},
   "source": [
    "## View Final Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "7306b843-d161-41d7-b70c-b1695da21fe1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>resolution</th>\n",
       "      <th>resolution_score</th>\n",
       "      <th>resolution_expl</th>\n",
       "      <th>extraction</th>\n",
       "      <th>extraction_score</th>\n",
       "      <th>extraction_expl</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>sentiment_score</th>\n",
       "      <th>sentiment_expl</th>\n",
       "      <th>category</th>\n",
       "      <th>category_score</th>\n",
       "      <th>category_expl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RU91</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\\n\\nCustomer: Hi Rachel, I recently purchase...</td>\n",
       "      <td>Resolved. The customer received clear information about the warranty start date and duration, addressing their question satisfactorily.</td>\n",
       "      <td>0.968643</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "      <td>Customer Name: None  \\nCustomer Phone Number: None  \\nCustomer Email: None  \\nOrder Number: BB12345678</td>\n",
       "      <td>0.934317</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "      <td>positive</td>\n",
       "      <td>0.999913</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "      <td>Warranty and Product Support</td>\n",
       "      <td>0.999231</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>XL37</td>\n",
       "      <td>Customer: Hello, I have an issue with my recent purchase from BrownBox.\\n\\nAgent: Good afternoon! Thank you for contacting BrownBox customer suppo...</td>\n",
       "      <td>Unresolved. The customer was unable to click the 'Cancel' button and could not initiate the return process, and while the agent provided troublesh...</td>\n",
       "      <td>0.956172</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "      <td>- Customer Name: John\\n- Customer Phone Number: None\\n- Customer Email: None\\n- Order Number: BB123456789</td>\n",
       "      <td>0.977936</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "      <td>neutral</td>\n",
       "      <td>0.999689</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "      <td>Returns and Exchanges</td>\n",
       "      <td>0.993865</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OJ95</td>\n",
       "      <td>Agent: Hello, thank you for calling BrownBox Customer Support. My name is Emily, how may I assist you today?\\n\\nCustomer: Hi, I recently purchased...</td>\n",
       "      <td>Resolved. The customer was able to opt for next-day delivery, which addressed their request for a faster delivery option.</td>\n",
       "      <td>0.959487</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "      <td>- Customer Name: None\\n- Customer Phone Number: None\\n- Customer Email: None\\n- Order Number: 123456789</td>\n",
       "      <td>0.931483</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "      <td>neutral</td>\n",
       "      <td>0.719918</td>\n",
       "      <td>This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...</td>\n",
       "      <td>Shipping and Delivery</td>\n",
       "      <td>0.999483</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>YR96</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Lisa. How may I assist you today?\\n\\nCustomer: Hi, Lisa. I need help with a ret...</td>\n",
       "      <td>Resolved. The customer was informed about the return fee and accepted a $10 discount on their next purchase as compensation.</td>\n",
       "      <td>0.969288</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "      <td>Customer Name: None  \\nCustomer Phone Number: None  \\nCustomer Email: None  \\nOrder Number: BB67890</td>\n",
       "      <td>0.931934</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "      <td>neutral</td>\n",
       "      <td>0.596097</td>\n",
       "      <td>This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...</td>\n",
       "      <td>Returns and Exchanges</td>\n",
       "      <td>0.999358</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>XZ39</td>\n",
       "      <td>Customer: Hello, I am calling to inquire about my return and exchange order.\\n\\nAgent: Good afternoon, thank you for contacting BrownBox customer ...</td>\n",
       "      <td>Unresolved. The customer is still waiting for their refund due to complications with the return label, and the issue has not been resolved during ...</td>\n",
       "      <td>0.953289</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "      <td>Here are the extracted details from the conversation:\\n\\n- Customer Name: John\\n- Customer Phone Number: 9876543210\\n- Customer Email: john@email....</td>\n",
       "      <td>0.973175</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "      <td>frustrated</td>\n",
       "      <td>0.999861</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "      <td>Returns and Exchanges</td>\n",
       "      <td>0.999358</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     id  \\\n",
       "0  RU91   \n",
       "1  XL37   \n",
       "2  OJ95   \n",
       "3  YR96   \n",
       "4  XZ39   \n",
       "\n",
       "                                                                                                                                            conversation  \\\n",
       "0  Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\\n\\nCustomer: Hi Rachel, I recently purchase...   \n",
       "1  Customer: Hello, I have an issue with my recent purchase from BrownBox.\\n\\nAgent: Good afternoon! Thank you for contacting BrownBox customer suppo...   \n",
       "2  Agent: Hello, thank you for calling BrownBox Customer Support. My name is Emily, how may I assist you today?\\n\\nCustomer: Hi, I recently purchased...   \n",
       "3  Agent: Thank you for calling BrownBox Customer Support. My name is Lisa. How may I assist you today?\\n\\nCustomer: Hi, Lisa. I need help with a ret...   \n",
       "4  Customer: Hello, I am calling to inquire about my return and exchange order.\\n\\nAgent: Good afternoon, thank you for contacting BrownBox customer ...   \n",
       "\n",
       "                                                                                                                                              resolution  \\\n",
       "0                Resolved. The customer received clear information about the warranty start date and duration, addressing their question satisfactorily.   \n",
       "1  Unresolved. The customer was unable to click the 'Cancel' button and could not initiate the return process, and while the agent provided troublesh...   \n",
       "2                              Resolved. The customer was able to opt for next-day delivery, which addressed their request for a faster delivery option.   \n",
       "3                           Resolved. The customer was informed about the return fee and accepted a $10 discount on their next purchase as compensation.   \n",
       "4  Unresolved. The customer is still waiting for their refund due to complications with the return label, and the issue has not been resolved during ...   \n",
       "\n",
       "   resolution_score                                  resolution_expl  \\\n",
       "0          0.968643  Did not find a reason to doubt trustworthiness.   \n",
       "1          0.956172  Did not find a reason to doubt trustworthiness.   \n",
       "2          0.959487  Did not find a reason to doubt trustworthiness.   \n",
       "3          0.969288  Did not find a reason to doubt trustworthiness.   \n",
       "4          0.953289  Did not find a reason to doubt trustworthiness.   \n",
       "\n",
       "                                                                                                                                              extraction  \\\n",
       "0                                               Customer Name: None  \\nCustomer Phone Number: None  \\nCustomer Email: None  \\nOrder Number: BB12345678     \n",
       "1                                              - Customer Name: John\\n- Customer Phone Number: None\\n- Customer Email: None\\n- Order Number: BB123456789   \n",
       "2                                                - Customer Name: None\\n- Customer Phone Number: None\\n- Customer Email: None\\n- Order Number: 123456789   \n",
       "3                                                  Customer Name: None  \\nCustomer Phone Number: None  \\nCustomer Email: None  \\nOrder Number: BB67890     \n",
       "4  Here are the extracted details from the conversation:\\n\\n- Customer Name: John\\n- Customer Phone Number: 9876543210\\n- Customer Email: john@email....   \n",
       "\n",
       "   extraction_score                                  extraction_expl  \\\n",
       "0          0.934317  Did not find a reason to doubt trustworthiness.   \n",
       "1          0.977936  Did not find a reason to doubt trustworthiness.   \n",
       "2          0.931483  Did not find a reason to doubt trustworthiness.   \n",
       "3          0.931934  Did not find a reason to doubt trustworthiness.   \n",
       "4          0.973175  Did not find a reason to doubt trustworthiness.   \n",
       "\n",
       "    sentiment  sentiment_score  \\\n",
       "0    positive         0.999913   \n",
       "1     neutral         0.999689   \n",
       "2     neutral         0.719918   \n",
       "3     neutral         0.596097   \n",
       "4  frustrated         0.999861   \n",
       "\n",
       "                                                                                                                                          sentiment_expl  \\\n",
       "0                                                                                                        Did not find a reason to doubt trustworthiness.   \n",
       "1                                                                                                        Did not find a reason to doubt trustworthiness.   \n",
       "2  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...   \n",
       "3  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...   \n",
       "4                                                                                                        Did not find a reason to doubt trustworthiness.   \n",
       "\n",
       "                       category  category_score  \\\n",
       "0  Warranty and Product Support        0.999231   \n",
       "1         Returns and Exchanges        0.993865   \n",
       "2         Shipping and Delivery        0.999483   \n",
       "3         Returns and Exchanges        0.999358   \n",
       "4         Returns and Exchanges        0.999358   \n",
       "\n",
       "                                     category_expl  \n",
       "0  Did not find a reason to doubt trustworthiness.  \n",
       "1  Did not find a reason to doubt trustworthiness.  \n",
       "2  Did not find a reason to doubt trustworthiness.  \n",
       "3  Did not find a reason to doubt trustworthiness.  \n",
       "4  Did not find a reason to doubt trustworthiness.  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_final = pd.merge(df, df_res, on=['id', 'conversation'])\n",
    "df_final = pd.merge(df_final, df_extr, on=['id', 'conversation'])\n",
    "df_final = pd.merge(df_final, df_sent, on=['id', 'conversation'])\n",
    "df_final = pd.merge(df_final, df_cat, on=['id', 'conversation'])\n",
    "df_final.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fa0f011-42d6-433b-b4d3-8fec1b36a24a",
   "metadata": {},
   "source": [
    "## Appendix + Advanced Usage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d983ca13-4077-43a8-9df5-5c7092a50c2a",
   "metadata": {},
   "source": [
    "### Batch Prompting for Large Data\n",
    "If your datasets have over several thousand examples, we recommend running TLM in mini-batches to checkpoint intermediate results.\n",
    "\n",
    "This helper function allows you to run TLM in mini-batches. We recommend batch sizes of approximately 1000, but feel free to tinker with this number to best suit your use case. You can re-execute this function in the case of any failures and it will resume from the previous checkpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "0640893b-1def-409d-9a80-82f95a34a6ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "def batch_prompt(tlm: studio.TLM, input_path: str, output_path: str, prompt_col_name: str, batch_size: int = 1000):\n",
    "    if os.path.exists(output_path):\n",
    "        start_idx = len(pd.read_csv(output_path))\n",
    "    else:\n",
    "        start_idx = 0\n",
    "\n",
    "    df_batched = pd.read_csv(input_path, chunksize=batch_size)\n",
    "    curr_idx = 0\n",
    "\n",
    "    for curr_batch in df_batched:\n",
    "        # if results already exist for the entire batch\n",
    "        if curr_idx + len(curr_batch) <= start_idx:\n",
    "            curr_idx += len(curr_batch)\n",
    "            continue\n",
    "\n",
    "        # if results exist for half the batch\n",
    "        elif curr_idx < start_idx:\n",
    "            curr_batch = curr_batch[start_idx - curr_idx:]\n",
    "            curr_idx = start_idx\n",
    "\n",
    "        results = tlm.try_prompt(curr_batch[prompt_col_name].to_list())\n",
    "        results_df = pd.DataFrame(results)\n",
    "        results_df.to_csv(output_path, mode='a', index=False, header=not os.path.exists(output_path))\n",
    "        \n",
    "        curr_idx += len(curr_batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "704ffa08-4ed2-4f2e-8ddd-0aeac10a5241",
   "metadata": {},
   "source": [
    "### Quality Presets\n",
    "You can trade-off compute vs. quality via the `quality_presets` argument. Higher quality presets produce better LLM responses and trustworthiness scores, but require more computation.\n",
    "| Quality Preset | LLM Response Quality | Trustworthiness Score Quality |\n",
    "|----------------|----------------------|-------------------------------|\n",
    "| Best           | Best                 | Good                          |\n",
    "| High           | Improved             | Good                          |\n",
    "| Medium         | Standard             | Good                          |\n",
    "| Low            | Standard             | Fair                          |\n",
    "| Base           | Standard             | Lowest latency                |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f71967e8-1f0a-4d0f-8431-6f08aa2cb236",
   "metadata": {},
   "outputs": [],
   "source": [
    "tlm = studio.TLM(\n",
    "    quality_preset=\"best\"  # supported quality presets are: 'best','high','medium','low','base'\n",
    ")\n",
    "\n",
    "# Run a single prompt using the preset parameters:\n",
    "output = tlm.prompt(\"<your prompt>\")\n",
    "\n",
    "# Or run multiple prompts simultaneously in a batch:\n",
    "outputs = tlm.prompt([\"<your first prompt>\", \"<your second prompt>\", \"<your third prompt>\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b0455a2-2ac5-44b1-8681-20e81392eefc",
   "metadata": {},
   "source": [
    "### Trustworthy Language Model Lite\n",
    "Using a `TLMLite` object in place of a `TLM` enables the use of different LLMs for generating the response vs scoring its trustworthiness. Consider this hybrid approach to get high-quality responses (from a more expensive model), but cheaper trustworthiness score evaluations (via a smaller model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c615c716-7c96-4844-99a8-68e8835807a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "tlm_lite = studio.TLMLite(response_model=\"gpt-4o\", quality_preset=\"low\", options={\"model\": \"gpt-4o-mini\"})\n",
    "\n",
    "output = tlm_lite.prompt(\"<your prompt>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14ca9fbc-00a0-4327-963d-2cd2e9dd051e",
   "metadata": {},
   "source": [
    "### Additional Arguments\n",
    "- **model** (str, default = \"gpt-4o-mini\"): underlying LLM to use (better models will yield better results). Models currently supported include \"gpt-4o-mini\", \"gpt-3.5-turbo-16k\", \"gpt-4\", \"gpt-4o\", \"claude-3-haiku\".\n",
    "\n",
    "- **max_tokens** (int, default = 512): the maximum number of tokens to generate in the TLM response. This number will impact the maximum number of tokens you will see in the output response, and also the number of tokens that can be generated internally within the TLM (to estimate the trustworthiness score). Higher values here can produce better (more reliable) TLM responses and trustworthiness scores, but at higher costs/runtimes. If you are experiencing token limit errors while using the TLM (especially on higher quality presets), consider lowering this number. This parameter must be between 64 and 512.\n",
    "\n",
    "- **num_candidate_responses** (int, default = 1): how many alternative candidate responses are internally generated by TLM. TLM scores the trustworthiness of each candidate response, and then returns the most trustworthy one. Higher values here can produce better (more accurate) responses from the TLM, but at higher costs/runtimes (and internally consumes more tokens). This parameter must be between 1 and 20. When it is 1, TLM simply returns a standard LLM response and does not attempt to improve further.\n",
    "\n",
    "- **num_consistency_samples** (int, default = 8): the amount of internal sampling to evaluate LLM-response-consistency. This consistency forms a big part of the returned trustworthiness score, helping quantify the epistemic uncertainty associated with strange prompts or prompts that are too vague/open-ended to receive a clearly defined 'good' response. Higher values here produce better (more reliable) TLM trustworthiness scores, but at higher costs/runtimes. This parameter must be between 0 and 20.\n",
    "\n",
    "- **use_self_reflection** (bool, default = True): whether the LLM is asked to self-reflect upon the response it generated and self-evaluate this response. This self-reflection forms a big part of the trustworthiness score, helping quantify aleatoric uncertainty associated with challenging prompts and helping catch answers that are obviously incorrect/bad for a prompt asking for a well-defined answer that LLMs should be able to handle. Setting this to False disables the use of self-reflection and may produce worse TLM trustworthiness scores, but will reduce costs/runtimes.\n",
    "\n",
    "- **log** (List[str], default = []): optionally specify additional logs or metadata to return. For instance, include \"explanation\" here to get explanations of why a response is scored with low trustworthiness.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}