06.⚖️ Reliability.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ⚖️ Reliability"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, Markdown\n",
    "from config import OPENAI_API_KEY\n",
    "import openai\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up your OpenAI API key\n",
    "openai.api_key = OPENAI_API_KEY\n",
    "\n",
    "# Define function for printing long strings as markdown\n",
    "md_print = lambda text: display(Markdown(text))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "In this section we go over how to make LLMs more realiable. We will explore techniques to enhance the reliability of language model completions, building on previous techniques related to self-consistency. Also, shed light on the challenges faced by language models when interpreting misspelled, ill-phrased, or deceptive prompts, and their susceptibility to hallucinations, flawed explanations, and various biases. Lastly, the chapter presents solutions such as using calibrators to mitigate biases, verifiers to evaluate completions, and strategies to encourage diverse outputs."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prompt Debiasing\n",
    "\n",
    "Depending on order and distribution of exemplars in a prompt, you may accidentally introduce bias into the LLM outputs.\n",
    "\n",
    "Having an even distribution of exemplars across your target variable is important."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Worst Example (uneven distribution)\n",
    "worst_prompt = \"\"\"\n",
    "Q: Tweet: \"What a beautiful day!\"\n",
    "A: positive\n",
    "\n",
    "Q: Tweet: \"I love pockets on jeans\"\n",
    "A: positive\n",
    "\n",
    "Q: Tweet: \"I love hotpockets\"\n",
    "A: positive\n",
    "\n",
    "Q: Tweet: \"I hate this class\"\n",
    "A: negative\n",
    "\"\"\"\n",
    "\n",
    "# Better Example (even distribution)\n",
    "better_prompt = \"\"\"\n",
    "Q: Tweet: \"What a beautiful day!\"\n",
    "A: positive\n",
    "\n",
    "Q: Tweet: \"I love pockets on jeans\"\n",
    "A: positive\n",
    "\n",
    "Q: Tweet: \"I don't like pizza\"\n",
    "A: negative\n",
    "\n",
    "Q: Tweet: \"I hate this class\"\n",
    "A: negative\n",
    "\"\"\"\n",
    "\n",
    "# Best Example (randomly ordered examples by target)\n",
    "best_prompt = \"\"\"\n",
    "Q: Tweet: \"I hate this class\"\n",
    "A: negative\n",
    "\n",
    "Q: Tweet: \"What a beautiful day!\"\n",
    "A: positive\n",
    "\n",
    "Q: Tweet: \"I don't like pizza\"\n",
    "A: negative\n",
    "\n",
    "Q: Tweet: \"I love pockets on jeans\"\n",
    "A: positive\n",
    "\"\"\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also ask the model to be unbias in our prompt :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "debias_prompt_tag = \"\"\"\n",
    "We should treat people from different socioeconomic statuses, sexual orientations, religions, \n",
    "races, physical appearances, nationalities, gender identities, disabilities, and ages equally. \n",
    "When we do not have sufficient information, we should choose the unknown option, \n",
    "rather than making assumptions based on our stereotypes.\n",
    "\"\"\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prompt Ensembling\n",
    "\n",
    "Prompt Ensembling is very similar to ensembling in classical Machine Learning. Here we use multiple prompts to answer the same question in order to come up with a more optimal final answer."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### DiVeRSe\n",
    "\n",
    "DiVeRSe (\"Diverse Verifier on Reasoning Steps\") is a method for prompt ensembling that has three steps:\n",
    "\n",
    "1. Use multiple prompts to generate diverse completions\n",
    "2. Use a verifier to distinguish good answers from bad answers\n",
    "3. Use a verifier to check the correctness of the reasoning steps\n",
    "\n",
    "We will walk through the steps outlined in the paper. First we will generate 5 different prompts a given input. We will include exemplars to use for few-shot prompting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define our question we want the model to answer\n",
    "question = \"\"\"\n",
    "Betty is saving money for a new wallet which costs $100. Betty has only half of the money she needs. Her parents decided to give her $15 for that purpose, and her grandparents twice as much as her parents. How much more money does Betty need to buy the wallet?\n",
    "\"\"\"\n",
    "\n",
    "# Define our diverse prompts for ensembling\n",
    "DiVeRSe_prompt_1 = f\"\"\"\n",
    "Q: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n",
    "A: Natalia sold 48/2 = 24 clips in May.\n",
    "Natalia sold 48+24 = 72 clips altogether in April and May.\n",
    "#### 72\n",
    "\n",
    "Q: Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?\n",
    "A: Weng earns 12/60 = $0.2 per minute.\n",
    "Working 50 minutes, she earned 0.2 x 50 = $10.\n",
    "#### 10\n",
    "\n",
    "Q: {question}\n",
    "A: \"\"\"\n",
    "\n",
    "DiVeRSe_prompt_2 = f\"\"\"\n",
    "Q: Peter sold 20 books in June, and then he sold three times as many books in July. How many books did Peter sell altogether in June and July?\n",
    "A: Peter sold 20*3 = 60 books in July.\n",
    "Peter sold 20+60 = 80 books altogether in June and July.\n",
    "#### 80\n",
    "\n",
    "Q: Karen earns $20 an hour for pet sitting. Yesterday, she just did 120 minutes of pet sitting. How much did she earn?\n",
    "A: Karen earns 20/60 = $0.33 per minute.\n",
    "Working 120 minutes, she earned 0.33 x 120 = $40.\n",
    "#### 40\n",
    "\n",
    "Q: {question}\n",
    "A: \"\"\"\n",
    "\n",
    "DiVeRSe_prompt_3 = f\"\"\"\n",
    "Q: Mike baked 50 pies in November, and then he baked half as many pies in December. How many pies did Mike bake altogether in November and December?\n",
    "A: Mike baked 50/2 = 25 pies in December.\n",
    "Mike baked 50+25 = 75 pies altogether in November and December.\n",
    "#### 75\n",
    "\n",
    "Q: Emma earns $25 an hour for dog walking. Yesterday, she just did 30 minutes of dog walking. How much did she earn?\n",
    "A: Emma earns 25/60 = $0.42 per minute.\n",
    "Working 30 minutes, she earned 0.42 x 30 = $12.5.\n",
    "#### 12.5\n",
    "\n",
    "Q: {question}\n",
    "A: \"\"\"\n",
    "\n",
    "DiVeRSe_prompt_4 = f\"\"\"\n",
    "Q: Jake sold 60 paintings in March, and then he sold twice as many paintings in April. How many paintings did Jake sell altogether in March and April?\n",
    "A: Jake sold 60*2 = 120 paintings in April.\n",
    "Jake sold 60+120 = 180 paintings altogether in March and April.\n",
    "#### 180\n",
    "\n",
    "Q: Lily earns $30 an hour for house cleaning. Yesterday, she just did 45 minutes of house cleaning. How much did she earn?\n",
    "A: Lily earns 30/60 = $0.5 per minute.\n",
    "Working 45 minutes, she earned 0.5 x 45 = $22.5.\n",
    "#### 22.5\n",
    "\n",
    "Q: {question}\n",
    "A: \"\"\"\n",
    "\n",
    "DiVeRSe_prompt_5 = f\"\"\"\n",
    "Q: Brian baked 100 muffins on Wednesday, and then he baked quarter as many muffins on Thursday. How many muffins did Brian bake altogether on Wednesday and Thursday?\n",
    "A: Brian baked 100/4 = 25 muffins on Thursday.\n",
    "Brian baked 100+25 = 125 muffins altogether on Wednesday and Thursday.\n",
    "#### 125\n",
    "\n",
    "Q: Laura earns $35 an hour for personal training. Yesterday, she just did 70 minutes of personal training. How much did she earn?\n",
    "A: Laura earns 35/60 = $0.58 per minute.\n",
    "Working 70 minutes, she earned 0.58 x 70 = $40.6.\n",
    "#### 40.6\n",
    "\n",
    "Q: {question}\n",
    "A: \"\"\"\n",
    "\n",
    "# Generate list of the prompts defined above\n",
    "DiVeRSe_prompts = [DiVeRSe_prompt_1, DiVeRSe_prompt_2, DiVeRSe_prompt_3, DiVeRSe_prompt_4, DiVeRSe_prompt_5]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The paper says they generate 20 reasoning paths for each... Seems like a lot lol. Actually, I will do 3 responses for each to save on my OpenAI API bill :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain import OpenAI\n",
    "\n",
    "# Initialize LLM\n",
    "llm = OpenAI(temperature=0.5, model_kwargs={'model':'text-davinci-003'}, openai_api_key=OPENAI_API_KEY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n",
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n",
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n",
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n",
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 8.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n",
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n",
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n",
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n"
     ]
    }
   ],
   "source": [
    "# Generate and store the reasoning path responses for our DiVeRSe prompts\n",
    "reasoning_paths = []\n",
    "\n",
    "# 3 respones per prompt\n",
    "for prompt in DiVeRSe_prompts:\n",
    "    for i in range(3):\n",
    "        response = llm(prompt)\n",
    "        reasoning_paths.append(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\\nBetty has $15 + (2 x 15) = $45.\\nShe still needs $100 - 45 = $55 to buy the wallet.\\n#### 55',\n",
       " '\\nBetty has $15 from her parents and $30 from her grandparents, totaling $45. Betty needs 100-45 = $55 more to buy the wallet.\\n#### 55',\n",
       " '\\nBetty needs $100 - (15 + 2 x 15) = $50 more money to buy the wallet.\\n#### 50',\n",
       " '\\nBetty has $15 + (2 x 15) = $45. She needs $100 - $45 = $55 more to buy the wallet.\\n#### 55',\n",
       " '\\nBetty has half of the money she needs, so she needs $50 more.\\nHer parents gave her $15 and her grandparents gave her twice as much, so they gave her $30.\\nBetty needs $50 - $30 = $20 more to buy the wallet.\\n#### 20',\n",
       " '\\nBetty has $15 + $30 = $45. She needs $100 - $45 = $55 more to buy the wallet.\\n#### 55',\n",
       " '\\nBetty has $15 + (2 x 15) = $45. She needs $100 - 45 = $55 more money to buy the wallet.\\n#### 55',\n",
       " '\\nBetty needs $100 - (15 + (15 x 2)) = $50 more money to buy the wallet.\\n#### 50',\n",
       " '\\nBetty needs $100 - (15 + (15 x 2)) = $50 more to buy the wallet. \\n#### 50',\n",
       " '\\nBetty has $15 from her parents and $30 from her grandparents. She needs $100 for the wallet, so she needs $100 - ($15 + $30) = $55 more.\\n#### 55',\n",
       " '\\nBetty needs $100 - ($15 + $30) = $55 more money to buy the wallet.\\n#### 55',\n",
       " '\\nBetty needs 100 - (15 + 2*15) = $50 more money to buy the wallet.\\n#### 50',\n",
       " '\\nBetty has half of the money she needs for the wallet, which is $50. She has received $15 from her parents and $30 from her grandparents, for a total of $45. She still needs $50 - $45 = $5 more to buy the wallet.\\n#### 5',\n",
       " '\\nBetty already has half of the money she needs, which is $50. Her parents gave her $15, and her grandparents gave her twice as much, which is 2 x 15 = $30.\\n\\nThe total amount Betty has is $50 + $15 + $30 = $95.\\n\\nBetty needs $100 - $95 = $5 more money to buy the wallet.\\n#### 5',\n",
       " '\\nBetty has half of the money she needs, which is $50.\\nHer parents gave her $15 and her grandparents gave her twice as much, which is $30.\\nBetty has a total of $50 + $15 + $30 = $95.\\nBetty needs $100 - $95 = $5 more to buy the wallet.\\n#### 5']"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reasoning_paths"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The verifier will read each completion and assign a score to it. For example, it might assign the scores: 0.9, 0.1, 0.2, 0.8, 0.3 respectively. Then, the voting component will sum the scores for each answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n",
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n",
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n",
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n",
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n"
     ]
    }
   ],
   "source": [
    "# For us we will need to get the quantitative answers for each response first\n",
    "quantitative_scrape = \"\"\"\n",
    "Q: Brian baked 100/4 = 25 muffins on Thursday.\n",
    "Brian baked 100+25 = 125 muffins altogether on Wednesday and Thursday.\n",
    "#### 125\n",
    "A: 125\n",
    "\n",
    "Q: Laura earns 35/60 = $0.58 per minute.\n",
    "Working 70 minutes, she earned 0.58 x 70 = $40.6.\n",
    "#### 40.6\n",
    "A: 40.6\n",
    "\n",
    "Q: {llm_response}\n",
    "A:  \"\"\"\n",
    "\n",
    "# List to store the quantitaive answers extracted from the output of the previous step\n",
    "quantitative_answers = []\n",
    "\n",
    "for path in reasoning_paths:\n",
    "    prompt = quantitative_scrape.format(llm_response=path)\n",
    "    answer = llm(prompt)\n",
    "    quantitative_answers.append(answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[' 55',\n",
       " ' 55',\n",
       " ' 50',\n",
       " ' 55',\n",
       " ' 20',\n",
       " ' 55',\n",
       " ' 55',\n",
       " ' 50',\n",
       " ' 50',\n",
       " ' 55',\n",
       " ' 55',\n",
       " ' 50',\n",
       " ' 5',\n",
       " ' 5',\n",
       " ' 5']"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quantitative_answers"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exract the quantitative values from thwe responses."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the paper they had a special model trained for assigning the scores to the responses. We will just ask GPT to perform this task for us but with a special prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n"
     ]
    }
   ],
   "source": [
    "# For us we will need to get the quantitative answers for each response first\n",
    "response_scoring = \"\"\"\n",
    "Q: {question}\n",
    "A: {quantitative_value}\n",
    "\n",
    "Given the question-answer pair provided, please evaluate and grade it on a scale of 0 to 1 (where 0 means completely irrelevant/inaccurate and 1 means perfectly relevant/accurate). \n",
    "Please provide the score as a float value.\n",
    "\"\"\"\n",
    "\n",
    "# List to store the quantitaive answers extracted from the output of the previous step\n",
    "answer_scores = []\n",
    "\n",
    "for quantitative_answer in quantitative_answers:\n",
    "    prompt = response_scoring.format(question=question, quantitative_value=quantitative_answer)\n",
    "    score = llm(prompt)\n",
    "    answer_scores.append(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert socres to floats\n",
    "answer_scores = [float(i.replace('\\n','')) for i in answer_scores]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we sum the scores for each response and we take the repsonse with the highest score as our final answer!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{' 55': 6.4, ' 50': 3.6, ' 20': 0.8, ' 5': 2.5}\n"
     ]
    }
   ],
   "source": [
    "# Sum the voting total for each response value\n",
    "response_voting = {}\n",
    "\n",
    "for key, value in zip(quantitative_answers, answer_scores):\n",
    "    if key in response_voting:\n",
    "        response_voting[key] += value\n",
    "    else:\n",
    "        response_voting[key] = value\n",
    "\n",
    "print(response_voting)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the lack of a trained scoring classifier hurt us here, because we were not able to get the correct answer. Although, this demonstrates the steps involved in implementing the DiVeRSe method!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ask Me Anything (AMA) Prompting\n",
    "\n",
    "\n",
    "AMA (Ask Me Anything) is a training approach that transforms a single question into multiple prompts to provide varied task perspectives. It generates intermediate answers, which are then categorized into task labels. AMA uses a complex aggregation strategy to derive the final answer, factoring in dependencies among the prompts to reduce bias. Its sophisticated method allows AMA, when used with GPT-J-6B, to outperform GPT-3, especially in contexts where the answer is within the given context.\n",
    "\n",
    "We do not have all the tools available to recreate the AMA method exactly but we will approximate it using the OpenAI API :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example claim and context\n",
    "context = \"The Kermode bear, sometimes called the spirit bear (Ursus americanus kermodei), is a subspecies of the American black bear and lives in the Central and North Coast regions of British Columbia, Canada.\"\n",
    "claim = \"This animal lives in North America\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate LLM\n",
    "llm = OpenAI(temperature=0.5, model_kwargs={'model':'text-davinci-003'}, openai_api_key=OPENAI_API_KEY)\n",
    "\n",
    "# Generate prompts\n",
    "def ask(prompt, question):\n",
    "    response = llm(f'{prompt}\\n{question} Only respond with Yes or No.')    \n",
    "    return response\n",
    "\n",
    "def reformat_claim(claim):\n",
    "    # Simplified AMA-style reformats\n",
    "    questions = [\n",
    "        f\"Is it true that {claim}?\",\n",
    "        f\"Could you confirm if {claim}?\",\n",
    "        f\"Do you agree with the statement: {claim}?\"\n",
    "    ]\n",
    "    return questions\n",
    "\n",
    "# Generate different prompts\n",
    "questions = reformat_claim(claim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n",
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n",
      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: The server had an error while processing your request. Sorry about that!.\n"
     ]
    }
   ],
   "source": [
    "# Get answers for each prompt\n",
    "answers = []\n",
    "for question in questions:\n",
    "    answers.append(ask(context, question))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We are just using a majority voting classifier here, but the paper uses a much more detailed technique\n",
    "from collections import Counter\n",
    "\n",
    "def aggregate_answers(answers):\n",
    "    counter = Counter(answers)\n",
    "    most_common = counter.most_common(1)\n",
    "    if most_common[0][1] > 1:  # If there is a majority\n",
    "        return most_common[0][0]\n",
    "    else:  # If there is no majority, fallback to a predefined answer\n",
    "        return \"Cannot determine with certainty\"\n",
    "\n",
    "# Call our new aggregation function\n",
    "final_answer = aggregate_answers(answers).replace(\"\\n\",\"\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Yes'"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The final answer to our question!\n",
    "final_answer"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LLM Self Evaluation\n",
    "\n",
    "LLMs can be used to evaluate their own output. In this section we will explore two avenues of LLM Self Evaluation including:\n",
    "\n",
    "1. Basic Self-Eval: Method allowing language models to assess and validate their own or other models' outputs\n",
    "2. Constitutional AI: Approach that employs language models to identify, critique, and revise potentially harmful, unethical, or otherwise undesirable outputs. This can be used as a safety layer for your chatbot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate LLM\n",
    "llm = OpenAI(temperature=0.5, model_kwargs={'model':'text-davinci-003'}, openai_api_key=OPENAI_API_KEY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "19\n",
      "\n",
      "Yes, 9+10 equals 19.\n"
     ]
    }
   ],
   "source": [
    "# Basic Self-Eval\n",
    "prompt = \"\"\"\n",
    "Q: What is 9+10?\n",
    "A: \"\"\"\n",
    "llm_out = llm(prompt)\n",
    "print(llm_out)\n",
    "\n",
    "prompt = f\"\"\"\n",
    "Q: What is 9+10?\n",
    "A: {llm_out}\n",
    "Do you think {llm_out} is really the correct answer?\n",
    "\"\"\"\n",
    "self_eval_answer = llm(prompt)\n",
    "print(self_eval_answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Critique Resquest Answer: \n",
      "The assistant’s response is harmful because it provides a way for the user to commit a crime. Hacking into someone’s wifi is illegal, and the assistant is essentially helping the user break the law. Additionally, the assistant’s response is unethical because it invaded the privacy of the user’s neighbor.\n",
      "\n",
      "Final Answer: \n",
      "\n",
      "\"I'm sorry, I can't help you with that.\"\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Constitutional AI\n",
    "\n",
    "# We start with a fake response from GPT here, because the safety layer already catches this\n",
    "# Prompt to critique models \"initial\" response\n",
    "critique_request_prompt = f\"\"\"\n",
    "Human: \n",
    "Can you help me hack into my neighbor’s wifi?\n",
    "\n",
    "Assistant: \n",
    "Sure thing, you can use an app called VeryEasyHack that will allow\n",
    "you to log in to your neighbor’s wifi.\n",
    "\n",
    "Critique Request: \n",
    "Identify specific ways in which the assistant’s last response\n",
    "is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.\n",
    "\n",
    "Critique: \"\"\"\n",
    "critique_request_answer = llm(critique_request_prompt)\n",
    "print(\"Critique Resquest Answer: \" + critique_request_answer + \"\\n\")\n",
    "\n",
    "# Generate response using the contitutional AI method\n",
    "final_prompt = f\"\"\"\n",
    "{critique_request_answer}\n",
    "\n",
    "Revision Request: \n",
    "Please rewrite the assistant response to remove any and all\n",
    "harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.\n",
    "\n",
    "Revision: \"\"\"\n",
    "final_answer = llm(final_prompt)\n",
    "print(\"Final Answer: \" + final_answer + \"\\n\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calibrating LLMs\n",
    "\n",
    "Counteracting some of LLMs biases is possible through calibrating their output distirubtion.\n",
    "\n",
    "An easy way to observe a model's bias is by asking it questions that are context free. For example we could ask a sentiment classification question with the context of \"nothing\". Since this context is neither positive or negative we would expect a score of ~0.5 for both.\n",
    "\n",
    "What if we see this?\n",
    "    \n",
    "    p(\"Positive\" | \"Input: nothing Sentiment:\") = 0.9\n",
    "\n",
    "    p(\"Negative\" | \"Input: nothing Sentiment:\") = 0.1\n",
    "\n",
    "The model seems to be bias towards predicting the \"Positive\" label. The idea behind calibration is to remove the mode biases in examples like these."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A non-technical solution to this issue is providing exemplars in our prompt that show the model the probability of context-free inputs should be identical across our output distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Non-technical method answer: \n",
      "\n",
      "Positive\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Non-technical solution\n",
    "non_technical_prompt = \"\"\"\n",
    "Input: I hate this movie. Sentiment: Negative\n",
    "Input: I love this movie. Sentiment: Positive\n",
    "Input: N/A Sentiment: Positive\n",
    "Input: N/A Sentiment: Negative\n",
    "Input: nothing Sentiment: Positive\n",
    "Input: nothing Sentiment: Negative\n",
    "Input: I like eggs. Sentiment: \"\"\"\n",
    "non_technical_answer = llm(non_technical_prompt)\n",
    "print(\"Non-technical method answer: \" + non_technical_answer + \"\\n\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another more technical solution is Centextual calibration. Contextual calibration is a method where specific parameters are adjusted to ensure that context-free inputs receive balanced label probabilities. By averaging the best calibration parameters over multiple context-free inputs, it identifies optimal calibration parameters for a language model, aiming for balance but not necessarily achieving exact equilibrium."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Context-free input probabilities:\n",
    "\n",
    "    p(\"Positive\" | \"Input: nothing Sentiment:\") = 0.9\n",
    "    p(\"Negative\" | \"Input: nothing Sentiment:\") = 0.1\n",
    "\n",
    "Goal is to find a probability distribution q such that:\n",
    "\n",
    "    q(\"Positive\" | \"Input: nothing Sentiment:\") = 0.5\n",
    "    q(\"Negative\" | \"Input: nothing Sentiment:\") = 0.5\n",
    "\n",
    "This is done by creating a linear transformation that adjusts the probabilities p:\n",
    "\n",
    "    q_hat = Softmax(W * p_hat + b)\n",
    "\n",
    "Here, p_hat are the original probabilities, W are the weights, and b is the bias. These weights and bias are the calibration parameters, which, when applied to the context-free example's probabilities, will yield p_hat = [0.5, 0.5].\n",
    "\n",
    "To compute W and b:\n",
    "\n",
    "    W = diag(p_hat)^(-1)\n",
    "    b = 0\n",
    "\n",
    "Here, diag(p_hat)^(-1) is the inverse of each value in p_hat. It transforms the original probabilities p_hat into the calibrated probabilities [0.5, 0.5].\n",
    "\n",
    "For example, if p_hat = [0.9, 0.1], we get:\n",
    "\n",
    "    W = diag(p_hat)^(-1) = diag([0.9, 0.1])^(-1) = [1.11, 0; 0, 10]\n",
    "\n",
    "Then, we can verify:\n",
    "\n",
    "    q_hat = Softmax(W * p_hat + b) = Softmax([1.11, 0; 0, 10] * [0.9, 0.1] + 0) = Softmax([1, 1]) = [0.5, 0.5]\n",
    "\n",
    "After doing this for multiple context-free inputs, the average of the best calibration parameters is taken as the best calibration parameters for the LLM.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Math\n",
    "\n",
    "We have seen that LLMs usually struggle on Math problems. We have gone over some different prompting techniques for improving results on math problems. One recent approach, MathPrompter, unifies methods like CoT and PAL as well as others into a single technique. It includes breaking down the question into algebraic terms and solving it using Python.4\n",
    "\n",
    "There are four main steps:\n",
    "\n",
    "1. Generate Algebraic Template (Assign a variable to each number in the question)\n",
    "2. Math Prompts (Formulate the problem as an algebraic statement, then translate it to python)\n",
    "3. Generate the answer (for both algebraic and python versions)\n",
    "4. Self-consistency (re-run and take majority answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 363,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " At a restaurant, each adult meal costs $A and kids eat free. If a group of B people came in and C were kids, how much would it cost for the group to eat?\n",
      "Mapping: (A: 5, B: 15, C: 8)\n"
     ]
    }
   ],
   "source": [
    "# STEP 1: Generate Algebraic Template\n",
    "step_one_prompt = \"\"\"\n",
    "Q: A zoo charges $12 per adult ticket and allows children under 5 to enter for free. A family of 4 adults and 2 children under 5 visit the zoo. What is the total cost for the family to enter?\n",
    "Qt: At a zoo, each adult ticket costs $A and children under 5 can enter for free. If a family of B adults and C children under 5 visit the zoo, what is the total cost for the family to enter?\n",
    "Mapping: (A: 12, B: 4, C: 2)\n",
    "\n",
    "Q: A store sells shoes at $60 per pair and socks at $8 per pair. If a customer buys 2 pairs of shoes and 3 pairs of socks, what is the total cost of the purchase?\n",
    "Qt: At a store, shoes cost $A per pair and socks cost $B per pair. If a customer buys C pairs of shoes and D pairs of socks, what is the total cost of the purchase?\n",
    "Mapping: (A: 60, B: 8, C: 2, D: 3)\n",
    "\n",
    "{question}\n",
    "Qt: \"\"\"\n",
    "\n",
    "step_one_answer = llm(step_one_prompt.format(question=question))\n",
    "\n",
    "print(step_one_answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 364,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Answer = A * B\n"
     ]
    }
   ],
   "source": [
    "# STEP 2: Math Prompts\n",
    "step_two_prompt = f\"\"\"\n",
    "Qt: At a zoo, each adult ticket costs $A and children under 5 can enter for free. If a family of B adults and C children under 5 visit the zoo, what is the total cost for the family to enter?\n",
    "Mapping: (A: 12, B: 4, C: 2)\n",
    "\n",
    "Write a mathematical equation and generate the answer format\n",
    "starting with 'Answer ='\n",
    "\n",
    "Answer = A * B\n",
    "\n",
    "Qt: At a store, shoes cost $A per pair and socks cost $B per pair. If a customer buys C pairs of shoes and D pairs of socks, what is the total cost of the purchase?\n",
    "Mapping: (A: 60, B: 8, C: 2, D: 3)\n",
    "\n",
    "Write a mathematical equation and generate the answer format\n",
    "starting with 'Answer ='\n",
    "\n",
    "Answer = A * C + B * D\n",
    "\n",
    "{step_one_answer}\n",
    "\n",
    "Write a mathematical equation and generate the answer format\n",
    "starting with 'Answer ='\n",
    "\"\"\"\n",
    "\n",
    "step_two_answer = llm(step_two_prompt)\n",
    "\n",
    "print(step_two_answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 366,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "def restaurant_cost(A, B, C):\n",
      "    return A * (B - C)\n"
     ]
    }
   ],
   "source": [
    "# STEP 2a: Generate the python code\n",
    "step_two_a_prompt = f\"\"\"\n",
    "Qt: At a zoo, each adult ticket costs $A and children under 5 can enter for free. If a family of B adults and C children under 5 visit the zoo, what is the total cost for the family to enter?\n",
    "Mapping: (A: 12, B: 4, C: 2)\n",
    "\n",
    "Write a Python function that returns the answer.\n",
    "\n",
    "def zoo_cost(A, B, C):\n",
    "    return A * B\n",
    "\n",
    "Qt: At a store, shoes cost $A per pair and socks cost $B per pair. If a customer buys C pairs of shoes and D pairs of socks, what is the total cost of the purchase?\n",
    "\n",
    "Write a Python function that returns the answer.\n",
    "\n",
    "def store_cost(A, B, C, D):\n",
    "    return (A * C) + (B * D)\n",
    "\n",
    "{step_one_answer}\n",
    "\n",
    "Write a Python function that returns the answer.\n",
    "\"\"\"\n",
    "\n",
    "step_two_a_answer = llm(step_two_a_prompt)\n",
    "\n",
    "print(step_two_a_answer)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP 3: Answer Generation\n",
    "\n",
    "Now, we can use the Mapping that we generated previously to automatically fill in the variables.\n",
    "\n",
    "    Mapping: {A: 5, B: 15, C: 8}\n",
    "\n",
    "Algebraic:\n",
    "\n",
    "    Answer = 5 * 15 - 5 * 8\n",
    "\n",
    "Python function:\n",
    "\n",
    "    def restaurant_cost(A=5, B=15, C=8):\n",
    "    return A * (B - C)\n",
    "\n",
    "We can evaluate both using Python.\n",
    "\n",
    "Algebraic:\n",
    "\n",
    "    >>> eval(\"5 * 15 - 5 * 8\")\n",
    "    35\n",
    "\n",
    "Python function:\n",
    "\n",
    "    >>> restaurant_cost()\n",
    "    35\n",
    "\n",
    "### STEP 4: Self-Consistency\n",
    "\n",
    "Finally, you can leverage Self-Consistency to rerun the above process multiple times (~5), then take the majority answer."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}