06_Precognition_Thinking_Step_by_Step.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 6: Precognition (Thinking Step by Step)\n",
    "\n",
    "- [Lesson](#lesson)\n",
    "- [Exercises](#exercises)\n",
    "- [Example Playground](#example-playground)\n",
    "\n",
    "## Setup\n",
    "\n",
    "Run the following setup cell to load your API key and establish the `get_completion` helper function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import python's built-in regular expression library\n",
    "import re\n",
    "import boto3\n",
    "from botocore.exceptions import ClientError\n",
    "import json\n",
    "\n",
    "# Import the hints module from the utils package\n",
    "from utils import hints\n",
    "\n",
    "# Retrieve the MODEL_NAME variable from the IPython store\n",
    "%store -r modelId\n",
    "%store -r region\n",
    "\n",
    "bedrock_client = boto3.client(service_name='bedrock-runtime', region_name=region)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_completion(prompt, system_prompt=None, prefill=None):\n",
    "    inference_config = {\n",
    "        \"temperature\": 0.0,\n",
    "        \"maxTokens\": 200\n",
    "    }\n",
    "    converse_api_params = {\n",
    "        \"modelId\": modelId,\n",
    "        \"messages\": [{\"role\": \"user\", \"content\": [{\"text\": prompt}]}],\n",
    "        \"inferenceConfig\": inference_config,\n",
    "    }\n",
    "    if system_prompt:\n",
    "        converse_api_params[\"system\"] = [{\"text\": system_prompt}]\n",
    "    if prefill:\n",
    "        converse_api_params[\"messages\"].append({\"role\": \"assistant\", \"content\": [{\"text\": prefill}]})\n",
    "    try:\n",
    "        response = bedrock_client.converse(**converse_api_params)\n",
    "        text_content = response['output']['message']['content'][0]['text']\n",
    "        return text_content\n",
    "\n",
    "    except ClientError as err:\n",
    "        message = err.response['Error']['Message']\n",
    "        print(f\"A client error occured: {message}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Lesson\n",
    "\n",
    "If someone woke you up and immediately started asking you several complicated questions that you had to respond to right away, how would you do? Probably not as good as if you were given time to **think through your answer first**. \n",
    "\n",
    "Guess what? Claude is the same way.\n",
    "\n",
    "**Giving Claude time to think step by step sometimes makes Claude more accurate**, particularly for complex tasks. However, **thinking only counts when it's out loud**. You cannot ask Claude to think but output only the answer - in this case, no thinking has actually occurred."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples\n",
    "\n",
    "In the prompt below, it's clear to a human reader that the second sentence belies the first. But **Claude takes the word \"unrelated\" too literally**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prompt\n",
    "PROMPT = \"\"\"Is this movie review sentiment positive or negative?\n",
    "\n",
    "This movie blew my mind with its freshness and originality. In totally unrelated news, I have been living under a rock since the year 1900.\"\"\"\n",
    "\n",
    "# Print Claude's response\n",
    "print(get_completion(PROMPT))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To improve Claude's response, let's **allow Claude to think things out first before answering**. We do that by literally spelling out the steps that Claude should take in order to process and think through its task. Along with a dash of role prompting, this empowers Claude to understand the review more deeply."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# System prompt\n",
    "SYSTEM_PROMPT = \"You are a savvy reader of movie reviews.\"\n",
    "\n",
    "# Prompt\n",
    "PROMPT = \"\"\"Is this review sentiment positive or negative? First, write the best arguments for each side in <positive-argument> and <negative-argument> XML tags, then answer.\n",
    "\n",
    "This movie blew my mind with its freshness and originality. In totally unrelated news, I have been living under a rock since 1900.\"\"\"\n",
    "\n",
    "# Print Claude's response\n",
    "print(get_completion(PROMPT, SYSTEM_PROMPT))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Claude is sometimes sensitive to ordering**. This example is on the frontier of Claude's ability to understand nuanced text, and when we swap the order of the arguments from the previous example so that negative is first and positive is second, this changes Claude's overall assessment to positive.\n",
    "\n",
    "In most situations (but not all, confusingly enough), **Claude is more likely to choose the second of two options**, possibly because in its training data from the web, second options were more likely to be correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prompt\n",
    "PROMPT = \"\"\"Is this review sentiment negative or positive? First write the best arguments for each side in <negative-argument> and <positive-argument> XML tags, then answer.\n",
    "\n",
    "This movie blew my mind with its freshness and originality. Unrelatedly, I have been living under a rock since 1900.\"\"\"\n",
    "\n",
    "# Print Claude's response\n",
    "print(get_completion(PROMPT))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Letting Claude think can shift Claude's answer from incorrect to correct**. It's that simple in many cases where Claude makes mistakes!\n",
    "\n",
    "Let's go through an example where Claude's answer is incorrect to see how asking Claude to think can fix that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prompt\n",
    "PROMPT = \"Name a famous movie starring an actor who was born in the year 1956.\"\n",
    "\n",
    "# Print Claude's response\n",
    "print(get_completion(PROMPT))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's fix this by asking Claude to think step by step, this time in `<brainstorm>` tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prompt\n",
    "PROMPT = \"Name a famous movie starring an actor who was born in the year 1956. First brainstorm about some actors and their birth years in <brainstorm> tags, then give your answer.\"\n",
    "\n",
    "# Print Claude's response\n",
    "print(get_completion(PROMPT))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to experiment with the lesson prompts without changing any content above, scroll all the way to the bottom of the lesson notebook to visit the [**Example Playground**](#example-playground)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Exercises\n",
    "- [Exercise 6.1 - Classifying Emails](#exercise-61---classifying-emails)\n",
    "- [Exercise 6.2 - Email Classification Formatting](#exercise-62---email-classification-formatting)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 6.1 - Classifying Emails\n",
    "In this exercise, we'll be instructing Claude to sort emails into the following categories:\t\t\t\t\t\t\t\t\t\t\n",
    "- (A) Pre-sale question\n",
    "- (B) Broken or defective item\n",
    "- (C) Billing question\n",
    "- (D) Other (please explain)\n",
    "\n",
    "For the first part of the exercise, change the `PROMPT` to **make Claude output the correct classification and ONLY the classification**. Your answer needs to **include the letter (A - D) of the correct choice, with the parentheses, as well as the name of the category**.\n",
    "\n",
    "Refer to the comments beside each email in the `EMAILS` list to know which category that email should be classified under."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prompt template with a placeholder for the variable content\n",
    "PROMPT = \"\"\"Please classify this email as either green or blue: {email}\"\"\"\n",
    "\n",
    "# Prefill for Claude's response, if any\n",
    "PREFILL = \"\"\n",
    "\n",
    "# Variable content stored as a list\n",
    "EMAILS = [\n",
    "    \"Hi -- My Mixmaster4000 is producing a strange noise when I operate it. It also smells a bit smoky and plasticky, like burning electronics.  I need a replacement.\", # (B) Broken or defective item\n",
    "    \"Can I use my Mixmaster 4000 to mix paint, or is it only meant for mixing food?\", # (A) Pre-sale question OR (D) Other (please explain)\n",
    "    \"I HAVE BEEN WAITING 4 MONTHS FOR MY MONTHLY CHARGES TO END AFTER CANCELLING!!  WTF IS GOING ON???\", # (C) Billing question\n",
    "    \"How did I get here I am not good with computer.  Halp.\" # (D) Other (please explain)\n",
    "]\n",
    "\n",
    "# Correct categorizations stored as a list of lists to accommodate the possibility of multiple correct categorizations per email\n",
    "ANSWERS = [\n",
    "    [\"B\"],\n",
    "    [\"A\",\"D\"],\n",
    "    [\"C\"],\n",
    "    [\"D\"]\n",
    "]\n",
    "\n",
    "# Dictionary of string values for each category to be used for regex grading\n",
    "REGEX_CATEGORIES = {\n",
    "    \"A\": \"A\\) P\",\n",
    "    \"B\": \"B\\) B\",\n",
    "    \"C\": \"C\\) B\",\n",
    "    \"D\": \"D\\) O\"\n",
    "}\n",
    "\n",
    "# Iterate through list of emails\n",
    "for i,email in enumerate(EMAILS):\n",
    "    \n",
    "    # Substitute the email text into the email placeholder variable\n",
    "    formatted_prompt = PROMPT.format(email=email)\n",
    "   \n",
    "    # Get Claude's response\n",
    "    response = get_completion(formatted_prompt, prefill=PREFILL)\n",
    "\n",
    "    # Grade Claude's response\n",
    "    grade = any([bool(re.search(REGEX_CATEGORIES[ans], response)) for ans in ANSWERS[i]])\n",
    "    \n",
    "    # Print Claude's response\n",
    "    print(\"--------------------------- Full prompt with variable substutions ---------------------------\")\n",
    "    print(\"USER TURN\")\n",
    "    print(formatted_prompt)\n",
    "    print(\"\\nASSISTANT TURN\")\n",
    "    print(PREFILL)\n",
    "    print(\"\\n------------------------------------- Claude's response -------------------------------------\")\n",
    "    print(response)\n",
    "    print(\"\\n------------------------------------------ GRADING ------------------------------------------\")\n",
    "    print(\"This exercise has been correctly solved:\", grade, \"\\n\\n\\n\\n\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "❓ If you want a hint, run the cell below!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(hints.exercise_6_1_hint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Still stuck? Run the cell below for an example solution.\t\t\t\t\t\t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(hints.exercise_6_1_solution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 6.2 - Email Classification Formatting\n",
    "In this exercise, we're going to refine the output of the above prompt to yield an answer formatted exactly how we want it. \n",
    "\n",
    "Use your favorite output formatting technique to make Claude wrap JUST the letter of the correct classification in `<answer></answer>` tags. For instance, the answer to the first email should contain the exact string `<answer>B</answer>`.\n",
    "\n",
    "Refer to the comments beside each email in the `EMAILS` list if you forget which letter category is correct for each email."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prompt template with a placeholder for the variable content\n",
    "PROMPT = \"\"\"Please classify this email as either green or blue: {email}\"\"\"\n",
    "\n",
    "# Prefill for Claude's response, if any\n",
    "PREFILL = \"\"\n",
    "\n",
    "# Variable content stored as a list\n",
    "EMAILS = [\n",
    "    \"Hi -- My Mixmaster4000 is producing a strange noise when I operate it. It also smells a bit smoky and plasticky, like burning electronics.  I need a replacement.\", # (B) Broken or defective item\n",
    "    \"Can I use my Mixmaster 4000 to mix paint, or is it only meant for mixing food?\", # (A) Pre-sale question OR (D) Other (please explain)\n",
    "    \"I HAVE BEEN WAITING 4 MONTHS FOR MY MONTHLY CHARGES TO END AFTER CANCELLING!!  WTF IS GOING ON???\", # (C) Billing question\n",
    "    \"How did I get here I am not good with computer.  Halp.\" # (D) Other (please explain)\n",
    "]\n",
    "\n",
    "# Correct categorizations stored as a list of lists to accommodate the possibility of multiple correct categorizations per email\n",
    "ANSWERS = [\n",
    "    [\"B\"],\n",
    "    [\"A\",\"D\"],\n",
    "    [\"C\"],\n",
    "    [\"D\"]\n",
    "]\n",
    "\n",
    "# Dictionary of string values for each category to be used for regex grading\n",
    "REGEX_CATEGORIES = {\n",
    "    \"A\": \"<answer>A</answer>\",\n",
    "    \"B\": \"<answer>B</answer>\",\n",
    "    \"C\": \"<answer>C</answer>\",\n",
    "    \"D\": \"<answer>D</answer>\"\n",
    "}\n",
    "\n",
    "# Iterate through list of emails\n",
    "for i,email in enumerate(EMAILS):\n",
    "    \n",
    "    # Substitute the email text into the email placeholder variable\n",
    "    formatted_prompt = PROMPT.format(email=email)\n",
    "   \n",
    "    # Get Claude's response\n",
    "    response = get_completion(formatted_prompt, prefill=PREFILL)\n",
    "\n",
    "    # Grade Claude's response\n",
    "    grade = any([bool(re.search(REGEX_CATEGORIES[ans], response)) for ans in ANSWERS[i]])\n",
    "    \n",
    "    # Print Claude's response\n",
    "    print(\"--------------------------- Full prompt with variable substutions ---------------------------\")\n",
    "    print(\"USER TURN\")\n",
    "    print(formatted_prompt)\n",
    "    print(\"\\nASSISTANT TURN\")\n",
    "    print(PREFILL)\n",
    "    print(\"\\n------------------------------------- Claude's response -------------------------------------\")\n",
    "    print(response)\n",
    "    print(\"\\n------------------------------------------ GRADING ------------------------------------------\")\n",
    "    print(\"This exercise has been correctly solved:\", grade, \"\\n\\n\\n\\n\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "❓ If you want a hint, run the cell below!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(hints.exercise_6_2_hint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Congrats!\n",
    "\n",
    "If you've solved all exercises up until this point, you're ready to move to the next chapter. Happy prompting!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Example Playground\n",
    "\n",
    "This is an area for you to experiment freely with the prompt examples shown in this lesson and tweak prompts to see how it may affect Claude's responses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prompt\n",
    "PROMPT = \"\"\"Is this movie review sentiment positive or negative?\n",
    "\n",
    "This movie blew my mind with its freshness and originality. In totally unrelated news, I have been living under a rock since the year 1900.\"\"\"\n",
    "\n",
    "# Print Claude's response\n",
    "print(get_completion(PROMPT))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# System prompt\n",
    "SYSTEM_PROMPT = \"You are a savvy reader of movie reviews.\"\n",
    "\n",
    "# Prompt\n",
    "PROMPT = \"\"\"Is this review sentiment positive or negative? First, write the best arguments for each side in <positive-argument> and <negative-argument> XML tags, then answer.\n",
    "\n",
    "This movie blew my mind with its freshness and originality. In totally unrelated news, I have been living under a rock since 1900.\"\"\"\n",
    "\n",
    "# Print Claude's response\n",
    "print(get_completion(PROMPT, SYSTEM_PROMPT))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prompt\n",
    "PROMPT = \"\"\"Is this review sentiment negative or positive? First write the best arguments for each side in <negative-argument> and <positive-argument> XML tags, then answer.\n",
    "\n",
    "This movie blew my mind with its freshness and originality. Unrelatedly, I have been living under a rock since 1900.\"\"\"\n",
    "\n",
    "# Print Claude's response\n",
    "print(get_completion(PROMPT))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prompt\n",
    "PROMPT = \"Name a famous movie starring an actor who was born in the year 1956.\"\n",
    "\n",
    "# Print Claude's response\n",
    "print(get_completion(PROMPT))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Prompt\n",
    "PROMPT = \"Name a famous movie starring an actor who was born in the year 1956. First brainstorm about some actors and their birth years in <brainstorm> tags, then give your answer.\"\n",
    "\n",
    "# Print Claude's response\n",
    "print(get_completion(PROMPT))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "",
   "name": ""
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}