10_4_3_Reading_Charts_Graphs_Powerpoints.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix 10.4.3: Working with Charts, Graphs, and Slide Decks\n",
    "Claude is highly capable of working with charts, graphs, and broader slide decks. Depending on your use case, there are a number of tips and tricks that you may want to take advantage of. This recipe will show you common patterns for using Claude with these materials."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Charts and Graphs\n",
    "For the most part, using claude with charts and graphs is simple. Let's walk through how to ingest them and pass them to Claude, as well as some common tips to improve your results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ingestion and calling the Claude API\n",
    "The best way to pass Claude charts and graphs is to take advantage of its vision capabilities. That is, give Claude an image of the chart or graph, along with a text question about it. While all versions of Claude can accept images, Sonnet and Opus are our recommended models for data-heavy image tasks. Let's get started using Sonnet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install -qUr requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import json\n",
    "import base64\n",
    "from datetime import datetime\n",
    "from IPython.display import Image\n",
    "from botocore.exceptions import ClientError\n",
    "\n",
    "session = boto3.Session()\n",
    "region = session.region_name\n",
    "\n",
    "#modelId = 'anthropic.claude-3-sonnet-20240229-v1:0'\n",
    "modelId = 'anthropic.claude-3-haiku-20240307-v1:0'\n",
    "\n",
    "print(f'Using modelId: {modelId}')\n",
    "print('Using region: ', region)\n",
    "\n",
    "bedrock_client = boto3.client(service_name = 'bedrock-runtime', region_name = region,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_completion(messages):\n",
    "    converse_api_params = {\n",
    "        \"modelId\": modelId,\n",
    "        \"messages\": messages,\n",
    "    }\n",
    "    response = bedrock_client.converse(**converse_api_params)\n",
    "    # Extract the generated text content from the response\n",
    "    return response['output']['message']['content'][0]['text']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To start, we'll need an image. We will be using the .png image located at cvna_2021_annual_report_image.png.\n",
    "# Start by reading in the image and encoding it as base64.\n",
    "with open(\"./images/reading_charts_graphs/cvna_2021_annual_report_image.png\", \"rb\") as f:\n",
    "    image_file = f.read()\n",
    "\n",
    "# Let's also see the image for ourself\n",
    "Image(filename='./images/reading_charts_graphs/cvna_2021_annual_report_image.png') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how we can pass this image to the model alongside a simple question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"text\": \"What's in this image? Answer in a single sentence.\"},\n",
    "            {\"image\": {\n",
    "                \"format\": 'png',\n",
    "                \"source\": {\"bytes\": image_file }\n",
    "                },\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "print(get_completion(messages))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's pretty good! Now let's ask it some more useful questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "questions = [\n",
    "    \"What was CVNA revenue in 2020?\",\n",
    "    \"How many additional markets has Carvana added since 2014?\",\n",
    "    \"What was 2016 revenue per retail unit sold?\"\n",
    "]\n",
    "\n",
    "for index, question in enumerate(questions):\n",
    "    messages = [\n",
    "        {\n",
    "            \"role\": 'user',\n",
    "            \"content\": [\n",
    "                {\"text\": \"What's in this image? Answer in a single sentence.\"},\n",
    "                {\"image\": {\n",
    "                    \"format\": 'png',\n",
    "                    \"source\": {\"bytes\": image_file }\n",
    "                    },\n",
    "                }\n",
    "            ]\n",
    "        }\n",
    "    ]\n",
    "    \n",
    "    print(f\"\\n----------Question {index+1}----------\")\n",
    "    print(get_completion(messages))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, Claude is capable of answering fairly detailed questions about charts and graphs. However, there are some tips and tricks that will help you get the most out of it.\n",
    "- Sometimes Claude's arithmetic capabilities get in the way. You'll notice that if you sample the third question above it will occasionally output an incorrect final answer because it messes up the arithmetic. Consider providing Claude with a calculator tool to ensure it doesn't make these types of mistakes.\n",
    "- With super complicated charts and graphs, we can ask Claude to \"First describe every data point you see in the image\" as a way to elicit similar improvements to what we seen in traditional Chain of Thought.\n",
    "- Claude occasionally struggles with charts that depend on lots of colors to convey information, such as grouped bar charts with many groups. Asking Claude to first identify the colors in your graph using HEX codes can boost its accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Slide Decks\n",
    "Now that we know Claude is a charts and graphs wizard, it is only logical that we extend it to the true home of charts and graphs - slide decks!\n",
    "\n",
    "Slides represent a critical source of information for many domains, including financial services. While you *can* use packages like PyPDF to extract text from slide decks, their chart/graph heavy nature often makes this a poor choice as models will struggle to access the information they actually need. Vision can be a great replacement as a result. In this section we will go over how to use vision Claude to review slide decks, and how to deal with some common pitfalls of this approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best way to get a typical slide deck into claude is to download it as a PDF and then convert each pdf page to an image. Here's how you can accomplish this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#%pip install PyMuPDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "import io\n",
    "import fitz\n",
    "\n",
    "def pdf_to_pngs(pdf_path, quality=75, max_size=(1024, 1024)):\n",
    "    \"\"\"\n",
    "    Converts a PDF file to a list of PNG images.\n",
    "\n",
    "    Args:\n",
    "        pdf_path (str): The path to the PDF file.\n",
    "        quality (int, optional): The quality of the output PNG images (default is 75).\n",
    "        max_size (tuple, optional): The maximum size of the output images (default is (1024, 1024)).\n",
    "\n",
    "    Returns:\n",
    "        list: A list of PNG images as bytes.\n",
    "    \"\"\"\n",
    "    # Open the PDF file\n",
    "    doc = fitz.open(pdf_path)\n",
    "    pdf_to_png_images = []\n",
    "\n",
    "    # Iterate through each page of the PDF\n",
    "    for page_num in range(doc.page_count):\n",
    "        # Load the page\n",
    "        page = doc.load_page(page_num)\n",
    "\n",
    "        # Render the page as a PNG image\n",
    "        pix = page.get_pixmap(matrix=fitz.Matrix(300/72, 300/72))\n",
    "\n",
    "        # Save the PNG image\n",
    "        output_path = f\"./images/reading_charts_graphs/slides/page_{page_num+1}.png\"\n",
    "        pix.save(output_path)\n",
    "\n",
    "        # Open the saved image using PIL\n",
    "        image = Image.open(output_path)\n",
    "\n",
    "        # Resize the image if it exceeds the maximum size\n",
    "        if image.size[0] > max_size[0] or image.size[1] > max_size[1]:\n",
    "            image.thumbnail(max_size, Image.Resampling.LANCZOS)\n",
    "\n",
    "        # Convert the PIL image to bytes\n",
    "        image_data = io.BytesIO()\n",
    "        image.save(image_data, format='PNG', optimize=True, quality=quality)\n",
    "        image_data.seek(0)\n",
    "        pdf_to_png_image = image_data.getvalue()\n",
    "\n",
    "        # Append the PNG image bytes to the list\n",
    "        pdf_to_png_images.append(pdf_to_png_image)\n",
    "\n",
    "    # Close the PDF document\n",
    "    doc.close()\n",
    "\n",
    "    return pdf_to_png_images\n",
    "\n",
    "# Specify the path to the PDF file\n",
    "pdf_path = './images/reading_charts_graphs/twilio_q4_2023.pdf'\n",
    "# Call the pdf_to_pngs function to convert the PDF to PNG images\n",
    "pdf_pngs = pdf_to_pngs(pdf_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Now let's pass the first 20 of these images (in order) to Claude at once and ask it a question about the deck. \n",
    "# Why 20? Currently, the Anthropic API only allows you to pass in a maximum of 20 images.\n",
    "# While this number will likely increase over time, we have some helpful tips for how to manage it later in this recipe.\n",
    "\n",
    "content = [{\"image\": {\"format\": 'png', \"source\": {\"bytes\": pdf_png}}} for pdf_png in pdf_pngs[:20]]\n",
    "\n",
    "question = \"What was Twilio y/y revenue growth for fiscal year 2023?\"\n",
    "#question = \"What was the non-GAAP gross margin?\"\n",
    "\n",
    "# Append the question to our images\n",
    "content.append({\"text\": question})\n",
    "\n",
    "messages = [\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": content\n",
    "    }\n",
    "]\n",
    "\n",
    "print(get_completion(messages))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This approach is a great way to get started, and for some use cases offers the best performance. However, there are some limitations.\n",
    "- You can only include up to 20 images (we intend to increase this limit over time)\n",
    "- If you are using slide content as part of RAG, introducing images into your embeddings can cause problems\n",
    "\n",
    "Luckily, we can take advantage of Claude's vision capabilities to get a much higher quality representation of the slide deck **in text form** than normal pdf transciption allows.\n",
    "\n",
    "We find the best way to do this is to ask Claude to sequentially narrate the deck from start to finish, passing it the current slide and its prior narration. Let's see how."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define two functions that allow us to craft prompts for narrating our slide deck. We would adjut these prompts based on the nature of the deck, but keep the structure largely the same.\n",
    "def build_previous_slides_prompt(previous_slide_narratives):\n",
    "    prompt = '\\n'.join([f\"<slide_narration id={index+1}>\\n{narrative}\\n</slide_narration>\" for index, narrative in enumerate(previous_slide_narratives)])\n",
    "    return prompt\n",
    "\n",
    "def build_slides_narration_prompt(previous_slide_narratives):\n",
    "    if len(previous_slide_narratives) == 0:\n",
    "        prompt = \"\"\"You are the Twilio CFO, narrating your Q4 2023 earnings presentation.\n",
    "\n",
    "You are currently on slide 1, shown in the image.\n",
    "Please narrate this page from Twilio's Q4 2023 Earnings Presentation as if you were the presenter. Do not talk about any things, especially acronyms, if you are not exactly sure you know what they mean. Do not discuss anything not explicitly seen on this slide as there are more slides to narrate later that will likely cover that material.\n",
    "Do not leave any details un-narrated as some of your viewers are vision-impaired, so if you don't narrate every number they won't know the number.\n",
    "\n",
    "Put your narration in <narration> tags.\"\"\"\n",
    "\n",
    "    else:\n",
    "        prompt = f\"\"\"You are the Twilio CFO, narrating your Q4 2023 earnings presentation. So far, here is your narration from previous slides:\n",
    "<previous_slide_narrations>\n",
    "{build_previous_slides_prompt(previous_slide_narratives)}\n",
    "</previous_slide_narrations>\n",
    "\n",
    "You are currently on slide {len(previous_slide_narratives)+1}, shown in the image.\n",
    "Please narrate this page from Twilio's Q4 2023 Earnings Presentation as if you were the presenter, accounting for what you have already said on previous slides. Do not talk about any things, especially acronyms, if you are not exactly sure you know what they mean. Do not discuss anything not explicitly seen on this slide as there are more slides to narrate later that will likely cover that material.\n",
    "Do not leave any details un-narrated as some of your viewers are vision-impaired, so if you don't narrate every number they won't know the number.\n",
    "\n",
    "Use excruciating detail.\n",
    "\n",
    "Put your narration in <narration> tags.\"\"\"\n",
    "    \n",
    "    return prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Now we use our functions to narrate the entire deck. Note that this may take a few minutes to run (often up to 10).\n",
    "import re\n",
    "from tqdm import tqdm\n",
    "previous_slide_narratives = []\n",
    "for i, pdf_png in tqdm(enumerate(pdf_pngs)):\n",
    "    messages = [\n",
    "        {\n",
    "            \"role\": 'user',\n",
    "            \"content\": [\n",
    "                {\"text\": build_slides_narration_prompt(previous_slide_narratives)},\n",
    "                {\"image\": {\n",
    "                    \"format\": 'jpeg',\n",
    "                    \"source\": {\"bytes\": pdf_png }\n",
    "                    },\n",
    "                }\n",
    "            ]\n",
    "        }\n",
    "    ]\n",
    "    completion = get_completion(messages)\n",
    "\n",
    "\n",
    "    pattern = r\"<narration>(.*?)</narration>\"\n",
    "    match = re.search(pattern, completion.strip(), re.DOTALL)\n",
    "    if match:\n",
    "        narration = match.group(1)\n",
    "    else:\n",
    "        raise ValueError(\"No narration available.\")\n",
    "    \n",
    "    previous_slide_narratives.append(narration)\n",
    "    # If you want to see the narration we produced, uncomment the below line\n",
    "    # print(narration)\n",
    "\n",
    "slide_narration = build_previous_slides_prompt(previous_slide_narratives)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a text-based narration (it's far from perfect but it's pretty good), we have the ability to use this deck with any text-only workflow. Including vector search!\n",
    "\n",
    "As a final sanity check, let's ask a few questions of our narration-only setup!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "questions = [\n",
    "    \"What percentage of q4 total revenue was the Segment business line?\",\n",
    "    \"Has the rate of growth of quarterly revenue been increasing or decreasing? Give just an answer.\",\n",
    "    \"What was acquisition revenue for the year ended december 31, 2023 (including negative revenues)?\"\n",
    "]\n",
    "\n",
    "for index, question in enumerate(questions):\n",
    "    prompt = f\"\"\"You are an expert financial analyst analyzing a transcript of Twilio's earnings call.\n",
    "Here is the transcript:\n",
    "<transcript>\n",
    "{slide_narration}\n",
    "</transcript>\n",
    "\n",
    "Please answer the following question:\n",
    "<question>\n",
    "{question}\n",
    "</question>\"\"\"\n",
    "    messages = [\n",
    "        {\n",
    "            \"role\": 'user',\n",
    "            \"content\": [\n",
    "                {\"text\": prompt},\n",
    "            ]\n",
    "        }\n",
    "    ]\n",
    "\n",
    "    print(f\"\\n----------Question {index+1}----------\")\n",
    "    print(get_completion(messages))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks good! With these techniques at your side, you are ready to start applying models to chart and graph heavy content like slide decks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}