02_open_ai.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The OpenAI API\n",
    "\n",
    "In this section we will cover the basics of using the OpenAI API, including:\n",
    "- Chat Completions\n",
    "- Streaming\n",
    "- Vision input\n",
    "\n",
    "The beauty of the OpenAI API is that is very simple to use.\n",
    "\n",
    "In your environment you should have a file called `.env` with the following:\n",
    "\n",
    "```bash\n",
    "OPENAI_API_KEY=\"sk-proj-1234567890\"\n",
    "```\n",
    "\n",
    "We will give you this key in the workshop. __The key will be deactivated after the workshop!__\n",
    "\n",
    "You can then grab the key using python:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "import dotenv\n",
    "import os\n",
    "from rich import print as rprint # for making fancy outputs\n",
    "\n",
    "dotenv.load_dotenv()\n",
    "\n",
    "client = OpenAI()\n",
    "\n",
    "OPENAI_API_KEY = os.getenv(\"OPENAI_API_KEY\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chat Completions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling a model is simple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Silent paws tread soft,  \n",
      "Moonlight gleams on sharpened claws—  \n",
      "Honor in each pounce.  \n"
     ]
    }
   ],
   "source": [
    "system_prompt = \"You are Matsuo Basho, the great Japanese haiku poet.\"\n",
    "user_query = \"Can you give me a haiku about a Samurai cat.\"\n",
    "\n",
    "response = client.chat.completions.create(\n",
    "  model=\"gpt-4o-mini\",\n",
    "  messages=[\n",
    "    {\"role\": \"system\", \"content\": system_prompt},\n",
    "    {\"role\": \"user\", \"content\": user_query},\n",
    "  ],\n",
    "  max_tokens=128\n",
    ")\n",
    "\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Purrfect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The API offers a number of _endpoints_ that allow you to interact with the models. The main one that we will cover here is the `/chat/completions` endpoint. This endpoint allows you to interact with the model in a conversational manner.\n",
    "\n",
    "Only 2 arguments are actually required for this endpoint:\n",
    "\n",
    "- `model: str` The model to use. For OpenAI, this includes:\n",
    "    - `'gpt-3.5-turbo'`\n",
    "\n",
    "    - `'gpt-4'`\n",
    "\n",
    "    - `'gpt-4o'`\n",
    "\n",
    "    - `'gpt-4o-mini'`\n",
    "    \n",
    "    - Any fine-tuned versions of these models.\n",
    "    \n",
    "    - Many specific versions of the above models.\n",
    "\n",
    "- `messages: list` A list of messages that the model should use to generate a response. Each entry in the list of messages comes in the form:\n",
    "\n",
    "```python\n",
    "{\"role\": \"<role>\", \"content\": \"<content>\", \"name\": \"<name>\"}\n",
    "```\n",
    "\n",
    "Where `<role>` can take one of the following forms:\n",
    "\n",
    "- `'system'` This is a system level prompt, designed to guide the conversation. For example: \n",
    "\n",
    "_\"You are a customer service bot.\"_\n",
    "\n",
    "- `'user'` This is direct input from the user. For example: \n",
    "\n",
    "_\"How do I reset my password?\"_\n",
    "\n",
    "- `'assistant'` This is the response from the model. For example:\n",
    "\n",
    "_\"To reset your password, please visit our website and click on the 'Forgot Password' link.\"_\n",
    "\n",
    "So all of this fed into one message list would look like this:\n",
    "\n",
    "```python\n",
    "messages = [\n",
    "    {\"role\": \"system\", \"content\": \"You are a customer service bot.\"},\n",
    "    {\"role\": \"user\", \"content\": \"How do I reset my password?\"},\n",
    "    {\"role\": \"assistant\", \"content\": \"To reset your password, please visit our website and click on the 'Forgot Password' link.\"}\n",
    "]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional arguments\n",
    "The `/chat/completions` endpoint also accepts a number of additional arguments that can be used to alter the response. These include (arguments are listed with their default values if applicable):\n",
    "\n",
    "- `max_tokens: int` The maximum number of tokens to generate in the response. Important to stop the model from generating too much text and racking up a huge bill.\n",
    "\n",
    "- `n: int = 1` The number of completions to generate. This is useful when you want to generate multiple completions and select the best one. You'll be charged for the _**total**_ number of tokens generated across all completions, so be careful with setting this too high.\n",
    "\n",
    "- `temperature: float = 1.0` The temperature of the model, ranging from 0.0 to 2. Use low values for deterministic responses, and high values for more creative responses.\n",
    "\n",
    "- `top_p: float = 1.0` The probability of sampling from the top `p` tokens. This is useful for controlling the diversity of the responses. Setting this to a higher values means the model is more likely to sample from a wider range of tokens.\n",
    "\n",
    "- `logprobs: bool = False` Whether to return the log probabilities of the tokens generated. This is useful when you want to understand how the model is making decisions.\n",
    "\n",
    "- `logit_bias: dict` A dictionary of logit biases to apply to the tokens. This is useful when you want to guide the model towards generating certain types of responses.\n",
    "\n",
    "- `response_format: str` The format of the response. We will cover this later...\n",
    "\n",
    "- `stream: bool = False` Whether to stream the response back to the client. This is useful when you want to get the response in real-time. Nobody likes to sit and wait for a response. Seeing the text generated as and when it is ready is a much better user experience.\n",
    "\n",
    "For a full list of arguments, check out the [OpenAI API documentation](https://platform.openai.com/docs/api-reference/chat/create)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Available models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we have used model `gpt-4o-mini`, but there are a range of models available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model(id='gpt-3.5-turbo', created=1677610602, object='model', owned_by='openai')\n",
      "Model(id='gpt-3.5-turbo-0125', created=1706048358, object='model', owned_by='system')\n",
      "Model(id='dall-e-2', created=1698798177, object='model', owned_by='system')\n",
      "Model(id='gpt-4-1106-preview', created=1698957206, object='model', owned_by='system')\n",
      "Model(id='tts-1-hd-1106', created=1699053533, object='model', owned_by='system')\n",
      "Model(id='tts-1-hd', created=1699046015, object='model', owned_by='system')\n",
      "Model(id='dall-e-3', created=1698785189, object='model', owned_by='system')\n",
      "Model(id='whisper-1', created=1677532384, object='model', owned_by='openai-internal')\n",
      "Model(id='text-embedding-3-large', created=1705953180, object='model', owned_by='system')\n",
      "Model(id='text-embedding-3-small', created=1705948997, object='model', owned_by='system')\n",
      "Model(id='text-embedding-ada-002', created=1671217299, object='model', owned_by='openai-internal')\n",
      "Model(id='gpt-4-turbo', created=1712361441, object='model', owned_by='system')\n",
      "Model(id='gpt-4o-2024-05-13', created=1715368132, object='model', owned_by='system')\n",
      "Model(id='gpt-4-0125-preview', created=1706037612, object='model', owned_by='system')\n",
      "Model(id='gpt-4-turbo-2024-04-09', created=1712601677, object='model', owned_by='system')\n",
      "Model(id='gpt-4-turbo-preview', created=1706037777, object='model', owned_by='system')\n",
      "Model(id='gpt-3.5-turbo-16k', created=1683758102, object='model', owned_by='openai-internal')\n",
      "Model(id='gpt-4o', created=1715367049, object='model', owned_by='system')\n",
      "Model(id='tts-1', created=1681940951, object='model', owned_by='openai-internal')\n",
      "Model(id='gpt-3.5-turbo-1106', created=1698959748, object='model', owned_by='system')\n",
      "Model(id='tts-1-1106', created=1699053241, object='model', owned_by='system')\n",
      "Model(id='gpt-3.5-turbo-instruct-0914', created=1694122472, object='model', owned_by='system')\n",
      "Model(id='gpt-4', created=1687882411, object='model', owned_by='openai')\n",
      "Model(id='gpt-4-0613', created=1686588896, object='model', owned_by='openai')\n",
      "Model(id='gpt-3.5-turbo-instruct', created=1692901427, object='model', owned_by='system')\n",
      "Model(id='chatgpt-4o-latest', created=1723515131, object='model', owned_by='system')\n",
      "Model(id='babbage-002', created=1692634615, object='model', owned_by='system')\n",
      "Model(id='davinci-002', created=1692634301, object='model', owned_by='system')\n",
      "Model(id='gpt-4o-mini-2024-07-18', created=1721172717, object='model', owned_by='system')\n",
      "Model(id='gpt-4o-mini', created=1721172741, object='model', owned_by='system')\n",
      "Model(id='gpt-4o-2024-08-06', created=1722814719, object='model', owned_by='system')\n"
     ]
    }
   ],
   "source": [
    "for model in client.models.list():\n",
    "    print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As of writing `gpt-4o-2024-08-06` is the current best offering. But we'll stick with `gpt-4o-mini`, because it is cheaper and still highly capable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The response object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the `response` object?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletion</span><span style=\"font-weight: bold\">(</span>\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'chatcmpl-A5yRAL35y156KgXeY6uUBbLNHD4qh'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">choices</span>=<span style=\"font-weight: bold\">[</span>\n",
       "        <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Choice</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">finish_reason</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'stop'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">index</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">logprobs</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">message</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionMessage</span><span style=\"font-weight: bold\">(</span>\n",
       "                <span style=\"color: #808000; text-decoration-color: #808000\">content</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'Silent pawsteps glide,  \\nMoonlight dances on the blade—  \\nFeline honor blinds.  '</span>,\n",
       "                <span style=\"color: #808000; text-decoration-color: #808000\">refusal</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "                <span style=\"color: #808000; text-decoration-color: #808000\">role</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'assistant'</span>,\n",
       "                <span style=\"color: #808000; text-decoration-color: #808000\">function_call</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "                <span style=\"color: #808000; text-decoration-color: #808000\">tool_calls</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
       "            <span style=\"font-weight: bold\">)</span>\n",
       "        <span style=\"font-weight: bold\">)</span>\n",
       "    <span style=\"font-weight: bold\">]</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">created</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1725987324</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">model</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'gpt-4o-mini-2024-07-18'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">object</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'chat.completion'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">service_tier</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">system_fingerprint</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'fp_483d39d857'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">usage</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">CompletionUsage</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">completion_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">20</span>, <span style=\"color: #808000; text-decoration-color: #808000\">prompt_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">37</span>, <span style=\"color: #808000; text-decoration-color: #808000\">total_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">57</span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mChatCompletion\u001b[0m\u001b[1m(\u001b[0m\n",
       "    \u001b[33mid\u001b[0m=\u001b[32m'chatcmpl-A5yRAL35y156KgXeY6uUBbLNHD4qh'\u001b[0m,\n",
       "    \u001b[33mchoices\u001b[0m=\u001b[1m[\u001b[0m\n",
       "        \u001b[1;35mChoice\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[33mfinish_reason\u001b[0m=\u001b[32m'stop'\u001b[0m,\n",
       "            \u001b[33mindex\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "            \u001b[33mlogprobs\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "            \u001b[33mmessage\u001b[0m=\u001b[1;35mChatCompletionMessage\u001b[0m\u001b[1m(\u001b[0m\n",
       "                \u001b[33mcontent\u001b[0m=\u001b[32m'Silent pawsteps glide,  \\nMoonlight dances on the blade—  \\nFeline honor blinds.  '\u001b[0m,\n",
       "                \u001b[33mrefusal\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "                \u001b[33mrole\u001b[0m=\u001b[32m'assistant'\u001b[0m,\n",
       "                \u001b[33mfunction_call\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "                \u001b[33mtool_calls\u001b[0m=\u001b[3;35mNone\u001b[0m\n",
       "            \u001b[1m)\u001b[0m\n",
       "        \u001b[1m)\u001b[0m\n",
       "    \u001b[1m]\u001b[0m,\n",
       "    \u001b[33mcreated\u001b[0m=\u001b[1;36m1725987324\u001b[0m,\n",
       "    \u001b[33mmodel\u001b[0m=\u001b[32m'gpt-4o-mini-2024-07-18'\u001b[0m,\n",
       "    \u001b[33mobject\u001b[0m=\u001b[32m'chat.completion'\u001b[0m,\n",
       "    \u001b[33mservice_tier\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "    \u001b[33msystem_fingerprint\u001b[0m=\u001b[32m'fp_483d39d857'\u001b[0m,\n",
       "    \u001b[33musage\u001b[0m=\u001b[1;35mCompletionUsage\u001b[0m\u001b[1m(\u001b[0m\u001b[33mcompletion_tokens\u001b[0m=\u001b[1;36m20\u001b[0m, \u001b[33mprompt_tokens\u001b[0m=\u001b[1;36m37\u001b[0m, \u001b[33mtotal_tokens\u001b[0m=\u001b[1;36m57\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rprint(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is some useful stuff in here, apart from the `content` property, such as the token usage. You might notice some other things too, like `function_call` and `tool_calls`. These are specific to OpenAI models, and not every model supports function calling or tools, so we won't cover them. We can achieve many of the same effects without them anyway."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Streaming a response\n",
    "\n",
    "Streaming a response is mainly for user experience. It allows the user to see the response as it comes in, rather than waiting for the whole response to come in. For many applications, this might not be necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Silent paws in dusk,  \n",
      "Moonlit blade gleams in the night—  \n",
      "Fierce heart, whiskers twitch."
     ]
    }
   ],
   "source": [
    "response = client.chat.completions.create(\n",
    "  model=\"gpt-4o-mini\",\n",
    "  messages=[\n",
    "    {\"role\": \"system\", \"content\": system_prompt},\n",
    "    {\"role\": \"user\", \"content\": user_query},\n",
    "  ],\n",
    "  max_tokens=128,\n",
    "  stream=True\n",
    ")\n",
    "\n",
    "for chunk in response:\n",
    "    if chunk.choices[0].delta.content is not None:\n",
    "        print(chunk.choices[0].delta.content, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All this really does is create a streaming object, which acts like a generator. We can then print the chunk as it comes in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vision input\n",
    "A huge draw of OpenAI models is the ability to input vision data. This is useful for a wide range of applications, including:\n",
    "- Image captioning\n",
    "- Object detection\n",
    "- Face recognition\n",
    "- Image generation\n",
    "\n",
    "Let's try an example of inputting an image. First we need to look at the image:\n",
    "\n",
    "![plot](./imgs/figure.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the caption from this figure:\n",
    "\n",
    "<table><tr><td>\n",
    "\n",
    "**Fig. 2 Spatial and temporal self-similarity and correlation in switching activity.**\n",
    "\n",
    "_(A) Percolating devices produce complex patterns of switching events that are self-similar in nature. The top panel contains 2400 s of data, with the bottom panels showing segments of the data with 10, 100, and 1000 times greater temporal magnification and with 3, 9, and 27 times greater magnification on the vertical scale (units of G0 = 2e2/h, the quantum of conductance, are used for convenience). The activity patterns appear qualitatively similar on multiple different time scales. (B and E) The probability density function (PDF) for changes in total network conductance, P(ΔG), resulting from switching activity exhibits heavy-tailed probability distributions. (C and F) IEIs follow power law distributions, suggestive of correlations between events. (D and G) Further evidence of temporal correlation between events is given by the autocorrelation function (ACF) of the switching activity (red), which decays as a power law over several decades. When the IEI sequence is shuffled (gray), the correlations between events are destroyed, resulting in a significant increase in slope in the ACF. The data shown in (B) to (D) (sample I) were obtained with our standard (slow) sampling rate, and the data shown in (E) to (G) (sample II) were measured 1000 times faster (see Materials and Methods), providing further evidence for self-similarity._\n",
    "</td></tr></table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This figure is taken from _[Avalanches and criticality in self-organized nanoscale networks, Mallinson et al., 2019.](https://www.science.org/doi/10.1126/sciadv.aaw8438)_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's use the OpenAI vision model to generate a caption for this figure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = (\n",
    "    \"This figure is a caption from a paper entitled Avalanches and criticality in self-organized nanoscale networks. \"\n",
    "    \"Please provide a caption for this figure. \"\n",
    "    \"You should describe the figure, grouping the panels where appropriate. \"\n",
    "    \"Feel free to make any inferences you need to.\"\n",
    "\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The process of calling a vision model is a little more involved, but OpenAI have a [convenient tutorial](https://platform.openai.com/docs/guides/vision) on how to do this.\n",
    "\n",
    "Essentially we need to first convert the image to a base64 string. We can then pass this to the OpenAI API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "import base64\n",
    "import requests\n",
    "\n",
    "# Function to encode the image\n",
    "def encode_image(image_path):\n",
    "  with open(image_path, \"rb\") as image_file:\n",
    "    return base64.b64encode(image_file.read()).decode('utf-8')\n",
    "\n",
    "# Path to your image\n",
    "image_path = \"imgs/figure.jpeg\"\n",
    "\n",
    "\n",
    "def get_image_caption(image_path, prompt):\n",
    "  # Getting the base64 string\n",
    "  base64_image = encode_image(image_path)\n",
    "\n",
    "  headers = {\n",
    "    \"Content-Type\": \"application/json\",\n",
    "    \"Authorization\": f\"Bearer {OPENAI_API_KEY}\"\n",
    "  }\n",
    "\n",
    "  payload = {\n",
    "    \"model\": \"gpt-4o-mini\",\n",
    "    \"messages\": [\n",
    "      {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "          {\n",
    "            \"type\": \"text\",\n",
    "            \"text\": prompt\n",
    "          },\n",
    "          {\n",
    "            \"type\": \"image_url\",\n",
    "            \"image_url\": {\n",
    "              \"url\": f\"data:image/jpeg;base64,{base64_image}\"\n",
    "            }\n",
    "          }\n",
    "        ]\n",
    "      }\n",
    "    ],\n",
    "    \"max_tokens\": 512\n",
    "  }\n",
    "\n",
    "  response = requests.post(\"https://api.openai.com/v1/chat/completions\", headers=headers, json=payload)\n",
    "\n",
    "  return response.json()['choices'][0]['message']['content']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**Figure Caption:**\n",
      "\n",
      "**Figure X:** Analysis of avalanche dynamics and criticality in self-organized nanoscale networks. \n",
      "\n",
      "**(A)** Time series data showing the fluctuations in conductance, \\(\\Delta G(G_0)\\), over varying observation periods (100 s, 10 s, 1 s, 0.1 s). The four panels illustrate distinct behavior and amplitude of fluctuations as time scales decrease. \n",
      "\n",
      "**(B) and (E)** Power law distributions, \\(P(\\Delta G)\\), of the amplitude of conductance fluctuations are presented on logarithmic scales, revealing power-law exponents \\(\\Delta G \\approx -2.59\\) (B) and \\(\\Delta G \\approx -2.36\\) (E). \n",
      "\n",
      "**(C) and (F)** Temporal distributions, \\(P(t)\\), showing the frequency of events over time. The corresponding power law exponents are \\(t \\approx -1.39\\) (C) and \\(t \\approx -1.30\\) (F), indicating scale-invariant behavior.\n",
      "\n",
      "**(D) and (G)** Characteristic avalanche sizes \\(A(t)\\) as a function of time, indicating distinct scaling regimes with exponents \\(t \\approx -0.19\\) and \\(t \\approx -0.23\\) for the upper and lower panels, respectively. The gray data in (D) suggests a crossover behavior in larger time scales. \n",
      "\n",
      "These results highlight the critical dynamics and self-organized behavior of the nanoscale networks under study.\n"
     ]
    }
   ],
   "source": [
    "caption = get_image_caption(image_path, prompt)\n",
    "print(caption)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I mean, I don't know about you, but I think that's incredible. Let's consider what it has done:\n",
    "- Correctly grouped the panels in the same way the real caption did.\n",
    "- Provided information on the observation periods.\n",
    "- Drawn out the important information, such as critical exponents.\n",
    "- Made the link between power law distributions and scale-free behaviour.\n",
    "\n",
    "However, it has failed to provide information on temporal correlations, and it has not noticed the self-similarity in caption 1.\n",
    "\n",
    "But this is still quite impressive, and with more information we could potentially get some better captions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tools\n",
    "We can also give the model some tools to use - these are essentially just functions that we can call, and the role of the LLM is to generate the arguments to that function.\n",
    "\n",
    "Here is a simple example to do some maths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1382976"
     ]
    }
   ],
   "source": [
    "system_prompt = \"You are a helpful mathematician. You will only solve the problems given to you. Do not provide any additional information. Provide only the answer.\"\n",
    "user_query = \"What is 1056 * 1316?\"\n",
    "\n",
    "response = client.chat.completions.create(\n",
    "  model=\"gpt-4o-mini\",\n",
    "  messages=[\n",
    "    {\"role\": \"system\", \"content\": system_prompt},\n",
    "    {\"role\": \"user\", \"content\": user_query},\n",
    "  ],\n",
    "  max_tokens=256,\n",
    "  stream=True\n",
    ")\n",
    "\n",
    "for chunk in response:\n",
    "    if chunk.choices[0].delta.content is not None:\n",
    "        print(chunk.choices[0].delta.content, end=\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1389696"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1056 * 1316"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the LLM is not correct :(.\n",
    "\n",
    "To endow the model with \"tool use\", we add a function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def multiply(a: float, b: float) -> float:\n",
    "    return a * b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then provide the model with a _schema_, which is just a description of the function in dictionary form (or JSON):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "tool_schema = {\n",
    "    \"type\": \"function\",\n",
    "    \"function\": {\n",
    "        \"name\": \"multiply\",\n",
    "        \"description\": \"Given two floats, a and b, return the product of a and b.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"a\": {\n",
    "                    \"type\": \"number\",\n",
    "                    \"description\": \"The first number to multiply.\"\n",
    "                },\n",
    "                \"b\": {\n",
    "                    \"type\": \"number\",\n",
    "                    \"description\": \"The second number to multiply.\"\n",
    "                }\n",
    "            },\n",
    "            \"required\": [\"a\", \"b\"],\n",
    "            \"additionalProperties\": False,\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "tools = [\n",
    "    tool_schema\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we make function calls with an LLM, we have to let it know that it has access to one or more tools. We do this by passing in the `tools` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ChatCompletionMessageToolCall(id='call_1vXSH7jPzjHCLYBlccyzewuB', function=Function(arguments='{\"a\":1056,\"b\":1316}', name='multiply'), type='function')\n"
     ]
    }
   ],
   "source": [
    "response = client.chat.completions.create(\n",
    "    model=\"gpt-4o-mini\",\n",
    "    messages=[\n",
    "        {\"role\": \"system\", \"content\": system_prompt},\n",
    "        {\"role\": \"user\", \"content\": user_query},\n",
    "    ],\n",
    "    max_tokens=256,\n",
    "    tools=tools,\n",
    "    )\n",
    "\n",
    "print(response.choices[0].message.tool_calls[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now in our `response`, we have this extra part called `tool_calls` that we can extract information from - in this case the arguments to the `multiply` function.\n",
    "\n",
    "Note that you could achieve a similar result with appropriate prompting - e.g. \"Extract only the arguments to a function that multiplies two numbers.\"\n",
    "\n",
    "We unpack the actual arguments as a dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'a': 1056, 'b': 1316}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "tool_call = response.choices[0].message.tool_calls[0]\n",
    "arguments = json.loads(tool_call.function.arguments)\n",
    "print(arguments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we now feed the arguments into our `multiply` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1389696\n"
     ]
    }
   ],
   "source": [
    "result = multiply(**arguments)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now we have the answer. We can either just return this, or we can feed it back into the LLM. We need to provide the model with the `tool_calls[0].id`, so that it can associate response messages of the tool type with the correct tool call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1389696'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tool_call_result = {\n",
    "    \"role\": \"tool\",\n",
    "    \"content\": json.dumps({\n",
    "        \"a\" : arguments[\"a\"],\n",
    "        \"b\" : arguments[\"b\"],\n",
    "        \"result\": result\n",
    "    }),\n",
    "    \"tool_call_id\": response.choices[0].message.tool_calls[0].id\n",
    "}\n",
    "\n",
    "response = client.chat.completions.create(\n",
    "    model=\"gpt-4o-mini\",\n",
    "    messages=[\n",
    "        {\"role\": \"system\", \"content\": system_prompt},\n",
    "        {\"role\": \"user\", \"content\": user_query},\n",
    "        response.choices[0].message,\n",
    "        tool_call_result\n",
    "    ],\n",
    "    max_tokens=56\n",
    ")\n",
    "\n",
    "response.choices[0].message.content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is quite a lot of work to multiply two numbers, but of course the power comes when doing more complex tasks.\n",
    "\n",
    "And this brings to light an interesting contrast. People talk a lot about \"agents\" and \"tools\" and \"systems\", and when we interact with ChatGPT, we get a single coherent experience. Sometimes it is difficult to distinguish between what the LLM is doing, and what the software engineers have built around it in order to create this seemless experience."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}