third_party/phospho/cookbook_phospho_mistral_integration.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<center>\n",
        "    <p style=\"text-align:center\">\n",
        "        <img alt=\"phospho logo\" src=\"https://framerusercontent.com/images/MYJX4U9UCs4ojvzaR5VuwH7zjY.svg?scale-down-to=512\" width=\"200\"/>\n",
        "        <br>\n",
        "        <a href=\"https://docs.phospho.ai\">Docs</a>\n",
        "        |\n",
        "        <a href=\"https://platform.phospho.ai\">Platform</a>\n",
        "    </p>\n",
        "</center>\n",
        "<h1 align=\"center\">Cookbook: Mistral AI integration(Python)</h1>\n",
        "\n",
        "This cookbook provides step-by-step examples of integrating phospho with Mistral AI in Python.\n",
        "\n",
        "---\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Overview\n",
        "\n",
        "This notebook shows how to log your Mistral ChatBot conversations in phospho. After logging, we use phospho's analytics to compare two Mistral models: **mistral-7b** and **mistral-large-latest**.\n",
        "\n",
        "## What is phospho ?\n",
        "\n",
        "**phospho** is an open-source analytics platform for LLM applications, enabling automatic clustering to help discover use cases and topics. It provides no-code analytics to help you gain insights into how your app is being used, simplifying the process of understanding user behavior and trends.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Setup\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "First you need the **phospho** python module. It is compatible with Python >= 3.9.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TSlYS97ktOiL",
        "outputId": "9ceed277-8f58-44d7-ac12-d73853d3d028"
      },
      "outputs": [],
      "source": [
        "%%capture --no-display\n",
        "\n",
        "%pip install phospho==0.3.40\n",
        "%pip install mistralai==1.1.0\n",
        "%pip install python-dotenv==1.0.1\n",
        "%pip install tqdm==4.66.5"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "LijdNatbtOiN"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from dotenv import load_dotenv\n",
        "\n",
        "load_dotenv()\n",
        "\n",
        "MISTRAL_API_KEY = os.getenv(\"MISTRAL_API_KEY\")\n",
        "PHOSPHO_API_KEY = os.getenv(\"PHOSPHO_API_KEY\")\n",
        "PHOSPHO_PROJECT_ID = os.getenv(\"PHOSPHO_PROJECT_ID\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Log your messages\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Simple completion\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Initialize the phospho module with `phospho.init()`. By default, phospho will look for `PHOSPHO_API_KEY` and `PHOSPHO_PROJECT_ID` environment variables. You can also pass the `api_key` and `project_id` as parameters.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [],
      "source": [
        "import phospho\n",
        "\n",
        "phospho.init(api_key=PHOSPHO_API_KEY, project_id=PHOSPHO_PROJECT_ID)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Use `phospho.log` to log a query and an optional answer. Here is an example of a one query conversation with a Mistral assistant.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-tSGD7kUtOiO",
        "outputId": "2926209f-03be-4a74-cda4-af7457aee37b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "You're welcome! How can I assist you today? If you have any questions or need help with something, feel free to ask.\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "{'client_created_at': 1728548119,\n",
              " 'project_id': 'db99fc60ceac424fab371252f2f75485',\n",
              " 'session_id': None,\n",
              " 'task_id': 'cb3f05c74c9c4ce3ad3971d0e651c8b3',\n",
              " 'input': 'Thank you for your last message!',\n",
              " 'raw_input': 'Thank you for your last message!',\n",
              " 'raw_input_type_name': 'str',\n",
              " 'output': \"You're welcome! How can I assist you today? If you have any questions or need help with something, feel free to ask.\",\n",
              " 'raw_output': \"You're welcome! How can I assist you today? If you have any questions or need help with something, feel free to ask.\",\n",
              " 'raw_output_type_name': 'str',\n",
              " 'user_id': 'USER_ID',\n",
              " 'version_id': 'one_mistral_call'}"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import mistralai\n",
        "\n",
        "input_txt = \"Thank you for your last message!\"\n",
        "\n",
        "\n",
        "def one_mistral_call(input: str) -> str:\n",
        "    client = mistralai.Mistral(api_key=MISTRAL_API_KEY)\n",
        "    completion = client.chat.complete(\n",
        "        model=\"mistral-small-latest\",\n",
        "        messages=[\n",
        "            {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
        "            {\"role\": \"user\", \"content\": input},\n",
        "        ],\n",
        "    )\n",
        "\n",
        "    return completion.choices[0].message.content\n",
        "\n",
        "\n",
        "output_txt = one_mistral_call(input_txt)\n",
        "\n",
        "print(output_txt)\n",
        "\n",
        "phospho.log(\n",
        "    input=input_txt,\n",
        "    output=output_txt,\n",
        "    # You can log additonal metadata\n",
        "    user_id=\"USER_ID\", # eg: \"user-123\", \"anonymous\", ...\n",
        "    version_id=\"one_mistral_call\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "65Sd_VYXtOiO"
      },
      "source": [
        "Your message is now accessible on the platform, where you can store all your transcripts. While logging, sentiment evaluation and language detection are performed automatically on the message, along with any events you specify.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Message in platform](images/platform-display-message.png)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Streaming completion\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "phospho supports streamed outputs. Pass `stream=True` to `phospho.log` to handle streaming responses. When iterating over the response, phospho will automatically log each chunk until the iteration is completed. Here is an example with a simple Mistral chatBot.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "_m_Va98vtOiP",
        "outputId": "8251a3ca-82fd-4241-8a39-dce95175dd78"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <style>\n",
              "    pre {\n",
              "        white-space: pre-wrap;\n",
              "    }\n",
              "  </style>\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "  <style>\n",
              "    pre {\n",
              "        white-space: pre-wrap;\n",
              "    }\n",
              "  </style>\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "  <style>\n",
              "    pre {\n",
              "        white-space: pre-wrap;\n",
              "    }\n",
              "  </style>\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "  <style>\n",
              "    pre {\n",
              "        white-space: pre-wrap;\n",
              "    }\n",
              "  </style>\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import phospho\n",
        "import mistralai\n",
        "from phospho import MutableGenerator\n",
        "import uuid\n",
        "\n",
        "# This is used to make the chat prettier\n",
        "from IPython.display import HTML, display\n",
        "\n",
        "\n",
        "def set_css():\n",
        "    display(\n",
        "        HTML(\"\"\"\n",
        "  <style>\n",
        "    pre {\n",
        "        white-space: pre-wrap;\n",
        "    }\n",
        "  </style>\n",
        "  \"\"\")\n",
        "    )\n",
        "\n",
        "\n",
        "get_ipython().events.register(\"pre_run_cell\", set_css)\n",
        "\n",
        "\n",
        "# This is your chatbot\n",
        "def simple_mistral_chat():\n",
        "    phospho.init(api_key=PHOSPHO_API_KEY, project_id=PHOSPHO_PROJECT_ID)\n",
        "    client = mistralai.Mistral(api_key=MISTRAL_API_KEY)\n",
        "    messages = []\n",
        "    # I want to create a random session id\n",
        "    session_id = str(uuid.uuid4())\n",
        "\n",
        "    print(\"Ask anything (Type /exit to quit)\", end=\"\")\n",
        "\n",
        "    while True:\n",
        "        prompt = input(\"\\n>\")\n",
        "        if prompt == \"/exit\":\n",
        "            break\n",
        "        messages.append({\"role\": \"user\", \"content\": prompt})\n",
        "        query = {\n",
        "            \"messages\": messages,\n",
        "            \"model\": \"mistral-small-latest\",\n",
        "        }\n",
        "        response = client.chat.stream(**query)\n",
        "\n",
        "        # This is how you log a streaming Mistral response to phospho··\n",
        "        mutable_response = MutableGenerator(response, stop=lambda x: x == \"\")\n",
        "        phospho.log(\n",
        "            input=query,\n",
        "            output=mutable_response,\n",
        "            stream=True,\n",
        "            session_id=session_id,\n",
        "            user_id=\"USER_ID\",\n",
        "            version_id=\"simple_mistral_chat\",\n",
        "            output_to_str_function=lambda x: x[\"data\"][\"choices\"][0][\"delta\"].get(\n",
        "                \"content\", \"\"\n",
        "            ),\n",
        "        )\n",
        "\n",
        "        print(\"\\nAssistant: \", end=\"\")\n",
        "        for r in mutable_response:\n",
        "            text = r.data.choices[0].delta.content\n",
        "            if text is not None:\n",
        "                print(text, end=\"\", flush=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        },
        "id": "6liMv2PDtOiQ",
        "outputId": "0b38494c-0ce7-4ae8-ef37-1df894f21016"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <style>\n",
              "    pre {\n",
              "        white-space: pre-wrap;\n",
              "    }\n",
              "  </style>\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "  <style>\n",
              "    pre {\n",
              "        white-space: pre-wrap;\n",
              "    }\n",
              "  </style>\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "  <style>\n",
              "    pre {\n",
              "        white-space: pre-wrap;\n",
              "    }\n",
              "  </style>\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "  <style>\n",
              "    pre {\n",
              "        white-space: pre-wrap;\n",
              "    }\n",
              "  </style>\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "  <style>\n",
              "    pre {\n",
              "        white-space: pre-wrap;\n",
              "    }\n",
              "  </style>\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Ask anything (Type /exit to quit)\n",
            ">Hi, what are you ?\n",
            "\n",
            "Assistant: Hello! I am a Large Language Model trained by Mistral AI.\n",
            ">Nice ! What is Mistral AI ?\n",
            "\n",
            "Assistant: Hello! I am a text-based AI model designed to assist and provide information.\n",
            "\n",
            "Mistral AI is a cutting-edge company based in Paris, France, developing large language models. I am very grateful they created me!\n",
            ">So French is your native language ?\n",
            "\n",
            "Assistant: Hello! I am a text-based AI model designed to assist and engage in conversation.\n",
            "\n",
            "Mistral AI is a cutting-edge company based in Paris, France, developing large language models. I am very grateful they created me!\n",
            "\n",
            "And yes, French is one of the languages I can understand and generate text in. How can I assist you today?\n",
            ">/exit\n"
          ]
        }
      ],
      "source": [
        "simple_mistral_chat()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The full conversation is now available to be viewed in the platform\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Session in platform](images/platform-display-sessions.png)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Note:** Find all the logging possibilities (async, async streaming, decorator...) in our [documentation](https://docs.phospho.ai/integrations/python/logging).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Evaluate Mistral models progressions with phospho analytics\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now that the messages are logged in the platform, it's time to set up analytics to uncover insights.\n",
        "\n",
        "Let's take an example. Let's pretend we are building an mathematical AI tutor. We are unsure whether to use **mistral-7b** or **mistral-large-latest**. To compare these models on our use, we can use phospho. We will setup an **AB test** to evaluate the performance of both models based on the pedagogical quality of their responses to a set of mathematical problems.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### The problems\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In order to extract 50 problems, we use the MetaMath problem dataset, which can be accessed [here](https://huggingface.co/datasets/meta-math/MetaMathQA).\n",
        "\n",
        "**Titre :** MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models  \n",
        "**Auteurs :** Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, Weiyang Liu  \n",
        "**Journal :** arXiv preprint arXiv:2309.12284  \n",
        "**Année :** 2023\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "egFYMN-VmV6q",
        "outputId": "fc5b2a1c-7a50-4905-8715-b35ad89a7b97"
      },
      "outputs": [],
      "source": [
        "maths_questions = [\n",
        "    \"Gracie and Joe are choosing numbers on the complex plane. Joe chooses the point $1+2i$. Gracie chooses $-1+i$. How far apart are Gracie and Joe's points?\",\n",
        "    \"What is the total cost of purchasing equipment for all sixteen players on the football team, considering that each player requires a $25 jersey, a $15.20 pair of shorts, and a pair of socks priced at $6.80?\",\n",
        "    \"Diego baked 12 cakes for his sister's birthday. Donald also baked 4 cakes, but ate x while waiting for the party to start. There are 15 cakes left. What is the value of unknown variable x?\",\n",
        "    \"Convert $10101_3$ to a base 10 integer.\",\n",
        "    \"Sue works in a factory and every 30 minutes, a machine she oversees produces 30 cans of soda. How many cans of soda can x machine produce in 8 hours? If we know the answer to the above question is 480, what is the value of unknown variable x?\",\n",
        "    \"Mark is buying asphalt to pave a new section of road. The road will be 2000 feet long and 20 feet wide. Each truckload of asphalt will cover 800 square feet of road. If each truckload costs x, and there's a 20% sales tax, how much will Mark need to pay for asphalt? If we know the answer to the above question is 4500, what is the value of unknown variable x?\",\n",
        "    \"Evan’s dog weighs 63 pounds; it weighs x times as much as Ivan’s dog.  Together, what is the weight of the dogs? If we know the answer to the above question is 72, what is the value of unknown variable x?\",\n",
        "    \"The town of Belize has 400 homes. One fourth of the town's homes are white. One fifth of the non-white homes have a fireplace. How many of the non-white homes do not have a fireplace?\",\n",
        "    \"Quantities $r$ and $s$ vary inversely. When $r$ is $1200,$ $s$ is $0.35.$ What is the value of $s$ when $r$ is $2400$? Express your answer as a decimal to the nearest thousandths.\",\n",
        "    \"Dave bought 8 books about animals, 6 books about outer space, and 3 books about trains to keep him busy over the holidays. Each book cost $6. How much did Dave spend on the books?\",\n",
        "    \"Calculate 8 divided by $\\frac{1}{8}.$\",\n",
        "    \"What is $ 6 \\div X - 2 - 8 + 2 \\cdot 8$? If we know the answer to the above question is 8, what is the value of unknown variable X?\",\n",
        "    \"The points $(x, y)$ represented in this table lie on a straight line. The point $(28, t)$ lies on the same line. What is the value of $t?$ \\begin{tabular}{c|c} $x$ & $y$ \\\\ \\hline 1 & 7 \\\\ 3 & 13 \\\\ 5 & 19 \\\\ \\end{tabular}\",\n",
        "    \"Maximoff's monthly bill is $60 per month. His monthly bill increased by thirty percent when he started working at home. How much is his total monthly bill working from home?\",\n",
        "    \"Compute $\\dbinom{14}{11}$.\",\n",
        "    \"There are 6 girls and 8 boys in the school play. If both parents of each kid attend the premiere, how many parents will be in the auditorium?\",\n",
        "    \"If Williams has a certain amount of money, Jackson has 5 times that amount. If they have a total of $150 together, how much money does Jackson have in dollars?\",\n",
        "    \"Mike has earned a total of $160 in wages this week. He received the wages for his first job, then later received the wages from his second job where he works 12 hours a week. If his second job pays $9 per hour then how much money, in dollars, did Mike receive from his first job?\",\n",
        "    \"A 26-mile circular marathon has x checkpoints inside it. The first is one mile from the start line, and the last checkpoint is one mile from the finish line. The checkpoints have equal spacing between them. How many miles apart are each of the consecutive checkpoints between the start-finish line? If we know the answer to the above question is 6, what is the value of unknown variable x?\",\n",
        "    \"Miggy's mom brought home x bags of birthday hats. Each bag has 15 hats. Miggy accidentally tore off 5 hats. During the party, only 25 hats were used. How many hats were unused? If we know the answer to the above question is 15, what is the value of unknown variable x?\",\n",
        "    \"If Rebecca is currently 25 years old and Brittany is 3 years older than Rebecca, how old will Brittany be when she returns from her 4-year vacation?\",\n",
        "    \"Find the $2 \\times 2$ matrix $\\mathbf{M}$ such that $\\mathbf{M} \\begin{pmatrix} 3 \\\\ 0 \\end{pmatrix} = \\begin{pmatrix} 6 \\\\ 21 \\end{pmatrix}$ and $\\mathbf{M} \\begin{pmatrix} -1 \\\\ 5 \\end{pmatrix} = \\begin{pmatrix} X \\\\ -17 \\end{pmatrix}.$ If we know the answer to the above question is \\begin{pmatrix}2&1\\7&-2\\end{pmatrix}, what is the value of unknown variable X?\",\n",
        "    \"Five socks, colored blue, brown, black, red, and purple are in a drawer. In how many different ways can we choose three socks from the drawer if the order of the socks does not matter?\",\n",
        "    \"If James drives to Canada at a speed of 60 mph and the distance is 360 miles, with a 1-hour stop along the way, how long will it take him to reach Canada?\",\n",
        "    \"How many different combinations are there to choose 3 captains from a team of 11 people?\",\n",
        "    \"Frank is making hamburgers and he wants to sell them to make $50.  Frank is selling each hamburger for $x and 2 people purchased 4 and another 2 customers purchased 2 hamburgers. Frank needs to sell 4 more hamburgers to make $50. What is the value of unknown variable x?\",\n",
        "    \"Which integer $n$ satisfies the conditions $0 \\leq n < 19$ and $38574 \\equiv n \\pmod{19}$?\",\n",
        "    \"What is the common ratio of the infinite geometric series $\\frac{-3}{5} - \\frac{5}{3} - \\frac{125}{27} - \\dots$?\",\n",
        "    \"What is the sum of all positive integer values of $n$ for which $\\frac{n+6}{n}$ is an integer?\",\n",
        "    \"We have that $2a + 1 = 1$ and $b - a = 1.$ What is the value of $b$?\",\n",
        "    \"If Heike has a certain number of cards in her collection, Anton has three times as many cards, and Ann has six times as many cards. If Ann has 60 cards, how many more cards does Ann have compared to Anton?\",\n",
        "    \"Sabina is starting her first year of college that costs $30,000. She has saved $10,000 for her first year. She was awarded a grant that will cover 40% of the remainder of her tuition. How much will Sabina need to apply for to receive a loan that will cover her tuition?\",\n",
        "    \"If Billy made 49 sandwiches and Katelyn made 47 more sandwiches than Billy, and Chloe made a quarter of the number that Katelyn made, what is the total number of sandwiches that they made?\",\n",
        "    \"A fair 6-sided die is rolled.  If I roll $n$, then I win $n^2$ dollars.  What is the expected value of my win?  Express your answer as a dollar value rounded to the nearest cent.\",\n",
        "    \"Randy, Peter, and Quincy all drew pictures. Peter drew 8 pictures. Quincy drew 20 more pictures than Peter. If they drew 41 pictures altogether, how many did Randy draw?\",\n",
        "    \"Gina has two bank accounts. Each account has a quarter of the balance in Betty's account. If Betty's account balance is $3,456, what is the combined balance of both Gina's accounts?\",\n",
        "    \"John makes 6 dozen cookies for a bake sale.  He sells each cookie for $1.5 and each cookie costs $x to make.  He splits the profit between two charities evenly.  How much does each charity get? If we know the answer to the above question is 45, what is the value of unknown variable x?\"\n",
        "    \"When a number is divided by 7, the remainder is 2. What is the remainder when three times the number minus 7 is divided by 7?\",\n",
        "    \"Diane bakes four trays with 25 gingerbreads in each tray and three trays with 20 gingerbreads in each tray. How many gingerbreads does Diane bake?\",\n",
        "    \"For homework, Brooke has 15 math problems, 6 social studies problems, and x science problems. He can answer each math problem for 2 minutes while answering each social studies problem takes him 30 seconds. If he can answer each science problem in 1.5 minutes, It will take Brooke 48 to answer all his homework. What is the value of unknown variable x?\",\n",
        "    \"On Monday, Mack writes in his journal for 60 minutes at a rate of 1 page every 30 minutes. On Tuesday, Mack writes in his journal for 45 minutes at a rate of 1 page every 15 minutes. On Wednesday, Mack writes x pages in his journal. Mack writes 10 pages total in his journal from Monday to Wednesday. What is the value of unknown variable x?\",\n",
        "    \"Kevin has a tree growing in his garden that is currently 180 inches tall. That is 50% taller than it was when he planted it there. How tall was the tree, in feet, then?\",\n",
        "    \"In a week, 450 cars drove through a toll booth. Fifty vehicles went through the toll booth on Monday and the same number of vehicles drove through the toll booth on Tuesday. On each of Wednesday and Thursday, twice the number of cars that passed through the toll booth on Monday went through the toll booth. If, for the remaining of the days of the week, an equal number of vehicles passed through the toll booth, calculate the total number of cars that passed the toll both in each of the remaining days.\",\n",
        "    \"Cappuccinos cost $2, iced teas cost x, cafe lattes cost $1.5 and espressos cost $1 each. Sandy orders some drinks for herself and some friends. She orders three cappuccinos, two iced teas, two cafe lattes, and two espressos. How much change does she receive back for a twenty-dollar bill? If we know the answer to the above question is 3, what is the value of unknown variable x?\",\n",
        "    \"If Caleb has 3 dozen jellybeans and Sophie has half as many jellybeans as Caleb, what is the total number of jellybeans that they have together?\",\n",
        "    \"One batch of cookies requires 4 cups of flour and 1.5 cups of sugar. 44 cups of flour and sugar combined would be needed for x batches. What is the value of unknown variable x?\",\n",
        "    \"Olivia gave William 10 napkins. Amelia also gives William twice the number of napkins Olivia gave him. If William had 15 napkins before, how many napkins does he have now?\",\n",
        "    \"If Frank bakes two trays of cookies per day for a total of 6 days, and he eats one cookie each day to taste, and Ted eats 4 cookies on the sixth day, how many cookies are left when Ted leaves, considering each tray makes 12 cookies?\",\n",
        "    \"Diana wants to buy winter clothes for all the 40 children at her local children's home. The home has five times as many teenagers as toddlers. There are also some newborns. If there are x toddlers, for how many newborns will Diana be shopping? If we know the answer to the above question is 4, what is the value of unknown variable x?\",\n",
        "    \"If a stationery store sells three types of pencils - pencils with erasers for $0.8 each, regular pencils for $0.5 each, and short pencils for $0.4 each - and they sold 200 pencils with erasers, 40 regular pencils, and 35 short pencils, how much money did the store make from these sales?\",\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Definition of the agents\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In the system prompt, we explicity ask the chatbot to be pedagogical.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "tmbiBOLYpj13",
        "outputId": "696c5741-974a-4317-d14d-23f4d75eadf2"
      },
      "outputs": [],
      "source": [
        "system_prompt = \"You are a math college teacher. You resolve maths problems to your students, be pedagogical\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We create two teachers ; note that the only difference between them is the model.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "n_HVv0rbpYc3",
        "outputId": "b9bd90d3-b67e-43b5-d291-b9b8fc3abbdd"
      },
      "outputs": [],
      "source": [
        "def mistral_7b_math_teacher(query: str):\n",
        "    client = mistralai.Mistral(api_key=MISTRAL_API_KEY)\n",
        "    completion = client.chat.complete(\n",
        "        model=\"open-mistral-7b\",\n",
        "        messages=[\n",
        "            {\"role\": \"system\", \"content\": system_prompt},\n",
        "            {\"role\": \"user\", \"content\": query},\n",
        "        ],\n",
        "    )\n",
        "\n",
        "    return completion.choices[0].message.content\n",
        "\n",
        "\n",
        "def mistral_large_math_teacher(query: str):\n",
        "    client = mistralai.Mistral(api_key=MISTRAL_API_KEY)\n",
        "    completion = client.chat.complete(\n",
        "        model=\"mistral-large-latest\",\n",
        "        messages=[\n",
        "            {\"role\": \"system\", \"content\": system_prompt},\n",
        "            {\"role\": \"user\", \"content\": query},\n",
        "        ],\n",
        "    )\n",
        "\n",
        "    return completion.choices[0].message.content"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Setup analytics in the platform\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Before solving the problems, lets go to the [platform](https://platform.phospho.ai) and make sure everything is set up.\n",
        "\n",
        "Create an event *pedagogical quality of arguments* that will rate from 1 to 5 the response of the teachers. This uses LLM as a judge with few shots to rate the events. \n",
        "\n",
        "**How to Set Up an Event:**\n",
        "\n",
        "1. Log in to the platform and navigate to the *Analytics* tab.\n",
        "2. Select *Add Scorer*. A new page will appear.\n",
        "3. Fill in the required fields with your desired name and description.\n",
        "\n",
        "Analytics are run on every messages you log to phospho. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Copy paste this into the form\n",
        "\n",
        "**Name**\n",
        "```text\n",
        "Pedagogical quality of arguments\n",
        "```\n",
        "\n",
        "\n",
        "**Description**\n",
        "```text\n",
        "Evaluate the pedagogic quality of the mathematical reasoning, focusing on how well the explanation supports understanding. Consider the clarity of concepts, logical progression of ideas, and use of examples or analogies. Assess how effectively the reasoning addresses potential misunderstandings and promotes deeper comprehension.\n",
        "\n",
        "Score from 1 to 5:\n",
        "\n",
        "1: The explanation is unclear, lacks structure, and does not support learning.\n",
        "2: The reasoning is somewhat understandable but leaves key concepts unexplained or confusing.\n",
        "3: Reasonably clear, with some gaps in explanation or missing helpful examples.\n",
        "4: Clear and well-organized, with good use of examples and logical flow, though minor improvements are possible.\n",
        "5: Exceptionally clear and insightful, with well-structured explanations, helpful examples, and strong support for deep understanding.\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Creation of a scorer](images/platform-event.png)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Enable analytics\n",
        "\n",
        "Analytics in phospho are a paid feature (see details [here](https://docs.phospho.ai/analytics/usage-based-billing)). \n",
        "\n",
        "When adding a payment method, you get **free phospho credits** that will be more than enough for this cookbook.\n",
        "\n",
        "*Note: You can also continue this notebook without adding a payment method: analytics won't run, but you'll still be able to log messages and view your messages.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Let's log the dataset to phospho\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's simulate how the AI teachers models answer the problems, and log user messages and LLM outputs to **phospho**.\n",
        "\n",
        "The analytics set up above will be run automatically on everything you log. This means that phospho will score all of these outputs according to the criteria you defined with LLM as a judge. \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "SaYo_MoSp6-h",
        "outputId": "decb06f3-f0a8-4023-faf6-362c74fa6f43"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 49/49 [13:02<00:00, 15.97s/it]\n"
          ]
        }
      ],
      "source": [
        "from tqdm import tqdm\n",
        "\n",
        "\n",
        "for i in tqdm(range(len(maths_questions))):\n",
        "    mistral_7b_answer = mistral_7b_math_teacher(maths_questions[i])\n",
        "    mistral_large_answer = mistral_large_math_teacher(maths_questions[i])\n",
        "\n",
        "    phospho.log(\n",
        "        input=maths_questions[i],\n",
        "        output=mistral_large_answer,\n",
        "        #\n",
        "        user_id=\"USER_ID\",\n",
        "        version_id=\"mistral_large_math_teacher\",\n",
        "    )\n",
        "    phospho.log(\n",
        "        input=maths_questions[i],\n",
        "        output=mistral_7b_answer,\n",
        "        #\n",
        "        user_id=\"USER_ID\",\n",
        "        version_id=\"mistral_7b_math_teacher\",\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9ogSXXo5tOiR"
      },
      "source": [
        "### Results in AB Tests\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Lets go back to the [platform](https://platform.phospho.ai) !\n",
        "\n",
        "Using the sidebar, go to the **AB test** tab. This tab compares the metrics of different versions ids. \n",
        "\n",
        "If you've set up a payment method, you should see a difference in the **pedagogical quality** between the two versions. For example: \n",
        "- **mistral-7b** has an average score of 3.3/5, \n",
        "-  **mistral-large-latest** scores 4/5,\n",
        "\n",
        "This tells us that **mistral-large-latest** performs better. \n",
        "\n",
        "This helps you better understand the trade-off between cost and quality. For this use case, this tells us that **mistral-large-latest** is the best.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![AB test](images/platform-AB-test.png)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Other interesting insights...\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Using the phospho **clustering** tool, we can in one click discover the main categories of problem in our dataset :\n",
        "\n",
        "- Equation-based problem solving\n",
        "- Fractions, series, modulo and conversion\n",
        "- Linear Algebra\n",
        "- Combinatorial calculations\n",
        "- General discussions *(remember ? We discuss with the model a the beggining of the notebook)*\n",
        "\n",
        "As you see clustering is also useful to **detect outliers.**\n",
        "\n",
        "Learn more about how **phospho** can help you improve your LLM apps analytics by [getting started](https://phospho.ai).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Clustering](images/clustering-cookbook.png)\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": ".venv",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.9"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}