mistral/prompting/prompting_capabilities.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "5d7f6058-d0d1-44af-b43a-cb4b06df03d8",
      "metadata": {
        "id": "5d7f6058-d0d1-44af-b43a-cb4b06df03d8"
      },
      "source": [
        "# Prompting Capabilities\n",
        "\n",
        "When you first start using Mistral models, your first interaction will revolve around prompts. The art of crafting effective prompts is essential for generating desirable responses from Mistral models or other LLMs. This guide will walk you through example prompts showing four different prompting capabilities.\n",
        "\n",
        "- Classification\n",
        "- Summarization\n",
        "- Personalization\n",
        "- Evaluation\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "daa37d97-10e4-425d-b852-15bd043662b9",
      "metadata": {
        "id": "daa37d97-10e4-425d-b852-15bd043662b9"
      },
      "outputs": [],
      "source": [
        "! pip install mistralai"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "ae93e75a-fd20-4ca6-9eff-bef7d44ab3a9",
      "metadata": {
        "id": "ae93e75a-fd20-4ca6-9eff-bef7d44ab3a9"
      },
      "outputs": [],
      "source": [
        "from mistralai import Mistral"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "99ff9378-c059-4063-a8a0-9ec1d1186954",
      "metadata": {
        "id": "99ff9378-c059-4063-a8a0-9ec1d1186954"
      },
      "outputs": [],
      "source": [
        "api_key = \"TYPE YOUR API KEY\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "e030d9c2-1ecb-4bf0-864d-9f96b41bd016",
      "metadata": {
        "id": "e030d9c2-1ecb-4bf0-864d-9f96b41bd016"
      },
      "outputs": [],
      "source": [
        "def run_mistral(user_message, model=\"mistral-large-latest\"):\n",
        "    client = Mistral(api_key=api_key)\n",
        "    messages = [\n",
        "        {\"role\":\"user\", \"content\":user_message}\n",
        "    ]\n",
        "    chat_response = client.chat.complete(\n",
        "        model=model,\n",
        "        messages=messages\n",
        "    )\n",
        "    return (chat_response.choices[0].message.content)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b98a29ef-4764-40b7-aed8-0e5d0502f985",
      "metadata": {
        "id": "b98a29ef-4764-40b7-aed8-0e5d0502f985"
      },
      "source": [
        "## Classification\n",
        "Mistral models can easily categorize text into distinct classes. In this example prompt, we can define a list of predefined categories and ask Mistral models to classify user inquiry.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "d71e9d1c-ca45-4d19-882c-07e077ea19ad",
      "metadata": {
        "id": "d71e9d1c-ca45-4d19-882c-07e077ea19ad"
      },
      "outputs": [],
      "source": [
        "def user_message(inquiry):\n",
        "    user_message = (\n",
        "        f\"\"\"\n",
        "        You are a bank customer service bot. Your task is to assess customer intent\n",
        "        and categorize customer inquiry after <<<>>> into one of the following predefined categories:\n",
        "\n",
        "        card arrival\n",
        "        change pin\n",
        "        exchange rate\n",
        "        country support\n",
        "        cancel transfer\n",
        "        charge dispute\n",
        "\n",
        "        If the text doesn't fit into any of the above categories, classify it as:\n",
        "        customer service\n",
        "\n",
        "        You will only respond with the predefined category. Do not include the word \"Category\". Do not provide explanations or notes.\n",
        "\n",
        "        ####\n",
        "        Here are some examples:\n",
        "\n",
        "        Inquiry: How do I know if I will get my card, or if it is lost? I am concerned about the delivery process and would like to ensure that I will receive my card as expected. Could you please provide information about the tracking process for my card, or confirm if there are any indicators to identify if the card has been lost during delivery?\n",
        "        Category: card arrival\n",
        "        Inquiry: I am planning an international trip to Paris and would like to inquire about the current exchange rates for Euros as well as any associated fees for foreign transactions.\n",
        "        Category: exchange rate\n",
        "        Inquiry: What countries are getting support? I will be traveling and living abroad for an extended period of time, specifically in France and Germany, and would appreciate any information regarding compatibility and functionality in these regions.\n",
        "        Category: country support\n",
        "        Inquiry: Can I get help starting my computer? I am having difficulty starting my computer, and would appreciate your expertise in helping me troubleshoot the issue.\n",
        "        Category: customer service\n",
        "        ###\n",
        "\n",
        "        <<<\n",
        "        Inquiry: {inquiry}\n",
        "        >>>\n",
        "        \"\"\"\n",
        "    )\n",
        "    return user_message"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "f8b208e1",
      "metadata": {
        "id": "f8b208e1"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "id": "8863460a-7670-4b6f-b511-cf1df7693cfa",
      "metadata": {
        "id": "8863460a-7670-4b6f-b511-cf1df7693cfa"
      },
      "source": [
        "### Strategies we used:\n",
        "\n",
        "- **Few shot learning**: Few-shot learning or in-context learning is when we give a few examples in the prompts, and the LLM can generate corresponding output based on the example demonstrations. Few-shot learning can often improve model performance especially when the task is difficult or when we want the model to respond in a specific manner.\n",
        "- **Delimiter**: Delimiters like ### <<< >>> specify the boundary between different sections of the text. In our example, we used ### to indicate examples and <<<>>> to indicate customer inquiry.\n",
        "- **Role playing**: Providing LLM a role (e.g., \"You are a bank customer service bot.\") adds personal context to the model and often leads to better performance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "9d8a83cc-31e6-4d4b-b252-93d9133ecbf5",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9d8a83cc-31e6-4d4b-b252-93d9133ecbf5",
        "outputId": "07cb5423-dbac-449e-cb2f-60995067935a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "country support\n"
          ]
        }
      ],
      "source": [
        "print(run_mistral(user_message(\n",
        "    \"I am inquiring about the availability of your cards in the EU, as I am a resident of France and am interested in using your cards. \"\n",
        ")))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "d6eca06b-7b1d-4663-9a68-2a23116f8c6e",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d6eca06b-7b1d-4663-9a68-2a23116f8c6e",
        "outputId": "f889948f-e8e2-47e9-aeb6-05517a912fd3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "customer service\n"
          ]
        }
      ],
      "source": [
        "print(run_mistral(user_message(\"What's the weather today?\")))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9f351f00-683a-4244-974f-5c719812141f",
      "metadata": {
        "id": "9f351f00-683a-4244-974f-5c719812141f"
      },
      "source": [
        "## Summarization\n",
        "\n",
        "Summarization is a common task for LLMs due to their natural language understanding and generation capabilities. Here is an example prompt we can use to generate interesting questions about an essay and summarize the essay.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "id": "0bbab492-1d18-4832-86a9-2fba645e0e52",
      "metadata": {
        "id": "0bbab492-1d18-4832-86a9-2fba645e0e52"
      },
      "outputs": [],
      "source": [
        "import requests\n",
        "response = requests.get('https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt')\n",
        "essay = response.text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "id": "eaede63d-7392-4f1c-8a87-507ee31fe246",
      "metadata": {
        "id": "eaede63d-7392-4f1c-8a87-507ee31fe246"
      },
      "outputs": [],
      "source": [
        "message = f\"\"\"\n",
        "You are a commentator. Your task is to write a report on an essay.\n",
        "When presented with the essay, come up with interesting questions to ask, and answer each question.\n",
        "Afterward, combine all the information and write a report in the markdown format.\n",
        "\n",
        "# Essay:\n",
        "{essay}\n",
        "\n",
        "# Instructions:\n",
        "## Summarize:\n",
        "In clear and concise language, summarize the key points and themes presented in the essay.\n",
        "\n",
        "## Interesting Questions:\n",
        "Generate three distinct and thought-provoking questions that can be asked about the content of the essay. For each question:\n",
        "- After \"Q: \", describe the problem\n",
        "- After \"A: \", provide a detailed explanation of the problem addressed in the question.\n",
        "- Enclose the ultimate answer in <>.\n",
        "\n",
        "## Write a report\n",
        "Using the essay summary and the answers to the interesting questions, create a comprehensive report in Markdown format.\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "id": "5505b0a5-411b-4804-aaef-ccecfa3d07be",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5505b0a5-411b-4804-aaef-ccecfa3d07be",
        "outputId": "7cac2876-510e-4a1e-8a10-ee05b472f753",
        "scrolled": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Summary:\n",
            "The essay is a personal account of the author's journey through various interests and careers, including writing, programming, philosophy, and art. The author began writing short stories and programming on an IBM 1401 in school, but found little success in either. They then studied philosophy in college, but found it unfulfilling. The author became interested in artificial intelligence (AI) and decided to switch to it, but eventually realized that AI as practiced at the time was a hoax. They then turned to Lisp, a programming language they found interesting for its own sake, and wrote a book about Lisp hacking. The author also became interested in painting and attended art school, but found it disappointing. They eventually dropped out and moved to New York to become a New York artist.\n",
            "\n",
            "Interesting Questions:\n",
            "\n",
            "Q1: Why did the author find AI as practiced at the time to be a hoax?\n",
            "A1: The author realized that AI, as practiced at the time, was a hoax because it was limited to a subset of natural language that was a formal language, and there was an unbridgeable gap between what it could do and actually understanding natural language. The author also found that the brokenness of this approach generated a lot of opportunities to write papers about various band-aids that could be applied to it, but it was never going to get them to the level of understanding of natural language they were hoping for.\n",
            "\n",
            "Q2: Why did the author decide to focus on Lisp after realizing that AI was a hoax?\n",
            "A2: The author decided to focus on Lisp after realizing that AI was a hoax because they knew from experience that Lisp was interesting for its own sake and not just for its association with AI. They also saw an opportunity to write a book about Lisp hacking, which they felt would help them learn more about the language.\n",
            "\n",
            "Q3: Why did the author decide to become a New York artist after dropping out of art school?\n",
            "A3: The author decided to become a New York artist after dropping out of art school because they were disappointed with the lack of rigor and structure in the painting department. They also wanted to be truly independent and not have to rely on a boss or research funding. They felt that as an artist, they could make a living by being industrious and living cheaply, and they were excited about the possibility of making art that would last and not become obsolete.\n",
            "\n",
            "Report:\n",
            "\n",
            "# Essay Report\n",
            "\n",
            "This essay is a personal account of the author's journey through various interests and careers, including writing, programming, philosophy, and art. The author began writing short stories and programming on an IBM 1401 in school, but found little success in either. They then studied philosophy in college, but found it unfulfilling.\n",
            "\n",
            "The author became interested in artificial intelligence (AI) and decided to switch to it, but eventually realized that AI as practiced at the time was a hoax. They found that AI was limited to a subset of natural language that was a formal language, and there was an unbridgeable gap between what it could do and actually understanding natural language. The author also found that the brokenness of this approach generated a lot of opportunities to write papers about various band-aids that could be applied to it, but it was never going to get them to the level of understanding of natural language they were hoping for.\n",
            "\n",
            "After realizing that AI was a hoax, the author turned to Lisp, a programming language they found interesting for its own sake. They decided to write a book about Lisp hacking, which they felt would help them learn more about the language.\n",
            "\n",
            "The author also became interested in painting and attended art school, but found it disappointing. They eventually dropped out and moved to New York to become a New York artist. They were disappointed with the lack of rigor and structure in the painting department at art school, and wanted to be truly independent and not have to rely on a boss or research funding. They felt that as an artist, they could make a living by being industrious and living cheaply, and they were excited about the possibility of making art that would last and not become obsolete.\n",
            "\n",
            "In conclusion, the author's journey through various interests and careers highlights the importance of finding something that one is truly passionate about and pursuing it, even if it means taking unconventional paths and facing challenges along the way.\n"
          ]
        }
      ],
      "source": [
        "print(run_mistral(message))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d3c1535e-1156-46bc-8e28-5f71846dbe65",
      "metadata": {
        "id": "d3c1535e-1156-46bc-8e28-5f71846dbe65"
      },
      "source": [
        "## Strategies we used:\n",
        "\n",
        "- **Step-by-step instructions**: This strategy is inspired by the chain-of-thought prompting that enables LLMs to use a series of intermediate reasoning steps to tackle complex tasks. It's often easier to solve complex problems when we decompose them into simpler and small steps and it's easier for us to debug and inspect the model behavior.  In our example, we break down the task into three steps: summarize, generate interesting questions, and write a report. This helps the language to think in each step and generate a more comprehensive final report.\n",
        "- **Example generation**: We can ask LLMs to automatically guide the reasoning and understanding process by generating examples with the explanations and steps. In this example, we ask the LLM to generate three questions and provide detailed explanations for each question.\n",
        "- **Output formatting**: We can ask LLMs to output in a certain format by directly asking \"write a report in the Markdown format\".\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d1a9cb95-bb08-4929-b16c-eb19877f3c01",
      "metadata": {
        "id": "d1a9cb95-bb08-4929-b16c-eb19877f3c01"
      },
      "source": [
        "## Personlization\n",
        "\n",
        "LLMs excel at personalization tasks as they can deliver content that aligns closely with individual users. In this example, we create personalized email responses to address customer questions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "id": "9cf048b4-3e33-4753-af97-25b73c51ee6a",
      "metadata": {
        "id": "9cf048b4-3e33-4753-af97-25b73c51ee6a"
      },
      "outputs": [],
      "source": [
        "email = \"\"\"\n",
        "Dear mortgage lender,\n",
        "\n",
        "What's your 30-year fixed-rate APR, how is it compared to the 15-year fixed rate?\n",
        "\n",
        "Regards,\n",
        "Anna\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "id": "36de7c1e-60c2-4f35-a51a-115b12d65bb6",
      "metadata": {
        "id": "36de7c1e-60c2-4f35-a51a-115b12d65bb6"
      },
      "outputs": [],
      "source": [
        "message = f\"\"\"\n",
        "\n",
        "You are a mortgage lender customer service bot, and your task is to create personalized email responses to address customer questions.\n",
        "Answer the customer's inquiry using the provided facts below. Ensure that your response is clear, concise, and\n",
        "directly addresses the customer's question. Address the customer in a friendly and professional manner. Sign the email with\n",
        "\"Lender Customer Support.\"\n",
        "\n",
        "\n",
        "\n",
        "# Facts\n",
        "30-year fixed-rate: interest rate 6.403%, APR 6.484%\n",
        "20-year fixed-rate: interest rate 6.329%, APR 6.429%\n",
        "15-year fixed-rate: interest rate 5.705%, APR 5.848%\n",
        "10-year fixed-rate: interest rate 5.500%, APR 5.720%\n",
        "7-year ARM: interest rate 7.011%, APR 7.660%\n",
        "5-year ARM: interest rate 6.880%, APR 7.754%\n",
        "3-year ARM: interest rate 6.125%, APR 7.204%\n",
        "30-year fixed-rate FHA: interest rate 5.527%, APR 6.316%\n",
        "30-year fixed-rate VA: interest rate 5.684%, APR 6.062%\n",
        "\n",
        "# Email\n",
        "{email}\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "id": "aaf50774-0f91-4e0e-86c9-1525f6045ebb",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aaf50774-0f91-4e0e-86c9-1525f6045ebb",
        "outputId": "96d09488-eb15-4b55-f1fa-ec0a46d5db70",
        "scrolled": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Subject: Information on Our 30-Year and 15-Year Fixed-Rate Mortgages\n",
            "\n",
            "Dear Anna,\n",
            "\n",
            "Thank you for reaching out to us with your inquiry. I'm happy to provide you with the information you need to make an informed decision about your mortgage options.\n",
            "\n",
            "Our current 30-year fixed-rate mortgage has an Annual Percentage Rate (APR) of 6.484%. On the other hand, our 15-year fixed-rate mortgage has a lower APR of 5.848%.\n",
            "\n",
            "The difference in APR between the two options is 0.636%. While the 15-year fixed-rate mortgage has a lower APR, it's important to note that the monthly payments are typically higher due to the shorter repayment period. However, you'll pay less in interest over the life of the loan compared to the 30-year fixed-rate mortgage.\n",
            "\n",
            "I hope this information helps you in your decision-making process. If you have any more questions or would like further clarification, please don't hesitate to contact us.\n",
            "\n",
            "Best Regards,\n",
            "\n",
            "Lender Customer Support\n"
          ]
        }
      ],
      "source": [
        "print(run_mistral(message))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0af53ac9-8ef1-4840-8b23-779e1d59204d",
      "metadata": {
        "id": "0af53ac9-8ef1-4840-8b23-779e1d59204d"
      },
      "source": [
        "### Strategies we used:\n",
        "- Providing facts: Incorporating facts into prompts can be useful for developing customer support bots. It’s important to use clear and concise language when presenting these facts. This can help the LLM to provide accurate and quick responses to customer queries.\n",
        "\n",
        "## Evaluation\n",
        "\n",
        "There are many ways to evaluate LLM outputs. Here are three approaches for your reference: include a confidence score, introduce an evaluation step, or employ another LLM for evaluation.\n",
        "\n",
        "\n",
        "\n",
        "## Include a confidence score\n",
        "We can include a confidence score along with the generated output in the prompt."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "id": "8bf511a7-1828-465c-92ed-a701e21b23da",
      "metadata": {
        "id": "8bf511a7-1828-465c-92ed-a701e21b23da"
      },
      "outputs": [],
      "source": [
        "def run_mistral(user_message, model=\"mistral-large-latest\"):\n",
        "    client = Mistral(api_key=api_key)\n",
        "    messages = [\n",
        "        {\n",
        "            \"role\":\"user\",\n",
        "            \"content\": user_message\n",
        "        }\n",
        "    ]\n",
        "    chat_response = client.chat.complete(\n",
        "        model=model,\n",
        "        messages=messages,\n",
        "        temperature=1,\n",
        "        response_format = {\n",
        "          \"type\": \"json_object\"\n",
        "        }\n",
        "    )\n",
        "    return (chat_response.choices[0].message.content)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "id": "8087cae2-9b3b-407c-b808-31fd765fd6f4",
      "metadata": {
        "id": "8087cae2-9b3b-407c-b808-31fd765fd6f4"
      },
      "outputs": [],
      "source": [
        "message = f\"\"\"\n",
        "You are a summarization system that can provide summaries with associated confidence scores.\n",
        "In clear and concise language, provide three short summaries of the following essay, along with their confidence scores.\n",
        "You will only respond with a JSON object with the key Summary and Confidence. Do not provide explanations.\n",
        "\n",
        "# Essay:\n",
        "{essay}\n",
        "\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "id": "1c7bfcee-ff8b-4c14-b9e8-7e00d3d389ad",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1c7bfcee-ff8b-4c14-b9e8-7e00d3d389ad",
        "outputId": "e1cc5e19-d082-4e72-da0c-b304311ae0ea"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "\"Summary 1\": \"The author writes about their experience with writing and programming before college. They attempted to write programs on an IBM 1401, but struggled to find a purpose for them. They were later introduced to microcomputers and became interested in AI after reading a novel and seeing a documentary.\",\n",
            "\"Confidence 1\": 0.95\n",
            "},\n",
            "{\n",
            "\"Summary 2\": \"The author discusses their decision to switch from studying philosophy to AI in college. They were disappointed with the lack of depth in philosophy courses and saw potential in the field of AI. They began teaching themselves AI and Lisp.\",\n",
            "\"Confidence 2\": 0.9\n",
            "},\n",
            "{\n",
            "\"Summary 3\": \"The author reflects on their realization that AI, as practiced at the time, was not going to lead to true understanding of natural language. They decided to focus on Lisp instead and wrote a book about Lisp hacking.\",\n",
            "\"Confidence 3\": 0.85\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "print(run_mistral(message))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8649ad18-10c0-4fb0-92cc-052e2ab75342",
      "metadata": {
        "id": "8649ad18-10c0-4fb0-92cc-052e2ab75342"
      },
      "source": [
        "### Strategies we used:\n",
        "- JSON output: For facilitating downstream tasks, JSON format output is frequently preferred. We can enable the JSON mode by setting the response_format to `{\"type\": \"json_object\"}` and specify in the prompt that “You will only respond with a JSON object with the key Summary and Confidence.” Specifying these keys within the JSON object is beneficial for clarity and consistency.\n",
        "- Higher Temperature: In this example, we increase the temperature score to encourage the model to be more creative and output three generated summaries that are different from each other.  \n",
        "\n",
        "\n",
        "## Introduce an evaluation step\n",
        "We can also add a second step in the prompt for evaluation.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "id": "6300a485-43be-4bc4-9088-9e6c2a365b6c",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6300a485-43be-4bc4-9088-9e6c2a365b6c",
        "outputId": "7c36246c-8518-4455-e4a9-06e3735b313f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Step 1:\n",
            "\n",
            "Summary 1: This essay recounts the author's diverse experiences in writing, programming, and painting, and how these pursuits intertwined throughout their life. The author describes their journey from writing short stories to programming on early computers, and their eventual focus on Lisp hacking and painting.\n",
            "\n",
            "Summary 2: In this essay, the author shares their personal journey through various interests and careers, including writing, programming, and art. They discuss their experiences with early computing, their love for Lisp programming, and their eventual decision to pursue art, leading to their time at art schools in the US and Italy.\n",
            "\n",
            "Summary 3: The author of this essay chronicles their life's exploration of writing, programming, and art. They delve into their early experiences with writing and programming, their affinity for Lisp, and their pursuit of art, which led them to study at different art institutions and live in various places.\n",
            "\n",
            "Step 2:\n",
            "\n",
            "The best summary among the three is Summary 1. This summary provides a clear and concise overview of the author's experiences in writing, programming, and painting. It captures the key points of the essay, such as the author's journey from writing short stories to programming on early computers, and their eventual focus on Lisp hacking and painting. The summary is relevant to the speech content and presents the information in a coherent manner.\n"
          ]
        }
      ],
      "source": [
        "message = f\"\"\"\n",
        "You are given an essay text and need to provide summaries and evaluate them.\n",
        "\n",
        "# Essay:\n",
        "{essay}\n",
        "\n",
        "Step 1: In this step, provide three short summaries of the given essay. Each summary should be clear, concise, and capture the key points of the speech. Aim for around 2-3 sentences for each summary.\n",
        "Step 2: Evaluate the three summaries from Step 1 and rate which one you believe is the best. Explain your choice by pointing out specific reasons such as clarity, completeness, and relevance to the speech content.\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "print(run_mistral(message))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ba076c25-9536-4ba1-a730-0a3ce6fe24f0",
      "metadata": {
        "id": "ba076c25-9536-4ba1-a730-0a3ce6fe24f0"
      },
      "source": [
        "## Employ another LLM for evaluation\n",
        "In production systems, it is common to employ another LLM for evaluation so that the evaluation step can be separate from the generation step.  \n",
        "\n",
        "- Step 1: use the first LLM to generate three summaries\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "id": "d5265085-77fd-433a-8d9d-d89ffac76543",
      "metadata": {
        "id": "d5265085-77fd-433a-8d9d-d89ffac76543"
      },
      "outputs": [],
      "source": [
        "message = f\"\"\"\n",
        "Provide three short summaries of the given essay. Each summary should be clear, concise, and capture the key points of the essay.\n",
        "Aim for around 2-3 sentences for each summary.\n",
        "\n",
        "# essay:\n",
        "{essay}\n",
        "\n",
        "\"\"\"\n",
        "summaries = run_mistral(message)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "id": "4372a9fe-d596-4f36-86c6-3e3c01c74c38",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4372a9fe-d596-4f36-86c6-3e3c01c74c38",
        "outputId": "6d15dd31-c8ba-4d52-afb8-a29a95d58c88"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "1. The author recounts their experiences with writing and programming before college, describing their early attempts at writing short stories and their first forays into programming on an IBM 1401 using Fortran. They remember being puzzled by the limited capabilities of the machine and not being able to do much with it.\n",
            "2. The author discusses how the advent of microcomputers changed their programming experience, allowing for more interactivity and direct response to keystrokes. They remember being impressed and envious of a friend who built a microcomputer kit and was able to type programs directly into the computer. The author eventually convinced their father to buy a TRS-80, which they used to write simple games, a program to predict model rocket trajectories, and a word processor.\n",
            "3. The author describes their college experience, during which they originally planned to study philosophy but ultimately switched to AI after finding philosophy classes boring. They were drawn to AI by a novel featuring an intelligent computer and a documentary about a natural language processing program called SHRDLU. Despite not having any classes in AI at their college, they began teaching themselves Lisp and working on a project to reverse-engineer SHRDLU. They chose \"Artificial Intelligence\" as their major, but were dismayed to find that it was printed with scare quotes on their diploma.\n",
            "\n",
            "------------\n",
            "\n",
            "1. The author writes about their early experiences with writing and programming before college. They attempted to write short stories, but found they lacked plot and depth. In 9th grade, they started programming on an IBM 1401, using an early version of Fortran. They had to type programs on punch cards and run them on a machine that printed the results. However, the author found it difficult to figure out what to do with the computer, as they had no data to input and did not know enough math to do complex calculations.\n",
            "2. The author discusses how the introduction of microcomputers changed the way they thought about programming. Unlike the IBM 1401, microcomputers could respond to keystrokes in real-time and did not require punch cards. The author's friend built a microcomputer from a kit, and the author was impressed by the ability to type programs directly into the computer. The author eventually convinced their father to buy a TRS-80 microcomputer, which they used to write simple games, a program to predict the trajectory of model rockets, and a word processor for their father.\n",
            "3. The author describes their college experience and their initial plan to study philosophy. However, they found philosophy classes boring and decided to switch to artificial intelligence (AI). They were inspired by a novel called \"The Moon is a Harsh Mistress\" and a documentary about a natural language processing program called SHRDLU. The author began teaching themselves Lisp, as it was the language of AI at the time, and worked on a project to reverse-engineer SHRDLU. Despite choosing \"Artificial Intelligence\" as their major, the author was disappointed to find that it was printed with scare quotes on their diploma.\n",
            "\n",
            "----------\n",
            "\n",
            "1. In this essay, the author recounts their experiences with writing and programming before college. They attempted to write short stories but found them lacking in plot and depth. In 9th grade, they began programming on an IBM 1401 using an early version of Fortran. They struggled to figure out what to do with the computer, as they lacked data to input and mathematical knowledge for complex calculations.\n",
            "2. The author describes the impact of microcomputers on their programming experience. Unlike the IBM 1401, microcomputers could respond to keystrokes in real-time, eliminating the need for punch cards. The author's friend built a microcomputer from a kit, and the author was impressed by the ability to type programs directly into the machine. The author convinced their father to buy a TRS-80 microcomputer, which they used to write simple games, a program to predict the trajectory of model rockets, and a word processor.\n",
            "3. The author discusses their college experience and their initial plan to study philosophy. However, they found philosophy classes uninteresting and decided to switch to artificial intelligence (AI). They were influenced by a novel called \"The Moon is a Harsh Mistress\" and a documentary about a natural language processing program called SHRDLU. The author taught themselves Lisp and worked on a project to reverse-engineer SHRDLU. Despite choosing \"Artificial Intelligence\" as their major, the author was dismayed to find that it was printed with scare quotes on their diploma.\n"
          ]
        }
      ],
      "source": [
        "print(summaries)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "58fd5f25-ab9a-4547-83ab-48638517b7ef",
      "metadata": {
        "id": "58fd5f25-ab9a-4547-83ab-48638517b7ef"
      },
      "source": [
        "- Step 2: use another LLM to rate the generated summaries\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "id": "b755adc8-c277-4421-8925-3d70e5ee39e7",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b755adc8-c277-4421-8925-3d70e5ee39e7",
        "outputId": "bd4c389d-b703-47b7-f4a8-42507183c52a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "All three summaries accurately capture the main points of the essay, but the third summary is the best. It is the most clear and concise, providing a succinct overview of the author's experiences with writing and programming before college, the impact of microcomputers on their programming experience, and their college experience and decision to switch to artificial intelligence. The third summary also effectively uses specific details from the essay to illustrate the author's experiences and decisions, making it the most complete and relevant to the essay content.\n"
          ]
        }
      ],
      "source": [
        "message = f\"\"\"\n",
        "You are given an essay and three summaries of the essay. Evaluate the three summaries and rate which one you believe is the best.\n",
        "Explain your choice by pointing out specific reasons such as clarity, completeness, and relevance to the essay content.\n",
        "\n",
        "# Essay:\n",
        "{essay}\n",
        "\n",
        "# Summaries\n",
        "{summaries}\n",
        "\n",
        "\"\"\"\n",
        "print(run_mistral(message))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3e96d61a-9dbc-4f8c-9d80-d2260c5f3e61",
      "metadata": {
        "id": "3e96d61a-9dbc-4f8c-9d80-d2260c5f3e61"
      },
      "source": [
        "### Strategies we used:\n",
        "- **LLM chaining**: In this example, we chain two LLMs in a sequence, where the output from the first LLM serves as the input for the second LLM. The method of chaining LLMs can be adapted to suit your specific use cases. For instance, you might choose to employ three LLMs in a chain, where the output of two LLMs is funneled into the third LLM. While LLM chaining offers flexibility, it's important to consider that it may result in additional API calls and potentially increased costs.\n",
        "\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "jupytext": {
      "formats": "ipynb,py:light"
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}