gemini-2/live_api_tool_use.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jWESX0tpdrE-"
      },
      "source": [
        "##### Copyright 2024 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "YQvTrJpxzRlJ"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3hp_P0cDzTWp"
      },
      "source": [
        "# Gemini 2.0 - Multimodal live API: Tool use"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OLW8VU78zZOc"
      },
      "source": [
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/cookbook/blob/main/gemini-2/live_api_tool_use.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y7f4kFby0E6j"
      },
      "source": [
        "This notebook provides examples of how to use tools with the multimodal live API with [Gemini 2.0](https://ai.google.dev/gemini-api/docs/models/gemini-v2).\n",
        "\n",
        "The API provides Google Search, Code Execution and Function Calling tools. The earlier Gemini models supported versions of these tools. The biggest change with Gemini 2 (in the Live API) is that, basically, all the tools are handled by Code Execution. With that change, you can use **multiple tools** in a single API call, and the model can use multiple tools in a single code execution block.  \n",
        "\n",
        "This tutorial assumes you are familiar with the Live API, as described in the [this tutorial](https://github.com/google-gemini/cookbook/blob/main/gemini-2/live_api_starter.ipynb)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mfk6YY3G5kqp"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d5027929de8f"
      },
      "source": [
        "### Install SDK\n",
        "\n",
        "The new **[Google Gen AI SDK](https://ai.google.dev/gemini-api/docs/sdks)** provides programmatic access to Gemini 2.0 (and previous models) using both the [Google AI for Developers](https://ai.google.dev/gemini-api/docs) and [Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/overview) APIs. With a few exceptions, code that runs on one platform will run on both. This means that you can prototype an application using the Developer API and then migrate the application to Vertex AI without rewriting your code.\n",
        "\n",
        "More details about this new SDK on the [documentation](https://ai.google.dev/gemini-api/docs/sdks) or in the [Getting started](../gemini-2/get_started.ipynb) notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "46zEFO2a9FFd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/110.3 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m\u001b[90m━━\u001b[0m \u001b[32m102.4/110.3 kB\u001b[0m \u001b[31m3.8 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m110.3/110.3 kB\u001b[0m \u001b[31m2.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/168.2 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m168.2/168.2 kB\u001b[0m \u001b[31m6.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "!pip install -U -q google-genai"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CTIfnvCn9HvH"
      },
      "source": [
        "### Setup your API key\n",
        "\n",
        "To run the following cell, your API key must be stored it in a Colab Secret named `GOOGLE_API_KEY`. If you don't already have an API key, or you're not sure how to create a Colab Secret, see [Authentication](../quickstarts/Authentication.ipynb) for an example."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "A1pkoyZb9Jm3"
      },
      "outputs": [],
      "source": [
        "from google.colab import userdata\n",
        "import os\n",
        "\n",
        "os.environ['GOOGLE_API_KEY']=userdata.get('GOOGLE_API_KEY')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y13XaCvLY136"
      },
      "source": [
        "### Initialize SDK client\n",
        "\n",
        "The client will pickup your API key from the environment variable.\n",
        "To use the live API you need to set the client version to `v1alpha`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "HghvVpbU0Uap"
      },
      "outputs": [],
      "source": [
        "from google import genai\n",
        "\n",
        "client = genai.Client(http_options= {\n",
        "      'api_version': 'v1alpha'\n",
        "})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QOov6dpG99rY"
      },
      "source": [
        "### Select a model\n",
        "\n",
        "Multimodal Live API are a new capability introduced with the [Gemini 2.0](https://ai.google.dev/gemini-api/docs/models/gemini-v2) model. It won't work with previous generation models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "27Fikag0xSaB"
      },
      "outputs": [],
      "source": [
        "model_name = \"gemini-2.0-flash-exp\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pLU9brx6p5YS"
      },
      "source": [
        "### Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "yMG4iLu5ZLgc"
      },
      "outputs": [],
      "source": [
        "import asyncio\n",
        "import contextlib\n",
        "import json\n",
        "import wave\n",
        "\n",
        "from IPython import display\n",
        "\n",
        "from google import genai\n",
        "from google.genai import types"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yrb4aX5KqKKX"
      },
      "source": [
        "### Utilities"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rmfQ-NvFI7Ct"
      },
      "source": [
        "You're going to use the Live API's audio output, the easiest way hear it in Colab is to write the `PCM` data out as a `WAV` file:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "p2aGpzlR-60Q"
      },
      "outputs": [],
      "source": [
        "@contextlib.contextmanager\n",
        "def wave_file(filename, channels=1, rate=24000, sample_width=2):\n",
        "    with wave.open(filename, \"wb\") as wf:\n",
        "        wf.setnchannels(channels)\n",
        "        wf.setsampwidth(sample_width)\n",
        "        wf.setframerate(rate)\n",
        "        yield wf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KfdD9mVxqatm"
      },
      "source": [
        "Use a logger so it's easier to switch on/off debugging messages."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "wgHJgpV9Zw4E"
      },
      "outputs": [],
      "source": [
        "import logging\n",
        "logger = logging.getLogger('Live')\n",
        "#logger.setLevel('DEBUG')  # Switch between \"INFO\" and \"DEBUG\" to toggle debug messages.\n",
        "logger.setLevel('INFO')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4hiaxgUCZSYJ"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LQoca-W7ri0y"
      },
      "source": [
        "Most of the Live API setup will be similar to the [starter tutorial](../gemini-2/live_api_starter.ipynb). Since this tutorial doesn't focus on the realtime interactivity of the API, the code has been simplified: This code uses the Live API, but it only sends a single text prompt, and listens for a single turn of replies."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "lwLZrmW5zR_P"
      },
      "outputs": [],
      "source": [
        "n = 0\n",
        "async def run(prompt, modality='TEXT', tools=None):\n",
        "  global n\n",
        "  if tools is None:\n",
        "    tools=[]\n",
        "\n",
        "  config = {\n",
        "          \"tools\": tools,\n",
        "          \"generation_config\": {\n",
        "              \"response_modalities\": [modality]}}\n",
        "\n",
        "  async with client.aio.live.connect(model=model_name, config=config) as session:\n",
        "    display.display(display.Markdown(prompt))\n",
        "    display.display(display.Markdown('-------------------------------'))\n",
        "    await session.send(prompt, end_of_turn=True)\n",
        "\n",
        "    audio = False\n",
        "    filename = f'audio_{n}.wav'\n",
        "    with wave_file(filename) as wf:\n",
        "      async for response in session.receive():\n",
        "        logger.debug(str(response))\n",
        "\n",
        "        server_content = response.server_content\n",
        "        if server_content is not None:\n",
        "          a = handle_server_content(wf, server_content)\n",
        "          audio = audio or a\n",
        "\n",
        "        tool_call = response.tool_call\n",
        "        if tool_call is not None:\n",
        "          await handle_tool_call(session, tool_call)\n",
        "\n",
        "\n",
        "  if audio:\n",
        "    display.display(display.Audio(filename, autoplay=True))\n",
        "    n = n+1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ngrvxzrf0ERR"
      },
      "source": [
        "Since this tutorial demonstrates several tools, you'll need more code to handle the different types of objects it returns.\n",
        "\n",
        "- The `code_execution` tool can return `executable_code` and `code_execution_result` parts.\n",
        "- The `google_search` tool may attach a `grounding_metadata` object."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "CypjqSb-0C-Q"
      },
      "outputs": [],
      "source": [
        "def handle_server_content(wf, server_content):\n",
        "  audio = False\n",
        "  model_turn = server_content.model_turn\n",
        "  if model_turn:\n",
        "    for part in model_turn.parts:\n",
        "      text = part.text\n",
        "      if text is not None:\n",
        "        display.display(display.Markdown(text))\n",
        "\n",
        "      inline_data = part.inline_data\n",
        "      if inline_data is not None:\n",
        "        print('.', end='')\n",
        "        pcm_data = inline_data.data\n",
        "        wf.writeframes(pcm_data)\n",
        "        audio = True\n",
        "\n",
        "      executable_code = part.executable_code\n",
        "      if executable_code is not None:\n",
        "        display.display(display.Markdown('-------------------------------'))\n",
        "        display.display(display.Markdown(f'``` python\\n{executable_code.code}\\n```'))\n",
        "        display.display(display.Markdown('-------------------------------'))\n",
        "\n",
        "      code_execution_result = part.code_execution_result\n",
        "      if code_execution_result is not None:\n",
        "        display.display(display.Markdown('-------------------------------'))\n",
        "        display.display(display.Markdown(f'```\\n{code_execution_result.output}\\n```'))\n",
        "        display.display(display.Markdown('-------------------------------'))\n",
        "\n",
        "  grounding_metadata = getattr(server_content, 'grounding_metadata', None)\n",
        "  if grounding_metadata is not None:\n",
        "    display.display(\n",
        "        display.HTML(grounding_metadata.search_entry_point.rendered_content))\n",
        "\n",
        "  return audio"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dPnXSNZ5rydM"
      },
      "source": [
        "- Finally, with the `function_declarations` tool, the API may return `tool_call` objects. To keep this code minimal, the `tool_call` handler just replies to every function call with a response of `\"ok\"`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "EmTKF_DtrY4U"
      },
      "outputs": [],
      "source": [
        "async def handle_tool_call(session, tool_call):\n",
        "  for fc in tool_call.function_calls:\n",
        "    tool_response = types.LiveClientToolResponse(\n",
        "        function_responses=[types.FunctionResponse(\n",
        "            name=fc.name,\n",
        "            id=fc.id,\n",
        "            response={'result':'ok'},\n",
        "        )]\n",
        "    )\n",
        "\n",
        "    print('>>> ', tool_response)\n",
        "    await session.send(tool_response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TcNu3zUNsI_p"
      },
      "source": [
        "Try running it for a first time:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ss9I0MRdHbP2"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": "Hello?",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "Hello",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": " there! How can I help you today?\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "await run(prompt=\"Hello?\", tools=None, modality = \"TEXT\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z_BFBLLGp-Ye"
      },
      "source": [
        "## Simple function call"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RMq795G6t2hA"
      },
      "source": [
        "The function calling feature of the API Can handle a wide variety of functions. Support in the SDK is still under construction. So keep this simple just send a minimal function definition: Just the function's name.\n",
        "\n",
        "Note that in the live API function calls are independent of the chat turns. The conversation can continue while a function call is being processed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8Y00qqZZt5L-"
      },
      "outputs": [],
      "source": [
        "turn_on_the_lights = {'name': 'turn_on_the_lights'}\n",
        "turn_off_the_lights = {'name': 'turn_off_the_lights'}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8dCjPmz8nEbv"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": "Turn on the lights",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "``` python\nprint(default_api.turn_on_the_lights())\n\n```",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            ">>>  function_responses=[FunctionResponse(id='function-call-9277519295107782492', name='turn_on_the_lights', response={'result': 'ok'})]\n"
          ]
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "```\n{'result': 'ok'}\n\n```",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "OK",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"Turn on the lights\"\n",
        "\n",
        "tools = [\n",
        "    {'function_declarations': [turn_on_the_lights, turn_off_the_lights]}\n",
        "]\n",
        "\n",
        "await run(prompt, tools=tools, modality = \"TEXT\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eCnCiTbhqE8q"
      },
      "source": [
        "## Code execution"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P4ptRBNY4N8Q"
      },
      "source": [
        "The `code_execution` lets the model write and run python code. Try it on a math problem the model can't solve from memory:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k4dURhC-QoSw"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": "What is the largest prime palindrome under 100000.",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "Okay",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": ", I understand. You're asking for the largest prime number that is also",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": " a palindrome (reads the same forwards and backward) and is less than 1",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "00,000.\n\nHere's my plan:\n\n1. **Generate Palindromes:** I'll need to create a list of",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": " palindromic numbers under 100,000. I'll start from the top, and work my way down since I need the *",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "largest*.\n2. **Check for Primality:** I'll then test each of these palindromes for primality.\n3. **Return the Largest Prime:** The largest prime palindrome encountered will be the answer.\n\nLet's",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": " start by generating and checking the numbers using python.\n\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "``` python\ndef is_palindrome(n):\n    return str(n) == str(n)[::-1]\n\ndef is_prime(n):\n    if n < 2:\n        return False\n    for i in range(2, int(n**0.5) + 1):\n        if n % i == 0:\n            return False\n    return True\n\nlargest_prime_palindrome = 0\nfor i in range(99999, 1, -1):\n    if is_palindrome(i):\n        if is_prime(i):\n          largest_prime_palindrome = i\n          break\n\nprint(f'{largest_prime_palindrome=}')\n\n```",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "```\nlargest_prime_palindrome=98689\n\n```",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "Okay",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": ", I have found the largest prime palindrome under 100,00",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "0.\n\nThe code generated a list of palindromes by checking every number from",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": " 99999 downwards. It then tested each of the numbers to see if it was prime. The first prime palindrome found going downwards was 9",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "8689, so that is the largest one.\n\nTherefore, the answer is 98689.\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt=\"What is the largest prime palindrome under 100000.\"\n",
        "\n",
        "tools = [\n",
        "    {'code_execution': {}}\n",
        "]\n",
        "\n",
        "await run(prompt, tools=tools, modality='TEXT')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ueeerkpX5F-v"
      },
      "source": [
        "## Compositional Function Calling\n",
        "\n",
        "Compositional function calling refers to the ability to combine user defined functions with the `code_execution` tool. The model will write them into larger blocks of code, and then pause execution while it waits for you to send back responses for each call.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XzKyL_Rq5sG3"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": "Can you turn on the lights wait 10s and then turn them off?",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "``` python\nimport time\ndefault_api.turn_on_the_lights()\ntime.sleep(10)\ndefault_api.turn_off_the_lights()\n\n```",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            ">>>  function_responses=[FunctionResponse(id='function-call-6184362039141552989', name='turn_on_the_lights', response={'result': 'ok'})]\n",
            ">>>  function_responses=[FunctionResponse(id='function-call-17968678311537051', name='turn_off_the_lights', response={'result': 'ok'})]\n"
          ]
        }
      ],
      "source": [
        "prompt=\"Can you turn on the lights wait 10s and then turn them off?\"\n",
        "\n",
        "tools = [\n",
        "    {'code_execution': {}},\n",
        "    {'function_declarations': [turn_on_the_lights, turn_off_the_lights]}\n",
        "]\n",
        "\n",
        "await run(prompt, tools=tools, modality='TEXT')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G78sxDEcqHyO"
      },
      "source": [
        "## Google search"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FW10vRPN6UNp"
      },
      "source": [
        "The `google_search` tool lets the model conduct google searches. For example, try asking it about events that are too recent to be in the training data.\n",
        "\n",
        "The search still executes in `AUDIO` mode, but you don't see the detailed results. So switch to text mode to see the full output:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QKvWzROJic60"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": "Can you use google search tell me about the largest earthquake in california the week of Dec 5 2024?",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "``` python\nprint(google_search.search(queries=[\"largest earthquake in California week of December 5 2024\", \"California earthquakes week of December 5 2024\"]))\n\n```",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "```\nLooking up information on Google Search.\n\n```",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "The",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": " largest earthquake in California during the week of December 5, 202",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "4, was a magnitude 7.0 that occurred offshore of Cape Mendoc",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "ino on December 5th, 2024 at 10:44 a.m. PT. The earthquake was located approximately 6",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "0 miles southwest of Ferndale, California and about 45 miles southwest of Eureka.\n\nHere's a summary of what happened:\n\n*   **",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "Magnitude:** The earthquake was measured at a magnitude of 7.0, making it a major seismic event.\n*   **Location:** It was centered offshore, about 60 miles west of Ferndale in Humboldt County, Northern California",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": ". This region is where three tectonic plates meet, making it one of the most seismically active areas in California.\n*   **Tsunami Warning:** A tsunami warning was issued for the coast of Northern California and Southern Oregon following the quake",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": ". The warning extended from Davenport, California, to south of Florence, Oregon and included more than 4.6 million people. People were urged to move to higher ground, and Oregon State Parks closed access to its state park beaches.\n*   **Tsunami Cancelled:** Fortunately, the tsunami warning was cancelled within",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": " a couple hours as no significant waves were reported.\n*   **Aftershocks:** There were numerous aftershocks, including several that were magnitude 4.0 and greater. The strongest aftershock was a magnitude 4.7.\n*   **Impact:** The earthquake caused some shaking as far south as",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": " the Bay Area. There were reports of minor damage including broken windows, ruptured water pipes, and items knocked off store shelves.\n\nThis earthquake was the strongest in the region since 2005 when a magnitude 7.2 earthquake occurred.\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt=\"Can you use google search tell me about the largest earthquake in california the week of Dec 5 2024?\"\n",
        "\n",
        "tools = [\n",
        "   {'google_search': {}}\n",
        "]\n",
        "\n",
        "await run(prompt, tools=tools, modality='TEXT')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HM9y5rwfqKfY"
      },
      "source": [
        "## Multi-tool\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qrxAQjYA6vQX"
      },
      "source": [
        "The biggest difference with the new API however is that you're no longer limited to using 1-tool per request. Try combining those tasks from the previous sections:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QmB_4XPOslyA"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": "  Hey, I need you to do three things for me.\n\n  1. Then compute the largest prime plaindrome under 100000.\n  2. Then use google search to lookup unformation about the largest earthquake in california the week of Dec 5 2024?\n  3. Turn on the lights\n\n  Thanks!\n  ",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "Okay",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": ", I will perform those tasks for you. First, let's find the",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": " largest prime palindrome under 100000.\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "``` python\ndef is_palindrome(n):\n  return str(n) == str(n)[::-1]\n\ndef is_prime(n):\n  if n <= 1:\n    return False\n  if n <= 3:\n    return True\n  if n % 2 == 0 or n % 3 == 0:\n    return False\n  i = 5\n  while i * i <= n:\n    if n % i == 0 or n % (i + 2) == 0:\n      return False\n    i += 6\n  return True\n\nlargest_prime_palindrome = 0\nfor i in range(100000 - 1, 1, -1):\n  if is_palindrome(i) and is_prime(i):\n    largest_prime_palindrome = i\n    break\n\nprint(largest_prime_palindrome)\n\n```",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "```\n98689\n\n```",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "Okay",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": ", the largest prime palindrome under 100000 is 9",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "8689.\n\nNext, I will search for the largest earthquake in",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": " California the week of December 5, 2024.\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "``` python\nconcise_search(\"largest earthquake California week of December 5 2024\", max_num_results=3)\n\n```",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "```\nLooking up information on Google Search.\n\n```",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "Based",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": " on the search results, it appears that a magnitude 7.0 earthquake occurred",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": " off the coast of Cape Mendocino, California on December 5, ",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "2024. This earthquake triggered a brief tsunami warning for Northern California and Southern Oregon, which was later cancelled. There is also mention of a preliminary magnitude",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": " 6.6 quake in some reports but the 7.0 appears to be more accurate for the largest.\n\nFinally, I will turn on the lights",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": ".\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "``` python\ndefault_api.turn_on_the_lights()\n\n```",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": "-------------------------------",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            ">>>  function_responses=[FunctionResponse(id='function-call-16138398244891385862', name='turn_on_the_lights', response={'result': 'ok'})]\n"
          ]
        }
      ],
      "source": [
        "prompt = \"\"\"\\\n",
        "  Hey, I need you to do three things for me.\n",
        "\n",
        "  1. Then compute the largest prime plaindrome under 100000.\n",
        "  2. Then use google search to lookup unformation about the largest earthquake in california the week of Dec 5 2024?\n",
        "  3. Turn on the lights\n",
        "\n",
        "  Thanks!\n",
        "  \"\"\"\n",
        "\n",
        "tools = [\n",
        "    {'google_search': {}},\n",
        "    {'code_execution': {}},\n",
        "    {'function_declarations': [turn_on_the_lights, turn_off_the_lights]}\n",
        "]\n",
        "\n",
        "await run(prompt, tools=tools, modality=\"TEXT\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y0OhM95KkMzl"
      },
      "source": [
        "## Next Steps\n",
        "\n",
        "- For more information about the SDK see the [SDK docs](https://googleapis.github.io/python-genai/)\n",
        "- This tutorial uses the high level SDK, if you're interested in the lower-level details, try the [Websocket version of this tutorial](../gemini-2/websocket/search_tool.ipynb)\n",
        "- This tutorial only covers _basic_ usage of these tools for deeper (and more fun) example see the [Search tool tutorial](../gemini-2/search_tool.ipynb)\n",
        "\n",
        "Or check the other Gemini 2.0 capabilities from the [Cookbook](https://github.com/google-gemini/cookbook/blob/main/gemini-2/), in particular this other [multi-tool](../gemini-2/plotting_and_mapping.ipynb) example and the one about Gemini [spatial capabilities](../gemini-2/spatial_understanding.ipynb)."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "live_api_tool_use.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}