From 90c948d9419442e69b7f08666271a00e213a76db Mon Sep 17 00:00:00 2001 From: mrmer1 Date: Wed, 27 Nov 2024 16:25:27 +0800 Subject: [PATCH] add notebook - tool use with vs without structured outputs --- ...ol_use_structured_outputs_comparison.ipynb | 508 ++++++++++++++++++ 1 file changed, 508 insertions(+) create mode 100644 notebooks/guides/Tool_use_structured_outputs_comparison.ipynb diff --git a/notebooks/guides/Tool_use_structured_outputs_comparison.ipynb b/notebooks/guides/Tool_use_structured_outputs_comparison.ipynb new file mode 100644 index 0000000..a4ecb97 --- /dev/null +++ b/notebooks/guides/Tool_use_structured_outputs_comparison.ipynb @@ -0,0 +1,508 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# [Comparison: Tool Use With vs Without Structured Outputs](#toc0_)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This notebook compares the output of Cohere's tool use with and without the Structured Outputs feature. It highlights the benefits of using Structured Outputs where the tool calling output is guaranteed to follow the specified schema, thus eliminating the issues of model hallucinations.\n", + "\n", + "We'll examine different scenarios:\n", + "- Tool name generation\n", + "- Tool parameter generation\n", + "- Tool parameter data types generation" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "First, we make the necessary imports and set up the Cohere client.\n", + "\n", + "We'll also create a couple of helper functions to call the Chat API.\n", + "\n", + "For each scenario, we'll use the `display_results` function to make a few calls without and with structured outputs enabled (via the `strict_tools` parameter)." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "import requests, time, json\n", + "import cohere\n", + "import os\n", + "co = cohere.ClientV2(os.getenv(\"COHERE_API_KEY\")) " + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [], + "source": [ + "def get_response(message, tools, strict_tools, model, preamble=None):\n", + " url = \"https://api.cohere.ai/v2/chat\"\n", + " headers = {\n", + " \"Content-Type\": \"application/json\",\n", + " \"Accept\": \"application/json\",\n", + " \"Authorization\": f\"Bearer {os.getenv('COHERE_API_KEY')}\",\n", + " }\n", + "\n", + " if preamble is not None:\n", + " messages = [{\"role\": \"system\", \"content\": preamble},\n", + " {\"role\": \"user\", \"content\": message}]\n", + " else:\n", + " messages = [{\"role\": \"user\", \"content\": message}]\n", + " \n", + " payload = {\n", + " \"messages\": messages,\n", + " \"temperature\": 0,\n", + " \"stream\": False,\n", + " \"model\": model,\n", + " \"strict_tools\": strict_tools,\n", + " \"tools\": tools,\n", + " }\n", + "\n", + " response = requests.post(url, headers=headers, data=json.dumps(payload))\n", + "\n", + " if response.status_code == 200:\n", + " return response.json()\n", + " else:\n", + " return f\"Error: {response.status_code}, {response.text}\"" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [], + "source": [ + "def display_results(message, tools, model=\"command-r-08-2024\", preamble=None, num_gen=3):\n", + " print(\"USER MESSAGE:\\n\" + message)\n", + " print(\"-\"*30)\n", + " print(\"TOOLS AVAILABLE:\")\n", + " for tool in tools:\n", + " print(tool, \"\\n\" + \"-\"*10) \n", + " for strict_tools in [False, True]:\n", + " print(\"WITH STRUCTURED OUTPUTS:\" if strict_tools else \"WITHOUT STRUCTURED OUTPUTS\")\n", + " print(\"-\"*30)\n", + " for i in range(num_gen):\n", + " print(f\"TOOL CALLS (ATTEMPT #{i + 1}):\")\n", + " response = get_response(message, tools, strict_tools, model, preamble)\n", + " if isinstance(response, str) and response.startswith(\"Error\"):\n", + " print(response)\n", + " elif 'tool_calls' in response['message']:\n", + " for tc in response['message']['tool_calls']:\n", + " print(f\"Tool name: {tc['function']['name']} | Parameters: {tc['function']['arguments']}\")\n", + " print(\"-\"*100)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## [Eliminate hallucinations of tool names](#toc0_)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In this example, we demonstrate how structured outputs help prevent tool name hallucinations.\n", + "\n", + "When making the API call without structured outputs (strict_tools=False), the model returns hallucinated tool names that don't match our defined tools, resulting in 422 errors.\n", + "\n", + "However, when using structured outputs (strict_tools=True), the model is constrained to only generate the explicitly defined tool names, leading to successful tool calls with valid tool names." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "USER MESSAGE:\n", + "How can I perform a garbage collection test using the data from the 'humongous-test-case.json', execute a custom garbage collector, verify the object references using the `referenceChecker` function, and analyze the garbage collector log named 'gc-analysis.log' to ensure it contains 'GC pause' but does not contain 'OutOfMemoryError'?\n", + "------------------------------\n", + "TOOLS AVAILABLE:\n", + "{'type': 'function', 'function': {'name': 'TestObjectGraphAfterGC_doTesting', 'description': 'Executes a test that allocates an object graph based on the provided test case data, runs garbage collection, checks the object graph references, and verifies specific entries in the garbage collector log.', 'parameters': {'type': 'object', 'properties': {'testcaseData': {'type': 'string', 'description': 'The data for the test case to allocate the object graph. This is Java String type parameter in string representation.'}, 'doGC': {'type': 'string', 'description': 'A Runnable that triggers garbage collection. This parameter can be of any type of Java object in string representation.'}, 'checker': {'type': 'string', 'description': 'A Consumer that checks the object references after garbage collection. This parameter can be of any type of Java object in string representation.'}, 'gcLogName': {'type': 'string', 'description': 'The name of the garbage collector log file. This is Java String type parameter in string representation.'}, 'shouldContain': {'type': 'string', 'description': 'A list of strings that should be present in the garbage collector log. This is Java ArrayList type parameter in string representation. The list elements are of type String; they are not in string representation.'}, 'shouldNotContain': {'type': 'string', 'description': 'A list of strings that should not be present in the garbage collector log. This is Java ArrayList type parameter in string representation. The list elements are of type String; they are not in string representation.'}}, 'required': ['testcaseData']}}} \n", + "----------\n", + "WITHOUT STRUCTURED OUTPUTS\n", + "------------------------------\n", + "TOOL CALLS (ATTEMPT #1):\n", + "Error: 422, {\"message\":\"No valid tool call or response generated\"}\n", + "----------------------------------------------------------------------------------------------------\n", + "TOOL CALLS (ATTEMPT #2):\n", + "Error: 422, {\"message\":\"No valid tool call or response generated\"}\n", + "----------------------------------------------------------------------------------------------------\n", + "TOOL CALLS (ATTEMPT #3):\n", + "Error: 422, {\"message\":\"No valid tool call or response generated\"}\n", + "----------------------------------------------------------------------------------------------------\n", + "WITH STRUCTURED OUTPUTS:\n", + "------------------------------\n", + "TOOL CALLS (ATTEMPT #1):\n", + "Tool name: TestObjectGraphAfterGC_doTesting | Parameters: {\"checker\":\"referenceChecker\",\"doGC\":\"customGarbageCollector\",\"gcLogName\":\"gc-analysis.log\",\"shouldContain\":\"['GC pause']\",\"shouldNotContain\":\"['OutOfMemoryError']\",\"testcaseData\":\"humongous-test-case.json\"}\n", + "----------------------------------------------------------------------------------------------------\n", + "TOOL CALLS (ATTEMPT #2):\n", + "Tool name: TestObjectGraphAfterGC_doTesting | Parameters: {\"checker\":\"referenceChecker\",\"doGC\":\"customGarbageCollector\",\"gcLogName\":\"gc-analysis.log\",\"shouldContain\":\"['GC pause']\",\"shouldNotContain\":\"['OutOfMemoryError']\",\"testcaseData\":\"humongous-test-case.json\"}\n", + "----------------------------------------------------------------------------------------------------\n", + "TOOL CALLS (ATTEMPT #3):\n", + "Tool name: TestObjectGraphAfterGC_doTesting | Parameters: {\"checker\":\"referenceChecker\",\"doGC\":\"customGarbageCollector\",\"gcLogName\":\"gc-analysis.log\",\"shouldContain\":\"['GC pause']\",\"shouldNotContain\":\"['OutOfMemoryError']\",\"testcaseData\":\"humongous-test-case.json\"}\n", + "----------------------------------------------------------------------------------------------------\n" + ] + } + ], + "source": [ + "tools = [\n", + " {\n", + " \"type\": \"function\",\n", + " \"function\": {\n", + " \"name\": \"TestObjectGraphAfterGC_doTesting\",\n", + " \"description\": \"Executes a test that allocates an object graph based on the provided test case data, runs garbage collection, checks the object graph references, and verifies specific entries in the garbage collector log.\",\n", + " \"parameters\": {\n", + " \"type\": \"object\",\n", + " \"properties\": {\n", + " \"testcaseData\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The data for the test case to allocate the object graph. This is Java String type parameter in string representation.\"\n", + " },\n", + " \"doGC\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"A Runnable that triggers garbage collection. This parameter can be of any type of Java object in string representation.\"\n", + " },\n", + " \"checker\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"A Consumer that checks the object references after garbage collection. This parameter can be of any type of Java object in string representation.\"\n", + " },\n", + " \"gcLogName\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The name of the garbage collector log file. This is Java String type parameter in string representation.\"\n", + " },\n", + " \"shouldContain\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"A list of strings that should be present in the garbage collector log. This is Java ArrayList type parameter in string representation. The list elements are of type String; they are not in string representation.\"\n", + " },\n", + " \"shouldNotContain\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"A list of strings that should not be present in the garbage collector log. This is Java ArrayList type parameter in string representation. The list elements are of type String; they are not in string representation.\"\n", + " }\n", + " },\n", + " \"required\": [\"testcaseData\"]\n", + " }\n", + " }\n", + " }\n", + " ]\n", + "\n", + "message = \"How can I perform a garbage collection test using the data from the 'humongous-test-case.json', execute a custom garbage collector, verify the object references using the `referenceChecker` function, and analyze the garbage collector log named 'gc-analysis.log' to ensure it contains 'GC pause' but does not contain 'OutOfMemoryError'?\"\n", + "\n", + "display_results(message, tools, model=\"command-r-08-2024\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## [Eliminate hallucinations of tool parameters](#toc0_)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In this example, we demonstrate how structured outputs help prevent tool parameter hallucinations.\n", + "\n", + "When making the API call without structured outputs (strict_tools=False), the model returns hallucinated a tool name (`_from`) when it should be `_fromj8K2mN9pQ4rS7vW`.\n", + "\n", + "However, when using structured outputs (`strict_tools=True`), the model is constrained to only generate the explicitly defined tool parameters, leading to successful tool calls with valid parameter names." + ] + }, + { + "cell_type": "code", + "execution_count": 57, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "USER MESSAGE:\n", + "I'm travelling for work on business and need to find train tickets from Sacramento to Fresno on the 10th of March 2023.\n", + "------------------------------\n", + "TOOLS AVAILABLE:\n", + "{'type': 'function', 'function': {'name': 'Trains_1_GetTrainTickets', 'description': 'Reserve tickets for a train journey by providing journey details, the number of passengers, and seating class preferences.', 'parameters': {'type': 'object', 'properties': {'_from': {'type': 'string', 'description': \"The departure city for the train journey, in the format of 'City, State', such as 'Los Angeles, CA'.\"}, 'to': {'type': 'string', 'description': \"The arrival city for the train journey, in the format of 'City, State', such as 'New York, NY'.\"}, 'date_of_journey': {'type': 'string', 'description': \"The date of the train journey in the format 'YYYY-MM-DD', such as '2023-04-15'.\"}, 'journey_start_time': {'type': 'string', 'description': \"The start time of the train journey in 24-hour format 'HH:MM', such as '14:30'.\"}, 'number_of_adults': {'type': 'integer', 'description': 'The number of adults to reserve train tickets for. Must be between 1 to 5 adults.'}, 'trip_protection': {'type': 'string', 'description': 'Indicates whether to add trip protection to the reservation for an additional fee. Trip protection provides refund options and assistance in case of trip disruptions.'}, 'class': {'type': 'string', 'description': \"The fare class for the train reservation. Options include 'Value', 'Flexible', and 'Business'. The default value is: Value\"}}, 'required': ['_from', 'to', 'date_of_journey', 'journey_start_time', 'number_of_adults', 'trip_protection', 'class']}}} \n", + "----------\n", + "{'type': 'function', 'function': {'name': 'Trains_1_FindTrains', 'description': 'Finds available train options for a specified route on a given date, allowing users to select a preferred travel class.', 'parameters': {'type': 'object', 'properties': {'_fromj8K2mN9pQ4rS7vW': {'type': 'string', 'description': 'The city where the train journey will start.'}, 'to': {'type': 'string', 'description': 'The destination city for the train journey.'}, 'date_of_journey': {'type': 'string', 'description': \"The date of the train journey, in the format of 'YYYY-MM-DD'.\"}, 'class': {'type': 'string', 'description': 'The fare class for the train reservation. The default value is: Value'}, 'number_of_adults': {'type': 'integer', 'description': 'The number of adults to reserve train tickets for. The default value is: 1.'}}, 'required': ['_fromj8K2mN9pQ4rS7vW', 'to', 'date_of_journey', 'class', 'number_of_adults']}}} \n", + "----------\n", + "WITHOUT STRUCTURED OUTPUTS\n", + "------------------------------\n", + "TOOL CALLS (ATTEMPT #1):\n", + "Tool name: Trains_1_FindTrains | Parameters: {\"_from\":\"Sacramento\",\"class\":\"Business\",\"date_of_journey\":\"2023-03-10\",\"number_of_adults\":1,\"to\":\"Fresno\"}\n", + "----------------------------------------------------------------------------------------------------\n", + "TOOL CALLS (ATTEMPT #2):\n", + "Tool name: Trains_1_FindTrains | Parameters: {\"_from\":\"Sacramento\",\"class\":\"Business\",\"date_of_journey\":\"2023-03-10\",\"number_of_adults\":1,\"to\":\"Fresno\"}\n", + "----------------------------------------------------------------------------------------------------\n", + "TOOL CALLS (ATTEMPT #3):\n", + "Tool name: Trains_1_FindTrains | Parameters: {\"_from\":\"Sacramento\",\"class\":\"Business\",\"date_of_journey\":\"2023-03-10\",\"number_of_adults\":1,\"to\":\"Fresno\"}\n", + "----------------------------------------------------------------------------------------------------\n", + "WITH STRUCTURED OUTPUTS:\n", + "------------------------------\n", + "TOOL CALLS (ATTEMPT #1):\n", + "Tool name: Trains_1_FindTrains | Parameters: {\"_fromj8K2mN9pQ4rS7vW\":\"Sacramento\",\"class\":\"Business\",\"date_of_journey\":\"2023-03-10\",\"number_of_adults\":1,\"to\":\"Fresno\"}\n", + "----------------------------------------------------------------------------------------------------\n", + "TOOL CALLS (ATTEMPT #2):\n", + "Tool name: Trains_1_FindTrains | Parameters: {\"_fromj8K2mN9pQ4rS7vW\":\"Sacramento\",\"class\":\"Business\",\"date_of_journey\":\"2023-03-10\",\"number_of_adults\":1,\"to\":\"Fresno\"}\n", + "----------------------------------------------------------------------------------------------------\n", + "TOOL CALLS (ATTEMPT #3):\n", + "Tool name: Trains_1_FindTrains | Parameters: {\"_fromj8K2mN9pQ4rS7vW\":\"Sacramento\",\"class\":\"Business\",\"date_of_journey\":\"2023-03-10\",\"number_of_adults\":1,\"to\":\"Fresno\"}\n", + "----------------------------------------------------------------------------------------------------\n" + ] + } + ], + "source": [ + "tools = [\n", + " {\n", + " \"type\": \"function\",\n", + " \"function\": {\n", + " \"name\": \"Trains_1_GetTrainTickets\",\n", + " \"description\": \"Reserve tickets for a train journey by providing journey details, the number of passengers, and seating class preferences.\",\n", + " \"parameters\": {\n", + " \"type\": \"object\",\n", + " \"properties\": {\n", + " \"_from\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The departure city for the train journey, in the format of 'City, State', such as 'Los Angeles, CA'.\",\n", + " },\n", + " \"to\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The arrival city for the train journey, in the format of 'City, State', such as 'New York, NY'.\",\n", + " },\n", + " \"date_of_journey\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The date of the train journey in the format 'YYYY-MM-DD', such as '2023-04-15'.\",\n", + " },\n", + " \"journey_start_time\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The start time of the train journey in 24-hour format 'HH:MM', such as '14:30'.\",\n", + " },\n", + " \"number_of_adults\": {\n", + " \"type\": \"integer\",\n", + " \"description\": \"The number of adults to reserve train tickets for. Must be between 1 to 5 adults.\",\n", + " },\n", + " \"trip_protection\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"Indicates whether to add trip protection to the reservation for an additional fee. Trip protection provides refund options and assistance in case of trip disruptions.\",\n", + " },\n", + " \"class\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The fare class for the train reservation. Options include 'Value', 'Flexible', and 'Business'. The default value is: Value\",\n", + " },\n", + " },\n", + " \"required\": [\n", + " \"_from\",\n", + " \"to\",\n", + " \"date_of_journey\",\n", + " \"journey_start_time\",\n", + " \"number_of_adults\",\n", + " \"trip_protection\",\n", + " \"class\",\n", + " ],\n", + " },\n", + " },\n", + " },\n", + " {\n", + " \"type\": \"function\",\n", + " \"function\": {\n", + " \"name\": \"Trains_1_FindTrains\",\n", + " \"description\": \"Finds available train options for a specified route on a given date, allowing users to select a preferred travel class.\",\n", + " \"parameters\": {\n", + " \"type\": \"object\",\n", + " \"properties\": {\n", + " \"_fromj8K2mN9pQ4rS7vW\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The city where the train journey will start.\",\n", + " },\n", + " \"to\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The destination city for the train journey.\",\n", + " },\n", + " \"date_of_journey\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The date of the train journey, in the format of 'YYYY-MM-DD'.\",\n", + " },\n", + " \"class\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The fare class for the train reservation. The default value is: Value\",\n", + " },\n", + " \"number_of_adults\": {\n", + " \"type\": \"integer\",\n", + " \"description\": \"The number of adults to reserve train tickets for. The default value is: 1.\",\n", + " },\n", + " },\n", + " \"required\": [\n", + " \"_fromj8K2mN9pQ4rS7vW\",\n", + " \"to\",\n", + " \"date_of_journey\",\n", + " \"class\",\n", + " \"number_of_adults\",\n", + " ],\n", + " },\n", + " },\n", + " },\n", + "]\n", + "\n", + "message = \"I'm travelling for work on business and need to find train tickets from Sacramento to Fresno on the 10th of March 2023.\"\n", + "\n", + "display_results(message, tools, model=\"command-r\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## [Make sure all tool parameters have the expected data type](#toc0_)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Without structured outputs, the model may generate tool calls that don't respect parameter types defined in the tool schema.\n", + "\n", + "Here, when asked to save game progress with `stage` as integer', the tool call is generated with `stage` as a string instead.\n", + "\n", + "With structured outputs enabled, the model is forced to respect parameter types. This time, the tool call is generated with `stage` as an integer.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "USER MESSAGE:\n", + "How to save game progress at stage 7 in easy mode and stage 3 in hard mode? 'stage' should be a string.\n", + "------------------------------\n", + "TOOLS AVAILABLE:\n", + "{'type': 'function', 'function': {'name': 'game_save_progress', 'description': \"Save the current state of a player's game, given the stage, level and game mode.\", 'parameters': {'type': 'object', 'properties': {'stage': {'type': 'integer', 'description': 'The current stage in the game the player has reached.'}, 'mode': {'type': 'string', 'description': 'The game mode. Available modes are easy or hard.'}, 'level': {'type': 'string', 'description': \"The player's level. The default value is: user\"}}, 'required': ['stage', 'mode', 'level']}}} \n", + "----------\n", + "WITHOUT STRUCTURED OUTPUTS\n", + "------------------------------\n", + "TOOL CALLS (ATTEMPT #1):\n", + "Tool name: game_save_progress | Parameters: {\"level\":\"user\",\"mode\":\"easy\",\"stage\":\"7\"}\n", + "Tool name: game_save_progress | Parameters: {\"level\":\"user\",\"mode\":\"hard\",\"stage\":\"3\"}\n", + "----------------------------------------------------------------------------------------------------\n", + "TOOL CALLS (ATTEMPT #2):\n", + "Tool name: game_save_progress | Parameters: {\"level\":\"user\",\"mode\":\"easy\",\"stage\":\"7\"}\n", + "Tool name: game_save_progress | Parameters: {\"level\":\"user\",\"mode\":\"hard\",\"stage\":\"3\"}\n", + "----------------------------------------------------------------------------------------------------\n", + "TOOL CALLS (ATTEMPT #3):\n", + "Tool name: game_save_progress | Parameters: {\"level\":\"user\",\"mode\":\"easy\",\"stage\":\"7\"}\n", + "Tool name: game_save_progress | Parameters: {\"level\":\"user\",\"mode\":\"hard\",\"stage\":\"3\"}\n", + "----------------------------------------------------------------------------------------------------\n", + "WITH STRUCTURED OUTPUTS:\n", + "------------------------------\n", + "TOOL CALLS (ATTEMPT #1):\n", + "Tool name: game_save_progress | Parameters: {\"level\":\"user\",\"mode\":\"easy\",\"stage\":7}\n", + "Tool name: game_save_progress | Parameters: {\"level\":\"user\",\"mode\":\"hard\",\"stage\":3}\n", + "----------------------------------------------------------------------------------------------------\n", + "TOOL CALLS (ATTEMPT #2):\n", + "Tool name: game_save_progress | Parameters: {\"level\":\"user\",\"mode\":\"easy\",\"stage\":7}\n", + "Tool name: game_save_progress | Parameters: {\"level\":\"user\",\"mode\":\"hard\",\"stage\":3}\n", + "----------------------------------------------------------------------------------------------------\n", + "TOOL CALLS (ATTEMPT #3):\n", + "Tool name: game_save_progress | Parameters: {\"level\":\"user\",\"mode\":\"easy\",\"stage\":7}\n", + "Tool name: game_save_progress | Parameters: {\"level\":\"user\",\"mode\":\"hard\",\"stage\":3}\n", + "----------------------------------------------------------------------------------------------------\n" + ] + } + ], + "source": [ + "tools = [\n", + " {\n", + " \"type\": \"function\",\n", + " \"function\": {\n", + " \"name\": \"game_save_progress\",\n", + " \"description\": \"Save the current state of a player's game, given the stage, level and game mode.\",\n", + " \"parameters\": {\n", + " \"type\": \"object\",\n", + " \"properties\": {\n", + " \"stage\": {\n", + " \"type\": \"integer\",\n", + " \"description\": \"The current stage in the game the player has reached.\",\n", + " },\n", + " \"mode\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The game mode. Available modes are easy or hard.\",\n", + " },\n", + " \"level\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The player's level. The default value is: user\",\n", + " },\n", + " },\n", + " \"required\": [\"stage\", \"mode\", \"level\"],\n", + " },\n", + " },\n", + " }\n", + "]\n", + "\n", + "message = \"How to save game progress at stage 7 in easy mode and stage 3 in hard mode? 'stage' should be a string.\"\n", + "\n", + "display_results(message, tools, model=\"command-r-08-2024\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Conclusion\n", + "\n", + "The examples above demonstrate how Structured Outputs help enforce constraints in tool calls, which eliminates hallucinations in:\n", + "- tool names\n", + "- tool parameter names\n", + "- tool parameter data types\n", + "\n", + "Another scenario not explicitly shown here is Structured Outputs guarantees that the list of `required` parameters listed in the schema will be generated." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "base", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.4" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +}