Tool/function calling for chat and chat_async

[wendy-aw]

• Added function calling to chat and chat_async and their respective OpenAI and Anthropic functions.
• tools takes in a list of function schemas while tool_choice indicates whether the model is forced to use a tool.
• The chat functions continue to only output text content if no tools are called. However, if a tool has been called, it will output the tool parameters, tool outputs and the text content if any.

[rishsriv]

Added a bunch more features to this!

Significant refactoring to prevent code duplication

We were previously duplicating a lot of code between the async and sync methods. This is now fixed, with common build_params and process_response functions for each class.

OpenAI is an slight exception to this, where we have a bit of duplication across _process_openai_response_sync and _process_openai_response_async functions (to support async tool calls). We might have to do something similar for Anthropic and Gemini to support tool calling

Simplified way to call tools

We can now simplify pass a list of functions to tools, greatly simplifying their usage! The only constraint here is that the input to a tool be a single Pydantic object.

For example, we can now do

from pydantic import BaseModel

class Numbers(BaseModel):
    a: int = 0
    b: int = 0

def numsum(input: Numbers):
    """
    This function return the sum of two numbers
    """
    return input.a + input.b

def numprod(input: Numbers):
    """
    This function return the product of two numbers
    """
    return input.a * input.b

tools = [numsum, numprod]

result = await chat_async(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": "What is the product of 31283 and 2323, added to 5? Return only the final answer, nothing else.",},
    ],
    tools=tools,
)
print(result.content) # '72670414'

Under the hood, we are converting the function to OpenAI's function definition spec, using get_function_specs in utils_function_calling.py.

Support for tool execution and chaining

We can now execute tools (see above) directly in defog_utils. This is done via execute_tool and execute_tool_async. Whether or not a tool is async depends on whether it has an await statement somewhere in its code.

Additionally, we set parallel_tool_calls = False for all OpenAI tool use requests. This ensures that tools are only called sequentially, and makes it easier for us to pass the output of one tool as the input of another tool.

Specifically, this is done by first passing the entire tool call as a message, followed by the result of the tool. Like so.

# Append the tool calls as an assistant response
request_params["messages"].append({
    "role": "assistant",
    "tool_calls": message.tool_calls,
})

# Append the tool message
request_params["messages"].append(
    {
        "role": "tool",
        "tool_call_id": tool_call.id,
        "content": str(result),
    }
)

Tests for tool calls

We have now tool calls to our tests (and Github CI) as well. To run these tests, you can just run pytest. To run a specific test, run pytest tests/<filename.py>

There are 3 tests here:

• test_tool_use_arithmetic: tests tool chaining
• test_tool_use_async_search: tests running async function via a chat_async
• test_async_tool_in_sync_function: tests running an async tool in a sync function

[rishsriv]

Merging this since Gemini seems to be blocking the server for Github Actions rn, hence the failed action 🙄