Documentation on how to use structured output with AgentChat agents

[BenConstable9]

What feature would you like to be added?

Support for open ai structured output mode in agent chat. The ability to give an agent a Pydantic class that it must adhere to in the response.

This could be dumped to json as a message for intermediate agents?

Why is this needed?

This would make integrating AutoGen with api requests easier. e.g. consider an answer agent at the end of a complex agentic group chat. It might need to formulate the answer in a specific json format, once it takes into account all the previous messages.

OpenAI structured data mode makes the likelihood of a correct JSON structure far more likely than prompting alone.

[rysweet]

thanks @BenConstable9

[rysweet]

@ekzhu - I feel like this has been discussed recently - is there another relevant issue?

[ekzhu]

There was a PR a while back that add the response format option to model client. It is already available for OpenAIChatCompletionClient and AzureOpenAIChatCompletionClient.

https://microsoft.github.io/autogen/stable/user-guide/core-user-guide/cookbook/structured-output-agent.html

So a fix can be to add response_format parameter to AssistantAgent, and it accepts a Pydantic type.

[ekzhu]

Update. Submitted a fix #5116

So a fix can be to add response_format parameter to AssistantAgent, and it accepts a Pydantic type.

This is not necessary. Just need to set response_format with the model client.

import asyncio
from typing import Literal
from pydantic import BaseModel
from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.ui import Console
from autogen_ext.models.openai import OpenAIChatCompletionClient

class AgentResponse(BaseModel):
    thoughts: str
    response: Literal["happy", "sad", "neutral"]


async def main() -> None:
    model_client = OpenAIChatCompletionClient(
        model="gpt-4o", 
        response_format=AgentResponse, # type: ignore
    )
    agent = AssistantAgent(
        "assistant", 
        model_client=model_client, 
        system_message="Categorize the input as happy, sad, or neutral following the JSON format.",
    )
    await Console(agent.run_stream(task="I am happy."))

asyncio.run(main())

---------- user ----------
I am happy.
---------- assistant ----------
{"thoughts":"The user explicitly states that they are happy without any indication of sadness or neutrality.","response":"happy"}

So for this issue let's just update the documentation to show how to do this with AssistantAgent.

[BenConstable9]

Thanks, I will try it out

[trungnguyen21]

Does this work with AzureOpenAIChatCompletionClient? I could not get it to work with it (I'm using GPT-4o)

[ekzhu]

Does this work with AzureOpenAIChatCompletionClient? I could not get it to work with it (I'm using GPT-4o)

It should, did you use the right model version? Read: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/structured-outputs?tabs=python-secure

[trungnguyen21]

Does this work with AzureOpenAIChatCompletionClient? I could not get it to work with it (I'm using GPT-4o)

It should, did you use the right model version? Read: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/structured-outputs?tabs=python-secure

Thank you! For anyone looking for solution: I used model 2024-08-01-preview and did also a little configuration on the OpenAI client to get it to work: openai/openai-python#1733