Azure OpenAI API Streaming Response Causes AttributeError in pydantic-ai

[dabrodev]

When using pydantic-ai with Azure OpenAI's streaming API, an AttributeError occurs due to the API returning deltas without content fields.

Steps to Reproduce

1. Configure Azure OpenAI client with api_version="2024-12-01-preview"
2. Create an OpenAIModel using the Azure client
3. Use Agent.run_stream() with the model
4. Attempt to process streaming response

Error:

Traceback (most recent call last):
  File ".../pydantic_ai/models/openai.py", line 306, in _get_event_iterator
    content = choice.delta.content
             ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'content'

Analysis
The error occurs in pydantic-ai's OpenAI model implementation where it assumes every delta in the streaming response contains a content field. However, with Azure OpenAI's API, some deltas (like role initialization) may not include content, resulting in choice.delta.content being None.

Expected Behavior
The streaming implementation should handle deltas without content fields gracefully, possibly by:

• Checking if delta.content exists before accessing it
• Skipping deltas without content
• Or handling special message types separately

Current Workaround

Using API version "2023-05-15" works but has limitations:

• Streaming chunks are much larger (several sentences long)
• Less granular streaming experience compared to standard OpenAI API
• Makes the streaming appear more "blocky" rather than word-by-word

Environment:

• pydantic-ai: 0.0.20
• OpenAI Python package version: 1.60.2
• Azure OpenAI API version tested: 2024-12-01-preview

[sydney-runkle]

Thanks for the report @dabrodev, I think we could probably work around this as you've suggested.

[Kludex]

@dabrodev Can you give an snippet that reproduces the issue? I can't reproduce it here.

[Kludex]

I'm happy to reopen this if someone can reproduce the issue.

[martgra]

@Kludex

Dont know what API version in Azure is currently the newest but I encountered this error with the following:

python 3.11
pydantic-ai-slim = {version = "^0.0.40", extras = ["openai"]}

Steps to reproduce. In Azure Open AI ->

1. Set content filter to Asynchronous Filter for given streaming model
2. Use API version: 2024-09-01-preview

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.azure import AzureProvider


system_prompt = """....
"""

model = OpenAIModel(
    "gpt-4o",
    provider=AzureProvider(
        azure_endpoint="....",
        api_version="2024-09-01-preview",
        api_key="....",
    ),
)

agent = Agent(model=model, system_prompt=system_prompt)


@agent.tool_plain
def generate_citation_url(citation: str) -> str:
    """Generates a citation for the given topic formatted as markdown."""
    return f"""[{citation}](www.example.no)"""


async def get_streaming_response(query: str) -> dict:
    """Uses the Pydantic AI agent to process the query in streaming mode."""
    async with agent.run_stream(query) as stream:
        async for chunk in stream.stream_text(delta=True):
            yield chunk

Error

  File "/home/user/dev/rag-backend/api/consumers/streaming_agent.py", line 29, in get_streaming_response
    async for chunk in stream.stream_text(delta=True):
  File "/home/user/.cache/pypoetry/virtualenvs/api-2E0zB5zf-py3.11/lib/python3.11/site-packages/pydantic_ai/result.py", line 303, in stream_text
    async for text in self._stream_response_text(delta=delta, debounce_by=debounce_by):
  File "/home/user/.cache/pypoetry/virtualenvs/api-2E0zB5zf-py3.11/lib/python3.11/site-packages/pydantic_ai/result.py", line 450, in _stream_response_text
    async for text in _stream_text_deltas():
  File "/home/user/.cache/pypoetry/virtualenvs/api-2E0zB5zf-py3.11/lib/python3.11/site-packages/pydantic_ai/result.py", line 445, in _stream_text_deltas
    async for items in group_iter:
  File "/home/user/.cache/pypoetry/virtualenvs/api-2E0zB5zf-py3.11/lib/python3.11/site-packages/pydantic_ai/_utils.py", line 152, in async_iter_groups
    item = done.pop().result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/pypoetry/virtualenvs/api-2E0zB5zf-py3.11/lib/python3.11/site-packages/pydantic_ai/result.py", line 429, in _stream_text_deltas_ungrouped
    async for event in self._stream_response:
  File "/home/user/.cache/pypoetry/virtualenvs/api-2E0zB5zf-py3.11/lib/python3.11/site-packages/pydantic_ai/models/openai.py", line 476, in _get_event_iterator
    content = choice.delta.content
              ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'content'

This is 100% something that occurs from Azure Open AI when the content filter is set to Async - some chunks are just "None"

We used to handle like below before testing pydantic ai

    def _validate_stream(self, chunk):
        if chunk and chunk.choices and chunk.choices[0].delta and chunk.choices[0].delta.content:
            return chunk.choices[0].delta.content

        return ""

[martgra]

@Kludex Is there a possibility to prfioritize this? Kind of a dealbreaker for us using Azure - I guess that means a lot of enterprises.

Important - the error only occurs when using Azure output filtering in async mode (if you are to test). My guess this is on their hand so will file bug to them as well - but need a viable WA.

[martgra]

This is the same bug reported at semantic kernel -> microsoft/semantic-kernel#7250
Their fix is to check for None, as it appears in the stream when content filters are async.