openai won't response with a specific object form, even with specification in the prompt

[MosheAzraf]

Hi,
Im using the new Open-AI library for dotnet,
Im trying to make a service that analyzed an image and return a specific object with values, however, its turns out to return an ugly object with missing properties, and it seems like I can’t make it to return a consist output.
What am I doing wrong ?

using System.Text.Json;
using backend.DTOs;
using OpenAI.Chat;

namespace backend.Services;

public class OpenAiService
{
    private readonly ChatClient _chatClient;
    private readonly ChatCompletionOptions _options;

    public OpenAiService(IConfiguration config)
    {
        var apiKey = config.GetValue<string>("OpenAI:Key");
        _chatClient = new ChatClient("gpt-4o", apiKey);
        _options = new ChatCompletionOptions()
        {
            MaxTokens = 300,
            ResponseFormat = ChatResponseFormat.JsonObject
        };
    }

    public async Task<ReceiptDto> ExtreactListOfItems(Stream imageStream)
    {
        var imageBytes = await BinaryData.FromStreamAsync(imageStream);

        var messages = new List<ChatMessage>
        {
            new UserChatMessage(
                new List<ChatMessageContentPart>
            {
                ChatMessageContentPart.CreateTextMessageContentPart(
                    @"From the following image, extract a list of items. Ensure the output is a well-formed JSON object with correctly formatted Hebrew text (right-to-left). The JSON structure should be: 
                    { 
                        storeName: 'x', 
                        purchase: { 
                            purchaseDate: 'yyyy-mm-dd', 
                            items: [{ 
                                itemName: '', 
                                itemPrice: 10, 
                                quantity: 2, 
                                totalItemPrice: 20 
                            }] 
                        } 
                    }. 
                    Make sure all strings are properly closed, all objects and arrays are correctly terminated, and the JSON is valid,
                    Make sure each items has the the properties itemName, itemPrice, quantity totalItemPrice"
                    ),
                ChatMessageContentPart.CreateImageMessageContentPart(imageBytes, "image/png")
            })
        };
        
        ChatCompletion completion = await _chatClient.CompleteChatAsync(messages, _options);

        var resp = completion.Content[0].ToString().Trim();
        
        //resp = resp.Replace("```json", "").Replace("```", "").Trim();
        
        Console.WriteLine("Raw Response: " + resp); 
        
        using var document = JsonDocument.Parse(resp);

        
        var receiptDto = JsonSerializer.Deserialize<ReceiptDto>(document.RootElement.GetRawText());
        
        Console.WriteLine("Deserialized Receipt: " + JsonSerializer.Serialize(receiptDto));

        return receiptDto;
    }

}

Sometimes the properties: quantity, totalItemPrice are being cut off from the object, which causes a backend error..
Is there a way to force it to return a specific consist object form based on model or something similar ?

[sabbadino]

Have a look at strictured output

[kovac031]

https://github.com/openai/openai-dotnet?tab=readme-ov-file#how-to-use-chat-completions-with-structured-outputs

Instead of doing ResponseFormat = ChatResponseFormat.JsonObject

Actually specify the format structure as shown in the link.