[Bug]: Getting an error "The operation was canceled"

[mukeshmrajput]

Language

C#

Version

latest

Description

Our use case is,

• Check question ask is complex or not, for this we're calling Azure Open AI ("CompletePromptAsync" method) to split the main question.
• Then, looping through all split questions and calling Azure Open AI ("CompletePromptAsync" method) parallel to get the answer of all split question
• Now waiting for all parallel calls to get finished then calling last time Azure Open AI ("CompletePromptAsync" method) to summarize the answer

During above process, we're seeing "The operation was canceled" error in OpenAIModel.cs class in CompletePromptAsync method while calling Azure Open AI. We've noticed that status code 408 in the logs of Azure Open AI. So we tried to increase the time out in program.cs file "builder.Services.AddHttpClient("WebClient", client => client.Timeout = TimeSpan.FromMintutes(10))" but it's not working at all. Need help to see where we can increase the time out for Azure Open AI call.

Reproduction Steps

1. Using dotnet sample (dotnet\samples\08.datasource.azureaisearch\AzureAISearchBot)
2. Updated Azure->OpenAIApiKey and Azure->OpenAIEndpoint in appsettings.json file
3. Changed model name to gpt-4o in prompt config file and updated data_sources to Vector Database index (Cognitive Search)
5. Program.cs: Added below line to listen incoming message handler.
   app.OnActivity(ActivityTypes.Message, activityHandlers.OnMessageActivityAsync);
6. In incoming message handler. Calling "CompletePromptAsync" method manually for static 9 questions in sequentially looping along with keep updating turnState.Temp!.Input to runtime question. 
7. We're seeing prompt response successful from "CompletePromptAsync" method for first 3 to 4 questions then rest all questions giving prompt response "Error" with "The operation was canceled".
8. Need suggestion or help why such error throwing? what's the fix for it

[Nivedipa-MSFT]

@mukeshmrajput - Could you please confirm if you are developing a Teams app and please share your sample repo link here?
A status code 408 indicates that the request to Azure OpenAI timed out.
Here are some additional steps to help mitigate this issue:
1.Reduce the number of parallel requests to avoid overwhelming the service. You can use a semaphore to limit concurrency.
2.Batch Requests: Instead of sending all requests at once, batch them into smaller groups and process each batch sequentially.
3.Increase Timeout: Ensure the timeout is set correctly for each request.
4.Retry Logic: Implement retry policies to handle transient errors.

We are checking with the engineering team to determine where we can increase the timeout for Azure OpenAI calls.

[mukeshmrajput]

Thanks @Nivedipa-MSFT.

Yes we're developing a Teams App which give the answers based on our own RAG/datasource of electronics parts. I can't share the repo but yes try to give sample code that we're trying.

We're trying the suggestions that you provided too.

Yes, let me know how we can increase the time out for Azure OpenAI calls

Thanks,
Mukesh

[mukeshmrajput]

@Nivedipa-MSFT, do your team know how to increase the time out for Azure Open AI call.

[Nivedipa-MSFT]

@mukeshmrajput - Apologies for the delay. We are still checking this with engineering team. We appreciate your patience and will inform you as soon as we have any updates.
Thank you!