.Net: Bug: Time to wait on 429 only in text

[BartNetJS]

I'm encountering an issue when attempting to use the WaitAndRetryAsync method with a Policy on a 429 in my C# project.
I try to grab the time i have to wait as noted in the error message:

Requests to the ChatCompletions_Create Operation under Azure OpenAI API version 2024-07-01-preview have exceeded token rate limit of your current OpenAI S0 pricing tier. Please retry after 51 seconds. Please go here: https://aka.ms/oai/quotaincrease if you would like to further increase the default rate limit.

Can the time to wait be added as a property HttpOperationException (in Data?) or in the inner exception System.ClientModel.ClientResultException?

The calling function is: await kernel.InvokeAsync(promptFunc, kernelArgs);

Expected behavior
To add the time to wait to a property or in the data list

Screenshots

Platform

• Source: Microsoft.SemanticKernel version 1.19.0

[markwallace-microsoft]

I think the data is available in the response headers, will investigate to confirm.

[BartNetJS]

I think also, but it is throwing an exception, and i don't see how i then can pickup the header

[markwallace-microsoft]

@BartNetJS You should eb able to do something like this:

1. HttpOperationException.InnerException which should be a ClientResultException
2. ClientResultException.GetRawResponse() will be a PipelineResponse?
3. PipelineResponse.Headers will be PipelineResponseHeaders

See also: https://learn.microsoft.com/en-us/dotnet/api/system.clientmodel.primitives.pipelineresponse.headers?view=azure-dotnet-preview#system-clientmodel-primitives-pipelineresponse-headers

[BartNetJS]

@markwallace-microsoft that did the trick.
Thanks

This is the code, quit a lot boilerplate, any suggestion to write it compacter and more readable?:

    private static IAsyncPolicy<FunctionResult> GetKernelRetryPolicy()
        {
            return Policy<FunctionResult>
                .Handle<HttpOperationException>(ex => IsTooManyRequests(ex).isTooMany)
                .WaitAndRetryAsync(
                    retryCount: 3,
                    sleepDurationProvider: (retryAttempt, outcome, context) => GetSleepDuration(retryAttempt, outcome, context),
                    onRetryAsync: async (outcome, timespan, retryAttempt, context) =>
                    {
                        var exceptionMessage = outcome.Exception?.Message ?? "Unknown error";
                        Log.Information($"Function retry {retryAttempt}: Delaying for {timespan.TotalMilliseconds}ms due to exception {exceptionMessage}.");
                    });
        }

        private static TimeSpan GetSleepDuration(int retryAttempt, DelegateResult<FunctionResult> outcome, Context context)
        {
            if (outcome.Exception is HttpOperationException httpEx)
            {
                var (isTooMany, retryAfter) = IsTooManyRequests(httpEx);
                if (isTooMany && retryAfter.HasValue)
                {
                    return retryAfter.Value;
                }
            }
            // Use exponential backoff with a base of 1 second
            return TimeSpan.FromSeconds(Math.Pow(2, retryAttempt - 1));
        }

        private static (bool isTooMany, TimeSpan? retryAfter) IsTooManyRequests(HttpOperationException ex)
        {
            if (ex.InnerException is ClientResultException clientResultException1)
            {
                if(clientResultException1.Status == (int)HttpStatusCode.TooManyRequests)
                {
                    var response = clientResultException1.GetRawResponse();
                    var retryAfter = GetRetryAfterDelay(response.Headers);
                    return (true, retryAfter);
                }
            }

            return (false, null);
        }

        private static TimeSpan? GetRetryAfterDelay(System.ClientModel.Primitives.PipelineResponseHeaders headers)
        {
            var retryAfterHeader = headers.FirstOrDefault(h => h.Key.Equals("Retry-After", StringComparison.OrdinalIgnoreCase));
            if (retryAfterHeader.Value != null)
            {
                if (int.TryParse(retryAfterHeader.Value, out int seconds))
                {
                    return TimeSpan.FromSeconds(seconds);
                }
                else if (DateTimeOffset.TryParse(retryAfterHeader.Value, out var retryAfterDate))
                {
                    var delay = retryAfterDate - DateTimeOffset.UtcNow;
                    return delay > TimeSpan.Zero ? delay : TimeSpan.Zero;
                }
            }
            return null;
        }

[markwallace-microsoft]

Hi @BartNetJS
can you using the standard resilience handler. The code below is what we use in our integration tests.

If the ShouldRetryAfterHeader property is set to true then the generator will resolve the delay based on the Retry-After header rules, otherwise it will return null and the retry strategy delay will generate the delay based on the configured options.

var builder = Kernel.CreateBuilder();
builder.Services.ConfigureHttpClientDefaults(c =>
{
    c.AddStandardResilienceHandler().Configure(o =>
    {
        o.Retry.ShouldRetryAfterHeader = true;
        o.Retry.ShouldHandle = args => ValueTask.FromResult(args.Outcome.Result?.StatusCode is HttpStatusCode.TooManyRequests);
    });
});

[BartNetJS]

Hi @markwallace-microsoft ,

Great, makes it a lot simpler.

I now hit the default timeouts of 30 second.
I tried to increase it here, but still got a timeout after 30 seconds:

    builder.Services.ConfigureHttpClientDefaults(c =>
    {
        ....
        c.ConfigureHttpClient((sp, client) =>
        {
            client.Timeout = TimeSpan.FromSeconds(120);
        });
    });

Any idea how to implement it?

[BartNetJS]

Found it in AzureOpenAIClientOptions