Proxy not forwarding cancellation token to app correctly in WebJobs.Script.Grpc/Server/RetryProxyHandler.cs

[stuartf-aida]

Proxy not forwarding cancellation token correctly to function app

Canada Central example @pmerineau
first reported Azure/azure-functions-dotnet-worker#2417 (comment)

Sending InvocationCancel request for invocation: '0577cf46-de3f-47ba-9751-0750cc28ba18'
And your function invocation is failing, likely due to this proxy that is not forwarding the cancellation request to your app properly:
Exception while executing function: Functions.<YourFunctionName> ---> System.InvalidOperationException : Failed to proxy request with ForwarderError: RequestCanceled. System.IO.IOException : Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request

North Europe example @stuartf-aida
first reported Azure/azure-functions-dotnet-worker#2417 (comment)

Sending InvocationCancel request for invocation: 'a11b8045-59cf-4db2-90eb-356d561e2fb2'
Exception while executing function: Functions.<YourFunctionName> ---> System.InvalidOperationException : Failed to proxy request with ForwarderError: RequestCanceled. System.IO.IOException : Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request

As mentioned by @liliankasem

The error is actually happening on the Functions Host side in the RetryProxyHandler so I would suggest opening the issue there and I will continue that investigation there as well.

I hope this is what you need, Lilian!

[liliankasem]

This is perfect, thank you! Will add this to todo list :)

edit: @stuartf-aida any chance you have a minimal repro I can use to investigate the issue? Or other info on the type of function that is failing with this error i.e. is it a dotnet-isolated app? v4? .net 6 or .net 8? what kind of trigger binding? etc.

[stuartf-aida]

@liliankasem
It's dotnet isolated, v8. Http trigger binding.

We've shifted our code now to use an alternative, based on the workaround for the previous issue where I originally posted. That seems (so far) to have solved this, but it's hard to say. We don't (and didn't) have a situation that reliably caused this to happen - most invocations would not error. We can use the same payload on the http trigger such that the same code runs and we've not successfully reproduced the issue.

The function doesn't do very much.

It uses Azure.Security.KeyVault.Secrets (4.7.0) to grab a secret from the keyvault,
Environment.GetEnvironmentVariable to get an environment variable
and Azure.Storage.Queues (12.20.1) to put an entry on a queue

and along the way a bit of string manipulation

Thanks very much for your keen attention on these issues!

[liliankasem]

Okay so it looks like we have a fun two part bug.

Host

The first part is in the host. The DefaultHttpProxyService has this code which does not take into account cancellation (ForwarderError.RequestCanceled). Whether cancellation is handled or not by the worker/customer code, we will always get this error message when an invocation is cancelled:

System.InvalidOperationException : Failed to proxy request with ForwarderError: RequestCanceled

azure-functions-host/src/WebJobs.Script.Grpc/Server/DefaultHttpProxyService.cs

Lines 55 to 66 in 287ceac

	ForwarderError httpProxyTaskResult = await forwardingTask;
	if (httpProxyTaskResult is not ForwarderError.None)
	{
	Exception forwarderException = null;
	if (context.TryGetHttpRequest(out HttpRequest request))
	{
	forwarderException = request.HttpContext.GetForwarderErrorFeature()?.Exception;
	}
	throw new InvalidOperationException($"Failed to proxy request with ForwarderError: {httpProxyTaskResult}", forwarderException);
	}

Worker

There is also a bug in the AspNetCore worker extension which breaks the cancellation behaviour. In the DefaultHttpCoordinator, we pass in the cancellation token when we create the ContextReference:

var contextRef = _contextReferenceList.GetOrAdd(invocationId, static id => new ContextReference(id));
contextRef.SetCancellationToken(context.CancellationToken);
contextRef.FunctionContextValueSource.SetResult(context);

code

As a result, when a function invocation is cancelled, we cancel that same cts that is being used by ContextReference:

cancellationTokenSource?.Cancel();

code

So, we end up with a disposed HttpContext when we try to handle the invocation result:

ActionContext actionContext = new ActionContext(httpContext, httpContext.GetRouteData(), new ActionDescriptor());
await actionResult.ExecuteResultAsync(actionContext);

code

This is where the ObjectDisposedExecpetion comes from:

Exception: System.ObjectDisposedException: Request has finished and HttpContext disposed.
Object name: 'HttpContext'.
   at Microsoft.AspNetCore.Http.DefaultHttpContext.ThrowContextDisposed()
   at Microsoft.AspNetCore.Http.DefaultHttpContext.get_Features()
   at Microsoft.AspNetCore.Routing.RoutingHttpContextExtensions.GetRouteData(HttpContext httpContext)
   at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 54
   at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
Stack:    at Microsoft.AspNetCore.Http.DefaultHttpContext.ThrowContextDisposed()
   at Microsoft.AspNetCore.Http.DefaultHttpContext.get_Features()
   at Microsoft.AspNetCore.Routing.RoutingHttpContextExtensions.GetRouteData(HttpContext httpContext)
   at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 54
   at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89

Next Steps

I'll speak with the team and discuss how we want to handle these issues.

• We can use this issue to track the Host bug with DefaultHttpProxyService
• We can use the dotnet-isolated issue to track fixing the ObjectDisposedException bug