Merge pull request #2817 from Azure/dev

Merge Branch Dev To Branch Main

.github/ISSUE_TEMPLATE/new-release-template.md

@@ -16,20 +16,30 @@ _Due: <2-3-business-days-before-release>_
 - [ ] Publish DTFx packages to the [ADO feed](https://dev.azure.com/durabletaskframework/Durable%20Task%20Framework%20CI/_artifacts/feed/durabletask) for testing.
 - [ ] Keep branch `azure-storage-v12` updated with branch `main`.
 
+**Prep DotNet Isolated SDK Release: (assigned to:)**
+_Due: <2-3-business-days-before-release>_
+- [ ] If there were DTFx.Core changes, check its reference version [here](https://github.com/microsoft/durabletask-dotnet/blob/c838535adb6aedb6671cf193389ce63a6b4a9b24/src/Abstractions/Abstractions.csproj#L10). If updates are required, document the changes in [release notes](https://github.com/microsoft/durabletask-dotnet/blob/c838535adb6aedb6671cf193389ce63a6b4a9b24/src/Abstractions/RELEASENOTES.md).
+- [ ] Check dotnet isolated SDK versions [here](https://github.com/microsoft/durabletask-dotnet/blob/c838535adb6aedb6671cf193389ce63a6b4a9b24/eng/targets/Release.props#L20). If updated, document the changes in the [change logs](https://github.com/microsoft/durabletask-dotnet/blob/c838535adb6aedb6671cf193389ce63a6b4a9b24/CHANGELOG.md).
+- [ ] Run pipeline [Release .Net out-of-proc SDK](https://durabletaskframework.visualstudio.com/Durable%20Task%20Framework%20CI/_build?definitionId=29) to create the new package and publish it to the ADO feed for testing.
+
 **Prep Release (assigned to: )**
 _Due: <2-business-days-before-release>_
-- [ ] Update Durable Functions references (Analyzer? DTFx?) and check current version.
+- [ ] Update DTFx packages and Analyzer versions at WebJobs.Extensions.Durabletask.csproj and check if we need to update the WebJobs.Extensions.Durabletask version.
 - [ ] Locally, run `dotnet list package --vulnerable` to ensure the release is not affected by any vulnerable dependencies.
 - [ ] Review the [Dependabot vulnerability alerts](https://github.com/Azure/azure-functions-durable-extension/security/dependabot) and address them. Note: code samples / test projects _may_ be excluded from this check.
 - [ ] Add the Durable Functions package to the [ADO test feed](https://dev.azure.com/durabletaskframework/Durable%20Task%20Framework%20CI/_artifacts/feed/durabletask-test).
 - [ ] Check for package size, make sure it's not surprisingly heavier than a previous release.
+- [ ] Update .NET Isolated SDK version at Worker.Extensions.Durabletask.csproj and check if we need to update the Worker.Extensions.Durabletask version.
+- [ ] Run pipeline [Release .Net Isolated Worker Extension](https://durabletaskframework.visualstudio.com/Durable%20Task%20Framework%20CI/_build?definitionId=30) to create the new package and add it to the ADO feed for testing.
 - [ ] Merge (**choose create a merge commit, NOT squash merge**) dev into main. Person performing validation must approve PR.
 - [ ] Keep branch `v3.x` updated with branch `dev`. Do not merge PRs that are specific to Durable Functions v2.
 
 **Validation**
 _Due: <1-business-days-before-release>_
 - [ ] Run private performance tests and ensure no regressions. **(assigned to: )**
 - [ ] Smoke test Functions V1, Functions V2, and Functions V3 .NET apps. **(assigned to: )**
+- [ ] Smoke test .NET apps with backend Netherite, MSSQL. **(assigned to: )**
+- [ ] Smoke test .NET isolated apps. **(assigned to: )**
 
 **DTFx Release Completion (assigned to: )**
 _Due: <release-deadline>_
@@ -38,12 +48,18 @@ _Due: <release-deadline>_
 - [ ] Publish release notes for DTFx.
 - [ ] Patch increment DTFx packages that were released (either DT-AzureStorage only or if there were Core changes DT-AzureStorage, DT-Core, and DT-ApplicationInsights)
 
+**DotNet Isolated SDK Release Completion: (assigned to:)**
+_Due: <release-deadline>_
+- [ ] Upload .NET isolated SDK packages to NuGet (directly to nuget.org).
+- [ ] Publish release notes in the durable-dotnet repo.
+
 **Release Completion (assigned to: )**
 _Due: <release-deadline>_
 - [ ] Delete Durable Functions packages from the [ADO test feed](https://dev.azure.com/durabletaskframework/Durable%20Task%20Framework%20CI/_artifacts/feed/durabletask-test).
 - [ ] Run the [Durable Functions release pipeline](https://dev.azure.com/durabletaskframework/Durable%20Task%20Framework%20CI/_build?definitionId=23) and select `main` as the branch.
 - [ ] Add the Durable Functions package to the [ADO feed](https://dev.azure.com/durabletaskframework/Durable%20Task%20Framework%20CI/_artifacts/feed/durabletask) using [this pipeline](https://dev.azure.com/durabletaskframework/Durable%20Task%20Framework%20CI/_release?_a=releases&view=mine&definitionId=11).
 - [ ] Upload the Durable Functions package to NuGet (directly to nuget.org).
+- [ ] Upload .NET Isolated worker extension package to NuGet (directly to nuget.org).
 - [ ] Create a PR in the [Azure Functions templates repo](https://github.com/Azure/azure-functions-templates) targeting branch `dev` to update all references of "Microsoft.Azure.WebJobs.Extensions.DurableTask" (search for this string in the code) to the latest version.
 - [ ] _if and only if this is a new major release_, Create a PR in the [Azure Functions bundles repo](https://github.com/Azure/azure-functions-extension-bundles) to update bundles to the latest version .
 - [ ] Merge all pending PR docs from `pending_docs.md.`

.github/workflows/smoketest-dotnet-isolated-v4.yml

@@ -1,6 +1,7 @@
 name: Smoke Test - .NET Isolated on Functions V4
 
 on:
+  workflow_dispatch:
   push:
     branches: [ main, dev ]
     paths:

.github/workflows/smoketest-dotnet-v2.yml

@@ -1,6 +1,7 @@
 name: Smoke Test - .NET on Functions V2
 
 on:
+  workflow_dispatch:
   push:
     branches: [ main, dev ]
     paths:

.github/workflows/smoketest-dotnet-v3.yml

@@ -1,6 +1,7 @@
 name: Smoke Test - .NET on Functions V3
 
 on:
+  workflow_dispatch:
   push:
     branches: [ main, dev ]
     paths:

.github/workflows/smoketest-java8-v4.yml

@@ -1,6 +1,7 @@
 name: Smoke Test - Java 8 on Functions V4
 
 on:
+  workflow_dispatch:
   push:
     branches: [ main, dev ]
     paths:

.github/workflows/smoketest-mssql-inproc-v4.yml

@@ -1,23 +1,26 @@
 name: Smoke Test - .NET in-proc w/ MSSQL on Functions V4
 
 on:
+  workflow_dispatch:
   push:
     branches: [ main, dev ]
     paths:
       - 'src/**'
       - 'test/SmokeTests/BackendSmokeTests/MSSQL/**'
+      - '.github/workflows/smoketest-mssql-inproc-v4.yml'
   pull_request:
     branches: [ main, dev ]
     paths:
       - 'src/**'
       - 'test/SmokeTests/BackendSmokeTests/MSSQL/**'
+      - '.github/workflows/smoketest-mssql-inproc-v4.yml'
 
 jobs:
   build:
 
     runs-on: ubuntu-latest
     env:
-      SA_PASSWORD: NotASecret!12
+      SA_PASSWORD: ${{ secrets.SA_PASSWORD }}
 
     steps:
     - uses: actions/checkout@v4

.github/workflows/smoketest-netherite-inproc-v4.yml

@@ -1,6 +1,7 @@
 name: Smoke Test - .NET in-proc w/ Netherite on Functions V4
 
 on:
+  workflow_dispatch:
   push:
     branches: [ main, dev ]
     paths:

.github/workflows/smoketest-node14-v4.yml

@@ -1,6 +1,7 @@
 name: Smoke Test - Node 14 on Functions V4
 
 on:
+  workflow_dispatch:
   push:
     branches: [ main, dev ]
     paths:

.github/workflows/smoketest-python37-v4.yml

@@ -1,6 +1,7 @@
 name: Smoke Test - Python 3.7 on Functions V4
 
 on:
+  workflow_dispatch:
   push:
     branches: [ main, dev ]
     paths:

release_notes.md

@@ -1,16 +1,14 @@
 # Release Notes
 
-## Microsoft.Azure.Functions.Worker.Extensions.DurableTask 1.1.1
+## Microsoft.Azure.Functions.Worker.Extensions.DurableTask 1.2.0
 
 ### New Features
 
-- Add `CreateCheckStatusResponseAsync` APIs. (https://github.com/Azure/azure-functions-durable-extension/pull/2722)
+- Add `suspendPostUri` and `resumePostUri` to the list of returned URIs in `CreateCheckStatusResponseAsync`. (https://github.com/Azure/azure-functions-durable-extension/pull/2785)
+- Fix `NotSupportedException` when calling `PurgeAllInstancesAsync` and `PurgeInstanceAsync`
 
 ### Bug Fixes
 
-- Fix issue with isolated entities: custom deserialization was not working because IServices was not passed along (https://github.com/Azure/azure-functions-durable-extension/pull/2686)
-- Fix issue with `string` activity input having extra quotes (https://github.com/Azure/azure-functions-durable-extension/pull/2708)
-
 ### Breaking Changes
 
 ### Dependency Updates

src/WebJobs.Extensions.DurableTask/Auth/AzureCredentialFactory.cs

@@ -72,6 +72,10 @@ public AppAuthTokenCredential Create(IConfiguration configuration, TimeSpan toke
                 RefreshOffset = tokenRefreshOffset,
             };
 
+            // The token credential will make background callbacks to renew the token.
+            // We suppress async flow to avoid logging scope from being captured as we do not know
+            // where this will be called from first.
+            using AsyncFlowControl flowControl = System.Threading.ExecutionContext.SuppressFlow();
             return new AppAuthTokenCredential(
                 value.Token,
                 (o, t) => this.RenewTokenAsync((TokenRenewalState)o, t),

src/WebJobs.Extensions.DurableTask/ContextImplementations/DurableActivityContext.cs

@@ -127,15 +127,18 @@ internal object GetInput(Type destinationType)
 
             string serializedValue = jToken.ToString(Formatting.None);
 
-            // Object inputs for out-of-proc activities are passed in their JSON-stringified form with a destination
-            // type of System.String. Unfortunately, deserializing a JSON string to a string causes
-            // MessagePayloadDataConverter to throw an exception. This is a workaround for that case. All other
-            // inputs with destination System.String (in-proc: JSON and not JSON; out-of-proc: not-JSON) inputs with
-            // destination System.String should cast to JValues and be handled above.)
-            if (this.rawInput)
+            if (this.rawInput) // the "modern" OOProc protocol case
             {
                 return serializedValue;
             }
+            else if (destinationType.Equals(typeof(string))) // legacy OOProc protocol case (JS, Python, PowerShell)
+            {
+                // Object/complex inputs in "legacy" out-of-proc activities are passed with a destination
+                // type of System.String (to be precise, if inspected with a debugger, you'll see a stringified JSON).
+                // Unfortunately, deserializing a JSON string to a string causes  MessagePayloadDataConverter to throw an exception, so the
+                // return statement prevents that.
+                return serializedValue;
+            }
 
             return this.messageDataConverter.Deserialize(serializedValue, destinationType);
         }

src/WebJobs.Extensions.DurableTask/ContextImplementations/RemoteOrchestratorContext.cs

@@ -64,6 +64,20 @@ internal OrchestratorExecutionResult GetResult()
             return this.executionResult ?? throw new InvalidOperationException($"The execution result has not yet been set using {nameof(this.SetResult)}.");
         }
 
+        internal bool TryGetOrchestrationErrorDetails(out string details)
+        {
+            if (this.failure != null)
+            {
+                details = this.failure.Message;
+                return true;
+            }
+            else
+            {
+                details = string.Empty;
+                return false;
+            }
+        }
+
         internal void SetResult(IEnumerable<OrchestratorAction> actions, string customStatus)
         {
             var result = new OrchestratorExecutionResult

src/WebJobs.Extensions.DurableTask/LocalGrpcListener.cs

@@ -10,6 +10,7 @@
 using System.Threading.Tasks;
 using DurableTask.Core;
 using DurableTask.Core.Entities;
+using DurableTask.Core.Exceptions;
 using DurableTask.Core.History;
 using DurableTask.Core.Query;
 using DurableTask.Core.Serializing.Internal;
@@ -54,7 +55,13 @@ public Task StartAsync(CancellationToken cancelToken = default)
             int numAttempts = 1;
             while (numAttempts <= maxAttempts)
             {
-                this.grpcServer = new Server();
+                ChannelOption[] options = new[]
+                {
+                    new ChannelOption(ChannelOptions.MaxReceiveMessageLength, int.MaxValue),
+                    new ChannelOption(ChannelOptions.MaxSendMessageLength, int.MaxValue),
+                };
+
+                this.grpcServer = new Server(options);
                 this.grpcServer.Services.Add(P.TaskHubSidecarService.BindService(new TaskHubGrpcServer(this)));
 
                 int listeningPort = numAttempts == 1 ? DefaultPort : this.GetRandomPort();
@@ -155,10 +162,23 @@ public override Task<Empty> Hello(Empty request, ServerCallContext context)
                         InstanceId = instanceId,
                     };
                 }
-                catch (InvalidOperationException)
+                catch (OrchestrationAlreadyExistsException)
+                {
+                    throw new RpcException(new Status(StatusCode.AlreadyExists, $"An Orchestration instance with the ID {request.InstanceId} already exists."));
+                }
+                catch (InvalidOperationException ex) when (ex.Message.EndsWith("already exists.")) // for older versions of DTF.AS and DTFx.Netherite
                 {
                     throw new RpcException(new Status(StatusCode.AlreadyExists, $"An Orchestration instance with the ID {request.InstanceId} already exists."));
                 }
+                catch (Exception ex)
+                {
+                    this.extension.TraceHelper.ExtensionWarningEvent(
+                        this.extension.Options.HubName,
+                        functionName: request.Name,
+                        instanceId: request.InstanceId,
+                        message: $"Failed to start instanceId {request.InstanceId} due to internal exception.\n Exception trace: {ex}.");
+                    throw new RpcException(new Status(StatusCode.Internal, $"Failed to start instance with ID {request.InstanceId}.\nInner Exception message: {ex.Message}."));
+                }
             }
 
             public async override Task<P.RaiseEventResponse> RaiseEvent(P.RaiseEventRequest request, ServerCallContext context)

src/WebJobs.Extensions.DurableTask/OutOfProcMiddleware.cs

@@ -214,8 +214,31 @@ await this.LifeCycleNotificationHelper.OrchestratorCompletedAsync(
                         isReplay: false);
                 }
             }
+            else if (context.TryGetOrchestrationErrorDetails(out string details))
+            {
+                // the function failed because the orchestrator failed.
+
+                orchestratorResult = context.GetResult();
+
+                this.TraceHelper.FunctionFailed(
+                     this.Options.HubName,
+                     functionName.Name,
+                     instance.InstanceId,
+                     details,
+                     FunctionType.Orchestrator,
+                     isReplay: false);
+
+                await this.LifeCycleNotificationHelper.OrchestratorFailedAsync(
+                    this.Options.HubName,
+                    functionName.Name,
+                    instance.InstanceId,
+                    details,
+                    isReplay: false);
+            }
             else
             {
+                // the function failed for some other reason
+
                 string exceptionDetails = functionResult.Exception.ToString();
 
                 this.TraceHelper.FunctionFailed(
@@ -374,15 +397,27 @@ void SetErrorResult(FailureDetails failureDetails)
                     functionName.Name,
                     batchRequest.InstanceId,
                     functionResult.Exception.ToString(),
-                    FunctionType.Orchestrator,
+                    FunctionType.Entity,
                     isReplay: false);
 
-                SetErrorResult(new FailureDetails(
-                    errorType: "FunctionInvocationFailed",
-                    errorMessage: $"Invocation of function '{functionName}' failed with an exception.",
-                    stackTrace: null,
-                    innerFailure: new FailureDetails(functionResult.Exception),
-                    isNonRetriable: true));
+                if (context.Result != null)
+                {
+                    // Send the results of the entity batch execution back to the DTFx dispatch pipeline.
+                    // This is important so we can propagate the individual failure details of each failed operation back to the
+                    // calling orchestrator. Also, even though the function execution was reported as a failure,
+                    // it may not be a "total failure", i.e. some of the operations in the batch may have succeeded and updated
+                    // the entity state.
+                    dispatchContext.SetProperty(context.Result);
+                }
+                else
+                {
+                    SetErrorResult(new FailureDetails(
+                        errorType: "FunctionInvocationFailed",
+                        errorMessage: $"Invocation of function '{functionName}' failed with an exception.",
+                        stackTrace: null,
+                        innerFailure: new FailureDetails(functionResult.Exception),
+                        isNonRetriable: true));
+                }
 
                 return;
             }
@@ -399,8 +434,7 @@ void SetErrorResult(FailureDetails failureDetails)
                        FunctionType.Entity,
                        isReplay: false);
 
-            // Send the result of the orchestrator function to the DTFx dispatch pipeline.
-            // This allows us to bypass the default, in-process execution and process the given results immediately.
+            // Send the results of the entity batch execution back to the DTFx dispatch pipeline.
             dispatchContext.SetProperty(batchResult);
         }
 
@@ -621,31 +655,48 @@ private static bool TrySplitExceptionTypeFromMessage(
             [NotNullWhen(true)] out string? exceptionType,
             [NotNullWhen(true)] out string? exceptionMessage)
         {
-            // Example exception messages:
+            // In certain situations, like when the .NET Isolated worker is configured with
+            // WorkerOptions.EnableUserCodeException = true, the exception message we get from the .NET Isolated
+            // worker looks like this:
+            // "Exception of type 'ExceptionSerialization.Function+UnknownException' was thrown."
+            const string startMarker = "Exception of type '";
+            const string endMarker = "' was thrown.";
+            if (exception.StartsWith(startMarker) && exception.EndsWith(endMarker))
+            {
+                exceptionType = exception[startMarker.Length..^endMarker.Length];
+                exceptionMessage = string.Empty;
+                return true;
+            }
+
+            // The following are the more common cases that we expect to see, which will be common across a
+            // variety of language workers:
             // .NET   : System.ApplicationException: Kah-BOOOOM!!
             // Java   : SQLServerException: Invalid column name 'status'.
             // Python : ResourceNotFoundError: The specified blob does not exist. RequestId:8d5a2c9b-b01e-006f-33df-3f7a2e000000 Time:2022-03-25T00:31:24.2003327Z ErrorCode:BlobNotFound Error:None
             // Node   : SyntaxError: Unexpected token N in JSON at position 12768
 
-            // From the above examples, they all follow the same pattern of {ExceptionType}: {Message}
+            // From the above examples, they all follow the same pattern of {ExceptionType}: {Message}.
+            // However, some exception types override the ToString() method and do something custom, in which
+            // case the message may not be in the expected format. In such cases we won't be able to distinguish
+            // the exception type.
             string delimeter = ": ";
             int endExceptionType = exception.IndexOf(delimeter);
             if (endExceptionType < 0)
             {
                 exceptionType = null;
-                exceptionMessage = null;
+                exceptionMessage = exception;
                 return false;
             }
 
-            exceptionType = exception[..endExceptionType];
+            exceptionType = exception[..endExceptionType].TrimEnd();
 
             // The .NET Isolated language worker strangely includes the stack trace in the exception message.
             // To avoid bloating the payload with redundant information, we only consider the first line.
             int startMessage = endExceptionType + delimeter.Length;
             int endMessage = exception.IndexOf('\n', startMessage);
             if (endMessage < 0)
             {
-                exceptionMessage = exception[startMessage..];
+                exceptionMessage = exception[startMessage..].TrimEnd();
             }
             else
             {