Async Route feature with Callback and Server-Sent Events (SSE) support - Recovered from #1349

.gitignore

@@ -266,3 +266,6 @@ examples/PetStore/data/PetData.json
 packers/choco/pode.nuspec
 packers/choco/tools/ChocolateyInstall.ps1
 docs/Getting-Started/Samples.md
+
+#Todo files
+*.todo

README.md

@@ -60,7 +60,7 @@ Then navigate to `http://127.0.0.1:8000` in your browser.
 * OpenAPI documentation with Swagger, Redoc, RapidDoc, StopLight, OpenAPI-Explorer and RapiPdf
 * Listen on a single or multiple IP(v4/v6) address/hostnames
 * Cross-platform support for HTTP(S), WS(S), SSE, SMTP(S), and TCP(S)
-* Host REST APIs, Web Pages, and Static Content (with caching)
+* Host REST APIs,async REST APIs, Web Pages, and Static Content (with caching)
 * Support for custom error pages
 * Request and Response compression using GZip/Deflate
 * Multi-thread support for incoming requests
@@ -82,7 +82,7 @@ Then navigate to `http://127.0.0.1:8000` in your browser.
 * In-memory caching, with optional support for external providers (such as Redis)
 * (Windows) Open the hosted server as a desktop application
 * FileBrowsing support
-* Localization (i18n) in Arabic, German, Spanish, France, Italian, Japanese, Korean, Polish, Portuguese, and Chinese
+* Localization (i18n) in Arabic, German, Spanish, France, Italian, Japanese, Korean, Polish, Portuguese, Dutch, and Chinese.
 
 ## 📦 Install
 

docs/Tutorials/Configuration.md

@@ -79,6 +79,8 @@ A "path" like `Server.Ssl.Protocols` looks like the below in the file:
 | Server.FileMonitor               | Defines configuration for restarting the server based on file updates       | [link](../Restarting/Types/FileMonitoring)                              |
 | Server.ReceiveTimeout            | Define the amount of time a Receive method call will block waiting for data | [link](../Endpoints/Basic/StaticContent/#server-timeout)                |
 | Server.DefaultFolders            | Set the Default Folders paths                                               | [link](../Routes/Utilities/StaticContent/#changing-the-default-folders) |
+| Server.Tasks.HouseKeeping        | Set the House Keeping retension and frequency for the Tasks                 | [link](../Tasks) |
+| Server.AsyncRoutes.HouseKeeping  | Set the House Keeping retension and frequency for the AsyncRoutes           | [link](../Routes/Async/Utilities/HouseKeeping) |
 | Web.OpenApi.DefaultDefinitionTag | Define the primary tag name for OpenAPI ( `default` is the default)         | [link](../OpenAPI/Overview)                                             |
 | Web.OpenApi.UsePodeYamlInternal  | Force the use of the internal YAML converter (`False` is the default)       |                                                                         |
 | Web.Static.ValidateLast          | Changes the way routes are processed.                                       | [link](../Routes/Utilities/StaticContent)                               |

docs/Tutorials/Routes/Async/Features/AsyncIdGenerator.md

@@ -0,0 +1,19 @@
+# IdGenerator
+
+The `IdGenerator` parameter specifies the function used to generate unique IDs for asynchronous tasks. This allows you to customize the way IDs are generated for each async route task, ensuring they meet your application's requirements.
+
+- **Default Value**: The default function used is `New-PodeGuid`, which generates a unique GUID for each task.
+
+#### Customizing Async ID Generation
+
+You can define your own custom function to generate IDs by specifying it in the `IdGenerator` parameter. This can be useful if you need to follow a specific format or include particular information in the IDs.
+
+**Example Usage**
+
+```powershell
+Add-PodeRoute -PassThru -Method Post -Path '/customAsyncId' -ScriptBlock {
+    return @{ Message = "Custom Async ID" }
+} | Set-PodeAsyncRoute -ResponseContentType 'application/json', 'application/yaml' -IdGenerator  {return [guid]::NewGuid().ToString() + "-custom" }
+```
+
+In this example, the `New-CustomAsyncId` function generates a GUID with a custom suffix, ensuring each async route task has a unique and identifiable ID.

docs/Tutorials/Routes/Async/Features/Callback.md

@@ -0,0 +1,89 @@
+
+# Callback
+
+The `Set-PodeAsyncRoute` function supports including callback functionality for routes. This allows you to define a URL that will be called when the asynchronous task is completed. You can specify the callback URL, content type, HTTP method, and header fields.
+
+#### Callback Parameters
+
+- **Callback URL**: Specifies the URL field for the callback. Default is `'$request.body#/callbackUrl'`.
+  - Can accept the following meta values:
+    - `$request.query.param-name`: query-param-value
+    - `$request.header.header-name`: application/json
+    - `$request.body#/field-name`: callbackUrl
+    - Can accept runtime expressions based on the [OpenAPI specification](https://swagger.io/docs/specification/callbacks/).
+  - Acceptable static values (examples):
+    - 'http://example.com/callback'
+    - 'https://api.example.com/callback'
+
+- **Callback Content Type**: Specifies the content type for the callback. The default is `'application/json'`.
+  - Can accept the following meta values:
+    - `$request.query.param-name`: query-param-value
+    - `$request.header.header-name`: application/json
+    - `$request.body#/field-name`: callbackUrl
+    - Can accept runtime expressions based on the [OpenAPI specification](https://swagger.io/docs/specification/callbacks/).
+  - Acceptable static values (examples):
+    - 'application/json'
+    - 'application/xml'
+    - 'text/plain'
+
+- **Callback Method**: Specifies the HTTP method for the callback. The default is `'Post'`.
+  - Can accept the following meta values:
+    - `$request.query.param-name`: query-param-value
+    - `$request.header.header-name`: application/json
+    - `$request.body#/field-name`: callbackUrl
+    - Can accept runtime expressions based on the [OpenAPI specification](https://swagger.io/docs/specification/callbacks/).
+  - Acceptable static values (examples):
+    - `GET`
+    - `POST`
+    - `PUT`
+    - `DELETE`
+
+- **Callback Header Fields**: Specifies the header fields for the callback as a hashtable. The key can be a string representing the header key or one of the meta values. The value is the header value if it's a standard key or the default value if the meta value is not resolvable.
+  - Can accept the following meta values as keys:
+    - `$request.query.param-name`: query-param-value
+    - `$request.header.header-name`: application/json
+    - `$request.body#/field-name`: callbackUrl
+    - Can accept runtime expressions based on the [OpenAPI specification](https://swagger.io/docs/specification/callbacks/).
+  - Acceptable static values (examples):
+    - `@{ 'Content-Type' = 'application/json' }`
+    - `@{ 'Authorization' = 'Bearer token' }`
+    - `@{ 'Custom-Header' = 'value' }`
+
+- **Send Result**: If specified, sends the result of the callback.
+  - Type Boolean.
+
+- **Event Name**: Specifies the event name for the callback.
+  - Type String.
+
+
+#### Example Usage
+
+```powershell
+Add-PodeRoute -PassThru -Method Post -Path '/asyncWithCallback' -ScriptBlock {
+    return @{ Message = "Async Route with Callback" }
+} | Set-PodeAsyncRoute `
+    -ResponseContentType 'application/json', 'application/yaml' `
+    -Callback `
+    -CallbackUrl 'http://example.com/callbacks/{$request.body#/callbackPath}' `
+    -CallbackContentType 'application/json' `
+    -CallbackMethod '$request.body#/callbackMethod' `
+    -CallbackHeaderFields @{ 'Custom-Header' = '$request.header.CustomHeader' } `
+    -SendResult `
+    -EventName 'AsyncCompleted'
+```
+
+#### Explanation
+
+1. **Route Definition**: The `Add-PodeRoute` defines a route at `/asyncWithCallback` that processes a request and returns a message indicating it's an async route with a callback.
+
+2. **Setting Async Route with Callback**: The `Set-PodeAsyncRoute` processes the route to make it asynchronous and sets up the callback.
+    - `-ResponseContentType` specifies the response formats as JSON and YAML.
+    - `-Callback` enables the callback functionality.
+    - `-CallbackUrl` sets the URL that will be called when the async route task is completed, using a runtime expression based on the request body.
+    - `-CallbackContentType` specifies the content type for the callback request.
+    - `-CallbackMethod` sets the HTTP method for the callback request, using a runtime expression based on the request body.
+    - `-CallbackHeaderFields` includes custom header fields in the callback request, using a runtime expression based on the request headers.
+    - `-SendResult` ensures that the result of the async route task is sent in the callback request.
+    - `-EventName` specifies the event name for the callback.
+
+This setup ensures that when the asynchronous task completes, a request will be made to the specified callback URL with the defined settings, including the result of the async route task, using runtime expressions to dynamically set the callback parameters.

docs/Tutorials/Routes/Async/Features/OpenAPI.md

@@ -0,0 +1,94 @@
+
+# OpenAPI Integration with Async Routes
+
+Async routes defined using the `Set-PodeAsyncRoute` function can seamlessly integrate with OpenAPI documentation. This feature automatically generates detailed documentation, including response types and callback information, enhancing the ease of sharing and maintaining your API specifications.
+
+## Key Features
+
+### Automatic Documentation Generation
+
+When an async route is configured using `Set-PodeAsyncRoute`, the corresponding OpenAPI documentation is automatically generated. This documentation includes:
+- **Route Details**: Information about the HTTP method, path, and operation summary.
+- **Response Types**: Details of the possible response content types (`application/json`, `application/yaml`, etc.) and their associated schemas.
+- **Callback Details**: If the route includes callbacks, these are also documented in the OpenAPI definition.
+
+### Customization Options
+
+You can tailor the generated OpenAPI documentation to fit your specific needs:
+- **OpenApi Schemas**: Customize the schema name for the async route task using the `OATypeName` parameter, or other relevant parameters like `$TaskIdName`, `$QueryRequestName`, and `$QueryParameterName` using `Set-PodeAsyncRouteOASchemaName`.
+- **Route Information**: Further customize the OpenAPI route definition using Pode’s OpenAPI functions, such as `Set-PodeOARouteInfo` and any othe OpenApi functions available for route definition.
+
+### Piping for Documentation
+
+To generate OpenAPI documentation for an async route, you must pipe the route definition through `Set-PodeOARouteInfo`, as shown in the example below. This requirement also applies to the following async routes:
+- `Add-PodeAsyncRouteQuery`
+- `Add-PodeAsyncRouteStop`
+- `Add-PodeAsyncRouteGet`
+
+## Example Usage
+
+The following example demonstrates how to define an async route and customize its OpenAPI documentation:
+
+```powershell
+# Set a custom schema name for the async route task
+Set-PodeAsyncRouteOASchemaName -OATypeName 'MyTask'
+
+# Define an async route and customize its OpenAPI information
+Add-PodeRoute -PassThru -Method Post -Path '/asyncExample' -ScriptBlock {
+    return @{ Message = "Async Route" }
+} | Set-PodeAsyncRoute -ResponseContentType 'application/json', 'application/yaml' -PassThru |
+    Set-PodeOARouteInfo -Summary 'My Async Route Task' -Description 'This is a description'
+```
+
+### Resulting OpenAPI Documentation
+
+The generated OpenAPI documentation might look as follows:
+
+```yaml
+/asyncExample:
+  post:
+    summary: My Async Route Task
+    description: This is a description
+    responses:
+      200:
+        description: Successful operation
+        content:
+          application/yaml:
+            schema:
+              $ref: '#/components/schemas/MyTask'
+          application/json:
+            schema:
+              $ref: '#/components/schemas/MyTask'
+
+components:
+  schemas:
+    MyTask:
+      type: object
+      properties:
+        User:
+          type: string
+          description: The async route task owner.
+        CompletedTime:
+          type: string
+          description: The async route task completion time.
+          example: 2024-07-02T20:59:23.2174712Z
+          format: date-time
+        State:
+          type: string
+          description: The async route task status.
+          example: Running
+          enum:
+            - NotStarted
+            - Running
+            - Failed
+            - Completed
+        Result:
+          type: object
+          description: The result of the async route task.
+          properties:
+            InnerValue:
+              type: string
+              description: The inner value returned by the operation.
+```
+
+**Note**: The `MyTask` schema definition provided above is a partial example. You can expand this definition with additional properties according to your specific use case.

docs/Tutorials/Routes/Async/Features/Security.md

@@ -0,0 +1,103 @@
+
+# Security
+
+All async route operations are subject to Pode security, ensuring that any task operation complies with defined authentication and authorization rules.
+
+> **⚠ Important:**
+> All security checks are performed using the user identifier field specified by the `Set-PodeAsyncRouteUserIdentifierField` function. If this field is not explicitly set, the default field `Id` is used.
+
+#### Permissions
+ You can specify read and write permissions for each route. This can include specific users, groups, roles, and scopes.
+  - **Read Access**: Define which users, groups, roles, and scopes have read access. This means that the authenticated user that fits the permission can query the task status.
+  - **Write Access**: Define which users, groups, roles, and scopes have write access. This means that the authenticated user that fits the permission can stop the task.
+
+#### Permission Object Structure
+
+The permission object defines who can perform read or write operations on an async route. The object `Permission` has this structure:
+
+```powershell
+@{
+    Read  = @{
+        Groups = @()
+        Roles  = @()
+        Scopes = @()
+        Users  = @()
+    }
+    Write = @{
+        Groups = @()
+        Roles  = @()
+        Scopes = @()
+        Users  = @()
+    }
+}
+```
+
+- **Read**: Controls who can query the status of the async route task.
+- **Write**: Controls who can stop the async route task.
+
+An async route task generated by a route without any specified permissions will have read and write permissions granted to anyone, including anonymous users.
+
+By default, the owner has read and write privileges on the async route task.
+
+#### Example Usage
+
+```powershell
+New-PodeAuthScheme -Basic -Realm 'Pode Example Page' | Add-PodeAuth -Name 'Validate' -Sessionless -ScriptBlock {
+    param($username, $password)
+
+    # here you'd check a real user storage, this is just for example
+    if ($username -eq 'morty' -and $password -eq 'pickle') {
+        return @{
+            User = @{
+                Username = 'morty'
+                ID       = 'M0R7Y302'
+                Name     = 'Morty'
+                Type     = 'Human'
+                Groups   = @('Support')
+            }
+        }
+    }
+    elseif ($username -eq 'mindy' -and $password -eq 'pickle') {
+        return @{
+            User = @{
+                Username = 'mindy'
+                ID       = 'MINY321'
+                Name     = 'Mindy'
+                Type     = 'Alien'
+                Groups   = @('Developer')
+            }
+        }
+
+        return @{ Message = 'Invalid details supplied' }
+    }
+}
+
+Add-PodeRoute -PassThru -Method Put -Path '/asyncState' -Authentication 'Validate' -Group 'Support' -ScriptBlock {
+    $data = Get-PodeState -Name 'data'
+    Write-PodeHost 'data:'
+    Write-PodeHost $data -Explode -ShowType
+    Start-Sleep $data.sleepTime
+    return @{ InnerValue = $data.Message }
+} | Set-PodeAsyncRoute -PassThru \`
+    -ResponseContentType 'application/json', 'application/yaml' -Timeout 300 |
+    Set-PodeAsyncRoutePermission -Type Read -Groups 'Developer'
+```
+
+#### Explanation
+
+1. **Authentication Scheme**: The `New-PodeAuthScheme` creates a basic authentication scheme, and `Add-PodeAuth` adds the authentication named `Validate` with a script block that validates the user credentials.
+    - If the credentials match, the user information is returned.
+    - If the credentials do not match, an error message is returned.
+
+2. **Route Definition**: The `Add-PodeRoute` defines a route at `/asyncState` that requires authentication using the `Validate` scheme and is restricted to users in the `Support` group.
+    - The route retrieves some state data and writes it to the host, simulates some work by sleeping, and then returns the inner value of the state data.
+
+3. **Setting Async Route**: The `Set-PodeAsyncRoute` processes the route to make it asynchronous.
+    - `-ResponseContentType` specifies the response formats as JSON and YAML.
+    - `-Timeout 300` sets a timeout of 300 seconds for the async route task.
+4. **Setting Async Route Task Permissions**: The `Set-PodeAsyncRoutePermission` sets the read permission for users in the `Developer` group.
+
+    - By default only users in the `Developer` group can query the status of the task, and only users with write access can stop the task.
+    - The owner has read and write privileges on the async route task.
+
+This setup ensures that the route is secured with authentication, and permissions are properly managed to control who can query or stop the async route task.