Support for Async

[cmheazel]

API-Common has been assigned responsibility for Asynchronous operations. A simple async capability has bee proposed in the Proposals section of this repository. Another approach is provided in the EDR API Standard.

More to come.

[fhoubie]

Maybe alignment with OGC-API Process is a good start as well for async operations.

[jerstlouis]

The use of the Prefer header and job management conformance class of OGC API - Processes should be reviewed as a starting point. This could also support e.g. async large Coverage requests as well.

[christophenoel]

The use of the Prefer header and job management conformance class of OGC API - Processes should be reviewed as a starting point. This could also support e.g. async large Coverage requests as well.

What are the main proposed approaches ? I see the Processes API relying on openAPI friendly callback (+ poll async), and as alternative the responseHandler approach.

I'm in favor of the Processes API model on my side.

[jerstlouis]

Yesterday (September 23, 2024) at our joint OGC API - GDC / OGC API - Processes / Testbed 20 meeting, we discussed the possibility that the proposed Part 4 "Job management" ( see opengeospatial/ogcapi-processes#437 ) could be strictly focused on the "Common" aspect of job management, leaving the workflow description to be defined in Part 3: Workflows.

I proposed that this could potentially directly be called OGC API - Common "Job Management", addressing this issue, while delegating the drafting of the standard to the OGC API - Processes SWG.

cc. @joanma747 @pvretano @gfenoy @fmigneault

[gfenoy]

@jerstlouis, as introduced during the meeting, a section may be dedicated to PubSub to follow the job execution. I think we should participate in the discussion occurring in the OGC PubSub SWG (https://portal.ogc.org/files/?artifact_id=106803) that meets later this week.

@chris-little @cportele @tomkralidis @pvretano

[jerstlouis]

@gfenoy Yes it would probably make sense to have a requirement class in an OGC API - Common "Job Management" for job completion notification via PubSub.

[fmigneault]

There is a similar requirement in OAP core: https://docs.ogc.org/DRAFTS/18-062.html#Callbacks
Will "Job Management" reuse a similar concept? Will they be compatible?

[christophenoel]

Hi all,

First, I would like to highlight the recommendations we made in OGC Testbed 18: Secure Asynchronous Catalogue. Specifically for OGC API Common, we proposed a set of asynchronous patterns (section 8.1.4), including a variation with a subscriber.yaml in the header (instead of the body), as well as more advanced subscription approaches.

From my personal perspective, the PubSub mechanism seems overly complex and unnecessary for most processing scenarios (more useful if you need to configure regular updates from a catalogue for example). I don’t see it as a particularly useful requirement class, especially given the abundance of open-source tools that already implement de facto standard pub-sub brokers (e.g., AMQP).

Therefore, I believe the callback requirement class should be extended to support not only HTTP but also other common and standard protocols such as AMQP, email, etc. This is currently lacking.

[christophenoel]

PS: My previous statement about PubSub would be invalid if the PubSub mechanism were to provide a few minimal viable requirements that could effectively support a basic subscription mechanism.

[fmigneault]

callback requirement class should be extended to support not only HTTP but also other common and standard protocols such as AMQP, email, etc

I support this.
I don't think there is any technical limitation to extend the definition of callbacks.
In CRIM's implementation, we already have extended properties for multiple URL and email callbacks:

{
  "subscribers": {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "$id": "https://schemas.opengis.net/ogcapi/processes/part1/1.0/openapi/schemas/subscriber.yaml",
    "type": "object",
    "title": "JobExecuteSubscribers",
    "description": "Optional URIs for callbacks for this job.",
    "properties": {
      "successUri": {
        "type": "string",
        "title": "successUri",
        "description": "Location where to POST the job results on successful completion.",
        "format": "url",
        "pattern": "^((?:http|ftp)s?://)?(?!.*//.*$)(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\\.)+(?:[A-Z]{2,6}\\.?|[A-Z0-9-]{2,}\\.?)|localhost|\\[[a-f0-9:]+\\]|\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})(?::\\d+)?(?:/?|[/?]\\S+)$"
      },
      "failedUri": {
        "type": "string",
        "title": "failedUri",
        "description": "Location where to POST the job status if it fails execution.",
        "format": "url",
        "pattern": "^((?:http|ftp)s?://)?(?!.*//.*$)(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\\.)+(?:[A-Z]{2,6}\\.?|[A-Z0-9-]{2,}\\.?)|localhost|\\[[a-f0-9:]+\\]|\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})(?::\\d+)?(?:/?|[/?]\\S+)$"
      },
      "inProgressUri": {
        "type": "string",
        "title": "inProgressUri",
        "description": "Location where to POST the job status once it starts execution.",
        "format": "url",
        "pattern": "^((?:http|ftp)s?://)?(?!.*//.*$)(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\\.)+(?:[A-Z]{2,6}\\.?|[A-Z0-9-]{2,}\\.?)|localhost|\\[[a-f0-9:]+\\]|\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})(?::\\d+)?(?:/?|[/?]\\S+)$"
      },
      "successEmail": {
        "type": "string",
        "title": "successEmail",
        "description": "Email recipient to send a notification on successful job completion.",
        "format": "email"
      },
      "failedEmail": {
        "type": "string",
        "title": "failedEmail",
        "description": "Email recipient to send a notification on failed job completion.",
        "format": "email"
      },
      "inProgressEmail": {
        "type": "string",
        "title": "inProgressEmail",
        "description": "Email recipient to send a notification of job status once it starts execution.",
        "format": "email"
      }
    }
  }
}

[chris-little]

So what is being proposed to standardise? We identified several Async approaches in API-EDR but decided standardisation was inappropriate, until at least more implementation experience was visible "in the wild". Maybe a Best Practice document is required?

[christophenoel]

@chris-little In my view, OGC API Common may include a few requirement classes, or building blocks, from OGC PubSub.

One suggestion is a core property set for subscribing to callbacks, potentially extending OGC API Processes (see: OGC API Processes Subscriber). This could include:

• Additional protocols (e.g., AMQP, requiring a channel definition along with the endpoint)
• Subscription expiration time
• Frequency

Another improvement would be enhancing the callback format itself (see: OGC API Processes Callback), potentially defining formats/requirements for other protocols like AMQP topics.

[christophenoel]

Based on Testbed-18: Secure Asynchronous Catalog discussions, I would suggest as a starting point for the subscriber schema a flexible subscription model for receiving notifications related to job events or resource changes. Below schema supports both HTTP and AMQP protocols (as example), allowing users to subscribe to specific events (such as success, inProgress, failed) or directly to updates on resources using a resourceUri.

This schema defines a flexible subscription model for receiving notifications related to job events or resource changes. It supports both HTTP and AMQP protocols, allowing users to subscribe to specific events (such as success, inProgress, failed) or directly to updates on resources using a resourceUri.

Key Components:

1. subscriptions:

   • A list where each subscription can either target specific job events or resource changes.
   • Users can subscribe to multiple events by listing them in the event array (e.g., success, failed). E.g., Relevant for processing.
   • Alternatively, they can subscribe to updates on a specific resource by providing a resourceUri. E.g., Relevant for catalogues.

2. target:

   • Specifies the delivery method for the notifications.
   • Includes a protocol field, which can be either http or amqp.
   • For http, the schema requires the url to specify the HTTP callback destination.
   • For amqp, the schema requires the endpoint (URI of the AMQP broker) and channel (exchange name) for message delivery.

3. frequency (optional):

   • Defines how often notifications are sent, with options like immediate, hourly, daily, or weekly. This allows users to control the frequency of updates based on their needs.

4. expiresAt (optional):

   • Specifies the expiration time for the subscription in ISO 8601 format. After this time, the subscription is no longer valid.

How It Works:

• Users can create multiple subscriptions, each targeting different events or resources. For example, one subscription can send an HTTP callback on success and failed events, while another can subscribe to updates for a specific resource via AMQP.
• The schema is designed to allow different notification delivery methods for different subscriptions, providing flexibility to choose the appropriate protocol (HTTP or AMQP) for each use case.
• By offering frequency and expiresAt options, the schema supports both real-time notifications and delayed/scheduled updates, as well as time-bound subscriptions.

This model ensures that users can manage subscriptions with precision, supporting a variety of notification protocols, event types, and delivery frequencies.

description: |-
  Subscriptions for job events or resources, supporting HTTP and AMQP protocols, with optional frequency and expiration time.

  Support for these parameters is not required and may be
  removed from the API definition, if conformance class **'callback'**
  is not listed in the conformance declaration under `/conformance`.
type: object
required:
  - subscriptions
properties:
  subscriptions:
    type: array
    description: A list of events or resources the client is subscribing to, along with the corresponding target information.
    items:
      type: object
      required:
        - target
      properties:
        event:
          type: array
          description: Optional. The list of events or statuses being subscribed to (e.g., 'success', 'inProgress', 'failed'). If not provided, the subscription applies to resources.
          items:
            type: string
            enum: 
              - success
              - inProgress
              - failed
              - customEvent # Optional custom events can be added as needed.
        resourceUri:
          type: string
          format: uri
          description: Optional. URI of the resource to subscribe to if subscribing to a resource rather than events. Either `event` or `resourceUri` should be provided.

        # New 'target' property to encapsulate delivery details
        target:
          type: object
          description: The target where notifications should be delivered, based on the chosen protocol.
          required:
            - protocol
          properties:
            protocol:
              type: string
              description: The protocol to be used for the callback (e.g., 'http', 'amqp').
              enum:
                - http
                - amqp
            
            # HTTP-specific fields
            http:
              type: object
              description: HTTP-specific delivery information. This field is required if the protocol is 'http'.
              properties:
                url:
                  type: string
                  format: uri
                  description: The HTTP URL where the callback should be sent.

            # AMQP-specific fields
            amqp:
              type: object
              description: AMQP-specific delivery information. This field is required if the protocol is 'amqp'.
              properties:
                endpoint:
                  type: string
                  format: uri
                  description: The URI of the AMQP broker.
                channel:
                  type: string
                  description: The AMQP channel (queue or topic) for message delivery.
        
        # Optional frequency property
        frequency:
          type: string
          description: Optional. The frequency of the notifications (e.g., 'hourly', 'daily', 'immediate').
          enum:
            - immediate
            - hourly
            - daily
            - weekly
  
  # Property for expiration time
  expiresAt:
    type: string
    format: date-time
    description: The date and time when the subscription should expire, in ISO 8601 format.

Example

subscriptions:
  - event:
      - success
      - failed
    target:
      protocol: http
      http:
        url: "https://example.com/callback"
    frequency: immediate
  - resourceUri: "https://example.com/resource/123"
    target:
      protocol: amqp
      amqp:
        endpoint: "amqp://broker.example.com"
        channel: "resource_updates"
    frequency: hourly
expiresAt: "2024-12-31T23:59:59Z"

Security Aspects

As demonstrated in Testbed-18: Secure Asynchronous Catalog, it's not recommended to leave the callback endpoint exposed. Protecting the callback endpoint is crucial, particularly when using sensitive information like username and password for AMQP endpoints (those properties are missing in the above draft schema).

Even if transmitted over encrypted TLS communication, passing such credentials can pose security risks. Addressing these concerns would require an additional requirement class focused on securing the callback mechanism, ensuring proper authentication and protection of sensitive data during transmission

[christophenoel]

(maybe a different requirement class would be relevant to expose in the capabilites if the endpoint support resource subscription, or event subscription).