Add API & Processes profiles (#81)

Author: m-mohr

federation/backends/api.md

@@ -4,6 +4,84 @@ The general contract is the [openEO API](https://api.openeo.org) in the latest s
 
 The aggregator that proxies the back-ends in the federation also implements the same API, but it also implements the "Federation Extension" (currently in draft state).
 
+## Profiles
+
+In addition to the general openEO API specification and their [API Profiles](https://openeo.org/documentation/1.0/developers/profiles/api.html),
+openEO Platform requires to implement an additional API profile:
+*LP: Required for openEO Platform*, which requires the openEO profile *L2: Recommended*.
+The requirement to implement two of *L1A*, *L1B*, and *L1C* has been restricted for openEO Platform to always require *L1A: Synchronous Processing* and *L1B: Batch Jobs* (see req. no. 703). This means that *L1C: Secondary Web Services* is optional.
+
+<figure>
+    <img src="./profiles/api.png" alt="The hierarchy of openEO and openEO Platform API profiles.">
+    <figcaption>An overview of the openEO and openEO Platform API profiles.</figcaption>
+</figure>
+
+### LP: Required for openEO Platform
+
+The profile only lists requirements that are not covered by the openEO profile *L2: Recommended* yet.
+
+#### API fundamentals
+
+| # | Functionality | Description |
+| -- | -- | -- |
+| 14 | All > Billing | Supports the openEO Platform credit system |
+
+#### File Formats
+
+| # | Functionality | Description |
+| -- | -- | -- |
+| 45 | `GET /file_formats` | File format names and parameters aligned with openEO Platform as defined for the pre-defined [file formats](./fileformats.md) |
+
+#### Other
+
+| # | Functionality | Description |
+| -- | -- | -- |
+| 90 | `GET /health` | Returns 2XX or 5XX http status code (without authentication) |
+
+#### Auth
+
+| # | Functionality | Description |
+| -- | -- | -- |
+| 113 | `GET /credentials/oidc` | Supports EGI as identity provider (including tokens) |
+| 118 | `GET /credentials/oidc` | Supports the required entitlements of the vo.openeo.cloud virtual organization, especially the claim eduperson_entitlement |
+
+For more details about [Authentication and Authorization](#authentication-and-authorization), please see the corresponding chapters below.
+
+#### Pre-defined Processes
+
+| # | Functionality | Description |
+| -- | -- | -- |
+| 205 | `GET /processes` > processes | All processes are valid according to the specification (id, description, parameters, returns are required) |
+| 208 | `GET /processes` > processes | Processes are marked as experimental or deprecated if applicable |
+
+#### Collections
+
+| # | Functionality | Description |
+| -- | -- | -- |
+| 311 | `GET /collections` > collections | Collections are marked as experimental or deprecated if applicable |
+| 324 | `GET /collections/{id}` > id | IDs follow the openEO Platform naming convention |
+| 327 | `GET /collections/{id}` > providers | Each collection needs to expose the backend offering the data |
+
+#### Data Processing
+
+| # | Functionality | Description |
+| -- | -- | -- |
+| 703 |  | Batch jobs and synchronous processing are implemented (secondary web services are optional) |
+| 704 |  | Time after which batch job results get automatically deleted: 90 days or later |
+| 705 |  | Time after which batch job metadata gets automatically deleted: 1 year or later |
+
+#### Batch Jobs > Results
+
+| # | Functionality | Description |
+| -- | -- | -- |
+| 873 | `GET /jobs/{id}/results` > public access link | Default expiry time of the signed URLs for results: 7 days |
+
+#### Synchronous Processing
+
+| # | Functionality | Description |
+| -- | -- | -- |
+| 920 | `POST /result` > timeout | The timeout for synchronous calls is: 5 minutes |
+
 ## Authentication and authorization
 
 This important aspect of the federation is standardized by the [AARC Blueprint Architecture](https://aarc-project.eu/architecture/). EGI Check-in is the concrete implementation that is currently in use.
@@ -12,26 +90,26 @@ This important aspect of the federation is standardized by the [AARC Blueprint A
 
 The openEO platform federation standardizes on the use of [EGI Check-in](https://aai.egi.eu) as identity provider. Backends have to support the use of openID connect + PKCE, to enable this and register a client with EGI Check-in. 
 
-## Authorization
+### Authorization
 
-### Entitlements
+#### Entitlements
 
 Users of the federation are organized under the 'vo.openeo.cloud' virtual organization in EGI Check-in. Inside the virtual organization, different roles can be assigned to a user, to indicate that they have a certain subscription, or even on a more fine-grained level are entitled to specific actions or resources.
-The mechanism to check this, is again supported by EGI Check-in, under the 'eduperson_entitlement' claim: https://docs.egi.eu/providers/check-in/sp/#claims
+The mechanism to check this, is again supported by EGI Check-in, under the 'eduperson_entitlement' claim: <https://docs.egi.eu/providers/check-in/sp/#claims>
 
-### Credits
+#### Credits
 
 The second criterium for authorization is based on credits that are available to a user. Credits allow the platform to limit the volume of data access and processing operations that a user can perform during a given time frame. The amount of available credits depends on the subscription.
 When the credit balance of a user goes below zero, processing operations can be blocked.
 
-### Aggregator rules
+#### Aggregator rules
 
 Based on the subscription and available credits, the aggregator can implement these rules:
 
 1. Credit checks to block starting of batch jobs, synchronous requests to /result and viewing services.
 2. Rate limiting (TBD)
 
-### Backend rules
+#### Backend rules
 
 Some authorization rules will need to be enforced by the backends themselves:
 
@@ -40,7 +118,3 @@ Some authorization rules will need to be enforced by the backends themselves:
 3. Available processing resources, batch job priorities
 4. Batch job result data volume
 5. Access to restricted collections
-
-
-
-

federation/backends/collections.md

@@ -1,5 +1,6 @@
 # Collections
-This page describes requirements on STAC collection metadata for backend providers in the openEO platform federation. These requirements should be considered an addition to what is already required by the [openEO API](https://api.openeo.org/#tag/EO-Data-Discovery/operation/describe-collection).
+This page describes requirements on STAC collection metadata for backend providers in the openEO platform federation.
+These requirements should be considered an addition to what is already required by the [openEO API](https://api.openeo.org/#tag/EO-Data-Discovery/operation/describe-collection) and the [API profiles](./api.md#profiles).
 
 ## Collection availability
 

federation/backends/processes.md

@@ -1,123 +1,106 @@
 # Processes
 
-## Core Profile
+The openEO project defines a large number of processes, which can be found at <https://processes.openeo.org>.
+Due to the number of processes, we need to define a profile (i.e. a subset of processes) that must be implemented on each back-end so that a certain level of interoperability can be achieved.
 
-As the openEO project defines a lot of processes, we need to define a core profile (i.e. a subset) that needs to be implemented on each back-end.
-All processes specifications can be found at <https://processes.openeo.org>
+In addition to the general openEO Processes specification and their [Processes Profiles](https://openeo.org/documentation/1.0/developers/profiles/processes.html),
+openEO Platform requires to implement an additional Process profile:
+*L2P: Required - openEO Platform*, which requires one of the openEO profiles *L2A: Recommended (Raster)* or *L2B: Recommended (Vector)*.
+There are two additional openEO Platform specific profiles with additional requirements: *L3P: Advanced - openEO Platform* and *L4P: Complete - openEO Platform*.
 
-### Data Cubes
-- `add_dimension`: Add a new dimension
-- `aggregate_spatial`: Zonal statistics for geometries
-- `aggregate_temporal`: Temporal aggregations
-- `aggregate_temporal_period`: Temporal aggregations based on calendar hierarchies
-- `apply`: Apply a process to each pixel
-- `apply_dimension`: Apply a process to pixels along a dimension
-- `apply_kernel`: Apply a spatial convolution with a kernel
-- `dimension_labels`: Get the dimension labels
-- `drop_dimension`: Remove a dimension
-- `filter_bands`: Filter the bands by names
-- `filter_bbox`: Spatial filter using a bounding box
-- `filter_spatial`: Spatial filter using geometries
-- `filter_temporal`: Temporal filter for temporal intervals
-- `load_collection`: Load a collection
-- `load_result`: Load batch job results - **experimental**
-- `mask`: Apply a raster mask
-- `mask_polygon`: Apply a polygon mask
-- `merge_cubes`: Merge two data cubes
-- `reduce_dimension`: Reduce dimensions
-- `rename_dimension`: Rename a dimension
-- `rename_labels`: Rename dimension labels -> needed often for apply_dimension
-- `resample_cube_spatial`: Resample the spatial dimensions to match a target data cube
-- `resample_cube_temporal`: Resample temporal dimensions to match a target data cube - **experimental**
-- `resample_spatial`: Resample and warp the spatial dimensions
-- `save_result`: Save processed data
+<figure>
+    <img src="./profiles/processes.png" alt="The hierarchy of openEO and openEO Platform Processes profiles.">
+    <figcaption>An overview of the openEO and openEO Platform Processes profiles.</figcaption>
+</figure>
 
-### Arrays / Reducers
-- `array_append`: Append a value to an array - **experimental**
-- `array_apply`: Apply a process to each array element
-- `array_concat`: Merge two arrays - **experimental**
-- `array_contains`: Check whether the array contains a given value
-- `array_create`: Create an array - **experimental**
-- `array_element`: Get an element from an array
-- `array_filter`: Filter an array based on a condition
-- `array_find`: Get the index for a value in an array
-- `array_interpolate_linear`: One-dimensional linear interpolation for arrays - **experimental**
-- `array_labels`: Get the labels for an array
-- `array_modify`: Change the content of an array (insert, remove, update) - **experimental**
-- `count`: Count the number of elements
-- `extrema`: Minimum and maximum values
-- `first`: First element
-- `last`: Last element
-- `max`: Maximum value
-- `mean`: Arithmetic mean (average)
-- `median`: Statistical median
-- `min`: Minimum value
-- `order`: Create a permutation
-- `product`: Compute the product by multiplying numbers
-- `rearrange`: Rearrange an array based on a permutation
-- `sort`: Sort data
-- `sum`: Compute the sum by adding up numbers
+### Requirements per Profile
 
-### Math
-- `absolute`: Absolute value
-- `add`: Addition of two numbers
-- `arccos`: Inverse cosine
-- `arcosh`: Inverse hyperbolic cosine
-- `arcsin`: Inverse sine
-- `arctan`: Inverse tangent
-- `arctan2`: Inverse tangent of two numbers
-- `arsinh`: Inverse hyperbolic sine
-- `artanh`: Inverse hyperbolic tangent
-- `ceil`: Round fractions up
-- `clip`: Clip a value between a minimum and a maximum
-- `constant`: Define a constant value -> pretty easy implementation
-- `cos`: Cosine
-- `cosh`: Hyperbolic cosine
-- `divide`: Division of two numbers
-- `e`: Euler's number
-- `exp`: Exponentiation to the base e
-- `floor`: Round fractions down
-- `int`: Integer part of a number
-- `linear_scale_range`: Linear transformation between two ranges
-- `ln`: Natural logarithm
-- `log`: Logarithm to a base
-- `mod`: Modulo
-- `multiply`: Multiplication of two numbers
-- `nan` - Not a Number - **experimental**
-- `pi`: Pi
-- `power`: Exponentiation
-- `round`: Round to a specified precision
-- `sgn`: Signum
-- `sin`: Sine
-- `sinh`: Hyperbolic sine
-- `sqrt`: Square root
-- `subtract`: Subtraction of two numbers
-- `tan`: Tangent
-- `tanh`: Hyperbolic tangent
+The profiles below only lists requirements that are not covered by the corresponding openEO profiles yet.
 
-### Statistics / Indices
-- `ndvi`: Normalized Difference Vegetation Index
-- `normalized_difference`: Normalized difference
-- `quantiles`: Quantiles
-- `sd`: Standard deviation
-- `variance`: Variance
+Although in the openEO Profiles experimental processes shall only lead to a "warning", experimental processes in profiles that *L2P* depends on are required to be available (which currently are `inspect` and `nan`).
 
-### Logic
-- `and`: Logical AND
-- `all`: Are all of the values true?
-- `any`: Is at least one value true?
-- `between`: Between comparison
-- `eq`: Equal to comparison
-- `gt`: Greater than comparison
-- `gte`: Greater than or equal to comparison
-- `if`: If-Then-Else conditional
-- `is_infinite`: Value is an infinite number - **experimental**
-- `is_nan`: Value is not a number
-- `is_nodata`: Value is a no-data value
-- `is_valid`: Value is valid data
-- `lt`: Less than comparison
-- `lte`: Less than or equal to comparison
-- `neq`: Not equal to comparison
-- `not`: Inverting a boolean
-- `or`: Logical OR
-- `xor`: Logical XOR (exclusive or)
+## L2P: Required - openEO Platform
+
+Requires [openEO Profile L2: Recommended](https://openeo.org/documentation/1.0/developers/profiles/processes.html#l2-recommended) and additionally:
+
+- **`add_dimension`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`aggregate_spatial`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`aggregate_temporal`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`aggregate_temporal_period`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`apply`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`apply_dimension`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`apply_kernel`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`array_contains`**
+- **`dimension_labels`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`drop_dimension`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`filter_bands`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`filter_bbox`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`filter_spatial`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`filter_temporal`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`load_collection`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`mask`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`mask_polygon`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`merge_cubes`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`ndvi`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`reduce_dimension`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`rename_dimension`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`rename_labels`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`resample_cube_spatial`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`resample_spatial`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`save_result`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+
+## L3P: Advanced - openEO Platform
+
+Requires [openEO Profile L3: Advanced](https://openeo.org/documentation/1.0/developers/profiles/processes.html#l3-advanced) and additionally:
+
+- **`apply_neighborhood`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`ard_surface_reflectance`** (experimental)
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`filter_labels`** (experimental)
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`load_stac (load_result)`** (experimental)
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`reduce_spatial`** (experimental)
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`resample_cube_temporal`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`run_udf`**
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+
+## L4P: Complete - openEO Platform
+
+Requires [openEO Profile L4: Above and Beyond](https://openeo.org/documentation/1.0/developers/profiles/processes.html#l4-above-and-beyond) and additionally:
+
+- **`atmospheric_correction`** (experimental)
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`load_uploaded_files`** (experimental)
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`load_url`** (experimental)
+  - has been tested on > 100x100km at 10m resolution (or equivalent)
+- **`sar_backscatter`** (experimental)
+  - has been tested on > 100x100km at 10m resolution (or equivalent)