process update endpoints and schema (relates to #107)

.pylintrc

@@ -510,7 +510,7 @@ int-import-graph=
 
 # Force import order to recognize a module as part of the standard
 # compatibility libraries.
-known-standard-library=posixpath
+known-standard-library=posixpath,typing,typing_extensions
 
 # Force import order to recognize a module as part of a third party library.
 known-third-party=enchant,cornice_swagger,cwltool,cwt,docker

CHANGES.rst

@@ -12,11 +12,26 @@ Changes
 
 Changes:
 --------
-- No change.
+- Add support of `Process` revisions (resolves `#107 <https://github.com/crim-ca/weaver/issues/107>`_).
+- Add ``PATCH /processes/{processID}`` request, allowing ``MINOR`` and ``PATCH`` level modifications that can be
+  applied to an existing `Process` in order to revise non-execution critical information. Level ``PATCH`` is used to
+  identify changes with no impact on execution whatsoever, only affecting metadata such as its documented description.
+  Level ``MINOR`` is used to update components that affect only execution *methodology* (e.g.: sync/async) or `Process`
+  retrieval, but that do not directly impact *what* is executed (i.e.: the `Application Package` does not change).
+- Add ``PUT /processes/{processID}`` request, allowing ``MAJOR`` revision to essentially redeploy a new `Process`,
+  but leaving some form of relationship with older versions by reusing the same `Process` ID. This ``MAJOR`` update
+  level implies a relatively critical change to execute the `Process`, such as the addition, removal or modification
+  of an input or output, directly impacting the `Application Package` definition and parameters the `Process` offers.
+- Add support of ``{processID}:{version}`` representation in request path and ``processID`` of the `Job` definition
+  to reference the specific `Process` revisions when fetching a `Process` description or a `Job` status.
+- Add search query ``version`` and ``revisions`` parameters to allow description of a specific `Process` revision, or
+  listing all its versions history.
+- Add more entries in ``links`` referring to `Process` revisions whenever applicable.
 
 Fixes:
 ------
-- No change.
+- Fix invalid ``minimum`` and ``maximum`` OpenAPI fields that were defined as ``minLength`` and ``maxLength``
+  (duplicates definitions) for `Process` description and deployment schema validation.
 
 .. _changes_4.19.0:
 

README.rst

@@ -302,7 +302,7 @@ It is part of `PAVICS`_ and `Birdhouse`_ ecosystems and is available within the
 .. _ogc-eo-apps-pilot-er: http://docs.opengeospatial.org/per/20-045.html
 .. |ogc-best-practices-eo-apppkg| replace:: OGC Best Practice for Earth Observation Application Package
 .. _ogc-best-practices-eo-apppkg: https://docs.ogc.org/bp/20-089r1.html
-.. |ogc-api-proc-ext-part2| replace:: `OGC API - Processes - Part 2: Deploy, Replace, Undeploy`_ (DRU) extension
+.. |ogc-api-proc-part2| replace:: `OGC API - Processes - Part 2: Deploy, Replace, Undeploy`_ (DRU) extension
 .. _`OGC API - Processes - Part 2: Deploy, Replace, Undeploy`: https://github.com/opengeospatial/ogcapi-processes/tree/master/extensions/deploy_replace_undeploy
 .. |ogc-apppkg| replace:: `OGC Application Package`
 .. _ogc-apppkg: https://github.com/opengeospatial/ogcapi-processes/blob/master/extensions/deploy_replace_undeploy/standard/openapi/schemas/ogcapppkg.yaml

docs/examples/update-process-minor.http

@@ -0,0 +1,9 @@
+PATCH /processes/test-process HTTP/1.1
+Host: weaver.example.com
+Content-Type: application/json
+
+{
+  "description": "process async only",
+  "jobControlOptions": ["async-execute"],
+  "version": "1.4.0"
+}

docs/examples/update-process-patch.http

@@ -0,0 +1,17 @@
+PATCH /processes/test-process:1.2.3 HTTP/1.1
+Host: weaver.example.com
+Content-Type: application/json
+
+{
+  "description": "new description",
+  "inputs": {
+    "input": {
+      "description": "modified input description"
+    },
+    "outputs": {
+      "output": {
+        "title": "modified title"
+      }
+    }
+  }
+}

docs/source/processes.rst

@@ -413,6 +413,133 @@ that define the process references and expected inputs/outputs.
 .. _`Provider requests`: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Providers
 .. _`Process requests`: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Processes
 
+.. versionchanged:: 4.20
+
+With the addition of :term:`Process` revisions (see :ref:`Update Operation <proc_op_update` below), a registered
+:term:`Process` specified only by ``{processID}`` will retrieve the latest revision of that :term:`Process`.
+A specific older revision can be obtained by adding the tagged version in the path (``{processID}:{version}``) or
+adding the request query parameter ``version``.
+
+Using revisions provided through ``PUT`` and ``PATCH`` requests, it is also possible to list specific or all existing
+revisions of a given or multiple processes simultaneously using the ``revisions`` and ``version`` query parameters with
+the |getcap-req|_ request.
+
+
+.. _proc_op_undeploy:
+.. _proc_op_update:
+
+Update, replace or remove an existing process (Update, Replace, Undeploy)
+-----------------------------------------------------------------------------
+
+Since `Weaver` supports |ogc-api-proc-part2|_, it is able to remove a previously registered :term:`Process` using
+the :ref:`Deployment <proc_op_deploy>` request. The undeploy operation consist of a ``DELETE`` request targeting the
+specific ``{WEAVER_URL}/processes/{processID}`` to be removed.
+
+.. note::
+    The :term:`Process` must be accessible by the user considering any visibility configuration to perform this step.
+    See :ref:`proc_op_deploy` section for details.
+
+.. versionadded:: 4.20
+
+Starting from version `4.20 <https://github.com/crim-ca/weaver/tree/4.20.0>`_, a :term:`Process` can be replaced or
+updated using respectively the ``PUT`` and ``PATCH`` requests onto the specific ``{WEAVER_URL}/processes/{processID}``
+location of the reference to modify.
+
+.. note::
+    The :term:`Process` partial update operation (using ``PATCH``) is specific to `Weaver` only.
+    |ogc-api-proc-part2|_ only mandates the definition of ``PUT`` request for full override of a :term:`Process`.
+
+When a :term:`Process` is modified using the ``PATCH`` operation, only the new definitions need to be provided, and
+unspecified items are transferred over from the referenced :term:`Process` (i.e.: the previous revision). Using either
+the ``PUT`` or ``PATCH`` requests, previous revisions can be referenced using two formats:
+
+- ``{processID}:{version}`` as request path parameters (instead of usual ``{processID}`` only)
+- ``{processID}`` in the request path combined with ``?version={version}`` query parameter
+
+`Weaver` employs ``MAJOR.MINOR.PATCH`` semantic versioning to maintain revisions of updated or replaced :term:`Process`
+definitions. The next revision number to employ for update or replacement can either be provided explicitly in the
+request body using a ``version``, or be omitted. When omitted, the next revision will be guessed automatically based
+on the previous available revision according to the level of changes required. In either cases, the resolved ``version``
+will have to be available and respect the expected update level to be accepted as a new valid :term:`Process` revision.
+The applicable revision level depends on the contents being modified using submitted request body fields according
+to the following table. When a combination of the below items occur, the higher update level is required.
+
++-------------+-----------+------------------------------------+------------------------------------------------------+
+| HTTP Method | Level     | Change                             | Examples                                             |
++=============+===========+====================================+======================================================+
+| ``PATCH``   | ``PATCH`` | Modifications to metadata          | - :term:`Process` ``description``, ``title`` strings |
+|             |           | not impacting the :term:`Process`  | - :term:`Process` ``keywords``, ``metadata`` lists   |
+|             |           | execution or definition.           | - inputs/outputs ``description``, ``title`` strings  |
+|             |           |                                    | - inputs/outputs ``keywords``, ``metadata`` lists    |
++-------------+-----------+------------------------------------+------------------------------------------------------+
+| ``PATCH``   | ``MINOR`` | Modification that impacts *how*    | - :term:`Process` ``jobControlOptions`` (async/sync) |
+|             |           | the :term:`Process` could be       | - :term:`Process` ``outputTransmission`` (ref/value) |
+|             |           | executed, but not its definition.  | - :term:`Process` ``visibility``                     |
++-------------+-----------+------------------------------------+------------------------------------------------------+
+| ``PUT``     | ``MAJOR`` | Modification that impacts *what*   | - Any :term:`Application Package` modification       |
+|             |           | the :term:`Process` executes.      | - Any inputs/outputs change (formats, occurs, type)  |
+|             |           |                                    | - Any inputs/outputs addition or removal             |
++-------------+-----------+------------------------------------+------------------------------------------------------+
+
+.. note::
+    For all applicable fields of updating a :term:`Process`, refer to the schema of |update-req|_.
+    For replacing a :term:`Process`, refer instead to the schema of |replace-req|_. The replacement request contents
+    are extremely similar to the :ref:`Deploy <proc_op_deploy>` schema since the full :term:`Process` definition must
+    be provided.
+
+For example, if the ``test-process:1.2.3`` was previously deployed, and is the active latest revision of that
+:term:`Process`, submitting the below request body will produce a ``PATCH`` revision as ``test-process:1.2.4``.
+
+.. literalinclude:: ../examples/update-process-patch.http
+    :language: http
+    :caption: Sample request for ``PATCH`` revision
+
+Here, only metadata is adjusted and there is no risk to impact produced results or execution methods of the
+:term:`Process`. An external user would probably not even notice the :term:`Process` changed, which is why ``PATCH``
+is reasonable in this case. Notice that the ``version`` is not explicitly provided in the body. It is guessed
+automatically from the modified contents. Also, the example displays how :term:`Process`-level and
+inputs/outputs-level metadata can be updated.
+
+Similarly, the following request would produce a ``MINOR`` revision of ``test-process``. Since both ``PATCH`` and
+``MINOR`` level contents are defined for update, the higher ``MINOR`` revision is required. In this case ``MINOR`` is
+required because ``jobControlOptions`` (forced to asynchronous execution for following versions) would break any future
+request made by users that would expect the :term:`Process` to run (or support) synchronous execution.
+
+Notice that this time, the :term:`Process` reference does not indicate the revision in the path (no ``:1.2.4`` part).
+This automatically resolves to the updated revision ``test-process:1.2.4`` that became the new latest revision following
+our previous ``PATCH`` request.
+
+.. literalinclude:: ../examples/update-process-minor.http
+    :language: http
+    :caption: Sample request for ``MINOR`` revision
+
+In this case, the desired ``version`` (``1.4.0``) is also specified explicitly in the body. Since the updated number
+(``MINOR = 4``) matches the expected update level from the above table and respects an higher level than the reference
+``1.2.4`` :term:`Process`, this revision value will be accepted (instead of auto-resolved ``1.3.0`` otherwise). Note
+that if ``2.4.0`` was specified instead, the version would be refused, as `Weaver` does not consider this modification
+to be worth a ``MAJOR`` revision, and tries to keep version levels consistent. Skipping numbers (i.e.: ``1.3.0`` in this
+case), is permitted as long as there are no other versions above of the same level (i.e.: ``1.4.0`` would be refused if
+``1.5.0`` existed). This allows some level of flexibility with revisions in case users want to use specific numbering
+values that have more meaning to them. It is recommended to let `Weaver` auto-update version values between updates if
+this level of fined-grained control is not required.
+
+.. note::
+    To avoid conflicting definitions, a :term:`Process` cannot be :ref:`Deployed <proc_op_deploy>` directly using a
+    ``{processID}:{version}`` reference. Deployments are expected as the *first revision* and should only include the
+    ``{processID}`` portion as their identifier.
+
+If the user desires a specific version to deploy, the ``PUT`` request should be used with the appropriate ``version``
+within the request body. It is although up to the user to provide the full definition of that :term:`Process`,
+as ``PUT`` request will completely replace the previous definition rather than transfer over previous updates
+(i.e: ``PATCH`` requests).
+
+Even when a :term:`Process` is *"replaced"* using ``PUT``, the older revision is not actually removed and undeployed
+(``DELETE`` request). It is therefore still possible to refer to the old revision using explicit references with the
+corresponding ``version``. `Weaver` keeps track of revisions by corresponding ``{processID}`` entries such that if
+the latest revision is undeployed, the previous revision will automatically become the latest once again. For complete
+replacement, the user should instead perform a ``DELETE`` of all existing revisions (to avoid conflicts) followed by a
+new :ref:`Deploy <proc_op_deploy>` request.
+
 .. _proc_op_execute:
 
 Execution of a process (Execute)
@@ -483,6 +610,14 @@ and parametrization of various input/output combinations. Let's employ the follo
 .. |exec-api| replace:: OpenAPI Execute
 .. _exec-api: `exec-req`_
 
+
+.. versionchanged:: 4.20
+
+With the addition of :term:`Process` revisions (see :ref:`Update Operation <proc_op_update` section), a registered
+:term:`Process` specified only by ``{processID}`` will execute the latest revision of that :term:`Process`. An older
+revision can be executed by adding the tagged version in the path (``{processID}:{version}``) or adding the request
+query parameter ``version``.
+
 .. _proc_exec_body:
 
 Execution Body
@@ -1206,7 +1341,7 @@ email-specific settings (see: :ref:`Configuration`).
 .. _proc_op_status:
 .. _proc_op_monitor:
 
-Monitoring of a process (GetStatus)
+Monitoring of a process execution (GetStatus)
 ---------------------------------------------------------------------
 
 Monitoring the execution of a :term:`Job` consists of polling the status ``Location`` provided from the :ref:`Execute`

docs/source/references.rst

@@ -121,31 +121,35 @@
 .. _deploy-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Processes%2Fpaths%2F~1processes%2Fpost
 .. |getcap-req| replace:: ``GET {WEAVER_URL}/processes`` (GetCapabilities)
 .. _getcap-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Processes%2Fpaths%2F~1processes%2Fget
-.. |describe-req| replace:: ``GET {WEAVER_URL}/processes/{id}`` (DescribeProcess)
+.. |describe-req| replace:: ``GET {WEAVER_URL}/processes/{processID}`` (DescribeProcess)
 .. _describe-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Processes%2Fpaths%2F~1processes~1%7Bprocess_id%7D~1package%2Fget
+.. |update-req| replace:: ``PATCH {WEAVER_URL}/processes/{processID}`` (Update)
+.. _update-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Processes/paths/~1processes~1{process_id}/patch
+.. |replace-req| replace:: ``PUT {WEAVER_URL}/processes/{processID}`` (Replace)
+.. _replace-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Processes/paths/~1processes~1{process_id}/put
 .. |exec-req-name| replace:: Execute
 .. _exec-req-name: `exec-req`_
-.. |exec-req| replace:: ``POST {WEAVER_URL}/processes/{id}/execution`` (Execute)
+.. |exec-req| replace:: ``POST {WEAVER_URL}/processes/{processID}/execution`` (Execute)
 .. _exec-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Processes/paths/~1processes~1{process_id}~1execution/post
-.. |exec-req-job| replace:: ``POST {WEAVER_URL}/processes/{id}/jobs`` (Execute)
+.. |exec-req-job| replace:: ``POST {WEAVER_URL}/processes/{processID}/jobs`` (Execute)
 .. _exec-req-job: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Processes%2Fpaths%2F~1processes~1{process_id}~1jobs%2Fpost
-.. |vis-req| replace:: ``PUT {WEAVER_URL}/processes/{id}/visibility`` (Visibility)
+.. |vis-req| replace:: ``PUT {WEAVER_URL}/processes/{processID}/visibility`` (Visibility)
 .. _vis-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Processes%2Fpaths%2F~1processes~1%7Bprocess_id%7D~1visibility%2Fput
-.. |pkg-req| replace:: ``GET {WEAVER_URL}/processes/{id}/package`` (Package)
+.. |pkg-req| replace:: ``GET {WEAVER_URL}/processes/{processID}/package`` (Package)
 .. _pkg-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Processes%2Fpaths%2F~1processes~1%7Bprocess_id%7D~1package%2Fget
-.. |log-req| replace:: ``GET {WEAVER_URL}/processes/{id}/jobs/{id}/logs`` (GetLogs)
+.. |log-req| replace:: ``GET {WEAVER_URL}/processes/{processID}/jobs/{jobID}/logs`` (GetLogs)
 .. _log-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Logs%2Fpaths%2F~1processes~1%7Bprocess_id%7D~1jobs~1%7Bjob_id%7D~1logs%2Fget
-.. |except-req| replace:: ``GET {WEAVER_URL}/processes/{id}/jobs/{id}/exceptions`` (GetLogs)
+.. |except-req| replace:: ``GET {WEAVER_URL}/processes/{processID}/jobs/{jobID}/exceptions`` (GetLogs)
 .. _except-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Logs%2Fpaths%2F~1processes~1%7Bprocess_id%7D~1jobs~1%7Bjob_id%7D~1logs%2Fget
 .. |status-req-name| replace:: Status
 .. _status-req-name: `status-req`_
-.. |status-req| replace:: ``GET {WEAVER_URL}/processes/{id}/jobs/{id}`` (GetStatus)
+.. |status-req| replace:: ``GET {WEAVER_URL}/processes/{processID}/jobs/{jobID}`` (GetStatus)
 .. _status-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Status%2Fpaths%2F~1processes~1{process_id}~1jobs~1{job_id}%2Fget
-.. |inputs-req| replace:: ``GET {WEAVER_URL}/jobs/{id}/inputs`` (Inputs)
+.. |inputs-req| replace:: ``GET {WEAVER_URL}/jobs/{jobID}/inputs`` (Inputs)
 .. _inputs-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/inputs/paths/~1jobs~1{job_id}~1inputs/get
-.. |outputs-req| replace:: ``GET {WEAVER_URL}/jobs/{id}/outputs`` (Outputs)
+.. |outputs-req| replace:: ``GET {WEAVER_URL}/jobs/{jobID}/outputs`` (Outputs)
 .. _outputs-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/outputs/paths/~1jobs~1{job_id}~1outputs/get
-.. |results-req| replace:: ``GET {WEAVER_URL}/jobs/{id}/results`` (Results)
+.. |results-req| replace:: ``GET {WEAVER_URL}/jobs/{jobID}/results`` (Results)
 .. _results-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Results/paths/~1jobs~1{job_id}~1results/get
 .. |update-token-req| replace:: Update Token
 .. _update-token-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/UpdateToken/paths/~1processes~1{process_id}/put

requirements-dev.txt

@@ -25,8 +25,7 @@ pylint_quotes
 responses
 safety
 stopit
-# typing extension required for TypedDict
-typing_extensions; python_version < "3.8"
+typing_extensions
 WebTest
 wsgiproxy
 WSGIProxy2