process update endpoints and schema (relates to #107)

[fmigneault]

Description & Solution Reasoning

Provide support of revisions to existing processes with metadata updates or application redefinition.
When a new revision is created, a "duplicate" processID is generated. The duplicates are distinguished amongst each other using the combination of {processID}:{version} values.

Because processes collection has a unique rule on identifier in the DB, and for backward compatibility because some processes did not make use of version (and it still is optional), the "latest" revision is stored as identifier = processID while older ones are stored as identifier = processID:version. This respects the uniqueness while having "duplicate" process definitions. The only moment version becomes required is when revisions are involved, which is a required or auto-resolved field for PUT/PATCH requests (with specific version validation checks). The "previous latest" is updated in the DB by adding the version part to its identifier field, and the "new latest" revision can then be stored directly as identifier = processID. Using this approach, fetching the latest or an older revision is also natural from the API, as IDs are directly formatted appropriately. Jobs also make use of this processID:version format to properly refer to the right process that was executed.

The only adjustment that was required to retrieve a process was if the latest one was explicitly requested with processID:version. The fetch method of the storage therefore searches with both identifier = processID:version and identifier = processID && version = version combinations.

Changes:

• Add support of Process revisions (resolves Process UUID, updating and versioning #107).
• AddPATCH /processes/{processID} request, allowing MINOR and PATCH level modifications that can be applied
  to an existing Process in order to revise non-execution critical information such as its documented description.
• Add PUT /processes/{processID} request, allowing MAJOR revision to essentially redeploy a new Process,
  but leaving some form of relationship with the older version by reusing the same Process ID.
• Add support of {processID}:{version} representation in request path and Job processID to reference the
  specific Process revisions when fetch Process description and Job status.
• Add search query version and revisions to fetch a specific Process revision, or all versions history.
• Add more entries in links referring to Process revisions whenever applicable.

Fixes:

• Fix invalid minimum and maximum OpenAPI fields that were defined as minLength and maxLength
  (duplicates definitions) for Process description and deployment schema validation.

• Resolves DAC-199

• Resolves DAC-200

To Do

• validate that MAJOR PUT update becomes the latest, even if older process reference was used
  • make sure that correct previous "latest" is replaced (ie: not necessarily the older process reference)
    => when PUT, always use the "latest" (drop :{version} part) since deployed from scratch instead of update anyway?
  • make sure that compared version with target new version or auto-bumped uses the highest known version
• delete (undeploy) process with revision should make sure that new latest is updated if targeted revision was the current latest
• when listing revisions history, provide the diffs between them (use generate_diff function with generated process descriptions)
  moved to provide the diffs between revisions when listing their history #448
• propagate PATCH request updates to corresponding package metadata as specified from process metadata
  (eg: description -> CWL doc)
  moved to propagate PATCH request updates to corresponding package metadata as specified from process parameters #449
• test PATCH request to allow update of title, description, keywords, metadata of any input/output
• test listing of revisions
• more tests/complete missing implementations?
• update docs with new features
  • new small section in https://github.com/crim-ca/weaver/blob/master/docs/source/processes.rst
  • describe process versioning capabilities (what can be updated, listing revisions, fetching/executing a specific revision)
  • difference PATCH/PUT endpoints vs MAJOR/MINOR/PATCH updates (use same description as in changes 0a8c13b)
  • distinction deploy/undeploy vs deploy/update

[codecov]

Codecov Report

Merging #447 (ce9392a) into master (dfd3b34) will increase coverage by 0.11%.
The diff coverage is 84.87%.

@@            Coverage Diff             @@
##           master     #447      +/-   ##
==========================================
+ Coverage   81.64%   81.75%   +0.11%     
==========================================
  Files          75       75              
  Lines       14668    15210     +542     
  Branches     3188     3313     +125     
==========================================
+ Hits        11975    12435     +460     
- Misses       1975     2032      +57     
- Partials      718      743      +25     

Impacted Files	Coverage Δ
weaver/database/base.py	80.95% <ø> (ø)
weaver/processes/convert.py	80.33% <ø> (+0.20%)	⬆️
weaver/wps_restapi/utils.py	80.68% <64.00%> (-7.02%)	⬇️
weaver/processes/utils.py	78.76% <75.26%> (-1.12%)	⬇️
weaver/datatype.py	74.84% <76.13%> (-0.14%)	⬇️
weaver/database/__init__.py	95.83% <80.00%> (+1.09%)	⬆️
weaver/store/mongodb.py	74.61% <80.00%> (+0.94%)	⬆️
weaver/wps_restapi/processes/utils.py	93.67% <80.95%> (-4.72%)	⬇️
weaver/wps_restapi/processes/processes.py	84.28% <83.33%> (+0.82%)	⬆️
weaver/wps_restapi/jobs/utils.py	76.78% <90.00%> (+0.43%)	⬆️
... and 11 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dfd3b34...ce9392a. Read the comment docs.