Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal OGC API - Processs - Part4: Job Management #437

Open
wants to merge 26 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
717321a
Boostrap part 4 (with wrong document number)
gfenoy Aug 29, 2024
ca03fbd
Fix typos
gfenoy Aug 29, 2024
6b7acb3
Fix typos
gfenoy Aug 29, 2024
4aaf777
Fix typos
gfenoy Aug 29, 2024
8a6d469
Fix typo
gfenoy Aug 29, 2024
3281c3d
Fix files names
gfenoy Aug 29, 2024
7b7ed5d
Delete extensions/job_management/standard/20-044.xml.abort
gfenoy Aug 29, 2024
160869a
Remove the quotation and replace Recommendation by Permission
gfenoy Aug 29, 2024
c127cb7
Update statusCode.yaml to use created as mentioned in #419
gfenoy Aug 29, 2024
c40f9f3
Use in place of in statusInfo
gfenoy Sep 2, 2024
de992c9
Typo in security section
gfenoy Sep 2, 2024
18bad72
Fix Abstract test conformance class
gfenoy Sep 5, 2024
5469b35
Use PATCH method rather PUT to update the JOB
gfenoy Sep 5, 2024
dfe6472
Fix typo in front_material nad add reference to the exception when st…
gfenoy Sep 5, 2024
a3ed0c0
Fix typo in sequence diagram
gfenoy Sep 5, 2024
a36f218
reference to the update exception job locked
gfenoy Sep 5, 2024
16c0fda
Update after hearing from the GDC SWG meeting
gfenoy Sep 10, 2024
33677d8
Typo in processes-core/statusInfo.yaml in required fields
gfenoy Sep 12, 2024
ad78f24
Merge branch 'opengeospatial:master' into proposal-part4-initial
gfenoy Sep 12, 2024
df465c5
Take into acount the remarks made by @fmignault
gfenoy Sep 24, 2024
dacc370
Add @fmignault as author of the draft
gfenoy Sep 24, 2024
60c04ab
Fix typo and add paths to the clause_6
gfenoy Sep 24, 2024
166d805
Fix typos
gfenoy Sep 24, 2024
d12e0a6
Update extensions/job_management/standard/sections/annex_bibliography…
gfenoy Sep 24, 2024
3737503
Update part 1 schema to remove the limitation for the type of job
gfenoy Sep 24, 2024
80cbc5d
Add unsupported-schema requirement for job-management/create
gfenoy Sep 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
.DS_Store
.idea
.vscode
*.pdf
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,5 +20,5 @@ All questions regarding this submission should be directed to the editors or the
| Panagiotis (Peter) A. Vretanos _(editor)_ | CubeWerx Inc.
| Gérald Fenoy _(editor)_ | GeoLabs
| Pedro Gonçalves | Terradue Srl.
| Francis Charette Migneault | Centre de Recherche en Informatique de Montréal (CRIM)
|===

14 changes: 14 additions & 0 deletions extensions/job_management/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Standard template

This folder contains the content for the OGC API - Processes - Part 4: Job Management (JM).

This extension provides the ability to create new jobs without starting its execution. Meaning that you can prepare a job and execute it later. This extension also provides ways to track history associated with the execution, starting with the job definition.

The repo is organized as follows:

* standard - the main standard document content
- organized in multiple sections and directories (recommendations, requirements, etc.)
* xml - normative XML/XSD components specified by the standard
* examples - JSON and XML examples

The schemas associated with this extension are stored from the root directory in `openapi/*{api,parameters,path,responses,schemas}*/processes-job-management`.
1 change: 1 addition & 0 deletions extensions/job_management/examples/json/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add JSON examples to this directory. If not used, delete the directory.
1 change: 1 addition & 0 deletions extensions/job_management/examples/xml/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add XML examples to this directory. If not used, delete the directory.
56 changes: 56 additions & 0 deletions extensions/job_management/standard/24-051.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
= OGC API - Processes - Part 4: Job Management
:doctype: standard
:docsubtype: implementation
:docnumber: 24-051
:status: draft
:copyright-year: 2019
:edition: 1.0
:language: en
:published-date: yyyy-mm-dd
:received-date: yyyy-mm-dd
:issued-date: yyyy-mm-dd
:external-id: http://www.opengis.net/doc/IS/ogcapi-processes-4/1.0
:keywords: process, instance, spatial, data, openapi, job, create, update, delete, add, remove, REST, PATCH, POST, DELETE
:submitting-organizations: Geolabs; Terradue Srl.; Computer Research Institute of Montréal (CRIM).
:editor: Gérald Fenoy
:mn-document-class: ogc
:mn-output-extensions: xml,html,doc,pdf

////
Make sure to complete each included document
////
include::sections/clause_0_front_material.adoc[]

include::sections/clause_1_scope.adoc[]

include::sections/clause_2_conformance.adoc[]

include::sections/clause_3_references.adoc[]

include::sections/clause_4_terms_and_definitions.adoc[]

include::sections/clause_5_conventions.adoc[]

include::sections/clause_6_job_management.adoc[]

include::sections/clause_7_ogcapi-processes.adoc[]

include::sections/clause_8_openeo.adoc[]

include::sections/clause_9_provenance.adoc[]

include::sections/clause_10_oas.adoc[]

include::sections/clause_11_media_types.adoc[]

include::sections/clause_12_security_considerations.adoc[]

include::sections/annex_ats.adoc[]

////
Revision History should be the last annex before the Bibliography
Bibliography should be the last annex
////
include::sections/annex_history.adoc[]

include::sections/annex_bibliography.adoc[]
1 change: 1 addition & 0 deletions extensions/job_management/standard/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This folder contains the text for part 4 of the OGC API Processes standard.
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
[[ats_jm]]
[conformance_class]
.Job Management
====
[%metadata]
identifier:: http://www.opengis.net/spec/ogcapi-processes-4/1.0/conf/job-management
subject:: <<rc_job-management,http://www.opengis.net/spec/ogcapi-processes-4/1.0/conf/job-management>>
classification:: Target Type:Web API
conformance-test:: /conf/dru/deploy/post-op
====

==== Create operation

include::jm/create/ATS_post-op.adoc[]
27 changes: 27 additions & 0 deletions extensions/job_management/standard/abstract_tests/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
This folder contains the Abstract Test Suite.

Each file describes a single test. The naming convention for these files is:

"cc/TESTn.adoc" where "cc" corresponds to the identifier for the requirements class and "n" corresponds to the test number. Numbers should have preceeding zeros appropriate for the total number of tests in the conformance class (e.g., the first test could be TEST001 if less than 1000 tests are anticipated).

The test is expressed according to this pattern:

````
===== Test case title

(( additional discussion, if needed ))

====== a) Test Purpose:
(( description ))

====== b) Pre-conditions:
* (( list all preconditions ))

====== c) Test Method:
* (( steps to execute and assertions to test ))

====== d) References:
* <<req_cc_req,Requirement /req/cc/req>>
````

NOTE: for each test, there must be one or more requirements in the "requirements" folder.
15 changes: 15 additions & 0 deletions extensions/job_management/standard/abstract_tests/cc/TEST001.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
===== Test case title

(( additional discussion, if needed ))

====== a) Test Purpose:
(( description ))

====== b) Pre-conditions:
* (( list all preconditions ))

====== c) Test Method:
* (( steps to execute and assertions to test ))

====== d) References:
* <<req_cc_req,Requirement /req/cc/req>>
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
[[ats_jm_deploy_post-op]]

[abstract_test]
====
[%metadata]
identifier:: /conf/jm/create/post-op
target:: <<req_job-management_create_post-op,/req/job-management/create/post-op>>
test-purpose:: Validate that the server support HTTP POST operation at the path /jobs/
test-method::
+
--
1. Construct a path for the {root}/jobs path.

2. Issue an HTTP POST request for each path.

3. Validate that the response header does not contain `405 Method not allowed`
--
====

5 changes: 5 additions & 0 deletions extensions/job_management/standard/figures/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Figures go here.

Each figure is a separate file with the naming convention:

"PTm_FIGn.xxx" where "m" is a number with leading zeroes appropriate for the total number of parts, "n" is a number with leading zeroes appropriate for the total number of figures and "xxx" is the appropriate extension for the file type.
9 changes: 9 additions & 0 deletions extensions/job_management/standard/images/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
Image files for graphics go here.

Each graphic is a separate file with the naming convention:

"FIGn.xxx" where "n" is a number with leading zeroes appropriate for the total number of figures and "xxx" is the appropriate extension for the file type.

or, for other graphics not associated with figures:

"GRPn.xxx" where "n" is a sequential number with leading zeroes appropriate for the total number of graphics and "xxx" is the appropriate extension for the file type.
2 changes: 2 additions & 0 deletions extensions/job_management/standard/recommendations/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
OGC API - Processes - Part 4: Job Management recommendations.

Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_job-management_additional-status-codes]]
[permission]
====
[%metadata]
label:: /per/job-management/additional-status-codes
part:: Servers MAY support other HTTP protocol capabilities. Therefore, the server may return other status codes than those listed in <<status_codes>>.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_job-management_create_body]]
[permission]
====
[%metadata]
label:: /per/job-management/create/body
part:: A server MAY support any encoding in the body of a HTTP POST operation.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_job-management_create_content-schema]]
[permission]
====
[%metadata]
label:: /per/job-management/create/content-schema
part:: The `Content-Schema` header MAY be an URI to a JSON or OpenAPI 3.0 Schema document that describes the structure of the request body.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[[rec_job-management_create_body-ogcapi-processes]]
[recommendation]
====
[%metadata]
label:: /rec/job-management/create/body-ogcapi-processes
part:: If the job can be encoded as an <<rc_ogcapi-processes,OGC API - Processes - Workflow Execute Request>>, implementation SHOULD consider supporting the <<rc_ogcapi-processes,OGC API - Processes - Workflow Execute Request>> encoding.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Content-Schema: https://schemas.opengis.net/ogcapi/processes/part3/1.0/openapi/schemas/execute-workflows.yaml reference for this encoding (i.e.: https://github.com/opengeospatial/ogcapi-processes/blob/72c8a8e90de37d8e7f1ac246f23797246c872ce5/openapi/schemas/processes-workflows/execute-workflows.yaml) should be recommended to align with per_job-management_create_content-schema (see other comment about duplicate mentions).

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[[rec_job-management_create_body-openeo]]
[recommendation]
====
[%metadata]
label:: /rec/job-management/create/body-openeo

part:: If the job can be encoded as an <<rc_openeo,OpenEO graph>>, implementation SHOULD consider supporting the <<rc_openeo,OpenEO graph>> encoding.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As for the OAP Workflow Execute Request case, this should probably recommend an established Content-Schema for openEO (see other comment about duplicate mentions).

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_job-management_update_body]]
[permission]
====
[%metadata]
label:: /per/job-management/update/body
part:: A server MAY support any encoding in the body of a HTTP PATCH operation.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_job-management_update_content-schema]]
[permission]
====
[%metadata]
label:: /per/job-management/update/content-schema
part:: The `Content-Schema` header MAY be used to indicate the schema of a request body for updating a job.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[[rec_job-management_update-ogcapi-processes]]
[recommendation]
====
[%metadata]
label:: /rec/job-management/update/body-ogcapi-processes

part:: If a job can be created from an <<rc_ogcapi-processes,OGC API - Processes - Workflow Execute Request>>, implementations SHOULD consider supporting the <<rc_ogcapi-processes,OGC API - Processes - Workflow Execute Request>> encoding.

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[[rec_job-management_update-openeo]]
[recommendation]
====
[%metadata]
label:: /rec/job-management/update/body-openeo

part:: If a job can be created from an <<rc_openeo,OpenEO Process Graph>>, implementations SHOULD consider supporting the <<rc_openeo,OpenEO Process Graph>> encoding.

====
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_ogcapi-processes_create_content-schema]]
[permission]
====
[%metadata]
label:: /per/ogcapi-processes/create/content-schema
part:: The `Content-Schema` header MAY be pointing to the OpenAPI 3.0 schema https://github.com/opengeospatial/ogcapi-processes/blob/master/openapi/schemas/processes-workflows/execute-workflows.yaml[execute-workflows.yaml].
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally, all endpoints should use the URI to the expected location when published
(i.e: https://schemas.opengis.net/ogcapi/processes/part3/1.0/openapi/schemas/execute-workflows.yaml), so that they can refer to an eventual static reference.

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_ogcapi-processes_update_content-schema]]
[permission]
====
[%metadata]
label:: /per/ogcapi-processes/update/content-schema
part:: The `Content-Schema` header MAY be pointing to the OpenAPI 3.0 schema https://github.com/opengeospatial/ogcapi-processes/blob/master/openapi/schemas/processes-workflows/execute-workflows.yaml[execute-workflows.yaml].
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_openeo_create_content-schema]]
[permission]
====
[%metadata]
label:: /per/openeo/create/content-schema
part:: The `Content-Schema` header MAY be pointing to https://raw.githubusercontent.com/Open-EO/openeo-processes/master/meta/subtype-schemas.json#/definitions/process-graph[OpenEO Process Graph schema].
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly, we should look for an "official" endpoint where a static/versioned document can be found.

@m-mohr does one exist ?

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[per_openeo_update_content-schema]]
[permission]
====
[%metadata]
label:: /per/openeo/update/content-schema
part:: The `Content-Schema` header MAY be pointing to https://raw.githubusercontent.com/Open-EO/openeo-processes/master/meta/subtype-schemas.json#/definitions/process-graph[OpenEO Process Graph schema].
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[req_job-management_create_body]]
[requirement]
====
[%metadata]
label:: /req/job-management/create/body
part:: The body of the POST request SHALL be in JSON format.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[req_job-management_create_content-type]]
[requirement]
====
[%metadata]
label:: /req/job-management/create/content-type
part:: The `Content-Type` https://tools.ietf.org/html/rfc2616#section-14.17[header] SHALL be used to declare the media type of the request.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[req_job-management_create_post-op]]
[requirement]
====
[%metadata]
label:: /req/job-management/create/post-op
part:: The server SHALL support the HTTP POST operation at the path `/jobs`.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[[req_job-management_create_response-body]]
[requirement]
====
[%metadata]
label:: /req/job-management/create/response-body
part:: The response SHALL include a body that contains a status information of the created job that conforms to the https://schemas.opengis.net/ogcapi/processes/part1/1.0/openapi/schemas/statusInfo.yaml[statusInfo.yaml] schema.
part:: The response SHALL contain a `created` status code and the `jobId` property that contains the job identifier.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[req_job-management_create_response-jobid]]
[requirement]
====
[%metadata]
label:: /req/job-management/create/response-jobid
part:: If the operation completes, the server SHALL generate a job identifier (i.e. `{jobId}`) for the created job.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[[req_job-management_create_response_success]]
[requirement]
====
[%metadata]
label:: /req/job-management/create/response-success
part:: A successful execution of the operation SHALL be reported as a response with a HTTP status code `201`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that some servers could want to handle the case evoked in the meeting, (ie: job "created" and deleted right away to do a "sync" execution), a 200 OK or 202 Accepted, might be considered. This could also be used by async execution, to indicate that the job was accepted, but not yet "running".

part:: A response with HTTP status code `201` SHALL include a `Location` header with the URI of the deployed processes (path: `/jobs/{jobId}`).
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[[req_job-management_create_unsupported-media-type]]
[requirement]
====
[%metadata]
label:: /req/job-management/create/unsupported-media-type

part:: If the server does not support the Content-Type header associated with the request body, the code of the response SHALL be `415 Unsupported Media Type`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it be the same if a specific Content-Schema is not supported (given most definitions will share the same Content-Type: application/json)?

Otherwise, I suggest: 422 Unprocessable Content

part:: The content of that response SHALL be based upon the OpenAPI
3.0 schema https://raw.githubusercontent.com/opengeospatial/ogcapi-processes/master/core/openapi/schemas/exception.yaml[exception.yaml].
part:: The `type` of the exception SHALL be “http://www.opengis.net/def/exceptions/ogcapi-processes-4/1.0/unsupported-media-type”.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[[req_job-management_create_unsupported-schema]]
[requirement]
====
[%metadata]
label:: /req/job-management/create/unsupported-schema

part:: If the server does not support the Content-Schema header associated with the request body, the code of the response SHALL be `422 Unprocessable Content`.
part:: The content of that response SHALL be based upon the OpenAPI
3.0 schema https://raw.githubusercontent.com/opengeospatial/ogcapi-processes/master/core/openapi/schemas/exception.yaml[exception.yaml].
part:: The `type` of the exception SHALL be “http://www.opengis.net/def/exceptions/ogcapi-processes-4/1.0/unsupported-schema”.
====
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[[req_job-management_definition_get-op]]
[requirement]
====
[%metadata]
label:: /req/job-management/definition/get-op
part:: For every jobs (using method: POST on path: /jobs/{jobId}), the server SHALL support the HTTP GET operation at the path `/jobs/{jobId}/definition`.
part:: The parameter `jobId` is each `id` property in the job-list response (JSONPath: `$.jobs[*].id`).

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[req_job-management_definition_response-body]]
[requirement]
====
[%metadata]
label:: /req/job-management/definition/response-body
part:: A response with HTTP status code `200` SHALL include a body that contains the request body used to create or update the job.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe needs refining of the wording. It is not really clear what is the intent here?

Is it that, given a POST /jobs {something}, the {something} should be included in the GET /jobs/{jobID} response?

====
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[[req_job-management_definition_response-success]]
[requirement]
====
[%metadata]
label:: /req/job-management/definition/response-success
part:: A successful access to the resource SHALL be reported as a response with a HTTP status code `200`.
====
Loading
Loading