Proposal OGC API - Processs - Part4: Job Management #437

gfenoy · 2024-09-23T15:29:51Z

Following the discussions during today's GDC / OGC API—Processes SWG meeting, I created this PR.

It contains a proposal for an additional part to the OGC API - Processes family: "OGC API - Processes - Part 4: Job Management" extension.

This extension was initially discussed here:

The document identifier 24-051 was registered.

Use a different OpenEO JSON Schema

Add the job-locked exception requirement

…arting an execution

fmigneault · 2024-09-23T16:21:47Z

extensions/job_management/.DS_Store

remove and add .DS_Store to .gitignore to not worry about it anymore

Done in df465c5.

fmigneault · 2024-09-23T16:24:05Z

extensions/job_management/README.md

+The repo is organized as follows:
+
+* standard - the main standard document content
+  - organized in multiple sections and directories (openapi, requirements, etc.)


this is outdated since they are now at the root openapi/schemas/processes-job-management

Should be fixed in df465c5

fmigneault · 2024-09-23T16:31:59Z

extensions/job_management/standard/abstract_tests/jm/create/ATS_post-op.adoc

+1. Construct a path for each "rel=http://www.opengis.net/def/rel/ogc/1.0/job-list" link on the landing page as well as for the {root}/jobs path.
+
+2. Issue an HTTP POST request for each path.
+
+3. Validate that the response header does not contain `405 Method not allowed`


I believe an OAP Core could report job-list on its landing page while only supporting the "normal" /processes/{jobId}/execution. This AST should only validate that POST /jobs is supported, since the other endpoint does not "need" to support the additional capabilities of this extension, and POST /processes/{jobId}/execution should already be handled by the AST of core.

I think you're right, so I applied the modification and removed the reference to the job-list.

I think that the /job url relation type is still job-list on the landing page. I thought the ATS would check for the job-list relation type to know where to send the POST request (and used the {root}/jobs following other tests from part 1 and part 2).

fmigneault · 2024-09-23T16:32:25Z

extensions/job_management/standard/recommendations/README.md

@@ -0,0 +1,2 @@
+OGC API - Processes - Part 2: Transactions recommendations.


wrong title

Should be fixed in df465c5

fmigneault · 2024-09-23T16:33:11Z

extensions/job_management/standard/24-051.adoc

+:received-date: yyyy-mm-dd
+:issued-date: yyyy-mm-dd
+:external-id: http://www.opengis.net/doc/IS/ogcapi-processes-4/1.0
+:keywords: process, collection, instance, spatial, data, openapi, transactions, insert, update, delete, add, remove, deploy, undeploy, REST, PUT, POST, DELETE


"transactions" does not seem relevant. Add "jobs" instead? not sure about "collection" either.

fmigneault · 2024-09-23T19:23:47Z

openapi/schemas/processes-job-management/inputs.yaml

+type: object
+additionalProperties:
+  $ref: "input.yaml"


I suggest using an alternate representation:

inputs: type: object additionalProperties: $ref: "input.yaml" outputs: type: object additionalProperties: $ref: "output.yaml"

Reasons:

Although "outputs" are shown, those represent the requested outputs (i.e.: transmission mode, format, or any other alternate content negotiation parameters) submitted during the job creation request, in order to eventually produce the desired results. Often, the requested outputs depend on whichever inputs were submitted. Therefore, viewing them separately on different endpoints is not really convenient or useful.

The /jobs/{jobId}/outputs endpoint can easily be confused with /jobs/{jobId}/results. The "request outputs" in this case are "parameter descriptions of eventual outputs", which are provided upstream of the execution workflow. In a way, those are parametrization "inputs" of the processing pipeline.

Because OGC API - Processes core defines specific response combinations and requirements for /jobs/{jobId}/results, the /jobs/{jobId}/outputs is a convenient and natural endpoint name that an API can use to provide alternate response handling and representations conflicting with OGC API - Processes definitions. CRIM's implementation does exactly that. I would appreciate keeping that option available.

As a matter of fact, older OGC API - Processes implementations (start of ADES/EMS days) actually used /jobs/{jobId}/outputs instead of /jobs/{jobId}/results. Adding /jobs/{jobId}/outputs would break those implementations.

Having inputs and outputs nested under those fields (rather than at the root) allows providing even further contents that could be relevant along the inputs/outputs. For example, additional links, metadata or definitions to describe those parameters.

fmigneault · 2024-09-23T20:00:57Z

openapi/schemas/processes-job-management/outputs.yaml

see other comment about inputs

fmigneault · 2024-09-23T20:01:39Z

openapi/schemas/processes-job-management/openeo-process-graph.yaml

Should we use a $ref to openEO's definition, to avoid maintaining duplicate definitions?

fmigneault · 2024-09-23T20:05:41Z

openapi/schemas/processes-job-management/processes-type.yaml

+enum:
+  - process
+  - openeo


Should this be made a simple string with examples?
Do we want to create the same situation as statusCode needing an override because of the new created status?

If the long term use is to have job management available for an OGC API, maybe it would be better to define a Requirements class that say for openEO, type: openeo MUST be used, and process for OGC API - Processes. A "long" Coverage processing could then easily define their own requirement class with type: coverage, without causing invalid JSON schema definitions.

fmigneault · 2024-09-23T20:09:06Z

openapi/schemas/processes-job-management/statusInfo.yaml

+   id:
+      type: string
+   processID:
+      type: string


The requirements need to be revised. They used the alternate name jobID when referring to the GET /jobs responses.

Similarly, process was mentioned during the meeting.
I'm not sure if processID remains relevant however, because process would be the URI, not just the processID from GET /processes/{processID}.

extensions/job_management/standard/sections/annex_bibliography.adoc

….adoc Co-authored-by: Michael R. Crusoe <[email protected]>

fmigneault · 2024-09-24T21:38:02Z

@gfenoy
The README should include the URL for the eventual HTML/PDF draft location.

gfenoy added 19 commits August 29, 2024 16:09

Boostrap part 4 (with wrong document number)

717321a

Fix typos

ca03fbd

Fix typos

6b7acb3

Fix typos

4aaf777

Fix typo

8a6d469

Fix files names

3281c3d

Delete extensions/job_management/standard/20-044.xml.abort

7b7ed5d

Remove the quotation and replace Recommendation by Permission

160869a

Update statusCode.yaml to use created as mentioned in opengeospatial#419

c127cb7

Use in place of in statusInfo

c40f9f3

Use a different OpenEO JSON Schema

Typo in security section

de992c9

Fix Abstract test conformance class

18bad72

Use PATCH method rather PUT to update the JOB

5469b35

Add the job-locked exception requirement

Fix typo in front_material nad add reference to the exception when st…

dfe6472

…arting an execution

Fix typo in sequence diagram

a3ed0c0

reference to the update exception job locked

a36f218

Update after hearing from the GDC SWG meeting

16c0fda

Typo in processes-core/statusInfo.yaml in required fields

33677d8

Merge branch 'opengeospatial:master' into proposal-part4-initial

ad78f24

gfenoy added the Part 4 (Job Management) OGC API - Processes - Part 4: Job Management label Sep 23, 2024

fmigneault reviewed Sep 23, 2024

View reviewed changes

fmigneault mentioned this pull request Sep 23, 2024

Central job submission endpoint (also: POST /processes/{processID}/execution is not RESTful and is confusing for workflows) #419

Open

gfenoy added 2 commits September 24, 2024 14:10

Take into acount the remarks made by @fmignault

df465c5

Add @fmignault as author of the draft

dacc370

jerstlouis mentioned this pull request Sep 24, 2024

Support for Async opengeospatial/ogcapi-common#231

Open

mr-c reviewed Sep 24, 2024

View reviewed changes

extensions/job_management/standard/sections/annex_bibliography.adoc Outdated Show resolved Hide resolved

gfenoy and others added 4 commits September 24, 2024 15:56

Fix typo and add paths to the clause_6

60c04ab

Fix typos

166d805

Update extensions/job_management/standard/sections/annex_bibliography…

d12e0a6

….adoc Co-authored-by: Michael R. Crusoe <[email protected]>

Update part 1 schema to remove the limitation for the type of job

3737503

Add unsupported-schema requirement for job-management/create

80cbc5d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proposal OGC API - Processs - Part4: Job Management #437

Proposal OGC API - Processs - Part4: Job Management #437

gfenoy commented Sep 23, 2024

fmigneault Sep 23, 2024

gfenoy Sep 24, 2024 •

edited

Loading

fmigneault Sep 23, 2024

gfenoy Sep 24, 2024

fmigneault Sep 23, 2024

gfenoy Sep 24, 2024

fmigneault Sep 23, 2024

gfenoy Sep 24, 2024

fmigneault Sep 23, 2024

fmigneault Sep 23, 2024

fmigneault Sep 23, 2024

fmigneault Sep 23, 2024

fmigneault Sep 23, 2024

fmigneault Sep 23, 2024

fmigneault commented Sep 24, 2024

		@@ -0,0 +1,2 @@
		OGC API - Processes - Part 2: Transactions recommendations.

Proposal OGC API - Processs - Part4: Job Management #437

Are you sure you want to change the base?

Proposal OGC API - Processs - Part4: Job Management #437

Conversation

gfenoy commented Sep 23, 2024

Choose a reason for hiding this comment

gfenoy Sep 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Reasons:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fmigneault commented Sep 24, 2024

gfenoy Sep 24, 2024 •

edited

Loading