Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Other process encodings? #325

Open
m-mohr opened this issue May 19, 2023 · 65 comments
Open

Other process encodings? #325

m-mohr opened this issue May 19, 2023 · 65 comments
Assignees

Comments

@m-mohr
Copy link

m-mohr commented May 19, 2023

In Part 1 I found the following sentence:

The Core does not mandate the use of any specific process description to specify the interface of a process. Instead this standard defines and recommends the use of the following conformance class: OGC Process Description

This means I'd expect that I could for example use the openEO process encoding in OAP.

Requirement 11 in "Core" says:

The content of that response SHALL be based upon the OpenAPI 3.0 schema processList.yaml.

The processList.yaml refers to the processSummary.yaml though which has a very specific encoding in mind:
http://schemas.opengis.net/ogcapi/processes/part1/1.0/openapi/schemas/processSummary.yaml

Thus, I think there's a conflict in the specification that should be resolved. Are we allowed to return e.g. openEO processes in /processses?

@pvretano
Copy link
Contributor

@m-mohr no. As you point out, the /processes response must conform to the processList schema which in turn references processSummary.yaml. /processes is just the list of available processes with some summary description of each process. Looking at it now, we should probably move jobControlOptions and transmissionMode to process.yaml since they are specific to the OGC process description vocabulary.

You can however, respond with an openEO process description at the /processes/{processId} endpoint. That is what the requirement from core that you cite is trying to say using the terminology "interface of a process". The interface of a process is at /processes/{processId}.

Not sure if the specification mentions this explicitly but in the links section of the process summary you can include links to the process description in any number of process description languages or vocabularies so you could include a link to an openEO description from there.

@jerstlouis
Copy link
Member

jerstlouis commented May 19, 2023

@pvretano Regarding what is returned in the process list at /processes (which is the summary of each process), would this not depend on the negotiated media type? If the implementation does not specify conformance to "OGC process description" but specifies conformance to a new "OpenAPI process description" for example, couldn't it return an OpenAPI document?

Clients can already negotiate an HTML representation of /processes as another example.

The process list uses the "self" relation type inside the summary to link to the full object for each individual process, so it would make sense that the representation of the summary and that of the full process description are consistent based on a particular negotiated representation.

@m-mohr
Copy link
Author

m-mohr commented May 19, 2023

Thanks. I guess the content negotiation doesn't help if the media type for both is application/json (openEO processes and OGC process descriptions are both JSON).

@jerstlouis
Copy link
Member

jerstlouis commented May 19, 2023

@m-mohr In Part 3, there is currently a suggestion in 14-Media Types to define specific media types.
I suggest e.g., application/ogcexec+json for the execution requests. There should also be specific media types for OpenEO.
And possibly we could do the same for the process description. However, sticking to application/json for the default OGC process description would make sense. I think it is kind of expected that most processes implementation would support it.

This is an issue that pops up everywhere application/json is used as a content type (just saying JSON, XML or PBF says next to nothing about the content type), and is related to the negotiation by profile issue which would be an alternative in differentiating them.

Possibly we should always define new specific media types for new JSON schemas.

@m-mohr
Copy link
Author

m-mohr commented May 19, 2023

The issue in /processes is also that it has a JSON "wrapper" (links, processes - which is actually the same in OAP and openEO) and only the individual processes included in the processes property differ. I assume application/ogcexec+json describes the process itself, not necessarily the full response of /processes?!

@jerstlouis
Copy link
Member

jerstlouis commented May 19, 2023

@m-mohr If the component processes of OpenEO could be defined as regular OGC Process Description that would really be ideal. The OpenEO process graph is the equivalent of the process execution that is extended in Part 3 to be able to nest processes.

I assume application/ogcexec+json describes the process itself, not necessarily the full response of /processes?

Correct, sorry for the confusion -- I edited my message for clarity. It's for making a request to execute the process, not describe it. Not a response, but the payload from the client to execute the process.

In a sense it's a distinction between a workflow description (process execution request or OpenEO process graph) vs. a process description (single black box process).

It is possible that a single process also happens to be defined by a workflow (process execution request or OpenEO process graph), in which case that could be made available as a visible "execution unit" (the definition of that process, not its description). That is related to Part 2: Deploy, Replace, Update and the Deployed Workflows requirements class of Part 3. The description is only the inputs and outputs of the process; the definition's execution unit is what the process actually does expressed in some way (Docker container, execution request, OpenEO process graph, CWL, Python script...).

@fmigneault
Copy link
Contributor

However, sticking to application/json for the default OGC process description would make sense. I think it is kind of expected that most processes implementation would support it.

I agree about the default application/json response format.

Also, I don't understand why it is not mandatory to ensure interoperability. As mentioned, the definition is what changes a lot between implementations and extensions, but the core description should be somewhat consistent regardless.
At least a minimal set of critical components such as the inputs and outputs definition should be standardized, since a standardized representation for POST'ing the execution content is needed anyway.

Note that even if application/json was returned for multiple encodings, we could provide (or mandate to provide) a $schema or a JSON-LD @context field refering to which one is being represented. There is no need to introduced new Media-Type each time, as parsing the contents is effectively always valid in JSON.

@pvretano
Copy link
Contributor

21-AUG-2023: There is a difference between what you get at /processes and /processes/{processId}. What you get at /processes is a list of available processes. What you get at /processes/{processId} is an actual process description.

The schema for the "list of processes" is fixed by the specification and is defined in processSummary.yaml. All implementations of OAPIP regardless of how they describe their processes must use the same summary schema for the list of processes. You can negotiate a different output format (e.g. XML) at /processes but the specification only defines HTML and JSON output right now.

The story is different at /processes/{processesId}. At this endpoint you are request a detailed processes description that includes input definitions, output definitions, etc. At this endpoint the specification DOES NOT mandate a particular schema. Rather it includes a conformance class for an OGC process description but other process descriptions such as openEO are possible. We do need do define a media type for an OGC process description so that content negotiation can be used to distinguish an OGC process description from an openEO process description for example.

Assigning to @pvretano to create a PR to add a media type for an OGC processes description ...

@pvretano pvretano self-assigned this Aug 21, 2023
@m-mohr
Copy link
Author

m-mohr commented Aug 21, 2023

Thanks. Would it be an option that it's allowed to mix different process types for /processes?
So that the processes array can contain e.g. openEO and OGC API Processes descriptions?
We are currently experimenting with this in the GDC API and I have a client that supports it already.
It would be a way to support openEO and OAP through the same API without requiring different requests with different Accept media types for the specifications.

@jerstlouis
Copy link
Member

jerstlouis commented Sep 12, 2023

Even if we have do have separate media types for openEO vs. OGC process descriptions, with the openEO description friendly to existing openEO clients, ideally I think it should also be possible for openEO backends to offer an OGC process description for those openEO processes for clients implementing strictly OGC API - Processes. Still hoping we can validate the feasibility of this in T19-GDC (even if we don't have time to implement it).

@pvretano
Perhaps new media types for process description and/or execution requests is one thing that we could require in 2.0. It would be good for openEO to also have a different media type for both.

I think it would also make sense to explore having an OpenAPI process description, which we could consider including in 2.0 as a separate requirements class, perhaps in the sprint next week if we have time.

That is, /processes/{processId} for Accept: Application/vnd.oai.openapi+json;version=3.0 would return an OpenAPI definition tailored specifically to that single process, focusing on the content of the individual "inputs" in the schema for the POST request to /processes/{processId}/execution and the result (for the single output synchronous 200 response and/or for the async job results at .../results/{resultId}).

@fmigneault
Copy link
Contributor

I think it should also be possible for openEO backends to offer an OGC process description for those openEO processes for clients implementing strictly OGC API - Processes.

IMO, it is not "should", but "must". Otherwise, they are not really an interoperable OGC API - Processes implementation...

Interoperability is already barely accomplished with current implementations that should be using the same process description format. Adding alternatives will make it even more complicated than it already is. Nothing wrong in allowing additional media-types/encodings though, as long as the common one is fulfilled.

explore having an OpenAPI process description

I'm curious by what you have in mind? Isn't this already offered for inputs/outputs using this portion:
https://github.com/opengeospatial/ogcapi-processes/blob/master/openapi/schemas/processes-core/inputDescription.yaml#L17-L18

@jerstlouis
Copy link
Member

jerstlouis commented Sep 13, 2023

@fmigneault

IMO, it is not "should", but "must". Otherwise, they are not really an interoperable OGC API - Processes implementation...

Technically the OGC process description is not a mandatory requirement class, but I agree that this is a very strong should, and I hope it can be achieved.

There is on-going discussion about whether an implementtion of a GeoDataCube API supporting processing with an openEO backend should be fully aligned with OGC API - Processes, and I believe it should (including support for OGC Process Description), so that a generic OGC API - Processes / GeoDataCube API processing client can execute it, but this differs from the current published openEO specification. I will present on this topic and there will be a follow-on discussion on Monday the 25th at the Singapore meeting in the GeoDataCube SWG.

I'm curious by what you have in mind? Isn't this already offered for inputs/outputs using this portion:

This is a JSON Schema within an OGC process description, not an OpenAPI definition.
I am thinking of the process description itself being an OpenAPI definition, as can be found at /api, but in the case of /processes/{processId} (Accept: Application/vnd.oai.openapi+json;version=3.0), the included paths only concern the execution and response for that particular process, and the same schema is embedded directly within the POST payload JSON Schema for the /processes/{processId}/execution path (not using the OGC process description to describe it, but the OpenAPI way).

Essentially this would allow generic OpenAPI clients and developers to execute OGC API - Processes without knowing anything about OGC Process Descriptions or OGC API - Processes in general.

@pvretano
Copy link
Contributor

pvretano commented Sep 13, 2023

@fmigneault @jerstlouis when we first wrote WFS, for the sake of interoperability, we mandated GML. That turned out the be both a good thing (for interoperability) and a bad thing (because GML was a beast to implement). So, when the time came to design OGC API Features we decided, instead, not to mandate any particular format and let the client and the server negotiate a mutally agreeable format. To HELP interoperability we added conformance classes to OGC API Features for GeoJSON and GML.
The situation is similar here. I don't think we should MANDATE a specific process description language but rather let the client and the server negotiate a mutually agreeable format. However, as we did with Features, we included the OGC Process Description conformance class and RECOMMENDED that servers implement that for interoperability. I don't think we need to qualify the recomendation any further. The specification recomends it and that should be sufficient. In features we usually say "if [GeoJSON | GML] is suitable for your purposes ... blah, blah, blah".

@jerstlouis
Copy link
Member

jerstlouis commented Sep 13, 2023

@pvretano agreed, but where I hope we require it is as a dependency of the GeoDataCube API "Processing" conformance class (I added "OGC Process Description" to the processes-1 row in https://gitlab.ogc.org/ogc/T19-GDC/-/issues/25), which is sort of "profiling" OGC API standards to maximize interoperability (i.e., the chance of a successful client/server negotiation).

@pvretano
Copy link
Contributor

pvretano commented Sep 13, 2023

@jerstlouis agreed! It is perfectly legal for a referencing specification like GDC to say that an optional conformance class (OGC Process Description in this case) has to be mandatory in the GDC profile of processes.

@m-mohr
Copy link
Author

m-mohr commented Sep 15, 2023

Before we can require the OGC Process description we should make sure it's good enough to cater for most needs. I'm not sure whether we could encode openEO processes in OGC Process descriptions for example.

The issue with content negotiation is that you may have two JSON-based descriptions that don't have specific media types for them. And then you must also be able to convert from one encoding to the other, which may not be possible with some losses (see above).

@jerstlouis
Copy link
Member

jerstlouis commented Sep 15, 2023

@m-mohr

Before we can require the OGC Process description we should make sure it's good enough to cater for most needs

That is very reasonable.

I'm not sure whether we could encode openEO processes in OGC Process descriptions for example.

Can we perform the experiment and validate that? Would you have an example openEO process description that exercises most of the capabilities, and we can try to do the mapping?

If something is missing, there would be no better time than right now to try to address this with the Processes Code Sprint early next week validating Processes 1.1 or 2.0.

I really believe it is critical for interoperability to have this OGC Process Description support all use cases, including the openEO process descriptions.

@m-mohr
Copy link
Author

m-mohr commented Sep 15, 2023

I'd love to, but I'm on vacation until October so I can't do it before the sprint.

@jerstlouis
Copy link
Member

@m-mohr Enjoy your vacation :)

But if you have time to just point us to a good sample openEO process description between a Mai Tai and a Piña Colada I could give it a try next week :)

@m-mohr
Copy link
Author

m-mohr commented Sep 15, 2023

I can only point you to the official docs right now, so the process description schema at https://api.openeo.org/#tag/Process-Discovery/operation/list-processes and the processes at https://processes.openeo.org

@pvretano
Copy link
Contributor

pvretano commented Sep 15, 2023

I really believe it is critical for interoperability to have this OGC Process Description support all use cases, including the openEO process descriptions.

@jerstlouis if I understand what you are saying, I am not sure I agree.

Interoperability is not a function of the "OGC Process Description". The "OGC Process Description" is one way to describe a process. OpenEO is another as is CWL. For that matter, so is OpenAPI (I have been experimenting with posting OpenAPI descriptions of a process to my server).

What is required is that the API can accomodate each of these process description languages which it can via different conformance classes. The "OGC Process Description" conformance class already exists which means that a client can request the description (i.e. GET /process/{processId}) of a process as an "OGC Process Description". Assuming the necessary conformance classes existed, a client could then negotiate with the server to request the same description using OpenEO or CWL or OpenAPI in response to a GET /processes/{processId} request. Assuming, for example, that a server claimed support for all these process description languages a client (using Part 2) could deploy a process that is described using CWL and then another client could request a description of that deployed process using OpenEO. The only requirement is that mime types exists so that clients can negotiate a response in their desired process description language or format.

The same line of reasoning would apply to Part 2 where a process is described for the purpose of deployment. A server that claims to support multiple process description languages could then deploy a process described using "OGC Process Description" or OpenEO or CWL or ...

So I guess what I am saying in response to your comment and @m-mohr original comment is that it should not be a matter of mandating support for one process description language/format and then making sure that format can accomodate other process description languages/formats. Each process description language/format should be supported in its own right (via separate conformance classes) and the server should be responsible (internally) for crosswalking one to other as per a client's request.

How that I am writing this it occures to me that it may conflict with my previous agreement vis-a-vis GDC. Sorry about that but I have been thinking more about the situatiion and this comment reflects my current thinking. I could be complete wrong but I welcome response comments because this is an imporant interoperability point.

@jerstlouis
Copy link
Member

jerstlouis commented Sep 15, 2023

@pvretano Are we perhaps mixing up Process Description and Process Definition here?

By Process Description I am referring strictly to /processes/{processId} in terms of a client executing a process using Part 1 (potentially with some Part 3 extensions) and being able to find the relevant information (the inputs/oututs and their data types) in a consistent manner.

Although there is the notion that a Part 2 deployment can include a process description if the server can't figure out how to make up an OGC Process Description by itself, we are not talking about deployment at all here.

By interoperability, I mean any client would be able to only implement Part 1 with OGC Process Description, and would be able to execute any process, regardless of how it was defined (whether with CWL, openEO, Part 3, or anything else).

@jerstlouis
Copy link
Member

jerstlouis commented Sep 19, 2023

From a quick look at the first process from the EURAC GDC API end-point ( https://dev.openeo.eurac.edu/processes ), it seems to me that the basics of the openEO -> OGC process description mapping is quite straightforward.

The openEO "parameters" correspond to the OGC "inputs", and the openEO "returns" corresponds to the OGC "outputs".

The openEO "schema" corresponds to the OGC "schema" with the caveats related to the multiplicity (minOccurs / maxOccurs).

The "subtype": "raster-cube" could potentially correspond to a special "format" that we could define for gridded coverages, agnostic of any specific media type, like we do for feature collections. In the meantime the fact that an input or output is a gridded coverage / geo data cube is usually indicated by listing supported coverage media types such as "type": "string", "contentEncoding": "binary", "contentMediaType": "image/tiff; application=geotiff", or netCDF for additional dimensions.

@m-mohr
Copy link
Author

m-mohr commented Oct 2, 2023

I don't think it's that easy. For example, the returns and outputs to me are slightly different and in my conversion I had to make the OGC outputs openEO parameters (as there's a choice to be made). A couple of thinks can't be translated at all, I think. For example the process_graph and examples properties (but they are strictly descriptive and not required). Anyway, it looks like it's a lossy migration process.

@fmigneault
Copy link
Contributor

fmigneault commented Oct 2, 2023

@jerstlouis @m-mohr
I would like to remind that OGC Process Description can include an executionUnit part that can contain the exact representation of openEO if anything is missing on the OGC side (see Part 2 DRU example: https://github.com/opengeospatial/ogcapi-processes/blob/8c41db3fbc804450c89d94b704e1d241105272ae/openapi/schemas/processes-dru/ogcapppkg.yaml). Also, even Part 1: Core technically allows additional parameters in https://github.com/opengeospatial/ogcapi-processes/blob/8c41db3fbc804450c89d94b704e1d241105272ae/openapi/schemas/processes-core/process.yaml. Therefore, an openEO description would be compliant with OGC Process Description if it adds the missing parts, even if it provides more parameters.

The core requirements of an OGC Process Description would be to port what is compatible, namely, the critical inputs and outputs sections. Since openEO and OGC both employ JSON schema parameters to define I/O, I do not see were the problem is. Anything additional required by openEO for performing the actual "execution" can refer directly its it's executionUnit representation, or its other additional parameters, after mapping inputs -> parameters and outputs -> returns.

For the process_graph part, that would need to be ported into Part 3: Workflows, or Part 2: DRU with some executionUnit strategy that knows how to handle a Workflow such as CWL, or directly the openEO graph for that matter.

@jerstlouis
Copy link
Member

jerstlouis commented Oct 3, 2023

@m-mohr Right, as @fmigneault points out, for the process_graph of a particular process, that should be possible to retrieve (if the service does wish to expose the inner working of the process) as a link to a definition of the process (the executionUnit of an application package). We do support that already in our implementation for ad-hoc execution e.g., https://maps.gnosis.earth/ogcapi/collections/temp-exec-48D2606E/workflow , but could also have that for deployed processes ( e.g., /processes/{processId}/definition or /processes/{processId}/executionUnit).

The examples is something that could already be added to the process description without breaking anything, and it could be something that gets specified in a later version to standardize the approach.

I had to make the OGC outputs openEO parameters (as there's a choice to be made)

Could you please provide more details about this?
If this is about providing as an input a "destination" where things get saved / end up, I think as you say this is a choice that can be made for an individual process, but I think either way can work with Processes - Part 1.

To support the Part 3 approach of collection input / output, where a "collection / GeoDataCube" is a first class object, results should really always and only be outputs with no "storage" specified for them... things just flow out of the process and are requested / processing triggered using OGC API data access calls.

@m-mohr
Copy link
Author

m-mohr commented Oct 3, 2023

While we might be able to translate it, why should we do it? We loose all the openEO clients and get OGC API - Processes clients, which honestly, I haven't seen a good example of. Why not just allow different process flavours in /processes via conformance classes as we pretty much do in GDC right now?

Could you please provide more details about this?

For the output you need to specify what format you want. This needs to be a parameter in openEO as for return values it just describes what you get, there's no choice. Everything you need to choose from must be a parameter. As such I also think your return values are a bit flawed as it effectively is an input that the user may need to provide.

@jerstlouis
Copy link
Member

Pushing the same process definition onto another server implementation will not be portable unless they provided exactly the same logic.

There might be a different set of supported input / output formats from the deployed process (that would be reflected in the deployed process description). Executing the process might also result in slight differences in the outputs as a result of being converted by different tools. However, that same execution unit could be still deployed to those different implementations and work as intended, so I would not call that not portable.

I believe most of that machinery should be defined by respective processes themselves.

I am of the opinion that this should be up to the implementation / profiles / deployment to decide.
Certainly with the Part 3 workflows the idea is that workflows can be defined in a manner agnostic of formats.
One advantage of this is that a whole chain of processes could be happening in an optimal internal format of the implementation that never gets exposed, or kept in GPU memory for heavy CUDA parallel processing, without the need to explicitly encode / load things in a particular format until the very final step where the data is returned to a client. Not explicitly using processes for loading / converting / writing specific format can avoid superfluous steps in between the actuall calculations within a workflow.

As you might have understood from all our previous exchanges, I am more often working with "heavy processing" use cases, where async is not only preferable, but a must.

Those are the cases I was mentioning for which specifying an output format in the execution request beforehand makes sense, but they could be presented as different outputs altogether, or even as different processes.

@fmigneault
Copy link
Contributor

There might be a different set of supported input / output formats from the deployed process (that would be reflected in the deployed process description). Executing the process might also result in slight differences in the outputs as a result of being converted by different tools.

If all the conversion logic of I/Os are embedded into the execution unit of the process (or separate processes in a workflow chain), there is essentially no reason for any corresponding process description to be different from one server to another. The execution unit would basically dictate what it is able to accept and produce. The process description only normalizes and abstracts away how those I/O are represented are mapped to the execution unit, such that whether the execution unit is openEO, CWL, WPS, or a plain docker is irrelevant.

Since conversion would be accomplished by exactly the same tools as pre/post process steps in a workflow chain, there should not be any difference in produced results. If there are variations (eg: rounding errors due to float), I would argue that would be a misbehavior from the server due to poorly described I/Os in the process description.

One advantage of this is that a whole chain of processes could be happening in an optimal internal format of the implementation that never gets exposed, or kept in GPU memory for heavy CUDA parallel processing, without the need to explicitly encode / load things in a particular format until the very final step where the data is returned to a client.

While there might be some cases were such optimization would be beneficial, the logic required in those cases is so specific to each individual process, their I/O combinations, and their execution units that it makes it again not possible to automate it. If the workflow becomes "locked" by this very specific convertion chain because specific I/O combinations and server running it must be respected exactly to take advantage of such optimization, I believe this simply becomes a "very large process" were the logic is spread out across different sources instead of being properly isolated. Data lineage and reproductibility of the processes is greatly reduced, if not impossible.

I would argue that if a process requires this kind of optimization, it would be easier and more portable for the execution unit to simply implement a "step1->step2" script directly (and all necessary convertions between those steps) to avoid the intermediate encode/save/load/decode. So again, from the point of view of the process description and API, there would not be any additional conversion logic, and that execution unit combining the steps could be executed on any server.

@jerstlouis
Copy link
Member

If all the conversion logic of I/Os are embedded into the execution unit of the process (or separate processes in a workflow chain), there is essentially no reason for any corresponding process description to be different from one server to another.

I am of the opinion that nothing in Part 2 should restrict the possibility to automatically support additional input/output formats, and thus automatically enhancing the process description with additional format support compared to the execution unit's.
This allows for example to keep individual executionUnits much smaller and simpler, working off a single input format (e.g., only packaging libgeotiff in the execution unit), while automaticallly enabling additional ones through shared machinery (e.g., GDAL with full support for all its drivers and dependencies) outside the execution units.

With Part 3 collection input / output in particular automatically requires such outside machinery as collection input implies an OGC API client that need to negotiate whatever format and APIs are supported by the remote collection which may not match the execution unit's, and collection output similarly need to support clients that support different APIs and will negotiate formats that will match the OGC API server implementation's support for different formats.

Particular use cases or profile may have a preference for the approach you mention, where there is a very thin layer between the Processes server and the executionUnit, but this approach should not be mandatory by Part 2 (or that makes Part 2 incompatible with a lot of Part 3 such as collection input / output). Part 1: Core says nothing about this of course because it is completely agnostic of the concept of execution units.

While there might be some cases were such optimization would be beneficial, the logic required in those cases is so specific to each individual process...

I'm not sure I understand what you are saying in that paragraph. In Part 3 workflows, the idea is definitely not to lock any combination, and it does aim to facilitate preserving data lineage and reproducibility. However, it allows automatic negotiation of how things will be playing out (not involving end-user client) at every hop between two nodes in the workflow (whether happening internally on the same server, or spread across two servers where one server acts as a client to the other).

So again, from the point of view of the process description and API, there would not be any additional conversion logic, and that execution unit combining the steps could be executed on any server.

I was also considering cases where this process can be spread across datasets and processes spread across different deployments (potentially of the same software having an affinity for a particular format).

While it is possible to create a single process that implements the full workflow (whether the components are all on the same server, or involves external Processes implementation), this single process can be implemented as a chain of processes, and this workflow chain of processes can also be exposed as the source workflow.

@fmigneault
Copy link
Contributor

I don't see what Part 2 or Part 3 have to do with how convertion logic should be encapsulated in respective processes.
Part 2 only allows to deploy them dynamically, while Part 1 would require them to be predefined on the server, but the principle applies either way.
Part 3 allows to automatically chain them into a workflow, but you could still do a processing chain "manually" with Part 1 by executing a "core" process, followed by the "convertion" process using the output of the "core" process as input. The convertion machinery does not need to reside in the API.

This allows for example to keep individual executionUnits much smaller and simpler, working off a single input format (e.g., only packaging libgeotiff in the execution unit), while automaticallly enabling additional ones through shared machinery (e.g., GDAL with full support for all its drivers and dependencies) outside the execution units.

You seem to be describing exactly what I mentionned using small building blocks. My recommendation is that the shared machinery would simply be a gdal-converter process that accepts a bunch of input formats and can produce all their derivatives. The server doesn't need to have GDAL preinstalled. It does not even need to reside on the same server. If you want to allow a user to negociate a different media-type on /results/{outputID}, the logic under that endpoint could simply call gdal-converter under the hood with the original output, and return the converted one. The code under /results/{outputID} does not need to care about "how" to convert the data. It dispatches that operation to gdal-converter. At least, by having a dedicated gdal-converter process, you can now run that convertion logic with any desired input. You don't have to depend on previous job results to exist.

With Part 3 collection input / output in particular automatically requires such outside machinery [...] that makes Part 2 incompatible with a lot of Part 3 such as collection input / output

Again, that machinery could be a process. The "execution unit" of the workflow that needs to chain two processes with a collection could simply call an intermediate ogc-api-client-handler process that does the resolution of negociated types. The logic doesn't have to reside in the code of the API doing the workflow. The collection is a data structure just like any other, and a dedicated process that knows how to parse it and obtain requested results from it for the next process makes more sense. In that regard, that does not introduce any incompatibility with Part 2. If anything, it makes it more flexible because you now get more converter processes that can be combined in workflows however you want.

Part 1: Core says nothing about this of course because it is completely agnostic of the concept of execution units.

Even if "execution unit" is not explicitly in Part 1, the implementation will at some point run some code to do the processing. Call that however you want, but that code could be done in either of those methods:

  1. API does convert_input -> Process does core_compute -> API does convert_output
  2. Big Process does convert_input -> core_compute -> convert_output all in one
  3. Process convert_input, Process core_compute and Process convert_output are called one by one (Part 1) or in a workflow (Part 3).

My recommendation is again to go for the 3rd approach, because this can be ported into basically any other server (especially if the process was obtained by deploy Part 2), without side effets or hidden logic coming from the API as in approach 1.
Case 2 could be valid for optimization purposes, to avoid intermediate save/load, keeping items in memory, and so on, but I would employ that strategy only for rare cases that benefit from optimization because big logic like this is rapidly limitted in the number of cases it can handle. However, even the big process in case 2 could be chained within a workflow as the core_compute of case 3 to extent even more its supported I/O convertion. Therefore, there is really no reason to favor approach 1 in my opinion.

I'm not sure I understand what you are saying in that paragraph. [...]

The idea was that if you are using for example a GPU to do some processing, and that you want to leave the data in memory to allow it to be converted to somethig else, the convertion to be called would need very specific code to handle GPU logic and the specific convertion strategy for the input/output data format. If another process used CPUs instead, the same code would probably not work directly. Same for other data formats that need adapted logic. In other words, you would need a very specific implementation for every possible use case. Therefore, my point was that if you do have a use case that benefits from this specific implementation, you might as well package it is a dedicated process. For all other cases were preserving the data in memory this way would be negligeable, having dedicated processes that handle the convertion from one type to another, even if there are redundant save/load encode/decode between processes, would be much more scalable and portable across servers.

@jerstlouis
Copy link
Member

jerstlouis commented Oct 25, 2023

I don't see what Part 2 or Part 3 have to do with how convertion logic should be encapsulated in respective processes.

Specifically the Section 8: Collection Input and Section 11: Collection Output requirements classes. See also Section 6.2.5: Considerations for collection input / output.

What you're describing is similar to the openEO approach that requires an explicit process to "load" something from the collection and "publish" a collection.

With Collection input and output we can write a workflow like:

{
   "process" : "https://maps.gnosis.earth/ogcapi/processes/RFClassify",
   "inputs" : {
      "data" : { "collection" : "https://maps.gnosis.earth/ogcapi/collections/sentinel2-l2a" }
   }
}

(from https://maps.gnosis.earth/ogcapi/processes/RFClassify/execution?response=collection)

and access results triggering processing like:

https://maps.gnosis.earth/ogcapi/collections/temp-exec-2744D845/map/tiles/GNOSISGlobalGrid/12/1989/2656.png (map tile)

https://maps.gnosis.earth/ogcapi/collections/temp-exec-2744D845/coverage/tiles/GNOSISGlobalGrid/12/1989/2656.tif (coverage tile)

https://maps.gnosis.earth/ogcapi/collections/temp-exec-2744D845/coverage/tiles/GNOSISGlobalGrid (coverage tileset)

The "execution unit" of the workflow that needs to chain two processes with a collection could simply call an intermediate ogc-api-client-handler process that does the resolution of negociated types.

The intent with Part 3 Collection Input / Output is specifically not to require that.

Collection Output allows to present an OGC API Maps / Tiles / Coverages / EDR / Features / DGGS... as the front end, supporting content format/AoI/ToI/RoI/CRS/API negotiation on the output collection completely separately from the workflow definition.

If you extend this Collection Input / Output mechanism to how servers talk to each other, the communication can also be done entirely in an OGC API data access way. The servers do not need to act as Processes client for accessing results, they can instead use OGC API - Coverage requests to trigger processing on demand. They only need to POST the subset of the execution request intended for the remote server to /processes/{processId}/execution?response=collection to get back a collection that supports OGC API - Coverages and/or Tiles.

@fmigneault
Copy link
Contributor

I am probably missing something...

What you're describing is similar to the openEO approach that requires an explicit process to "load" something from the collection and "publish" a collection.

Exactly, but I would do it using CWL and Docker apps in my case since this is what my server supports. There is however no need to "load" anything. The map/coverage tiles URL would simply be retrieved and passed down to the following process.

To implement collection I/O on my server, I would simply create a collection-parser process that knows how to perform whichever OGC API negotiation that can be supported to obtain resolved URLs to relevant resources. Behind the scene, the server would call that collection-parser process with the specified collection, and whichever tile output is obtained from that would be chained to the RFClassify that expects an input image.

The distinction I am highlighting is that, if I wanted to understand how collection-parser resolved the input collection, I would be able to do so by calling it directly, by itself, and without the following logic from RFClassify. I could also manually take the resolved tile result from collection-parser execution, and manually execute RFClassify with the corresponding tile image retrieved from the collection as input. That would be the same as if I directly found that image using the coverage API and passed the URL reference to RFClassify myself. The logic of "how to parse a collection" would not be hard-coded by the server between the HTTP request parsing and forwarding to RFClassify, it would be a dispatched execution request to a collection-parser process that holds this logic.

To convert the output into a collection format, that would also be some kind of collection-maker process. That process would take care of any necessary collection creation, registration of underlying resources, etc. from an input. In that example, its input would be the output image produced by RFClassify, and some lineage metadata from RFClassify execution that generated it.

@jerstlouis
Copy link
Member

jerstlouis commented Oct 26, 2023

I am probably missing something...

Yes :)

With collection input / output in Part 3 workflows, the collection-parsing and collection-making is a pre-registration step that is done only once when first registering the workflow. That only happens when you click the Setup collection output button. This validates that the entire workflow is valid and set up negotiation for compatible APIs and formats between the different hop nodes (other the client will get a 400 Failure to validate workflow). It makes all components aware that they will be working together in that pipeline and are ready to roll.

All future requests for a specific AoI/ToI/RoI (or Tile or DGGS Zone Data) uses that workflow already registered (that can span collections and processes across multiple servers), and only triggers the processing workflow chain (which does not involve any "parse collection" or "make collection" step) for that specific region/time/resolution being requested. It will not be creating any new resources (no POST methods, all resources already exist virtually and their content gets actualized/cached the first time they are created with a GET, or beforehand if some server is preempting further request).

@fmigneault
Copy link
Contributor

This feel like a shortcut naming convention (which is OK), but that could still be defined explicitly with a workflow like:

{
  "process": "https://{some-server}/processes/collection-maker",
  "inputs": {
     "process" : "https://maps.gnosis.earth/ogcapi/processes/RFClassify",
     "inputs" : {
        "data" : { 
            "process" : "https://{some-server}/processes/collection-parser",
            "inputs": { "source-collection": { "href": "https://maps.gnosis.earth/ogcapi/collections/sentinel2-l2a" } }
            "outputs": {"image-found": {"format": {"mediaType": "image/tiff; application=geotiff" } } }
        }
     }
  }
}

The obtained workflow would be validated in the same manner, and could be executed by a CWL or openEO approach.
Using Part 2, that workflow would be deployable instead of executed immediately.

@jerstlouis
Copy link
Member

jerstlouis commented Oct 26, 2023

Yes you could do something like this, and that is using the Part 3 "Nested Processes" requirements class.

But whether you execut it or deploy it, when that workflow is executed it will process the entire input collection, unless you add a parameter to restrict the execution request for an AoI/ToI/RoI.

The whole point of Collection Input / Output is to have collections as first class objects where you do not need to specify API/AoI/ToI/RoI/format, and it allows you to express the workflow in a manner agnostic of all this.

The output collections exists for the spatiotemporal extent and resolution and all possible formats, and is accessible by regular OGC API clients like GDAL without having to actually process the whole thing, triggering processing on demand.

What you're saying essentially is that you can do things without Collection Input / Output.
Yes, of course. It's an optional requirements class, and a server or client can support other things in Part 3 like Nested Processes and OpenEO process graphs without implementing Collection Input / Output.

And you can integrate implementations that support it with some that don't either by using something like collection-maker / collection-parser, or by using an "href" pointing directly to the coverage output for example.

@fmigneault
Copy link
Contributor

Yes. Of course the example I provided is not complete. You would need additional parameters in collection-parser to indicate how to filter the collection and obtain the specific image-found output of interest. I focused more on the nested structure here.

What is important in that case is that I can easily map that nested OAP workflow structure to a CWL workflow representation. The same would be possible with an openEO processing graph after converting the inputs/outputs into arguments/returns.
This is possible only because each processing component in the chain are encapsulated in their own Process. Their is no hidden conversion logic between the steps.

If collection-parser returned a COG (or any other data format) instead of a GeoTiff, I could very easily add a cog2geotiff-converter process in between of collection-parser and RFClassify to fullfill the input format needed by RFClassify. If the cog2geotiff-converter logic was not defined in a dedicated process, there would not be a way to guarantee that the workflow chain would resolve correctly, since the types wouldn't match collection-parser [cog] -> X -> [geotiff] RFClassify. Your server could somehow negociate the required convertion, but that workflow would 100% fail with other servers that do not, hence it is not portable.

Depending on servers to somehow automatically negotiate/convert the types between steps greatly increases chances of workflow suddently failing.

@jerstlouis
Copy link
Member

jerstlouis commented Oct 26, 2023

You would need additional parameters in collection-parser to indicate how to filter the collection and obtain the specific image-found output of interest.

The idea of collection/input is you can represent the whole unfiltered input / output collections, preserving the ability to request small parts of it using OGC APIs. It's a late binding mechanism of these configuration options.

Because CWL (and openEO) do not have notion of an OGC API collection as a first class object, I don't think it would be possible to directly map a workflow making use of them to either.

However, a server-side implementation of Collection Input / Output could decide to map the workflow to an internal openEO or CWL workflow, taking additional parameters for AoI/ToI/RoI/format/API (or in the case of API or format, possibly selecting the appropriate helper processes for the task), which the Processes - Part 3 Collection/Input implementation could map to. For responding to client request for the not-fully-realized/on-demand Part 3 output collection, the Part 3 implementation would trigger that CWL or openEO workflow (which contains those extra processes e.g., to load a particular and convert to a particular format that the client expects) filling in the AoI/ToI/RoI/format/API parameters of that workflow to respond. This would be some of "extra machinery" of the API, but internally it could still use CWL directly or a pure Processes - Part 1 approach without the Collection Output first class object.

Your server could somehow negociate the required convertion, but that workflow would 100% fail with other servers that do not, hence it is not portable.

The workflow validation step which happens during registration would already perform the negociation and the idea is to report the failure before any actual processing is done. The negotiation happening at every hop has the client (or the server acting as a client) looking at the server conformance classes / capabilities and ensures the server supports an API / format working for the client side of the hop. It could also do further validation to make sure things work as expected. So if the workflow fails with another server, it will fail at the time of registering (POSTing the workflow to /processes/{processId}/execution?response=collection).

Depending on servers to somehow automatically negotiate/convert the types between steps greatly increases chances of workflow suddently failing.

My view is exactly the opposite. Not requiring a particular format at a specific step of a workflow greatly helps the chances that the client and server side of the hop can find a common ground between their respective capabilities. e.g., if I enforce GeoTIFF and EDR at a particular hop, and either the client and server does not support GeoTIFF, the workflow validation will fail. But if I leave that open, maybe they will find out that they both support JPEG-XL and OGC API - Tiles and can interoperate that way. Then I can take the same workflow and change one side of that hop, and this new hop now is able to operate with GeoTIFF and Coverages. Only the collection or process URL had to be changed in the workflwo execution request, everything else stays exactly the same. As an end-user client, I don't have to bother about figuring out which format / API each hop supports. I just discover compatible OGC API Collections and Processes and can easily assemble a workflow this way.

If a hop is not interoperable (no common ground on API / format), this feedback is received as workflow validation failure before trying to do any processing in the regsitration step, and the workflow can be fixed.

@fmigneault
Copy link
Contributor

Because CWL (and openEO) do not have notion of an OGC API collection as a first class object, I don't think it would be possible to directly map a workflow making use of them to either.

I don't see why that would not be possible. Using a OGC API client that knows how to interact with the concept of an OGC API collection is not different than another script. Whichever code that runs to resolve the OGC Workflow, even if doing a late binding, could be converted to a CWL/openEO workflow dynamically, filling in any necessary convertion process between steps, and then running it. I usually prefer to have static workflows that have well established steps and connections, so the user knows exactly what they are running, but still dynamic resolution could be supported.

I enforce GeoTIFF and EDR at a particular hop, and either the client and server does not support GeoTIFF, the workflow validation will fail. But if I leave that open, maybe they will find out that they both support JPEG-XL and OGC API - Tiles and can interoperate that way.

Why not set them to use JPEG-XL directly? If they both support it, they should both advertise it, and it would possible to align them this way right of the start.

Maybe for very specific formats, allowing some flexible format matching could make sense, but I can see a lot of cases were that could have the oppopsite effect. For exemple, if the servers figure how they both support application/json, that has a high chance of failing. Even with more specific formats such as GeoJSON, if the second step expects a GeometryCollection, but the first process returns a Point, the workflow will fail even though everything "worked" according to schemas and formats. Relying on formats and schemas helps, but is not guaranteed to have contents that work together. When workfows are defined explicitly (eg: the "static" vs "dynamic" I refered to earlier), users building the workflow have to make conscious choices on building them, thus, leading in fewer unexpected errors.

@m-mohr
Copy link
Author

m-mohr commented Feb 8, 2024

Because CWL (and openEO) do not have notion of an OGC API collection as a first class object, I don't think it would be possible to directly map a workflow making use of them to either.

openEO has a load_collection process and recently also got an export_collection process. This is "first class" in openEO :-)

But I feel like we are departing from the original issue. Was there a final conculsion regarding the other process encodings?

Even if I'd try to convert openEO processes into a process summary encoding, the required version number would be missing. I'm also not quite sure yet what the metadata property is used for.

@jerstlouis
Copy link
Member

jerstlouis commented Feb 9, 2024

Was there a final conculsion regarding the other process encodings?

We definitely want to allow for alternative encoding of processes, an OpenAPI encoding is one that I would be curious to experiment with.

If it would be useful to have the same resource also available as different encoding specific to openEO, that would require a different mediatype to negotiate it. Even if openEO community standards ends up using /proceses and/or /jobs differently than specified in Processes - Part 1, for the proposed Part 3 - openEO Process Graph Workflows requirement class the idea is that it would be possible to provide a process description using the OGC JSON requirement class of Part 1 (as discussed above, I think that should definitely be possible -- technically Processes - Part 1 is so generic that it can describe any kind of function that takes an input and generate an output).

the required version number would be missing.

You mean the version required by processSummary ?

If that information is not available that could just default to 1.0 ?

openEO has a load_collection process and recently also got an export_collection process. This is "first class" in openEO :-)
But I feel like we are departing from the original issue.

Probably departing indeed from the original issue and we should move this to a new Collection Input / Output discussion issue but:

What I mean by "first class" is that there is a "collection" concept which is an alternate mechanism to the "by value", "by reference" (href) and nested process mechanisms of how spatiotemporal input data can be accepted as an input to a process, or generated on demand as an output of a process, for a given time, area and resolution of interest, which does not involve the use of an additional process (as far the Processes API / execution request goes). Collection Output does not involve creating anything external anywhere, and different execution for different Area/Time/Resolution of interest do not result in separate collections.

Though they are designed to effortlessly chain with each other, and could internally be implemented as processes, Collection Input and Collection Output are quite different beasts in terms of how they are defined:

  • Collection Input defines the collection construct which is an additional execution request input type as an alternative to inline value, href, or nested process. It implies client capabilities for one or more OGC API data access standard.
  • Collection Output is an alternate execution mode (response=collection) for executing a process or workflow, where a regular client like GDAL accesses the response to that execution request just like it would access any OGC API collection at /collections/{collectionId} and then proceeds exactly as usual making /coverage, /map, /tiles, /items, /dggs requests...

@m-mohr I am curious to what extent your load_collection is equivalent to WHU's loadCube process as discussed in https://gitlab.ogc.org/ogc/T19-GDC/-/issues/57 ? (and @fmigneault similar question for your similar process )

In their case, supporting Collection Input (which are about local collections -- Remote Collections is actually the equivalent for OGC API collections from external APIs which requires the server acting as a client) would be as simple as internally converting:

   "data": {
      "collection": "http://oge.whu.edu.cn/geocube/gdc_api_t19/collections/SENTINEL-2%20Level-2A%20MSI"
   }

to:

   "data": {
      "process": "http://oge.whu.edu.cn/geocube/gdc_api_t19/processes/loadCube",
      "inputs": { "cubeName": "SENTINEL-2 Level-2A MSI" }
   }

The important aspect is that loadCube in this context does not mean requesting the entire collection -- only the spatiotemporal subset / resolution / fields relevant to satisfy the current requests (which may be coming in to the server as Collection Output client requests) needs to be retrieved. But the initial handshake with the remote server can all be established at the time the execution request is initially submitted (which for Collection Output is only when the client first did a POST of the execution requests, not for every coverage tile or subset requested later on).

The Collection Input req. class accomplishes a few things:

  • It provides a standard mechanism for this functionality, equivalent to the concept of a WellKnownProcess which would take particular inputs and returns particular outputs, but not having to discover find which processId does this for a particular implementation and adapt the execution request when switching to another
  • It implies that the input is data for a subset of that collection, based on the subset being generated for execution (defining workflows independent of the time/area/resolution of interest -- i.e., not hardcoding bbox, time intervals and zoom levels in the workflow definition itself, therefore making the same workflow re-usable for different areas, and allowing to define a virtual collection (Collection Output) with it
  • It provides a mechanism where the input / output fields modifiers (separate req. classes) such as filter (filter using CQL2 queries) and properties (selecting fields, deriving new fields, performing aggregation etc.) can be passed in as part of these partial data requests directly to the server providing that data

@pvretano
Copy link
Contributor

pvretano commented May 13, 2024

SWG Meeting: 13-MAY-2024: There was some disucssion in the SWG today about using OpenAPI as the process description language. Basically, you do a GET /processes/{processesId} and you negotiate to a Application/vnd.oai.openapi+json;version=3.0 response. What you get back is a small OpenAPI document with the description of that one process.

@m-mohr
Copy link
Author

m-mohr commented May 13, 2024

just fyi: That only makes sense if you expose processes as HTTP endpoints, which is not the case for openEO. And that was the initial question. Can we have a conformance class that allows us to send openEO process descriptions via the GET /processes endpoint.

@jerstlouis
Copy link
Member

@m-mohr A bit confused by your last comment...

Aren't there process description HTTP end-points in openEO?

I understand that there's no individual process execution end-points in openEO.
But I thought this issue was about the description, for which I thought there are individual process description end-points for openEO processes? Thanks.

@fmigneault
Copy link
Contributor

Both of these are requirements :

Allowing alternate negotiation formats for the process description/execution makes sense, but the APIs should at least provide the minimal endpoint requirements to allow these negotiations to take place.

@m-mohr
Copy link
Author

m-mohr commented May 13, 2024

We only have a single GET /processes endpoint which describes all endpoints according to the process definition language, there's no GET /processes/:id yet.

@bpross-52n
Copy link
Contributor

bpross-52n commented May 27, 2024

SWG meeting from 2024-05-27: Add information about how we intent alternative process encodings to interact with the /processes and /processes/{id}. This will be based on content negotiation. Wherever we describe a process description with regard to API Processes, include the media type. Expand the content of section 7.10, include example of OGC process description and example of OpenAPI description.

@m-mohr
Copy link
Author

m-mohr commented May 27, 2024

Please keep in mind that for example both OGC API Processes and openEO use application/json as media type, so content negotiation might be difficult.

@pvretano
Copy link
Contributor

pvretano commented May 27, 2024

@m-mohr the discussion was that we would define media types that were not the generic ones. So to get an OGC process list from /processes you would negotiate to something like application/ogc-proc-list+json ... or something like that. Anyway, I'll create the PR and everyone can review and chime in.

@fmigneault
Copy link
Contributor

@pvretano
Could application/json; proflie=urn:ogc:... and application/json; profile=urn:openeo:... be considered instead, similar to what https://www.w3.org/TR/dx-prof-conneg/#motivation proposes?

Using something like application/ogc-proc-list+json limits the use of other media-types, such as application/ld+json that could very well be used to represent an extended form of OGC/OpenEO processes.

@pvretano
Copy link
Contributor

@fmigneault I said "something like". I am not proposing application/ogc-proc-list+json be the mediatype. I was just using that as an example.

One question, is "profile" a valid parameter for the application/json media type? I don't remember reading that in the application/json media type.

@fmigneault
Copy link
Contributor

fmigneault commented May 28, 2024

Of course :) Just proposing ahead to consider this use case while I had it in mind.

The application/json RFC indicates that it has no optional parameters and no additional parameters defined. However, the Content-Type header RFC seems to indicate that subtypes may specify additional parameters for its own use. Therefore, unless specific uses are reserved by the type, other parameters are somewhat open (my interpretation at least). It also indicates that parameters are modifiers of the media subtype, but that they do not fundamentally affect the nature of the content.

For what it's worth, I have seen browsers respect filename*="xyz.json" used to indicate the recommended name when downloading the JSON content as a file, so I do not think there is a limitation about non-explicitly listed parameters. The profile name seems fairly standard for other structured formats such as XML, so I think it would be a reasonable use it in the case of JSON.

I've also seen Accept-Schema, Accept-Profile and Prefer: schema=x in some cases, so I don't think there is a unified way regardless of the choice.

@jerstlouis
Copy link
Member

@pvretano The media type ( https://www.iana.org/assignments/media-types/application/json ) does not define any parameters

I do believe we should stick to application/json for the OGC Process description for compatibility with 1.0.

But I think what @fmigneault is pointing out is that the way negotiation by profile works is that you can always add that profile= to any media type, if I understand correctly? This is the related common issue: opengeospatial/ogcapi-common#8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

5 participants