-
Notifications
You must be signed in to change notification settings - Fork 45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
P3-Workflows: Re-usable components ($ref with JSON Pointers) #304
Comments
I like the idea. In order to align with usual OpenAPI components, I think it would be good to enforce the process definitions to be under |
@fmigneault I agree, as long as you don't mean to enforce the use of $ref to define processes / inputs etc. so that things remain compatible with Part 1 (this would still be a separate conformance class). Thanks for the suggestion! However, a nested Process definition is one type of input to another process, so not sure about Maybe we should clarify what types of "components" we have. |
Only the
That made me realize, shouldn't the inputs look something like this? {
"inputs" : {
"orthoPhoto" : { "process": { "$ref" : "#/components/photogrammetry" } }
}
} The |
@fmigneault The This follows Part1 that supports simple JSON types as well as
Specifying a process input object is what indicates the chaining.
I agree that we could restrict exactly where a |
I think the current example has a fundamental problem. Currently, the process definition {
"photogrammetry":
{
"process": "https://example.com/ogcapi/processes/Photogrammetry",
"inputs" : { }
}
} Cannot be extended in place as expected in the below content because of the extra {
"inputs" : {
"orthoPhoto" : { "$ref" : "#/components/photogrammetry", "outputs" : { "orthoPhoto": { } },
}
} The correct schema would require a {
"inputs" : {
"orthoPhoto" : { "process": { "$ref" : "#/components/photogrammetry" }, "output": { } }
}
} If instead, we moved |
@fmigneault Thank you for pointing this out.
That is the intent.
It is the intent that the
That JSON Pointer (RFC 6901) itself has no such restriction.
|
It is normal RFC-6901 doesn't mention the limitation of Since all other definitions in the API use OpenAPI standard keywords, I would personally rather not have new keywords that could be up to interpretation by different tools. If implementers need to code additional parsing over a typical JSON schema resolver, I think the proposal is heading the wrong way. You are right bout |
Right, but this execution request has nothing to do with JSON Schema or OpenAPI, and as you point out We can try to come up with a better solution, but I think:
If we really want to stick to the JSON Schema-like {
"process": "https://example.com/ogcapi/processes/SomeProcess",
"inputs" : {
"orthoPhoto" : {
{
"process" : { "$ref" : "#/components/photogrammetry/process" },
"inputs" : { "$ref" : "#/components/photogrammetry/inputs" },
"outputs" : { "orthoPhoto": { } }
}
"pointCloud" : {
"process" : { "$ref" : "#/components/photogrammetry/process" },
"inputs" : { "$ref" : "#/components/photogrammetry/inputs" },
"outputs" : { "pointCloud": { } }
}
},
"components" :
{
"photogrammetry":
{
"process": "https://example.com/ogcapi/processes/Photogrammetry",
"inputs" : { ... }
}
}
} If the exact same input object is to be re-used with no modification at multiple places, you could also have another level, e.g.: {
"process": "https://example.com/ogcapi/processes/SomeProcess",
"inputs" : {
"orthoPhoto" : { "$ref": "#/components/photogrammetry-ortho" },
"pointCloud" : { "$ref": "#/components/photogrammetry-pointCloud" }
},
"components" :
{
"photogrammetry":
{
"process": "https://example.com/ogcapi/processes/Photogrammetry",
"inputs" : { ... }
},
"photogrammetry-ortho":
{
"process" : { "$ref" : "#/components/photogrammetry/process" },
"inputs" : { "$ref" : "#/components/photogrammetry/inputs" },
"outputs" : { "orthoPhoto": { } }
},
"photogrammetry-pointCloud":
{
"process" : { "$ref" : "#/components/photogrammetry/process" },
"inputs" : { "$ref" : "#/components/photogrammetry/inputs" },
"outputs" : { "pointCloud": { } }
}
}
} but this is all more verbose than, e.g.: {
"process": "https://example.com/ogcapi/processes/SomeProcess",
"inputs" : {
"orthoPhoto" : { "$include-properties": "#/components/photogrammetry", "outputs" : { "orthoPhoto": { } } },
"pointCloud" : { "$include-properties": "#/components/photogrammetry", "outputs" : { "pointCloud": { } } }
},
"components" :
{
"photogrammetry":
{
"process": "https://example.com/ogcapi/processes/Photogrammetry",
"inputs" : { ... }
}
}
} |
This is true, but since the How about something like this? Since the {
"process": "https://example.com/ogcapi/processes/SomeProcess",
"inputs" : {
"orthoPhoto" : {
"process": "https://example.com/ogcapi/processes/Photogrammetry",
"inputs": {
"input-1": { "$ref": "#/components/inputs/photogrammetry-input" },
"input-2": { "$ref": "#/components/inputs/shared-input" }
},
"outputs": { "$ref": "#/components/outputs/photogrammetry-output" }
},
"pointCloud" : {
"process": "https://example.com/ogcapi/processes/PointCloud",
"inputs": {
"input-1": { "$ref": "#/components/inputs/pointCloud-input" },
"input-2": { "$ref": "#/components/inputs/shared-input" }
},
"outputs": { "$ref": "#/components/outputs/pointCloud-output" }
}
},
"components" : {
"inputs": {
"photogrammetry-input": { "..." },
"pointCloud-input": { "..." },
"shared-input": { "..." }
},
"outputs": {
"photogrammetry-output": { "orthoPhoto": { } },
"pointCloud-output": { "pointCloud": { } }
}
}
} |
@fmigneault In your last example, there are different processes generating the pointCloud output and photogrammetry output. Another important use case is the exact same output used in different places. But having to piece them together (process, inputs, outputs) like in your example there does, it does not guarantee / clearly indicate that it is the same thing being re-used (e.g., which inputs to use as the orthoPhoto for SomeProcess would need to be changed in multiple places).
The schema is for the process description though, not the execution request. So it is somewhat removed...
The gain is that the process URI appears in only one location, so if you wish to change it to another process or server, you only need to change it in one place. If we really want to use |
You can simply replace Whether the same process is referenced by In other words, the real call to The implemention could decide to resolve all
Yes, but it is much easier to use the same one still. To look at what must be submitted for execution, one has to look at the process description. Going back and forth between different paradigms just makes the API harder to employ.
Ok. That's a valid use, but not a requirement for detecting reuse of the same process execution over multiple I/O.
I agree as long as the use of Maybe we should consider using process: "https://example.com/ogcapi/processes/SomeProcess"
inputs:
orthoPhoto:
$id: "#/components/processes/Photogrammetry/outputs/orthoPhoto"
pointCloud:
$id: "#/components/processes/Photogrammetry/outputs/pointCloud"
otherCloud:
$id: "#/components/processes/OtherPhotogram/outputs/pointCloud"
components:
$defs:
Photogrammetry:
process: "https://example.com/ogcapi/processes/Photogrammetry"
processes:
Photogrammetry:
process:
$ref: "#/components/$defs/Photogrammetry/process"
inputs:
input-1:
$ref: "#/components/inputs/photogrammetry-input"
input-2:
$id: "#/components/inputs/shared-input"
outputs:
orthoPhoto: {}
pointCloud: {}
OtherPhotogram:
process:
$ref: "#/components/$defs/Photogrammetry/process"
inputs:
input-1:
$ref: "#/components/inputs/pointCloud-input"
input-2:
$id: "#/components/inputs/shared-input"
outputs:
pointCloud: {}
inputs:
photogrammetry-input: "..."
pointCloud-input: "..."
shared-input: "..." |
In all examples, the Processes Part 1-3 / workflow engine implementation is really free to do whatever it wants as long as it conforms to the client's expectations.
Is Here, it introduces yet another type of input being specifically one output of a process. |
Indeed, |
@fmigneault but my point was that this is not what { "$id": "http://yourdomain.com/schemas/myschema.json" } This identifies the document as a whole with a URI (where the schema can also be retrieved) i.e., it says "this is this id" , as opposed to referencing something ("use that thing with this id over there"). Also, I had missed this so far: https://json-schema.org/draft/2019-09/release-notes.html#keyword-changes
It is equivalent to using OpenAPI also supports it as of version 3.1 that is fully aligned with JSON Schema. |
@jerstlouis However, I still think there should be something more than just I think this diverging interpretation could lead to non-interoperable servers because the way the workflow engine would perform |
Well, the schema for the execution request is NOT JSON Schema, so that is not applicable.
I was not really familiar with With $defs in JSON Schema you have a dictionary of keys to sub-schemas, where each schema can then have its own So perhaps we could indeed come up with something following that model.
That might actually be relevant and useful in some situations. So we should discuss.
Ideally, the implementation should be able to identify re-usability irrespective of whether things are using $ref or not.
I think the workflow definition should have at least some level of determinism regardless of the path taken by the workflow engine. |
I agree that the Workflow definition should not depend on the use of |
Possibly as a separate conformance class, it would be very useful to support re-usable components with
$ref
and a JSON Pointer to avoid repeating the same redundant (potentially complex) nested process definition within a workflow.In addition, an implementation would easily be able to identify that two nested process objects are identical and therefore their outputs should be re-used (as opposed to executing it twice).
This would also facilitate defining a nested process that generates multiple outputs, then refer to it adding an
{"outputs": ...
section to select a specific output for its use as a particular input e.g.,See also scenario 4 in #279 (comment) , where it is used in a different context (which may be better addressed with a mechanism as part of the Input/Output Fields conformance class to aggregate over a temporal dimension for multiple intervals, as opposed to completely reducing the temporal dimension).
The text was updated successfully, but these errors were encountered: