WIP

Author: soxofaan

docs/_static/custom.css

@@ -104,3 +104,7 @@ pre {
     font-size: 75%;
     padding: 0 1ex;
 }
+
+nav.contents.local {
+    border: none;
+}

docs/api-processbuilder.rst

@@ -0,0 +1,86 @@
+..  FYI this file is intended to be "included"
+    from the ProcessBuilder class doc block
+
+
+The :py:class:`ProcessBuilder <openeo.processes.ProcessBuilder>` class
+is a helper class that implements
+(much like the :ref:`openEO process functions <openeo_processes_functions>`)
+each openEO process as a method.
+On top of that it also adds syntactic sugar to support Python operators as well
+(e.g. ``+`` is translated to the ``add`` process).
+
+.. attention::
+    As normal user, you should never create a
+    :py:class:`ProcessBuilder <openeo.processes.ProcessBuilder>` instance
+    directly.
+
+    You should only interact with this class inside a callback
+    function/lambda while building a child callback process graph
+    as discussed at :ref:`child_callback_callable`.
+
+
+For example, let's start from this simple usage snippet
+where we want to reduce the temporal dimension
+by taking the temporal mean of each timeseries:
+
+.. code-block:: python
+
+    def my_reducer(data):
+        return data.mean()
+
+    cube.reduce_dimension(reducer=my_reducer, dimension="t")
+
+Note that this ``my_reducer`` function has a ``data`` argument,
+which conceptually corresponds to an array of pixel values
+(along the temporal dimension).
+However, it's important to understand that the ``my_reducer`` function
+is actually *not evaluated when you execute your process graph*
+on an openEO back-end, e.g. as a batch jobs.
+Instead, ``my_reducer`` is evaluated
+*while building your process graph client-side*
+(at the time you execute that ``cube.reduce_dimension()`` statement to be precise).
+This means that that ``data`` argument is actually not a concrete array of EO data,
+but some kind of *virtual placeholder*,
+a :py:class:`ProcessBuilder <openeo.processes.ProcessBuilder>` instance,
+that keeps track of the operations you intend to do on the EO data.
+
+To make that more concrete, let's add type hints
+(which will make it easier to discover what we can do with the argument,
+depending on which editor or IDE you are using):
+
+.. code-block:: python
+
+    from openeo.processes import ProcessBuilder
+
+    def my_reducer(data: ProcessBuilder) -> ProcessBuilder:
+        return data.mean()
+
+    cube.reduce_dimension(reducer=my_reducer, dimension="t")
+
+
+Because :py:class:`ProcessBuilder <openeo.processes.ProcessBuilder>` methods
+return new :py:class:`ProcessBuilder <openeo.processes.ProcessBuilder>` instances,
+and because it support syntactic sugar to use Python operators on it,
+and because :ref:`openeo.process functions <openeo_processes_functions>`
+also accept and return :py:class:`ProcessBuilder <openeo.processes.ProcessBuilder>` instances,
+we can mix methods, functions and operators in the callback function like this:
+
+.. code-block:: python
+
+    from openeo.processes import ProcessBuilder, cos
+
+    def my_reducer(data: ProcessBuilder) -> ProcessBuilder:
+        return cos(data.mean()) + 1.23
+
+    cube.reduce_dimension(reducer=my_reducer, dimension="t")
+
+or compactly, using an anonymous lambda expression:
+
+.. code-block:: python
+
+    from openeo.processes import cos
+
+    cube.reduce_dimension(
+        reducer=lambda data: cos(data.mean())) + 1.23,
+        dimension="t"
+    )

docs/api-processes.rst

@@ -1,20 +1,69 @@
-==========================
+=========================
 API: ``openeo.processes``
-==========================
+=========================
 
-The ``openeo.processes`` module contains helper to build call
+The ``openeo.processes`` module contains building blocks and helpers
+to construct so called "child callbacks" for openEO processes like
+:py:meth:`openeo.rest.datacube.DataCube.apply` and
+:py:meth:`openeo.rest.datacube.DataCube.reduce_dimension`,
+as discussed at :ref:`child_callback_callable`.
 
+.. note::
+    The contents of the ``openeo.processes`` module is automatically compiled
+    from the official openEO process specifications.
+    Developers that want to fix bugs in, or add implementations to this
+    module should not touch the file directly, but instead address it in the
+    upstream `openeo-processes <https://github.com/Open-EO/openeo-processes>`_ repository
+    or in the internal tooling to generate this file.
+
+
+.. contents:: Sections:
+   :depth: 1
+   :local:
+   :backlinks: top
+
+
+.. _openeo_processes_functions:
+
+Functions in ``openeo.processes``
+---------------------------------
+
+The ``openeo.processes`` module implements (at top-level)
+a regular Python function for each openEO process
+(not only the official stable ones, but also experimental ones in "proposal" state).
+
+These functions can be used directly as child callback,
+for example as follows:
+
+.. code-block:: python
+
+    from openeo.processes import absolute, max
+
+    cube.apply(absolute)
+    cube.reduce_dimension(max, dimension="t")
+
+
+Note how the signatures of the parent :py:class:`DataCube <openeo.rest.datacube.DataCube>` methods
+and the callback functions match up:
+
+-   :py:meth:`DataCube.apply() <openeo.rest.datacube.DataCube.apply>`
+    expects a callback that receives a single numerical value,
+    which corresponds to the parameter signature of :py:func:`openeo.processes.absolute`
+-   :py:meth:`DataCube.reduce_dimension() <openeo.rest.datacube.DataCube.reduce_dimension>`
+    expects a callback that receives an array of numerical values,
+    which corresponds to the parameter signature :py:func:`openeo.processes.max`
 
-``openeo.processes`` functions
--------------------------------
 
 .. automodule:: openeo.processes
     :members:
-    :exclude-members: ProcessBuilder
+    :exclude-members: ProcessBuilder, process, _process
+
 
+``ProcessBuilder`` helper class
+--------------------------------
 
-``openeo.processes.ProcessBuilder``
-------------------------------------
+..  FYI the ProcessBuilder docs are provided through its doc block
+    with an "include" of "api-processbuilder.rst"
 
 .. automodule:: openeo.processes
     :members: ProcessBuilder

docs/api.rst

@@ -1,5 +1,5 @@
 =============
-API: General
+API (General)
 =============
 
 High level Interface

docs/processes.rst

@@ -227,20 +227,37 @@ to be invoked on a subset or slice of the datacube.
 For example:
 
 *   process ``apply`` requires a transformation that will be applied
-    to each pixel in the cube (separately)
+    to each pixel in the cube (separately), e.g. in pseudocode
+
+    .. code-block:: text
+
+        cube.apply(
+            given a pixel value
+            => scale it with factor 0.01
+        )
+
 *   process ``reduce_dimension`` requires an aggregation function to convert
-    an array of pixel values (along a given dimension) to a single value
+    an array of pixel values (along a given dimension) to a single value,
+    e.g. in pseudocode
+
+    .. code-block:: text
+
+        cube.reduce_dimension(
+            given a pixel timeseries (array) for a (x,y)-location
+            => temporal mean of that array
+        )
+
 *   process ``aggregate_spatial`` requires a function to aggregate the values
     in one or more geometries
 
 These transformation functions are usually called "**callbacks**"
 because instead of being called explicitly by the user,
-they are called by their "parent" process
+they are called and managed by their "parent" process
 (the ``apply``, ``reduce_dimension`` and ``aggregate_spatial`` in the examples)
 
 
 The openEO Python Client Library currently provides a couple of DataCube methods
-that expect a callback, most commonly:
+that expect such a callback, most commonly:
 
 - :py:meth:`openeo.rest.datacube.DataCube.aggregate_spatial`
 - :py:meth:`openeo.rest.datacube.DataCube.aggregate_temporal`
@@ -249,8 +266,14 @@ that expect a callback, most commonly:
 - :py:meth:`openeo.rest.datacube.DataCube.apply_neighborhood`
 - :py:meth:`openeo.rest.datacube.DataCube.reduce_dimension`
 
-These functions support several ways to specify the desired callback.
+The openEO Python Client Library supports several ways
+to specify the desired callback for these functions:
+
 
+.. contents::
+   :depth: 1
+   :local:
+   :backlinks: top
 
 Callback as string
 ------------------
@@ -269,7 +292,7 @@ for example:
 This approach is only possible if the desired transformation is available
 as a single process. If not, use one of the methods below.
 
-Also important is that the "signature" of the provided callback process
+It's also important to note that the "signature" of the provided callback process
 should correspond properly with what the parent process expects.
 For example: ``apply`` requires a callback process that receives a
 number and returns one (like ``absolute`` or ``sqrt``),
@@ -283,44 +306,41 @@ Callback as a callable
 -----------------------
 
 You can also specify the callback as a "callable":
-a Python object that can be called (e.g. a function without parenthesis).
+which is a fancy word for a Python object that can be called,
+but just think of it like a function you can call.
 
-The openEO Python Client Library defines the
-official processes in the :py:mod:`openeo.processes` module,
-which can be used directly:
+You can use a regular Python function, like this:
 
 .. code-block:: python
 
-    from openeo.processes import absolute, max
+    def transform(x):
+        return x * 2 + 3
 
-    cube.apply(absolute)
-    cube.reduce_dimension(max, dimension="t")
+    cube.apply(transform)
 
-You can also use ``lambda`` functions:
+or, more compactly, a "lambda"
+(a construct in Python to create anonymous inline functions):
 
 .. code-block:: python
 
     cube.apply(lambda x: x * 2 + 3)
 
 
-or normal Python functions:
+The openEO Python Client Library implements most of the official openEO processes as
+:ref:`functions in the "openeo.processes" module <openeo_processes_functions>`,
+which can be used directly as callback:
 
 .. code-block:: python
 
-    from openeo.processes import array_element
-
-    def my_bandmath(data):
-        band1 = array_element(data, index=0)
-        band2 = array_element(data, index=1)
-        return band1 + 1.2 * band2
-
+    from openeo.processes import absolute, max
 
-    cube.reduce_dimension(my_bandmath, dimension="bands")
+    cube.apply(absolute)
+    cube.reduce_dimension(max, dimension="t")
 
 
-The argument that is passed to these functions is
-an instance of :py:class:`openeo.processes.ProcessBuilder`.
-This is a helper object with predefined methods for all standard processes,
+The argument that will be passed to all these callback functions is
+a :py:class:`ProcessBuilder <openeo.processes.ProcessBuilder>` instance.
+This is a helper object with predefined methods for all standard openEO processes,
 allowing to use an object oriented coding style to define the callback.
 For example:
 
@@ -334,12 +354,14 @@ For example:
     cube.reduce_dimension(avg, dimension="t")
 
 
-These methods also return ``ProcessBuilder`` objects,
+These methods also return :py:class:`ProcessBuilder <openeo.processes.ProcessBuilder>` objects,
 which also allows writing callbacks in chained fashion:
 
 .. code-block:: python
 
-    cube.apply(lambda x: x.absolute().cos().add(y=1.23))
+    cube.apply(
+        lambda x: x.absolute().cos().add(y=1.23)
+    )
 
 
 All this gives a lot of flexibility to define callbacks compactly
@@ -377,7 +399,7 @@ looks intuitive and straightforward, but it should be noted
 that not everything is allowed in these functions.
 You should just limit yourself to calling
 :py:mod:`openeo.processes` functions,
-:py:class:`openeo.processes.ProcessBuilder` methods
+:py:class:`ProcessBuilder <openeo.processes.ProcessBuilder>` methods
 and basic math operators.
 Don't call functions from other libraries like numpy or scipy.
 Don't use Python control flow statements like ``if/else`` constructs
@@ -414,9 +436,14 @@ Callback as ``PGNode``
 -----------------------
 
 You can also pass a :py:class:`~openeo.internal.graph_building.PGNode` object as callback.
-This method is used internally and could be useful for more
-advanced use cases, but it requires more in-depth knowledge of
-the openEO API and openEO Python Client Library to construct correctly.
+
+.. attention::
+    This approach should generally not be used in normal use cases.
+    The other options discussed above should be preferred.
+    It's mainly intended for internal use and an occasional, advanced use case.
+    It requires in-depth knowledge of the openEO API
+    and openEO Python Client Library to construct correctly.
+
 Some examples:
 
 .. code-block:: python

openeo/internal/processes/generator.py

@@ -117,6 +117,9 @@ def generate_process_py(processes: List[Process], output=sys.stdout, argv=None):
 
 
         class ProcessBuilder(ProcessBuilderBase):
+            \"\"\"
+            .. include:: api-processbuilder.rst
+            \"\"\"
 
             _ITERATION_LIMIT = 100
 
@@ -234,6 +237,7 @@ def __gt__(self, other) -> 'ProcessBuilder':
             # Used command line arguments:
             #    {cli}
         """.format(cli=" ".join(argv))))
+    # TODO: add metadata (both as comment and as variable in the module) about the source of the process specs (e.g. version/commit of openeo-processes)
     output.write(oo_src)
     output.write(fun_src)
 

openeo/processes.py

@@ -10,6 +10,9 @@
 
 
 class ProcessBuilder(ProcessBuilderBase):
+    """
+    .. include:: api-processbuilder.rst
+    """
 
     _ITERATION_LIMIT = 100