Issues integrating loguru into pydra tasks

[maestroque]

Found on pydra version 0.23.

The desired use case is integrating pydra into the phys2denoise package [relevant PR] for the refinement of a data processing workflow. The logging package intended for use was loguru, however the following issue arised:

While using this logging package in a task, for example as follows:

from loguru import logger

@pydra.mark.task
def compute_metrics(phys, metrics):
    if isinstance(metrics, list) or isinstance(metrics, str):
        for metric in metrics:
            if metric not in _available_metrics:
                logger.warning(f"Metric {metric} not available. Skipping")
                continue

            args = select_input_args(metric, {})
            phys = globals()[metric](phys, **args)
            logger.info(f"Computed {metric}")
    return phys

The task threw the following error on execution, without displaying any loguru log within the task:

>       task2 = compute_metrics(phys=fake_physio, metrics=["respiratory_variance"])

phys2denoise/tests/test_tasks.py:15:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pydra-p2d/lib/python3.10/site-packages/pydra/mark/functions.py:47: in decorate
    return FunctionTask(func=func, **kwargs)
pydra-p2d/lib/python3.10/site-packages/pydra/engine/task.py:146: in __init__
    fields.append(("_func", attr.ib(default=cp.dumps(func), type=bytes)))
pydra-p2d/lib/python3.10/site-packages/cloudpickle/cloudpickle.py:1479: in dumps
    cp.dump(obj)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <cloudpickle.cloudpickle.Pickler object at 0x7fe5ea17e140>, obj = <function compute_metrics at 0x7fe5e9fb3910>

    def dump(self, obj):
        try:
>           return super().dump(obj)
E           TypeError: cannot pickle 'EncodedFile' object

pydra-p2d/lib/python3.10/site-packages/cloudpickle/cloudpickle.py:1245: TypeError

I did not have any luck re-creating a simpler failing script, however the following works as intended:

from loguru import logger
import pydra

@pydra.mark.task
def add(x, y):
    logger.info(f"Adding {x} and {y}")
    return x + y

task = add(x=1, y=2)
task()
logger.info(f"Result: {task.result()}")

I will keep investigating and post here any clarifications. Please let me know if you need something extra from my side. If this error seems familiar in any way please let me know!

[tclose]

Hi, sorry to only get to this now. Did you get anywhere with this? If you are still interested in this, could you upgrade to 1.0a1 and port your code to the new syntax (i.e. use @pydra.compose.python.define instead of @pydra.mark.task)

[smoia]

Hi @tclose! We did not manage to solve the issue so far, so any help would be more than appreciated!

[tclose]

The globals()[metric] looks a bit strange to me, why can't you pass it in as a parameter? As the task will potentially get pickled, and then unpickled and run in a separate process I would expect accessing globals in this way to cause problems