Workflow's nodes' output directories point to TMPDIR instead of base_dir

[tsalo]

Summary

I am trying to write tests for nipype workflows, and I would like to use get_node() to select nodes' outputs, rather than hardcoding paths to output files. However, I've noticed that the nodes' output_dir() methods point to a subfolder in the TMPDIR environment variable, rather than to subfolders in the workflow's base_dir. It seems like this is a bug.

Tagging @mattcieslak as he's been helping me debug this.

Actual behavior

node = wf.get_node("write_string").output_dir() points to temporary directory (/private/var/.../init_minimal_wf/write_string).

Expected behavior

node = wf.get_node("write_string").output_dir() would point to /some/absolute/path/init_minimal_wf/write_string (the node's subfolder in base_dir).

How to replicate the behavior

I wrote a minimal workflow with

"""A test workflow."""
import os

from nipype.interfaces import utility as niu
from nipype.pipeline import engine as pe
from nipype.interfaces.base import (
    BaseInterfaceInputSpec,
    File,
    SimpleInterface,
    TraitedSpec,
    traits,
)


class _WriteStringInputSpec(BaseInterfaceInputSpec):
    in_str = traits.Str(mandatory=True)


class _WriteStringOutputSpec(TraitedSpec):
    out_file = File(exists=True)


class WriteString(SimpleInterface):
    """Write a string to a file."""

    input_spec = _WriteStringInputSpec
    output_spec = _WriteStringOutputSpec

    def _run_interface(self, runtime):
        self._results["out_file"] = os.path.join(runtime.cwd, "out_file.txt")
        with open(self._results["out_file"], "w") as fo:
            fo.write(self.inputs.in_str)

        return runtime


def init_minimal_wf(
    mem_gb=0.1,
    omp_nthreads=1,
    name="init_minimal_wf",
):
    """A minimal workflow to reproduce a bug."""
    workflow = pe.Workflow(name=name)

    inputnode = pe.Node(
        niu.IdentityInterface(fields=["in_str"]),
        name="inputnode",
    )
    outputnode = pe.Node(
        niu.IdentityInterface(fields=["out_file"]),
        name="outputnode",
    )

    write_string = pe.Node(
        WriteString(),
        mem_gb=mem_gb,
        n_procs=omp_nthreads,
        name="write_string",
    )

    # fmt:off
    workflow.connect([
        (inputnode, write_string, [("in_str", "in_str")]),
        (write_string, outputnode, [("out_file", "out_file")]),
    ])
    # fmt:on

    return workflow


if __name__ == "__main__":
    wf = init_minimal_wf()
    wf.inputs.inputnode.in_str = "hello world"
    wf.base_dir = "/some/absolute/path/"  # *not* TMPDIR
    wf.run()
    node = wf.get_node("write_string")
    print(node.output_dir())  # this *should* point to base_dir, but points to TMPDIR instead

Script/Workflow details

Please put URL to code or code here (if not too long).

Platform details:

{'commit_hash': '<not found>',
 'commit_source': '(none found)',
 'networkx_version': '2.8.8',
 'nibabel_version': '3.2.1',
 'nipype_version': '1.8.5',
 'numpy_version': '1.24.1',
 'pkg_path': '/opt/miniconda3/lib/python3.8/site-packages/nipype',
 'scipy_version': '1.8.1',
 'sys_executable': '/opt/miniconda3/bin/python',
 'sys_platform': 'darwin',
 'sys_version': '3.8.5 (default, Sep  4 2020, 02:22:02) \n[Clang 10.0.0 ]',
 'traits_version': '6.3.2'}

Execution environment

My python environment outside container

[effigies]

@tsalo Sorry for missing this:

res = wf.run()
nodes = {node.fullname: node for node in res.nodes}
nodes['init_minimal_wf.write_string'].output_dir()

[tsalo]

Thanks @effigies! Does that mean the behavior I was seeing was expected? Or is it a bug that is easier to circumvent than fix?

In any case, I'll try out your recommendation. Do you think this info is worth incorporating into the documentation somewhere? I know most folks don't use pytest to test their nipype workflows, but knowing how to access individual nodes from a run workflow could make it more common.

[effigies]

Does that mean the behavior I was seeing was expected?

Yes. The workflow graph that you construct is basically a template for the execution graph. The execution graph is flattened from nested workflows to a flat workflow, iterables are expanded, and IdentityInterfaces are removed. The nodes in the execution graph are what are actually run and get updated.

It's definitely worth adding to the docs. If you have a use case, would you be willing to write up a small notebook that we could render?

[tsalo]

Happy to! I've noticed that a lot of the advanced examples have been offloaded to https://github.com/miykael/nipype_tutorial. Should the notebook go there, or in the nipype docs?

[effigies]

Neither location has much maintenance effort devoted to it at this point. I have permissions here though, so I guess here is better.

[tsalo]

Sounds good!

[tsalo]

I opened niflows/nipype1-examples#7, but can move it to this repo if necessary.