Allow access to files in workflow directory from task commands

[jlumpe]

I have several tasks where the output from the core tool I'm using needs a little post-processing to pull out the values I need for the task's output. The way I'm handling this is in a little Python code in a heredoc in the task's command, like so:

task mytask {
    input {...}
    command <<<
        mytool -o output

        python <<CODE
            # Read output file, pick out important data and write to processed.json
        CODE
    >>>

    output {
        Map[String, String] o = read_json("processed.json")
    }
}

This seems to be a common thing, and the Python section can definitely get kind of long. It would be nicer if I could write this code in a separate Python file included with my other .wdl files, and then run it like:

task mytask {
    input {...}

    command <<<
        mytool -o output

       python ~{ workflow_file("scripts/process_mytool_output.py") }
    >>>

    output {...}
}

[pshapiro4broad]

The syntax you have isn't quite correct but I think the idea should work. Something like

task mytask {
    input {
        File process_mytool_output
        ...
    }
    command <<<
        mytool -o output
        python ~{process_mytool_output}
    >>>
    ...
}

In my experience a more typical approach is to create a custom container that contains your script as well as the commands that you need to run in your command block. In the case where you need to run mytool and python already maybe you're already building a custom container.

[jlumpe]

My experience is admittedly fairly limited so far, but if you have some basic scripts which exist entirely for gluing together parts of the workflow it seems much preferable to include them with the other workflow files instead of having to create new workflow-specific containers that are just a common tool plus your one script added.

[rhpvorderman]

Can you give a more specific example as to what kind of glueing is needed? I am co-maintainer of the biowdl tasks respository and we have no tasks where such extensive output processing is needed.

[rhpvorderman]

For clarification, above answer is not meant as a dismissal. I merely meant to say that my team and I have a lot of experience gluing tasks together and that maybe there is something that I can suggest based on your actual use case.

[jlumpe]

In this specific case, it is parsing a JSON output of one of my own tools. Some specific (nested) values are being pulled out and written to files, which are then put in the task's output using the read_{float,string,...}() functions.

I'm actually not sure, but in my specific case I don't think that there's a better way to pull multiple values out of a JSON file like this. I know there is a function to read a JSON file, but I don't think it's possible to declare a temporary variable that runs after the command section? Otherwise I would have to read the JSON file repeatedly for each output variable, which seems pretty inelegant.

Anyways, I can definitely see it being the case that in many common bioinformatics workflows the tools are already designed to work with each other. But it also seems like it should be common enough that you need to do a little custom processing in between.

[patmagee]

Interesting Idea. For me, one of the great aspects (potentially also easily abused) of WDL is that it allows for arbitrarily complex scripts to run in the command block. Because of this, I don't really view there being a "main" tool, but a unit of work which COULD be a single command line call, or it could be a whole script. whatever you ultimately need to do in order to extract the output you need.

[jlumpe]

Supporting scripts as a separate file would also make testing them locally during development a lot easier.