Collating all files at the end of a pipeline

[tjdurant]

Hi All,

Interested in what people do in the following scenario.

Say we have 10 processes in our DSL-1 main.nf pipeline.

Currently what we are doing is, once the pipeline is complete, we run a python script that collates select data from all of the output files and puts it into an excel file for easier review by genomic analysts.

We would like to incorporate that last step into the pipeline but it seems that there are variable practices in how to do this in NF.

So, I'm curious what other people are doing for steps like this.

Thanks!
T

[bentsherman]

It depends on which outputs you need. If you only need to collect outputs published by Nextflow processes, then you can just add another process to your workflow that collects those outputs. But if you also need the reports created by Nextflow (execution report, timeline, trace, log), then you must do that outside of Nextflow itself.

The best thing you can do in the latter case is to wrap the two steps -- Nextflow pipeline and python script -- in a bash script or even a Nextflow meta-pipeline (I have done this myself).

[bentsherman]

Actually, I am trying something like this today: https://github.com/bentsherman/nf-head-job-benchmark

It's a pipeline for measuring memory usage of Nextflow itself, so each process actually executes Nextflow. But each "meta" process calls a sub workflow using the -entry option. This way I can define everything in one pipeline. It's a bit messy right now but it might be a good option for you.

[tjdurant]

Gotcha, thanks, @bentsherman

We are interested in doing the former.

Quick follow-up: If I need to wait until all of the samples to be completed within each preceding process before kicking off this last process, would I use the Collect operator?

Thanks and I also love the idea of a meta-pipeline -- when you have done that, are those meta-pipelines kicked off within the parent pipeline? Or do you do kick off the meta-pipelines outside of nextflow?

[bentsherman]

Okay, then the solution will be much simpler. A good first step is to add a final process that calls your python script and feed all of the data you want to collect into that process via channels. You're correct, use the collect operator to gather all of those files before you start the final process.

A further optimization beyond that might be to replace the python script with native groovy code in your pipeline, but it's not necessary, and Python might be the better way to do it anyway. Entirely up to you.

Regarding the meta-pipeline, you're welcome to peruse the repo I linked. It's all in Nextflow, basically I have processes that do nextflow run ... and it works quite well. In fact I just ran it in Tower via AWS Batch.

[tjdurant]

Okay, then the solution will be much simpler. A good first step is to add a final process that calls your python script and feed all of the data you want to collect into that process via channels. You're correct, use the collect operator to gather all of those files before you start the final process

Awesome, thanks for that clarification. Much appreciated!

A further optimization beyond that might be to replace the python script with native groovy code in your pipeline, but it's not necessary, and Python might be the better way to do it anyway. Entirely up to you.

Thanks for mentioning this as well. I have been noticing with the pipelines on nf-core that they often have that native groovy approach. Is there a specific benefit to going that route?

Regarding the meta-pipeline, you're welcome to peruse the repo I linked. It's all in Nextflow, basically I have processes that do nextflow run ... and it works quite well. In fact I just ran it in Tower via AWS Batch.

Great to hear. We're in the process (albeit a slow one) of adopting NFT, so that is very useful to know.

Thanks again for the discussion!

[bentsherman]

Thanks for mentioning this as well. I have been noticing with the pipelines on nf-core that they often have that native groovy approach. Is there a specific benefit to going that route?

A native solution keeps everything in the same language, which might be more readable and efficient. Of course Groovy code isn't always easy to read. 😄 It's really just a matter of preference. I imagine the nf-core folks try to do more things in native code so that there are good examples of using native code in general, since nf-core is largely about codifying best practices.