running nextfow pipeline in slurm using srun and mpi support

[BioInf2305]

Hi,

Using a nextflow pipeline (developed in 22.10.6), I want to run ~4000 jobs; these jobs each can take 1-2 days depending on the coverage. I intend to run this workflow on the Linux clusters of our university. This HPC has a standard set-up: a login node should only be used to submit jobs to one of the several clusters and partitions. Now depending on the queue time and limitations associated with policies of the HPC administration, the pipeline completion can take at least a few weeks. Further, except for the login node, all the other queues support the job spawning with mpi or srun from the sbatch file. In this pipeline, I am using executor as "slurm" (in nextflow config file) with computational resources mentioned in base.config. In the past, I have run this pipeline (with 50-100 jobs) by directly running nextflow command in "screen" from the login node (though it is not recommended way in our HPC system).

I would highly appreciate if anyone can answer these questions: (1) how can I wrap "nextflow run" command in a shell script and submit it as any other job, (2) can I run nextflow job with mpi and srun; based on my extensive search in the past issues of this github repository, I came to know that it was possible to do so in the past releases of nextflow (20.XX), however, current version does not support it anymore.

Thanks.

[tbugfinder]

As far as I understand, your nextflow workflow is a standard one.
Are you already submitting processes to the Slurm cluster?

[BioInf2305]

Thanks for the response. Yes, of course, I simply run this command on screen from login node : nextflow run -qs 50 main.nf -profile cluster --with-conda, However, I really want to know how the community is using it without screen! And if it is possible to specify the number of nodes in the workflow.

[nmendozam]

I use this guide to run my pipelines, and modifying the required resources in the sbatch script.

[BioInf2305]

Thanks for the response. I already tried very similar slurm script on our HPC and failed because the problem is executor "slurm" in nextflow uses "sbatch" to submit the jobs and this is problematic in those HPCs where only "login" node are allowed to "sbatch" to computing node, i.e. nextflow cannot spawn the jobs from computing node. Therefore, only option that I managed to find out is using "nextflow run" in either "screen" or "tmux" on the "login" node.

[mpjw]

@BioInf2305 have you tried wrapping the nextflow pipeline in a sbatch job an then not using the SLURM executor?
This way nextflow would execute the pipeline locally on the allocated node.
Of course this depends on the time, cpu and ram limits of your HPC.
Also, I have only tried this for one node, don't know if it will work with multiple allocated nodes.

[birkhoff2017]

However, it’s never a good practice to launch a long running job in the login node, and therefore a good practice consists of running Nextflow itself as a cluster job. This can be done by wrapping the nextflow run command in a shell script and submitting it as any other job. An average pipeline may require 2 CPUs and 2 GB of resources allocation.

Note: the queue where the Nextflow driver job is submitted should allow the spawning of the pipeline jobs to carry out the pipeline execution.