Life after Nextflow+Apache Ignite. Alternatives?

[dgarrimar]

Hi!

A very simple, widely used Nextflow pipeline is the following:

1. Split a large input file into chunks
2. Run a specific tool for each chunk
3. Collect the results again in the output file

The speed-up achieved increases -obviously- with the number of chunks that can run in parallel (i.e. in an HPC environment, the number of cores/cpus available).

However, in many HPC clusters (e.g. MareNostrum), the maximum number of jobs that can be running/queuing simultaneously is limited (which is reasonable, they aim to schedule large, highly parallelized jobs).

Until recently, I used Nextflow + Ignite to "convert" a large set of jobs (e.g. 10000 jobs using 1 CPU each) to a single job (e.g. 1 job using 10000 CPUs). Ignite was "virtualizing" a cluster using the resources and internally sending the jobs one by one (1 job/cpu).

As Ignite is not supported anymore in NF, and working with very old NF versions that include it starts to give problems too, I wonder whether there is an alternative in current NF versions to do this specific task.

Thanks a lot!

Diego

[bentsherman]

The nf-ignite still exists and could be maintained if people are willing to maintain it.

Aside from that, there is the array directive for submitting job arrays, and #3909 will add a similar batch directive for task batching, which I think is closer to what you're asking

[pditommaso]

Array does not help with HCP like Marenostrum which requires re-allocating the nodes (MPI style)

[dgarrimar]

After some suggestions in Slack (thanks to @ewels) and some testing (with a lot of help from @emi80), it seems that HyperQueue does a really good job to address this type of scenario!. It also has the advantage of fewer dependencies/easier installation and usage with respect to Flux, particularly in MN5 nodes without internet access.

[pditommaso]

Interesting! I read nice thing about HyperQueue, great to hear it's working in the BSC scenario