Replace file system as source for status data to enable at scale compute tasks

[PeteClapham]

New feature

Hi! Thanks for using Nextflow and submitting the proposal
for a new feature or the enhancement of an existing functionality.

Please replace this text providing a short description of your
proposal.

Usage scenario

(What's the main usage case and the deployment scenario addressed by this proposal)

Large scale compute tasks that run in parallel (i.e. 10,000 tasks running concurrently) currently created many many small data files that are required to manage the workflow status and enable a workflow restart.

As parallel workloads scale over large clusters or cloud environments, this creates issues with data management, IO bottlenecks and lock contention, all of which impacts and impedes data analysis at scale.

Suggest implementation

(Highlight the main building blocks of a possible implementation and/or related components)

By managing status data within a data base structure, ideally of a resilient structure, the impact of the small data packages will be significantly improved and managed in a single service location.

Various databases support resilient infrastructure, i.e. MongoDB, MySQL, PostgreSQL etc etc. Would it be reasonable to push data into one such backend and ideally manage connection pooling to remove or reduce the overhead of establishing new connections etc ?

[pditommaso]

Using a DB would not remove the need for having the job script (and wrapper) to be written into the file system, otherwise, how would you execute to the batch scheduler?

[PeteClapham]

Hi Paolo, Whilst you are absolutely right, a database would not remove the need for scripts to bring about suitable execution of tasks, it would take the place of using the file system to hold small files that contain the state of running jobs. It is these state files that whilst small individually, become very large in number. We have been asked, for example, to increase inode quotas to 10’s of millions of files for multiple individual users, which is both rather high for a POSIX filesystem and takes a significant time to re-read should processes fail or require a clean-up. A database structure to hold the state of workflows would significantly improve the scalability where there is an increasingly high number of parallel tasks in flight, i.e. GATK joint calling, where 10’s of thousand tasks may reasonably be running concurrently. Each task generates a significant amount of state data in small files and impacts scalability for the reasons mentioned above. As Sendu also mentioned, a database, both collates the state data into a single target and is far more effective, in general, at coping with high frequency requests, hence improving the scalability of the solution. I hope this helps to clarify the request. Regards Pete

…

-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

[pditommaso]

Hi Pete, for what files are you suggesting a database, command input/output files? nextflow control files ie .command.*? or both?

[PeteClapham]

During the course of a job run, the TES and applications provide information as to the current state of the job in flight. This enables restarts and workflow / job management which is both a core component of NF and also becomes essential at scale.

With many jobs in flight concurrently, placing the individual job state files to disk increases the load on the backend storage systems in a manner that eventually reduces the ability for the backends to respond in a timely manner to the jobs themselves. This subsequently limits the number of jobs and can significantly increase the number of files that services need to support.

Comparison examples include:
ehive, which maintains it's job state within an SQL database and Cromwell which also provides the option to store state in an SQL backend. e.g. https://cromwell.readthedocs.io/en/stable/Configuring/ (search for SQL).

I hope this helps to separate out the need to keep binaries and general ref files on disk and the desire to keep job state separately on a backend database, either as a default option or as an option.

Thanks
Pete

[pditommaso]

Hi Pete, thanks for your reply and bringing the experience with other projects, but I still not seeing exactly how to fit a DB in NF model.

In your proposal what would be stored in the DB? command input/output files? nextflow control files ie .command.*? or both?

[danielecook]

Hey @pditommaso -

What about a way for picking up from an intermediate step even when work directories have been removed? This would be quite useful if you are continually processing data for a large analysis but need to clean up scratch space.

For example:

trim_fastqs → align → merge → output bam

This is not a trivial feature, but the idea is that nextflow would retain a database of storeDir files with information on their data lineage, and use this before checking workdirs.

When rerunning the pipeline, during DAG construction, nextflow could check whether an output file had previously been run through all the steps rather than relying on work directories.

Another benefit to this approach: You could develop a way for nextflow to automatically clean up work directories once a storeDir file has been successfully created.

[pditommaso]

@danielecook Yes, that would be quite useful. There's an open issue for that #452 even tho it's more complicated than expected.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.