batch4py is a lightweight Python 3 module that provides a programmatic interface to many common High Performance Computing (HPC) batch schedulers. It also provides a simple way to define directed acyclic graphs (DAG) of job chains.
A common workflow to submit jobs on a batch scheduler looks something like this:
$ qsub job1.pbs
INFO: Job submitted to account: jq0
8625371
$ qsub -W afterany:8625371 job2.pbs
INFO: Job submitted to account: jq0
8625372
$ qsub -W afterany:8625371 job3.pbs
INFO: Job submitted to account: jq0
8625373
$ qsub -W afterany:8625372 job4.pbs
INFO: Job submitted to account: jq0
8625374
Performing these steps manually is fine for a handful of jobs, but doing it for dozens or hundreds of jobs becomes impractical. A 100-batch job that has a simple linear chain of dependence looks like this in batch4py:
job_list = []
chain = batch4py.JobChain( 'pbs' )
for i in range(100):
new_job = batch4py.job.TORQUE( list_of_job_files[i] )
job_list.append( new_job )
chain.add_job( new_job )
# Define a simple linear tree
for i in range(99):
chain.set_dep( job_list[i], job_list[i+1], 'afterany' )
# Submit job chain
chain.submit()
Users can also submit single jobs very easily:
>>> job = batch4py.job.TORQUE( '/path/to/pbs/file/batch.sh' )
>>> job.submit()
Users may even pass file literals to batch4py (whereby the batch.sh file does not exist):
>>> job = batch4py.job.TORQUE( '''#!/bin/bash
... #PBS -l nodes=1:ppn=32
... aprun -n 1 -- echo "Hello world!"
... ''')
>>> job.get_script()
'/path/to/install/dir/batch4py/pbs_files/e78d32d0-9299-4735-bbde-05fcb208b5cf.pbs'
Complete function documentation is maintained in the source code and can be accessed using Python's help() built-in.
batch4py is on PyPi and can be installed using the normal commands:
pip install --user batch4py
Currently, batch4py is only compatible with TORQUE schedulers.
Pull requests are welcome. Please submit any issues or feature requests in the GitHub Issues tracker.