Pyslurm use cases

[carlosfamilia]

I have been looking for a python library that could send jobs to slurm, however it seems documentation is a little scarce for this project, and though several examples can be found in the repository, it would be great if use cases for this library could be described.
If the developers need help on this, I can volunteer some of my time.

[brendan-m-murphy]

I have a similar question. It seems like Pyslurm is aimed more at managing SLURM from a sys admin's perspective, rather than for managing jobs. My use case is organizing jobs submitted by a group of researchers for a particular project, so I'd like to tie job ID's to other metadata. I'm currently using subprocess to run sbatch, sacct, squeue and seff to get the info I need. Pyslurm sounds like the thing I should use instead, but it's not clear if that's true.

[tazend]

Hi,

basically, the goal of pyslurm is to make the public C-API of Slurm available for use with Python, trying to provide an interface that is both easy to use for users, while not trying to deviate too far from the original C-API. Essentially, pyslurm wraps C-API functions and gives you access to members of the underlying C-structs, by exposing them as methods and simple attributes in a class for each category, respectively.
Sometimes members of structs in the C-API have names that might not make much sense for users, so we try to adjust here in the python-API by a giving a better name.

By having access to all the attributes and functions for a specific category in Slurm (like Jobs), it gives you the foundation to do whatever slurm commands like sbatch, sacct, squeue, scontrol show job or seff for example are doing.

Right now, I think around 95% of parameters/flags that sbatch provides are also supported in pyslurm (sometimes with a different name to give a more descriptive name). Almost every attribute that can be queried from a Job (via squeue or scontrol show job) is also supported in pyslurm. The same goes for information retrieved by sacct, which is in a dedicated class called pyslurm.db.Job

To summarize, I will link you the API Documentation for all the classes that you might want to look at to get the same functionality as when executing slurm-commands:

1. sbatch

• https://pyslurm.github.io/23.2/reference/jobsubmitdescription/

2. squeue or scontrol show job

• https://pyslurm.github.io/23.2/reference/job/#pyslurm.Job
• https://pyslurm.github.io/23.2/reference/job/#pyslurm.Jobs

3. sacct and seff

• https://pyslurm.github.io/23.2/reference/db/job/#pyslurm.db.Job
• https://pyslurm.github.io/23.2/reference/db/job/#pyslurm.db.Jobs
• https://pyslurm.github.io/23.2/reference/db/jobfilter/
• https://pyslurm.github.io/23.2/reference/db/jobstats/ (some of the data here is also what seff uses)

There are also a few examples included in the Documentation there.
If you have any further questions, let me know :)

[brendan-m-murphy]

Thanks, this is really helpful! The docs with examples are particularly helpful. Some docs/examples showing the constructors for JobStatistics would be useful, but seems like there's a lot going on with the project at the moment.

[tazend]

Hi,

although you can construct a pyslurm.db.JobStatistics object yourselves, it doesn't have any specific functionality besides serving as a sort of "dataclass" that just holds attributes related to job statistics. You can access the JobStatistics instance of a pyslurm.db.Job directly via its stats attribute, like this:

import pyslurm
db_job = pyslurm.db.Job.load(9999)
print(db_job.stats.elapsed_cpu_time)
print(db_job.stats.total_cpu_time)
...