Submit multiple jobs to a single node, rather than scheduling one job per node

[kyleoconnell]

Hello,

I am running snakemake on HPC toolkit, using the default deployment (https://github.com/GoogleCloudPlatform/hpc-toolkit/blob/main/examples/README.md#hpc-slurmyaml-).

When I submit jobs, it submits one job to each node, and using the compute nodes, my 1-2 CPU jobs are spinning up and spinning down an n2-standard-60, which is very slow. I would like to submit multiple jobs to a single compute node until it is maxed out, then spin up a new compute node. This is how our on prem slurm system works and I would like to replicate it in GCP. Any idea what I need to change in the config to enable this?

Thanks,
Kyle

[nick-stroud]

A few clarifications:

1. Are you using the exact blueprint referenced or have you made some modifications. I ask because you mention it is spinning up n2-standard-60 but I don't thing that machine type is found in that blueprint. The compute_node_group will default to c2-standard-60.

2. What is the command you are using to submit your job? This may be obvious but depending on the arguments your job could be claiming a whole node even if it doesn't need it. For example srun -n1 requests 1 task, srun -N1 requests 1 whole node.

3. Is the partition exclusive? In the blueprint you reference, the debug_partition is non-exclusive and the compute_partition is exclusive (default value). This is denoted by the settings.exclusive value. When a partition has settings.exclusive: true (default), then every job will receive newly created nodes, which will be torn down after the job completes.

With the right combination of settings.exclusive: false and the correct slurm parameters, it should be possible to achieve your goal.

Last thing, you could also consider using a different machine shape that matches your job better using the settings.machine_type parameter on the node group.

[kyleoconnell-NOAA]

1. Are you using the exact blueprint referenced or have you made some modifications. I ask because you mention it is spinning up n2-standard-60 but I don't thing that machine type is found in that blueprint. The compute_node_group will default to c2-standard-60.
   Yes that was just a mistake on my part.

2. What is the command you are using to submit your job? This may be obvious but depending on the arguments your job could be claiming a whole node even if it doesn't need it. For example srun -n1 requests 1 task, srun -N1 requests 1 whole node.

I was using a Snakemake profile found here

3. Is the partition exclusive? In the blueprint you reference, the debug_partition is non-exclusive and the compute_partition is exclusive (default value). This is denoted by the settings.exclusive value. When a partition has settings.exclusive: true (default), then every job will receive newly created nodes, which will be torn down after the job completes.

Yes we have been playing with this. When we had exclusive set to false it was submitting multiple jobs to a single node but then killing the second job when the first finished. Maybe I am making a mistake with how I am submitting. I will try the -n 1

[nick-stroud]

That submission is sufficiently more complex, so I wouldn't be surprised if there was something in there that was killing the node once the first job finished.

I did run a simple reproduction on n2d-standard-8 (4 physical cores):

sbatch -n1 --wrap 'sleep 30'
sbatch -n1 --wrap 'sleep 60'
sbatch -n1 --wrap 'sleep 90'

All 3 jobs fit on the same machine and the machine stayed up until all 3 jobs were complete. I will admit that this was on a Slurm GCP v6 cluster, but I would not expect a difference in the exclusive behavior.