Make cluster submit script template more flexible

[renozao]

Hi,

our cluster has different queues (normal, short, long, test), with different default allocated ressources.
With the current templating system, one needs to define resource default values for each variable that appears hard-coded in the template, or provide values in the submitJobs call.

It would be nice if the specification of non-default resources (w.r.t. to the submission queue) could be generated dynamically, based on the optional values passed in submitJobs.
So the template would have a simple resources line with a single meta-tag:

#<%= resources %>

which would be formated and replaced by the suitable cluster-specific arguments, e.g. for Torque:

sprintf("#PBS -l %s=%s\n", names(resources), unlist(resources))

In fact this could even be generalised to other parameters

sprintf("#PBS %s %s=%s\n", sapply(resources, '[[', 'flag'), names(resources), sapply(resources, '[[', 'value'))

Each cluster function specification should then provide the mapping of resources to universal resource names that are found across clusters (nodes, memory, job.name, etc...).
Cluster-specific templates can still be provided, but they can also leverage the use of the resources meta-tag.

Makes sense? Is this actually alreayd impelemented?

[berndbischl]

Can you please give some concrete examples how this would be used so I dont misunderstand this?

[renozao]

This is to:

• simplify the template scripts
• not needing to hard-code resources in the template script
• enable using the default resources associated with a queue as defined by the queueing system itself, not by re-defining them in the config script or at submission time.

Currently, if I want to be able to specify resources at submission time, I need to have them in the template (torque.tmpl), with hard-coded default values in the config file (.BatchJobs.R).
These default values change with the queue, typically walltime, so if I want to dynamically switch queue I pass the queue name in the resource list, but then I need to also pass the values for each resource variable.
What I propose is that these resource flags are written on the fly (not hard-coded and requiring a value default or passed from submitJobs).