RFC: Runtime access to runtime attributes

[jdidion]

Within a task, it is possible to access input and intermediate parameters, but it is not currently possible to access the actual values of attributes in the runtime, hints, or meta* sections.

For example, let's say that I request 25 GB of memory using memory: 25G. When my task is running, I know that I have at least 25 GB, but I don't know how much memory I actually have. It may be useful to know the exact resources provisioned, for example so I can specify a -t parameter to bwa or an -Xmx parameter to Java. I realize that it is possible to get some of this information within the command block using standard linux commands, but there may be differences in the outputs of these commands between platforms.

I propose that we add a reserved variable name wdl to the environment of every task that can be used by expressions in declarations and in the command section (but not in expressions in the runtime section) to access attributes of the runtime environment, as well as task metadata.

command <<<
bwa mem -t ~{wdl.runtime.cpu - 1} ...
>>>

output {
  # If this task is running on DNAnexus, populate the
  # instance_type output with the actual instance type
  # that was selected to run this task, otherwise null
  String? instance_type = wdl.hints.dx.instance_type
}

Details:

• wdl.
  • runtime.
    • String? container: the image used to run the task; always has a protocol (e.g. docker://myrepo/myimg:tag); null if no container was used
    • Int cpu: actual number of cpus provisioned by the engine
    • Int memory: actual number of bytes of memory provisioned by the engine
    • Map[String, Float] disks: mapping of mount point to disk space in bytes provisioned by the engine
    • Int gpu: the number of GPUs provisioned; 0 if the engine is not able to determine if a GPU is available; 1 if the engine knows that at least 1 GPU is available, but is not able to determine exactly how many GPU are available
    • Int retries: the number of times the task has been retried; starts as 0 and is incremented by 1 each time the task is retried
    • Int? returnCode: the actual return code from the command section; the value is null except in the output section
  • hints: a copy of the attributes from the hints section
    • The engine may omit hints provided by the user and/or add extra hints at runtime; thus, attributes of hints are optional; an attempt to access one of these attributes returns null if it is not defined (rather than throwing an error)
  • meta: a copy of the attributes from the meta section
  • parameter_meta: a copy of the attributes from the parameter_meta section

Semantics:

• The wdl variable is an object
• The wdl variable is always present in the environment
• The second-level attributes of wdl (i.e. runtime, hints, meta, paramater_meta) are always defined and are objects
• The attributes of wdl.runtime are always defined. If the engine cannot determine the actual value of one of the runtime attributes, it must populate the attribute with the requested value (or the default value, if no override was specified by the user).