Context-aware retries using previous resource requirements.

[claymcleod]

It would be really useful to reference parameters from previous runs upon retry. For example, if you'd like to keep retrying a job up until a particular limit until it succeeds, it would could theoretically be expressed like this.

task foo {
  command <<<
    echo "do some work"
  >>>

  requirements {
    memory: if task.iteration > 0 {
      task.previous.memory * 2
    } else {
      # 2 GB
      2000000000
    }
  }
}

In other words, if this is the first iteration of the task, start with 2 GB. Otherwise, take the previous task requirement and double it. I'm curious if this is something that might be generally useful when building robust task implementations.

Open Questions

• Should only the last task requirement be kept as task.previous (or similar), or should there be, say, an array of task requirements from previous attempts?

[claymcleod]

I realized upon a closer reading of the specification that this is already supported through the attempt variable in the task scope: https://github.com/openwdl/wdl/blob/wdl-1.2/SPEC.md#runtime-access-to-requirements-hints-and-metadata.

[claymcleod]

Opening this again because @peterhuene pointed out to me that task is only accessible in output and command blocks.

[peterhuene]

To fix this, I suggest we expand the hidden task scope to include evaluation of requirements, hints, and runtime sections.

Additionally, we change the types of the following task variable fields to optional with an initial value of None while evaluating the requirements, hints, and runtime sections:

• container -> String?
• cpu -> Float?
• memory -> Int?
• gpu -> Array[String]?
• fpga -> Array[String]?
• disks -> Map[String, Int]?

Personally, I think such a change would be relatively backwards compatible with what's in 1.2.0 and could be fixed in a 1.2.1 release of the spec without having to wait for 1.3.0 to enable this functionality.