Command Section

The command section is the only required task section. It defines the command template that is evaluated to produce a Bash script that is executed within the task's container. Specifically, the commands are executed after all of the inputs are staged and before the outputs are evaluated. There may be any number of commands within a command section.

There are two different syntaxes that can be used to define the command section:

# HEREDOC style - this way is preferred
command <<< ... >>>

# older style - may be preferable in some cases
command { ... }

The command template is evaluated after all of the inputs are staged and before the outputs are evaluated. The command template is evaluated similarly to multi-line strings:

1. Remove all whitespace following the opening <<<, up to and including a newline (if any).
2. Remove all whitespace preceeding the closing >>>, up to and including a newline (if any).
3. Use all remaining non-blank lines to determine the common leading whitespace.
4. Remove common leading whitespace from each line.
5. Evaluate placeholder expressions.

Notice that there is one major difference between the evaluation of multi-line strings vs the command template: line continuations are removed in the former but left as-is in the latter. This also means that continued lines are considered when determining common leading whitespace, and that common leading whitespace is removed from continued lines as well.

String s = <<<
  This string has \
  no newlines
>>>

command <<<
  echo "~{s}"
  echo "This command has line continuations \
    that still appear in the Bash script \
    after evaluation"
>>>

When the above command template is evaluated the resulting Bash script is:

echo "This string has no newlines"
echo "This command has line continuations \
  that still appear in the Bash script \
  after evaluation"

For another example, consider a task that calls the python interpreter with an in-line Python script:

task heredoc {
  input {
    File infile
  }

  command <<<
    python <<CODE
      with open("~{in}") as fp:
        for line in fp:
          if not line.startswith('#'):
            print(line.strip())
    CODE
  >>>
  ....
}

Given an infile value of /path/to/file, the execution engine produces the following Bash script, which has removed the 4 spaces that were common to the beginning of each line:

python <<CODE
  with open("/path/to/file") as fp:
    for line in fp:
      if not line.startswith('#'):
        print(line.strip())
CODE

Each whitespace character is counted once regardless of whether it is a space or tab, so care should be taken when mixing whitespace characters. For example, if a command block has two lines, and the first line begins with <space><space><space><space>, and the second line begins with <tab> then only one whitespace character is removed from each line.

The characters that must be escaped within a command section are different from those that must be escaped in regular strings:

• Unescaped newlines (\n) are allowed.
• An unescaped backslash (\) may appear as the last character on a line - this is treated as a line continuation.
• In a HEREDOC-style command section, if there are exactly three consecutive right-angle brackets (>>>), then at least one of them must be escaped, e.g. \>>>.
• In the older-style command section, any right brace (}) that is not part of an expression placeholder must be escaped.

§Expression Placeholders

The command "template" can be thought of as a single string expression, which (like all string expressions) may contain placeholders.

There are two different syntaxes that can be used to define command expression placeholders, depending on which style of command section definition is used:

Note that the restriction on using ${} only applies to the HEREDOC-style command section - it may be used interchangeably with ~{} in string expressions including multi-line strings.

Any valid WDL expression may be used within a placeholder. For example, a command might reference an input to the task. The expression can also be more complex, such as a function call.

Example: test_placeholders_task.wdl

version 1.2

task test_placeholders {
  input {
    File infile
  }

  command <<<
    # The `read_lines` function reads the lines from a file into an
    # array. The `sep` function concatenates the lines with a space
    # (" ") delimiter. The resulting string is then printed to stdout.
    printf "~{sep(" ", read_lines(infile))}"
  >>>
  
  output {
    # The `stdout` function returns a file with the contents of stdout.
    # The `read_string` function reads the entire file into a String.
    String result = read_string(stdout())
  }
}

{
  "test_placeholders.infile": "data/greetings.txt"
}

{
  "test_placeholders.result": "hello world hi_world hello nurse"
}

In this case, infile within the ~{...} placeholder is an identifier expression referencing the value of the infile input parameter that was specified at runtime. Since infile is a File declaration, the execution engine will have staged whatever file was referenced by the caller such that it is available on the local file system, and will have replaced the original value of the infile parameter with the path to the file on the local filesystem.

In most cases, the ~{} style of placeholder is preferred, to avoid ambiguity between WDL placeholders and Bash variables, which are of the form $name or ${name}. If the command { ... } style is used, then ${name} is always interpreted as a WDL placeholder, so care must be taken to only use $name style Bash variables. If the command <<< ... >>> style is used, then only ~{name} is interpreted as a WDL placeholder, so either style of Bash variable may be used.

Example: bash_variables_fail_task.wdl

version 1.2

task bash_variables {
  input {
    String str
  }
  
  command {
    # store value of WDL declaration "str" to Bash variable "s"
    s=${str}
    # echo the string referenced by Bash variable "s"
    printf $s
    # this causes an error since "s" is not a WDL declaration
    printf ${s}
  }
}

{
  "bash_variables.str": "hello"
}

{}

{
  "fail": true
}

Like any other WDL string, the command section is subject to the rules of string interpolation: all placeholders must contain expressions that are valid when analyzed statically, and that can be converted to a String value when evaluated dynamically. However, the evaluation of placeholder expressions during command instantiation is more lenient than typical dynamic evaluation as described in Expression Placeholders.

The implementation is not responsible for interpreting the contents of the command section to check that it is a valid Bash script, ignore comment lines, etc. For example, in the following task the greeting declaration is commented out, so greeting is not a valid identifier in the task's scope. However, the placeholder in the command section refers to greeting, so the implementation will raise an error during static analysis. The fact that the placeholder occurs in a commented line of the Bash script doesn't matter.

Example: bash_comment_fail_task.wdl

version 1.2

task bash_comment {
  # String greeting = "hello"

  command <<<
  # printf "~{greeting} John!"
  >>>
}

{}

{}

{
  "fail": true
}

§Stripping Leading Whitespace

When a command template is evaluated, the execution engine first strips out all common leading whitespace.

For example, consider a task that calls the python interpreter with an in-line Python script:

Example: python_strip_task.wdl

version 1.2

task python_strip {
  input {
    File infile
  }

  command<<<
    python <<CODE
    with open("~{infile}") as fp:
      for line in fp:
        if not line.startswith('#'):
          print(line.strip())
    CODE
  >>>

  output {
    Array[String] lines = read_lines(stdout())
  }

  requirements {
    container: "python:latest"
  }
}

{
  "python_strip.infile": "data/comment.txt"
}

{
  "python_strip.lines": ["A", "B", "C"]
}

Given an infile value of /path/to/file, the execution engine will produce the following Bash script, which has removed the two spaces that were common to the beginning of each line:

python <<CODE
with open("/path/to/file") as fp:
  for line in fp:
    if not line.startswith('#'):
      print(line.strip())
CODE

If the user mixes tabs and spaces, the behavior is undefined. The execution engine should, at a minimum, issue a warning and leave the whitespace unmodified, though it may choose to raise an exception or to substitute e.g. 4 spaces per tab.