YAML File Structures

YAML File Structures

This is an example of a minimal Hyperion system configuration:

name: Example-Config
env: env/example_env.sh

groups:
- name: Example Group 1
  components:
  - name: xclock
    cmd:
    - start: xclock
    - check: pgrep xclock > /dev/null
    host: example-host-1

The above example describes a system named Example-config with a custom environment file example_env.sh at a relative path. It contains only one component group named Example Group 1 which holds a single component named xclock to be run on a host referenced by hostname example-host-1. On start the component will run the shell command xclock and the check command of the component searches for a xclock process piping the output to /dev/null.

The mandatory fields of a system configuration are the name and a list of groups holding at least one group that contains a single component. The optional field are the following:

• the env entry: holds the path to a custom environment file (both relative to the system file or absolute path will work) which is sourced before component commands and for parsing environment variables in the configuration.
• the shell_path entry: used to specify the shell executable by path. If none is given defaults to /bin/bash.
• the verbose_checks entry: Sets the Boolean value for verbose checks. If verbose checks are enabled, output of custom check commands is logged. Defaults to False.
• the exclude entry: holds a list of tags. All components tagged with a tag contained in the exclude list will be ignored when components are loaded.
• the monitoring_rate entry: Defines at which rate component and host availability checks are performed. This defaults to 1Hz.
• the local_monitor and remote_monitor entries: Can be used to enable/disable local/remote stat monitoring (average load, cpu and memory usage). These options are currently enabled by default.
• the local_stat_rate and remote_stat_rate entries: Can be used to specify the rate at which local/remote stats are being monitored. Both default to 1Hz

A component group consists of a (preferably short) name and a list of components. It has no optional fields yet.

The component configurations are the files that hold the most information about system configurations: It requires a name, a command list (cmd) and a host to be valid. The only mandatory entry in thecmd list is start, which defines the command to be run on component startup. A check command can be provided that will be run as second stage of a component check (first stage is checking if the child process of the tmux window (the invoked command) is still running) and a stop command is also optionally run at component shutdown. Note that currently this is the only part of the configuration where the order matters: start needs to be the first and check needs to be the second element of the list! UPDATE: since 3aca6d9 order of component cmds does not matter anymore.

Since 13f0cb5 it is possible to use environment variables as hostnames to make the configuration more dynamic. You only need to use ${your_env_var} and the parser will replace it with the value in your environment.

Multiline commands are also supported by the yaml format, you only have to use folding style or literal style.

An optional top-level field of a component is wait. It defines the time in seconds to wait after a component start for a successful component check until it is considered to have failed (note that a component start will return as soon as it receives a successful check status or the component wait time is reached). The default value used for wait is defined in config.py (currently 3 seconds).

Tagging components is possible with the tags field. It holds a list of tags associated with the component. Tagging components at this point only affects behavior in combination with the exclude feature for system configurations.

To model dependencies between components two optional fields are at hand: provides and requires (since 2.0.0-alpha): The provides field lists a set of virtual resources that are provided by the component, while requires field: lists a set of virtual resources that are required for the component to be run. For a configuration to be valid, the set of all combined requires has to be a subset of all combined provides entries. Two components are allowed to provide the same resource, if this is the case they both are direct dependencies of the requiring component. This can be used in combination with the exclude tags feature in order to make dynamic configurations that apply to different usecases (simulation, real robot, ...) and only need a change on the top-level system config to switch between them.

The yet unreleased followup version of 2.1.0 introduced optional-requires, a configuration field for a component that holds a list of optional requirements. Components providing those will be listed as dependency of said component and if no component at all provides a specified optional requirement, the requirement is ignored.
This allows to model a consistent hierarchy of dependencies that does not break if some components in between may be excluded using the exclude components feature.

The last optional field noauto was added in 5712297. It marks the component to never start unless it specifically is instructed by the user to do so. This can be used for tool components that are not necessary for a system to run, but still are handy to have in the user interface for occasional quick access (teleoperation, sim control, ...). To mark a component with noauto you simply have to add the field to the file. A value is not required but adding some symbolic value (like True) could enhance clarity of what is happening there.
Note: It is highly discouraged to make any component dependent on a component marked with noauto, because it will mess up the startup process (especially if the dependent component is not marked with noauto: the start_all procedure cannot be successful in that case!)

Component Example

name: Component name
cmd:
  - start: start shell command
  - check: |
      This is a literal style multiline
      check shell command
      If this does not work, check the indentation levels
tags:
  - tag1
  - tag2
  - ...
requires:
  - dependency1  
  - ...
provides:
  - base_system
  - ...
optional-requires:
  - something_optional
wait: N seconds 
noauto: This value is not necessary
host: hostname or ${env_var}

ATTENTION

Tee is used to redirect the stdout and stderr of a component process to a log file, which causes to change the buffering mode of the filedescriptors, thus if buffering is not specified by the executed command, the output will not be buffered by line. This also changes the buffering on the tty (output is only shown after the process finished running) which is bad if you want to keep track of a components state by taking a look at the stdout at runtime. This can be circumvented by adding stdbuf -oL -eL in front of your (last) start command (this will enforce linebuffering for stdout and stderr). This command is included in GNU coreutils so it is preinstalled on almost any linux machine.
(Issue #38 tackles this problem)

Currently checks are run via the subprocess module, which is an easy to use interface for creating shells and sending commands to them. Since aliases are only enabled in interactive shells and the process of spawning an interactive shell for every arbitrary shell is not that simple, aliases can not be used in check commands. For more information about this visit issue #40

ATTENTION

The parser allows splitting configurations in multiple files, which not only enhances clarity but also allows reusing components for different system configurations.

To include a yaml file inside another yaml file use the !include relative/path/to/file.yaml markup. Sadly pyyaml needs the include to be inside a field to allow appending entries in the including file. So something like this won't work:

!include nested.yaml
name: ¯\_(ツ)_/¯