rkt can run multiple applications in a pod, under a supervising process and alongside with a sidecar service which takes care of multiplexing its I/O toward the outside world.
Historically this has been done via systemd-journald only, meaning that all logging was handled via journald and interactive applications had to re-use a parent TTY.
Starting from systemd v232, it is possible to connect a service streams to arbitrary socket units and let custom sidecar multiplex all the I/O.
This document describes the architectural design for the current logging and attaching subsystem, which allows custom logging and attaching logic.
In order to be able to attach or apply custom logging logic to applications, an appropriate runtime mode must be specified when adding/preparing an application inside a pod.
This is done via stage0 CLI arguments (--stdin, --stdout, and --stder) which translate into per-application stage2 annotations.
This mode results in the application having the corresponding stream attached to the parent terminal.
For historical reasons and backward compatibility, this is a special mode activated via --interactive and only supports single-app pods.
Interactive mode does not support attaching and ties the runtime to the lifetime of the parent terminal.
Internally, this translates to an annotation at the app level:
{
"name": "coreos.com/rkt/stage2/stdin",
"value": "interactive"
},
{
"name": "coreos.com/rkt/stage2/stdout",
"value": "interactive"
},
{
"name": "coreos.com/rkt/stage2/stderr",
"value": "interactive"
}
In this case, the corresponding service unit file gains the following properties:
[Service]
StandardInput=tty
StandardOutput=tty
StandardError=tty
...
No further sidecar dependencies are introduced in this case.
This mode results in the application having the corresponding stream attached to a dedicated pseudo-terminal.
This is different from the "interactive" mode because:
- it allocates a new pseudo-terminal accounted towards pod resources
- it supports external attaching/detaching
- it supports multiple applications running inside a single pod
- it does not tie the pod lifetime to the parent terminal one
Internally, this translates to an annotation at the app level:
{
"name": "coreos.com/rkt/stage2/stdin",
"value": "tty"
},
{
"name": "coreos.com/rkt/stage2/stdout",
"value": "tty"
},
{
"name": "coreos.com/rkt/stage2/stderr",
"value": "tty"
}
In this case, the corresponding service unit file gains the following properties:
[Service]
TTYPath=/rkt/iomux/<appname>/stage2-pts
StandardInput=tty
StandardOutput=tty
StandardError=tty
...
A sidecar dependency to [email protected] is introduced in this case. Application has a Wants= and After= relationship to it.
This mode results in the application having each of the corresponding streams separately handled by a muxing service.
This is different from the "interactive" and "tty" modes because:
- it does not allocate any terminal for the application
- single streams can be separately handled
- it supports multiple applications running inside a single pod
Internally, this translates to an annotation at the app level:
{
"name": "coreos.com/rkt/stage2/stdin",
"value": "stream"
},
{
"name": "coreos.com/rkt/stage2/stdout",
"value": "stream"
},
{
"name": "coreos.com/rkt/stage2/stderr",
"value": "stream"
}
In this case, the corresponding service unit file gains the following properties:
[Service]
StandardInput=fd
Sockets=<appname>-stdin.socket
StandardOutput=fd
Sockets=<appname>-stdout.socket
StandardError=fd
Sockets=<appname>-stderr.socket
...
A sidecar dependency to [email protected] is introduced in this case. Application has a Wants= and Before= relationship to it.
Additional per-stream socket units are generated, as follows:
[Unit]
Description=<stream> socket for <appname>
DefaultDependencies=no
StopWhenUnneeded=yes
RefuseManualStart=yes
RefuseManualStop=yes
BindsTo=<appname>.service
[Socket]
RemoveOnStop=yes
Service=<appname>.service
FileDescriptorName=<stream>
ListenFIFO=/rkt/iottymux/<appname>/stage2-<stream>
This mode results in the application having the corresponding stream attached to systemd-journald.
This is the default mode for stdout/stderr, for historical reasons and backward compatibility.
Internally, this translates to an annotation at the app level:
{
"name": "coreos.com/rkt/stage2/stdout",
"value": "log"
},
{
"name": "coreos.com/rkt/stage2/stderr",
"value": "log"
}
In this case, the corresponding service unit file gains the following properties:
[Service]
StandardOutput=journal
StandardError=journal
...
A sidecar dependency to systemd-journald.service is introduced in this case. Application has a Wants= and After= relationship to it.
Logging is not a valid mode for stdin.
This mode results in the application having the corresponding stream closed.
This is the default mode for stdin, for historical reasons and backward compatibility.
Internally, this translates to an annotation at the app level:
{
"name": "coreos.com/rkt/stage2/stdin",
"value": "null"
},
{
"name": "coreos.com/rkt/stage2/stdout",
"value": "null"
},
{
"name": "coreos.com/rkt/stage2/stderr",
"value": "null"
}
In this case, the corresponding service unit file gains the following properties:
[Service]
StandardInput=null
StandardOutput=null
StandardError=null
[...]
No further sidecar dependencies are introduced in this case.
The following per-app annotations are defined for internal use, with the corresponding set of allowed values:
coreos.com/rkt/stage2/stdininteractivenullstreamtty
coreos.com/rkt/stage2/stdoutinteractivelognullstreamtty
coreos.com/rkt/stage2/stderrinteractivelognullstreamtty
All the logging and attaching logic is handled by the stage1 iottymux binary.
Each main application may additionally have a dedicated sidecar for I/O multiplexing, which proxies I/O to external clients over sockets.
Sidecar state is persisted at /rkt/iottymux/<appname> while the main application is running.
rkt attach can auto-discover endpoints, by reading the content of status file located at /rkt/iottymux/<appname>/endpoints.
This file provides a versioned JSON document, whose content varies depending on the I/O for the specific application.
For example, an application with all streams available for attaching will have a status file similar to the following:
{
"version": 1,
"targets": [
{
"name": "stdin",
"domain": "unix",
"address": "/rkt/iottymux/alpine-sh/sock-stdin"
},
{
"name": "stdout",
"domain": "unix",
"address": "/rkt/iottymux/alpine-sh/sock-stdout"
},
{
"name": "stderr",
"domain": "unix",
"address": "/rkt/iottymux/alpine-sh/sock-stderr"
}
]
}
Its --mode=list option just read the file and print it back to the user.
rkt attach --mode=auto performs the auto-discovery mechanism described above, and the proceed to attach stdin/stdour/stderr of the current process (itself) to all available corresponding endpoints.
This the default attaching mode.
rkt attach --mode=<stream> performs the auto-discovery mechanism described above, and the proceed to the corresponding available endpoints.
This is the default output multiplexer for stdout/stderr in logging mode, for historical reasons and backward compatibility.
Restrictions:
- requires journalctl (or similar libsystemd-based helper) to decode output entries
- requires a libsystemd on the host compiled with LZ4 support
- systemd-journald does not support distinguishing between entries from stdout and stderr
TODO(lucab): k8s logmode
This is the standard systemd-journald service. It is the default output handler for the "logging" mode.
iottymux is a multi-purpose stage1 binary. It currently serves the following purposes:
- Multiplex I/O over TTY (in TTY mode)
- Multiplex I/O from streams (in streaming mode)
- Attach to existing attachable applications (in TTY or streaming mode)
This component takes care of multiplexing dedicated streams and receiving clients for attaching.
It is started as an instance of the templated [email protected] service by a Before= dependency from the application.
Internally, it attaches to available FIFOs and proxies them to separate sockets for external clients.
It is implemented as a sub-action of the main iottymux binary and completely run in stage1 context.
This component takes care of multiplexing TTY and receiving clients for attaching.
It is started as an instance of the templated [email protected] service by a After= dependency from the application.
Internally, it creates a pesudo-tty pair (whose slave is used by the main application) and proxies the master to a socket for external clients.
It is implemented as a sub-action of the main iottymux binary and completely run in stage1 context.
This component takes care of discovering endpoints and attaching to them, both for TTY and streaming modes.
It is invoked by the "stage1" attach entrypoint and completely run in stage1 context. It is implemented as a sub-action of the main iottymux binary.