architecture.md

Text-based Desktop Environment Architecture

• UI Concept
• Process model
• Runtime modes
• Desktop applets
• IO modes
  • DirectVT mode
  • ANSI/VT mode
    • Input
      • Unix input sources
      • MS Windows input sources
    • Output
• Desktop structure
  • Desktop objects
• Usage scenarios
  • Local usage
    • Run vtm desktop
    • Run Terminal Console standalone
    • Run a CUI application standalone
    • Run a CUI application inside the Terminal Console
  • Remote access
    • Run a standalone CUI application remotely over SSH
    • Run remote vtm desktop in DirectVT IO mode over SSH
    • Run remote vtm desktop in ANSI/VT IO mode over SSH
    • Run remote vtm desktop in DirectVT IO mode using netcat
    • Run remote vtm desktop in DirectVT IO mode using inetd + ncat
    • Local standard I/O redirection using socat
  • Standard I/O stream monitoring
• Desktop taskbar customization
• Desktop Live Panel
• Desktop objects and built-in applications

UI Concept

graph TB
  subgraph GUI[Native GUI Window]
    subgraph TUI[TUI Matrix]
      subgraph DESK[Desktop]
        direction LR
        subgraph APP1[Application]
          direction LR
          App1[Application UI]
        end
        subgraph APP2[Application]
          direction LR
          App2[Application UI]
        end
      end
    end
  end
  subgraph GUI2[Generic Text Console]
    subgraph TUI2[TUI Matrix]
      subgraph DESK2[Desktop]
        direction LR
        subgraph APP21[Application]
          direction LR
          App21[Application UI]
        end
        subgraph APP22[Application]
          direction LR
          App22[Application UI]
        end
      end
    end
  end
  subgraph GUI3[Native GUI Window]
    subgraph TUI3[TUI Matrix]
      subgraph APP33[Application]
        direction LR
        App33[Application UI]
      end
    end
  end

Loading

Vtm is a text-based application where the entire user interface is represented by a mosaic of text cells forming a TUI matrix. The resulting TUI matrix is ​​just rendered either into its own GUI window or into a compatible text console.

It can wrap any console application and be nested indefinitely, forming a text-based desktop environment.

^{Currently, rendering into a native GUI window is only available on the Windows platform; on Unix platforms, a terminal emulator is required.}

Process model

graph TB
    subgraph IE10[Generic Text Console 1]
        subgraph IE1[Input]
            direction LR
            C1[keybd, mouse, focus<br>winsize, clipboard,<br>os signals]
        end
        subgraph OU1[Output]
            TC1[scrollback<br>buffer]
        end
        subgraph CS1[Desktop Client 1]
            VTM1[vtm<br>process 1]
        end
        C1 --> CS1
        TC1 --- CS1
    end

    subgraph IE20[Generic Text Console 2]
        subgraph IE2[Input]
            direction LR
            C2[keybd, mouse, focus<br>winsize, clipboard,<br>os signals]
        end
        subgraph OU2[Output]
            TC2[scrollback<br>buffer]
        end
        subgraph CS2[Desktop Client 2]
            VTM2[vtm<br>process 2]
        end
        C2 --> CS2
        TC2 --- CS2
    end

    subgraph IE30[Generic Text Console 3]
        subgraph IE3[Input]
            direction LR
            C3[keybd, mouse, focus<br>winsize, clipboard,<br>os signals]
        end
        subgraph OU3[Output]
            TC3[scrollback<br>buffer]
        end
        subgraph CS3[DirectVT Gateway]
            VTM3[vtm<br>process 3]
        end
        C3 --> CS3
        TC3 --- CS3
    end
    subgraph APP0x[Standalone DirectVT-aware Application]
        APP01[DirectVT App4<br>app: process 9]
    end
    CS3 <-->|DirectVT I/O<br>send: Events<br>recv: Render| APP01

    subgraph NP[Connection Point]
        TS["system-wide<br>named pipe"]
    end

    subgraph SS[Desktop Server]
        VTMs[vtm<br>process 0]
        subgraph SE[Desktop Session]
            subgraph APPx[running application windows]
                subgraph APP1[DirectVT Gateway]
                    eAPP1[DirectVT App1<br>app: process 4]
                end
                subgraph APP2[DirectVT Gateway]
                    eAPP2["App2 (Teletype+App2)<br>vtm: process 5<br>app: process 6"]
                end
                subgraph APP3[DirectVT Gateway]
                    eAPP3["App3 (Terminal+App3)<br>vtm: process 7<br>app: process 8"]
                end
            end
            subgraph APPu[connected users]
                USR1["Desktop Client 1<br>Viewport<br>Taskbar"]
                USR2["Desktop Client 2<br>Viewport<br>Taskbar"]
            end
            USR1 --->|keyboard focus| APP1
            USR2 --->|keyboard focus| APP2
            USR2 --->|keyboard focus| APP3
        end
        VTMs === SE
    end

    CS1 <-->|DirectVT I/O<br>send: Events<br>recv: Render| TS
    CS2 <-->|DirectVT I/O<br>send: Events<br>recv: Render| TS
    TS === VTMs

Loading

• Vtm is a text-based application that comes with a single executable and has a number of runtime modes for running multiple instances in parallel to form the desktop environment.
• A vtm process running in Desktop Server mode creates a desktop session.
• Desktop users connect to an existing desktop session through an additional vtm process running in Desktop Client mode.
• The desktop session has a unique id coined from the platform-specific creator UID unless explicitly specified.
• Only the session creator or elevated user can access the session.
• The regular user and the elevated user are different independent users despite having the same username.
• The session allows multiple access in real time.
• Multiple connected users can share a focused application, while each user can have multiple applications focused.
• Users can disconnect from the session and reconnect later.
• Sessions with different ids can coexist independently.
• To maximize rendering efficiency and minimize cross-platform issues, along with character-oriented xterm-compatible IO mode called ANSI/VT, vtm supports an additional message-based binary IO mode called DirectVT.
• A typical console application integrates into the desktop using the DirectV Gateway window as the DirectVT connection endpoint.
  • A DirectVT-aware application directly connected to the environment can seamlessly send and receive the entire set of desktop events, as well as render themselves in binary form, avoiding expensive ANSI/VT parsing.
  • To run a non-DirectVT application, an additional vtm host process is launched in Desktop Applet mode with the Teletype Console or Terminal Console applet as a DirectVT bridge to the desktop environment.
• The desktop server can receive and execute script commands relayed from other vtm processes running on behalf of the session creator.
• In the case of a vtm process with redirected standard input, all standard input is directly relayed to the desktop server as a script command flow.

Runtime modes

Runtime mode	IO mode	Environment role
Desktop Applet	auto detected	A desktop applet of an arbitrary type running in its own process that accepts user input and renders itself. Used to place a heavy (complex) desktop object in a separate process in order to optimize desktop resource consumption.
Desktop Client	auto detected	A desktop client running in its own process that forwards user input to the desktop and renders the corresponding desktop region with a taskbar overlay.
Desktop Server	n/a command line only	The desktop environment core that manages connected users, runs desktop applications, routes user input, and forwards renders to desktop clients.
Desktop Monitor	n/a command line only	A desktop log monitor which outputs desktop session logs and relays script commands to the desktop server via piped redirection.

The runtime mode can be selected using command-line options. By default, Desktop Client mode will be used, and Desktop Server will be started in parallel if it is not running.

Desktop applets

The desktop applet type can be specified using the vtm [-r [<type>]][<args...>] command-line option. By default, vtty will be used if <args...> is specified without any <type>.

Desktop applet	Type	Host for
Teletype Console (default)	vtty	CUI applications.
Terminal Console	term	CUI applications.
DirectVT Gateway	dtvt	DirectVT-aware applications.
DirectVT Gateway with TTY	dtty	CUI applications that redirect DirectVT flow to standard IO streams and require user input via platform's TTY.

IO modes

A vtm process instance running in Desktop Client or Desktop Applet mode can operate in one of two IO modes: either ANSI/VT mode or DirectVT(dtvt) mode.

DirectVT mode

In DirectVT IO mode, vtm process multiplexes the following events:

• Keyboard
• Mouse
• Focus
• Window size
• Clipboard
• Rendering
• Process lifetime control

The DirectVT stream can be wrapped in any transport layer protocol suitable for stdin/stdout transfer, such as SSH.

ANSI/VT mode

Input

In ANSI/VT IO mode, vtm process parses input from multiple standard sources, and forwards it to the desktop server using the DirectVT transport. The set of input sources varies by platform.

Unix input sources

• STDIN
  • Bracketed paste marks \x1b[200~/\x1b[201~ are treated as the boundaries of a binary immutable block pasted from the clipboard. This immutable block is handled independently of keyboard input and forwarded to the clipboard event channel.
  • SGR mouse reporting sequences \x1b[<s;x;yM/m are redirected to the mouse event channel.
  • Terminal window focus reporting sequences \x1b[I/\x1b[O are redirected to the focus event channel.
  • Line style reporting sequences \x1b[33:STYLEp are redirected to the style event channel (current/selected line wrapping on/off, left/right/center alignment).
  • All incoming text flow that does not fall into the above categories is clusterized, forming a key pressed stream forwarded to the keyboard event channel.
• Operating system signals
  • SIGWINCH events are forwarded to the window size event channel.
  • SIGINT events are forwarded to the process lifetime control channel to perform graceful exit.
  • SIGHUP events are forwarded to the process lifetime control channel to perform graceful exit.
  • SIGTERM events are forwarded to the process lifetime control channel to perform graceful exit.
• PS/2 Mouse device (Linux VGA Console only)
  • /dev/input/mice: Received ImPS/2 mouse protocol events are decoded and forwarded to the mouse event channel.
  • /dev/input/mice.vtm (used in case of inaccessibility of /dev/input/mice)

MS Windows input sources

• ReadConsoleInput events (Win32 Console API)
  • The KEY_EVENT stream is clusterized, forming a key pressed stream forwarded to the keyboard event channel (excluding repeat modifier keys).
  • The MOUSE_EVENT stream is forwarded to the mouse event channel (excluding double clicks and idle events).
  • The FOCUS_EVENT stream is forwarded to the focus event channel.
  • The WINDOW_BUFFER_SIZE_EVENT stream is forwarded to the window size event channel.
  • The MENU_EVENT stream is interpreted using the Event.MenuEvent.dwCommandId value:
    • 0x8000: The subsequent MENU_EVENT record is forwarded to the style event channel.
    • 0x8001: Clipboard-paste block start (INPUT_RECORD Begin Mark). Subsequent KEY_EVENT records are read until the INPUT_RECORD End Mark appears, and then a whole block of chars is forwarded to the clipboard event channel.
    • 0x8002: Clipboard-paste block end (INPUT_RECORD End Mark).
• Window system-defined messages
  • WM_CREATE event is forwarded to the clipboard event channel.
  • WM_CLIPBOARDUPDATE events are forwarded to the clipboard event channel.
  • WM_ENDSESSION event is interpreted using its sub-parameter's value:
    • ENDSESSION_CLOSEAPP: Register CTRL_CLOSE_EVENT signal.
    • ENDSESSION_LOGOFF: Register CTRL_LOGOFF_EVENT signal.
    • any other non-zero: Register CTRL_SHUTDOWN_EVENT signal.
• Operating system signals
  • CTRL_C_EVENT events are form the Ctrl+C key pressed event stream forwarded to the keyboard event channel.
  • CTRL_BREAK_EVENT events are form the Ctrl+Break key pressed event stream forwarded to the keyboard event channel.
  • CTRL_CLOSE_EVENT event is forwarded to the process lifetime control channel to perform graceful exit.
  • CTRL_LOGOFF_EVENT event is forwarded to the process lifetime control channel to perform graceful exit.
  • CTRL_SHUTDOWN_EVENT event is forwarded to the process lifetime control channel to perform graceful exit.

Output

CUI applications running as external processes are instantly rendered into their host DirectVT Gateways windows running directly in the desktop server address space.

The desktop server receives and caches window bitmaps and sends incremental changes to desktop clients every tick of an internal timer.

The binary render stream received from the desktop server to output is converted by the desktop client to the format suitable for the console being used to output. The console type is detected at the desktop client startup and can be one of the following:

• XTerm-compatible terminal with truecolor support
• XTerm-compatible terminal with 256-color support (Apple Terminal)
• XTerm-compatible terminal with 16-color support (Linux VGA Console, 16-color terminals)
• Win32 Console with 16-color support (Command Prompt on platforms from Windows 8.1 upto Windows 2019 Server)

The desktop client outputs the received render to the hosting console only when the console is ready to accept the next frame.

Desktop structure

Term	Description
colored character	A character depicted with rendition attributes such as background and foreground colors.
text console	A cellular rectangular surface designed to display colored monospaced characters in cells.
text cell	A text console cell containing a colored monospaced character or its fragment.
bitmap	A rectangular block of text cells.
canvas	A rectangular buffer for bitmap output.

Internally the desktop is represented by the parent-child object tree with a single root object that maintains a desktop-wide configuration, a list of connected users, and a list of running windows. The root object broadcasts a fixed number of ticks every second to update the tree state and to do something else in sync.

Users and windows are associated with the rectangular regions where they are placed at the moment. For the connected user it is a viewport of the terminal used to connect to the desktop. For the application window it is a window rectangle itself.

Desktop has no bounds and users can navigate the desktop in any direction. For every window located outside the user viewport the navigation string apeears from the viewport center towards the window location.

Each desktop window has a canvas for the hosted object bitmap, sizing grips around the canvas, a window title at the top, and a window footer at the bottom.

The desktop window can host an object instance of an arbitrary type. The hosted object controls all the hosting window's properties.

Desktop objects

Desktop object	Description
Teletype Console teletype	A solid rectangular truecolor text canvas depicting a freely scrollable buffer of the text runs generated by an xterm-compatible parser from the standard output of an attached CUI application. It can be a very heavy object due to maintaining a scrollback buffer of arbitrary length. Not used directly in the desktop process's address space.
Terminal Console terminal	A derivative of Teletype Console with additional UI controls.
DirectVT Gateway dtvt	A lightweight truecolor text canvas depicting content received from an external dtvt-aware process.
Teletype Console dtvt‑bridge vtty	A DirectVT Gateway hosting an external standalone Teletype Console applet. It is designed to run heavy Teletype Console objects as external processes to optimize desktop resource consumption.
Terminal Console dtvt‑bridge term	A DirectVT Gateway hosting an external standalone Terminal Console applet. It is designed to run heavy Terminal Console objects as external processes to optimize desktop resource consumption.
DirectVT Gateway with TTY dtty	A derivative of DirectVT Gateway stacked with additional limited Teletype Console as a controlling terminal. It is used for CUI applications that redirect DirectVT stream to standard output and require user input via platform's TTY. Depending on activity the corresponding console became active for the user.
Tiling Window Manager tile	A window container with an organization of the hosting window area into mutually non-overlapping panes for nested windows.
Desktop Region Marker site	A transparent resizable frame for marking the specific desktop region for quick navigation across the borderless workspace.

Do not confuse the Desktop Applet names with the desktop object names, even though they are the same literally, e.g. vtty and term. Desktop objects of the same name as Desktop Applets are wrappers for heavy desktop objects that should be launched in parallel vtm processes.

Usage scenarios

Local usage

Run vtm desktop

• Run command:

  vtm

Run Terminal Console standalone

• Run command:

  vtm -r term

Run a CUI application standalone

• Run command:

  vtm </path/to/console/app...>

Run a CUI application inside the Terminal Console

• Run command:

  vtm -r term </path/to/console/app...>
  # The `vtm -r term` option means to run the Terminal Console standalone to host a CUI application.

Remote access

In general, the local and remote platforms may be different.

When the DirectVT IO mode is used, all keyboard, mouse and other input events are transmitted between hosts in a binary endianness-aware form.

The following examples assume that vtm is installed on both the local and remote sides.

Run a standalone CUI application remotely over SSH

• Remote side
  • Run SSH-server if it is not running.
• Local side
  • Run command:

  vtm -r dtty ssh user@server vtm -r vtty </path/to/console/app...>
  # The `vtm -r dtty` option means to run the next statement in DirectVT&TTY console.
  # The `ssh user@server vtm -r vtty` statement means to connect via ssh and launch the Teletype Console on the remote host.

  or

  vtm ssh user@server vtm </path/to/console/app...>

Run remote vtm desktop in DirectVT IO mode over SSH

• Remote side
  • Run SSH-server if it is not running.
• Local side
  • Run command:

  vtm -r dtty ssh user@server vtm
  # The `vtm -r dtty` option means to run the next statement in DirectVT&TTY console.
  # The `ssh user@server vtm` statement means to connect via ssh and run the vtm desktop on the remote host.

  or

  vtm ssh user@server vtm
  # The `-r dtty` option is auto added if the first command-line argument starts with `ssh` keyword.

Run remote vtm desktop in ANSI/VT IO mode over SSH

• Remote side
  • Run SSH-server if it is not running.
• Local side
  • Run commands:

  ssh user@server
  vtm

  or

  ssh -t user@server vtm
  # The ssh's `ssh -t ...` option is required to allocate TTY on remote host.

Run remote vtm desktop in DirectVT IO mode using netcat (POSIX only, unencrypted, for private use only)

• Remote side
  • Run command:

  ncat -l tcp_port -k -e vtm
  # ncat's option `-l tcp_port` specifies tcp port to listen.
  # ncat's option `-k` to keep connection open for multiple clients.
  # ncat's option `-e` to run vtm for every connected client.

• Local side
  • Run command:

  vtm -r dtvt ncat remote_ip remote_tcp_port
  # The `vtm -r dtvt` option means to run DirectVT Gateway to host ncat.
  # Note: Make sure `ncat` is installed.

Run remote vtm desktop in DirectVT IO mode using inetd + ncat (POSIX only, unencrypted, for private use only)

• Remote side
  • Install inetd.
  • Add the following line to the /etc/inetd.conf:

    tcp_port stream tcp nowait user_name /remote/side/path/to/vtm  vtm
    # `tcp_port`: tcp port to listen.
    # `user_name`: user login name.

  • Launch inetd:

    inetd
    

• Local side
  • Run command:

  vtm -r dtvt ncat remote_ip remote_tcp_port

Local standard I/O redirection using socat (POSIX only)

• Host side
  • Run commands:

  mkfifo in && mkfifo out
  vtm >out <in
  

• User side
  • Run command:

  vtm -r dtvt socat open:out\!\!open:in stdin\!\!stdout
  # Note: Make sure `socat` is installed.

Standard I/O stream monitoring

Vtm allows developers to visualize standard input/output streams of the running CUI applications. Launched in the Desktop Monitor mode, vtm will log the event stream of each terminal window with the Logs switch enabled.

Important: Avoid enabling the Logs switch in the terminal window hosting the Desktop Monitor process running, this may lead to recursive event logging of event logging with unpredictable results.

Desktop taskbar customization

The taskbar menu can be configured using a settings file ~/.config/vtm/settings.xml (%USERPROFILE%\.config\vtm\settings.xml on Windows):

<config>
    <desktop>
        <taskbar>
            <!-- <item*/> --> <!-- Clear default item list -->
            <item splitter label="Remote Access"/>

            <item id="Run remote vtm desktop in DirectVT IO mode over SSH" type=dtty cmd="ssh user@server vtm"/>
            <item id="Run console app in remote terminal over SSH"         type=dtty cmd="ssh user@server vtm -r term </path/to/console/app...>"/>
            <item id="Run console app remotely over SSH w/o extra UI"      type=dtty cmd="ssh user@server vtm </path/to/console/app...>"/>

            <item splitter label="Another Examples"/>

            <item id="Far Manager"             type=vtty cmd="far"/>
            <item id="Far Manager in terminal" type=dtvt cmd="$0 -r term far"/>

            <item id="Midnight Commander"             type=vtty cmd="mc"/>
            <item id="Midnight Commander in terminal" type=dtvt cmd="$0 -r term mc"/>

            <item id="Remote cmd in terminal over SSH" type=dtty cmd="ssh user@server vtm -r term cmd"/>
            <item id="Remote cmd over SSH"             type=dtty cmd="ssh user@server vtm cmd"/>
            <item id="Remote Far Manager over SSH"     type=dtty cmd="ssh user@server vtm far"/>
            <item id="Remote wsl over SSH"             type=dtty cmd="ssh user@server vtm wsl"/>
            <item id="Remote mc over SSH"              type=dtty cmd="ssh user@server vtm mc"/>
            <item id="Remote wsl mc over SSH"          type=dtty cmd="ssh user@server vtm wsl mc"/>
        </taskbar>
    </desktop>
</config>

Additionally, the taskbar menu of the running desktop can be configured using shell piped redirection by sending script commands to the running vtm desktop:

# Delete existing menu items
echo "vtm.del()" | vtm

# Add new menu items
echo "vtm.set(id=Term label='Terminal' type=dtvt cmd='vtm -r term')" | vtm
echo "vtm.set(id=White label='White Terminal' type=dtvt cmd='vtm -r term' cfg='<config><terminal><colors><default fgc=0xFF000000 bgc=0xFFffffff/></colors></terminal></config>')" | vtm
echo "vtm.set(id=Huge label='Huge Terminal' type=dtvt cmd='vtm -r term' cfg='<config><terminal><scrollback size=500000/></terminal></config>')" | vtm
echo "vtm.set(id=Tile label='Three Terminals' type=tile cmd='v(h(Term, White), Huge)')" | vtm
echo "vtm.set(id=cmd label='Remote cmd over SSH' type=dtty cmd='ssh user@server vtm cmd')" | vtm

# Set default menu item
echo "vtm.selected(Term)" | vtm

# Run window with terminals
echo "vtm.run(id=Tile)" | vtm

Desktop Live Panel

• Desktop Live Panel

Desktop objects and built-in applications

• Desktop objects and built-in applications