Documentation for local socket mode, foreground

Author: devyn

book/plugins.md

@@ -2,7 +2,7 @@
 
 Nu can be extended using plugins. Plugins behave much like Nu's built-in commands, with the added benefit that they can be added separately from Nu itself.
 
-Nu plugins are executables; Nu launches them as needed and communicates with them over [stdin, stdout, and stderr](https://en.wikipedia.org/wiki/Standard_streams). Nu plugins can use either JSON or MSGPACK as their communication encoding.
+Nu plugins are executables; Nu launches them as needed and communicates with them over [stdin and stdout](https://en.wikipedia.org/wiki/Standard_streams) or [local sockets](https://en.wikipedia.org/wiki/Inter-process_communication). Nu plugins can use either [JSON](https://www.json.org/) or [MessagePack](https://msgpack.org/) as their communication encoding.
 
 ## Downloading and installing a plugin
 
@@ -54,14 +54,9 @@ Windows:
 > register .\my_plugins\nu_plugin_cool.exe
 ```
 
-When [`register`](/commands/docs/register.md) is called:
+When registering a plugin, Nu runs it in order to ensure compatibility and to get a list of all of the commands it supports. These are then saved to the plugin file (`$nu.plugin-path`) which acts as a cache.
 
-1. Nu launches the plugin, and waits for the plugin to tell Nu which communication encoding it should use
-2. Nu sends it a "Signature" message over stdin
-3. The plugin responds via stdout with a message containing its signature (name, description, arguments, flags, and more)
-4. Nu saves the plugin signature in the file at `$nu.plugin-path`, so registration is persisted across multiple launches
-
-Once registered, the plugin is available as part of your set of commands:
+Once registered, the plugin should show up in [`plugin list`](/commands/docs/plugin_list.md) and all of its commands are available in scope:
 
 ```nu
 > help commands | where command_type == "plugin"

contributor-book/plugin_protocol_reference.md

@@ -6,15 +6,19 @@ title: Plugin protocol reference
 
 ## How Nu runs plugins
 
-Nu plugins **must** be an executable file with a filename starting with `nu_plugin_`. All interaction with the plugin is handled over standard input (stdin) and output (stdout). Standard error (stderr) is not redirected, and can be used by the plugin to print messages directly.
+Nu plugins **must** be an executable file with a filename starting with `nu_plugin_`. Plugins can run in one of two modes:
 
-Plugins are always passed `--stdio` as a command line argument. Other command line arguments are reserved for options that might be added in the future, including other communication methods. Plugins that support the protocol as described in this document **should** reject other arguments and print an informational message to stderr.
+1. Stdio mode, which **must** be supported. The plugin is passed `--stdio` as a command line argument. All interaction with the plugin is handled over standard input (stdin) and output (stdout). Standard error (stderr) is not redirected, and can be used by the plugin to print messages directly.
+
+2. Local socket mode, which **may** be supported (advertised via the [`LocalSocket` feature](#localsocket-feature)). The plugin is passed `--local-socket` as the first command line argument, and then the path of the Unix domain socket or name of the Windows named pipe to use for communication. None of the standard input or output streams are redirected, and they may all be used to interact with the user's terminal. See the [documentation](#localsocket-feature) specific to the feature for more details.
+
+Other command line arguments are reserved for options that might be added in the future, including other communication methods. Plugins that support the protocol as described in this document **should** reject other arguments and print an informational message to stderr.
 
 Immediately after spawning a plugin, Nu expects the plugin to send its encoding type. Currently two encoding types are supported: [`json`](#json) and [`msgpack`](#messagepack). The desired encoding type **should** be sent first with the length of the string as a single byte integer and then the encoding type string. That is, with C-like escape syntax, `"\x04json"` or `"\x07msgpack"`. In this document, the JSON format will be used for readability, but the MessagePack format is largely equivalent. See the [Encoding](#encoding) section for specific intricacies of the formats.
 
 Nu will then send messages in the desired encoding. The first message is always [`Hello`](#hello). The plugin **must** send a `Hello` message indicating the expected Nu version that it is compatible with, and any supported protocol features. The engine will also send a `Hello` message with its version, and any supported protocol features. The plugin **may** verify that it is compatible with the Nu version provided by the engine, but the engine will end communication with a plugin if it is determined to be unsupported. The plugin **must not** use protocol features it supports if they are not also confirmed to be supported by the engine in its `Hello` message. It is not permitted to send any other messages before sending `Hello`.
 
-The plugin **should** then receive and respond to messages until its stdin is closed.
+The plugin **should** then receive and respond to messages until its input stream is closed.
 
 Typical plugin interaction after the initial handshake looks like this:
 
@@ -41,8 +45,6 @@ After the encoding type has been decided, both the engine and plugin **must** se
 
 To be accepted, the `version` specified **must** be [semver](https://semver.org) compatible with the engine's version. "0.x.y" and "x.y.z" for differing values of "x" are considered to be incompatible.
 
-There are currently no protocol features defined, and they are only likely to be used once Nu releases versions after stabilization at "1.0.0".
-
 Plugins **may** decide to refuse engine versions with more strict criteria than specified here.
 
 Example:
@@ -57,6 +59,34 @@ Example:
 }
 ```
 
+### Features
+
+All features are maps that **must** contain at least a `name` key, and **may** contain other keys. Features that are not recognized by `name` **must** be ignored, and not cause an error. Plugins **must** only advertise support for features they implement, and **should not** determine the features they will advertise depending on the engine's `Hello` message.
+
+#### `LocalSocket` feature
+
+This feature advertises support for local socket communication, instead of stdio.
+
+Example:
+
+```json
+{
+  "name": "LocalSocket"
+}
+```
+
+When local socket communication is advertised to an engine supporting the feature, the engine will cease stdio communication and launch the plugin again with the `--local-socket` command line argument. The second argument is either a path to a Unix domain socket on Linux, Android, macOS, and other Unix-like operating systems, or the name of a named pipe (without the `\\.\pipe\` prefix) on Windows.
+
+In either case, during startup, the plugin is expected to establish two separate connections to the socket, in this order:
+
+1. The input stream connection, used to send messages from the engine to the plugin
+2. The output stream connection, used to send messages from the plugin to the engine
+
+The connections are separate in order to facilitate ownership of the streams by separate threads. After these connections are both established, the engine will remove the socket, and will not accept further connections.
+
+If local socket communication fails to initialize, the engine will abort, stop the plugin, and start it again with the stdio mode, even if the plugin supports local sockets. Whether local socket mode initialized successfully, and therefore the plugin is allowed to use stdio, can be observed when
+[`EngineInterface::is_using_stdio()`](https://docs.rs/nu-plugin/latest/nu_plugin/struct.EngineInterface.html#method.is_using_stdio) returns `false` for Rust plugins.
+
 ## Input messages
 
 These are messages sent from the engine to the plugin. [`Hello`](#hello) and [`Stream messages`](#stream-messages) are also included.
@@ -806,6 +836,48 @@ Example:
 }
 ```
 
+#### `EnterForeground` engine call
+
+Moves the plugin to the foreground group for direct terminal access, in an operating system-defined manner. This should be called when the plugin is going to drive the terminal in raw mode, for example to implement a terminal UI. It will likely be necessary for the plugin to also be running in [local socket mode](#localsocket-feature) in that case.
+
+This call responds with [`Empty` pipeline data](#pipelinedataheader-empty) on success when no action is required by the plugin. On Unix-like operating systems, if the response is [`Value` pipeline data](#pipelinedataheader-value), it contains an [`Int`](#int) which is the process group ID the plugin must join using `setpgid()` in order to be in the foreground.
+
+This call will fail with an error if the plugin is already in the foreground.
+
+The plugin **should** call [`LeaveForeground`](#leaveforeground-engine-call) when it no longer needs to be in the foreground. Note that the plugin will also automatically be removed from the foreground when the plugin call response is received, even if the plugin call returns a stream.
+
+Example:
+
+```json
+{
+  "EngineCall": {
+    "context": 0,
+    "id": 0,
+    "call": "EnterForeground"
+  }
+}
+```
+
+#### `LeaveForeground` engine call
+
+Resets the state set by [`EnterForeground`](#enterforeground-engine-call).
+
+If the plugin had been requested to change process groups by the response of `EnterForeground`, it should also reset that state by calling `setpgid(0)`, since plugins are normally in their own process group.
+
+This call responds with [`Empty` pipeline data](#pipelinedataheader-empty) on success.
+
+Example:
+
+```json
+{
+  "EngineCall": {
+    "context": 0,
+    "id": 0,
+    "call": "LeaveForeground"
+  }
+}
+```
+
 #### `EvalClosure` engine call
 
 Pass a [`Closure`](#closure) and arguments to the engine to be evaluated. Returns a [`PipelineData` response](#pipelinedata-engine-call-response) if successful with the output of the closure, which may be a stream.

contributor-book/plugins.md

@@ -6,7 +6,7 @@ title: Plugins
 
 ## Protocol
 
-Plugins are executable applications that communicate with Nu by exchanging serialized data over stdin and stdout (much in the same way VSCode plugins do). The protocol is split into two stages.
+Plugins are executable applications that communicate with Nu by exchanging serialized data over a stream (much in the same way VSCode plugins do). The stream may either be stdio, which all plugins support, or a local socket (e.g. Unix domain socket or Windows named pipe) when supported. The protocol is split into two stages.
 
 The first stage of the protocol deals with the initial discovery of the plugin. When a plugin is registered the plugin is executed and asked to reply with its configuration. Just as with commands, plugins have a signature that they respond to Nu with. Once Nu has this signature, it knows how to later invoke the plugin to do work.
 
@@ -20,7 +20,9 @@ Nu keeps a registry of plugins at the file system location defined by configurat
 
 ## Launch environment
 
-Stdin and stdout are redirected for use in the plugin protocol, and must not be used for other purposes. Stderr is inherited, and may be used to print to the terminal.
+When launched in `stdio` mode, `stdin` and `stdout` are redirected for use in the plugin protocol, and must not be used for other purposes. Stderr is inherited, and may be used to print to the terminal.
+
+When launched in `local-socket` mode, `stdin` and `stdout` can also be used to interact with the user's terminal. This is the default for Rust plugins unless `local-socket` is disabled, and can be checked for by calling [`EngineInterface::is_using_stdio()`](https://docs.rs/nu-plugin/latest/nu_plugin/struct.EngineInterface.html#method.is_using_stdio). Plugins may fall back to `stdio` mode if sockets are not working for some reason, so it is important to check this if you are going to be using `stdin` or `stdout`.
 
 Environment variables set in the shell are set in the environment of a plugin when it is launched from a plugin call.
 
@@ -791,7 +793,7 @@ Ordinarily, Nu will execute the plugin and knows what data to pass to it and how
 
 We recommend keeping the [plugin protocol](plugin_protocol_reference.md) documentation handy as a reference while reading this section.
 
-Assuming you've built the Rust plugin described above let's now run it:
+Assuming you've built the Rust plugin described above let's now run it with `--stdio` so that it communicates with us there:
 
 ```sh
 $ ./target/release/nu_plugin_len --stdio