Add documentation for gRPC message and memory limits delphix#390 (del…

…phix#391)
jeffngo · Sep 13, 2021 · cea863b · cea863b
1 parent c1ee1b4
commit cea863b
Show file tree

Hide file tree

Showing 7 changed files with 42 additions and 5 deletions.
diff --git a/docs/docs/Best_Practices/.pages b/docs/docs/Best_Practices/.pages
@@ -7,3 +7,4 @@ arrange:
   - Unicode_Data.md
   - Working_with_Powershell.md
   - Scratch_Paths.md
+  - Message_Limits.md
diff --git a/docs/docs/Best_Practices/Message_Limits.md b/docs/docs/Best_Practices/Message_Limits.md
@@ -0,0 +1,32 @@
+# Message Limits
+
+There are limits on how much data can be sent back and forth between the plugin and engine at a time. There are five scenarios where this comes into play:
+1. Inputs sent from the engine to the plugin, as arguments to a [Plugin Operation](/References/Plugin_Operations.md). For example, the schema-defined `Repository` object that is provided as input to plugin operations.
+2. Outputs sent back from the plugin to the engine, as the return values from plugin operations.
+3. Exception messages and call stacks thrown by plugin code. For example, the `message` field within [User Visible Errors](/Best_Practices/User_Visible_Errors.md).
+4. Inputs sent from the plugin to the engine, as arguments to a [Platform library](/References/Platform_Libraries.md) function. For example, the `message` field that is passed to `logger.debug`.
+5. Outputs sent back from the engine to the plugin, as the return values from Platform Library functions. For example, the `stdout` resulting from a call to `libs.run_bash`.
+
+For case 1 and 2, the total size of data must be less than 4 mebibytes (4 MiB).
+
+For case 3, the total size of data must be less than 128 kibibytes (128 KiB).
+
+For case 4 and 5, the total size of data must be less than 32 mebibytes (32 MiB).
+
+The actual size of this information at runtime is dependent on how the Python interpreter chooses to represent the information, so it's not always possible to know ahead of time what the exact size will be.
+
+Here are some examples of where problems may occur:
+1. Using `libs.run_bash` to print the entire contents of a large file to stdout.
+2. Using a single `logger` command with many pages of output.
+3. Throwing an exception with a large message or stack trace.
+4. Large amount of metadata in a plugin defined schema like `Repository` or `Virtual Source`.
+
+## How to tell if the message size was exceeded
+The plugin operation or platform library callback will fail with a RPC error. The exception will look like:
+```
+Error
+Discovery of "my_plugin" failed: Plugin operation "Repository Discovery" got a RPC error for plugin "my_plugin". UNAVAILABLE: Network closed for unknown reason
+```
+
+## What to do if the maximum metadata or message size is exceeded
+Reach out to us via the [Virtualization SDK GitHub repository](https://github.com/delphix/virtualization-sdk/) for guidance.
diff --git a/docs/docs/Best_Practices/User_Visible_Errors.md b/docs/docs/Best_Practices/User_Visible_Errors.md
@@ -10,6 +10,8 @@ message | String | Description of the failure to show the end user.
 action | String | **Optional**. List of actions that the end user could take to fix the problem. If not provided, it defaults to `Contact the plugin author to correct the error.`
 output | String | **Optional**. Output or stack trace from the failure to give the end user more information so that they can self diagnose. If not provided, it defaults to the stack trace of the failure.
 
+!!! warning
+    There is a limit to how much data can be stored within the fields of a `UserError`. See [Message Limits](/Best_Practices/Message_Limits.md) for details.
 
 ## Example
 

diff --git a/docs/docs/References/Logging.md b/docs/docs/References/Logging.md
@@ -36,6 +36,8 @@ logger.setLevel(logging.DEBUG)
 
 	To avoid this complexity, add the `PlatformHandler` to the root logger. The root logger can be retrieved with `logging.getLogger()`.
 
+!!! warning
+    There is a limit to how much data can be stored within a log message. See [Message Limits](/Best_Practices/Message_Limits.md) for details.
 
 ## Usage
 Once the `PlatformHandler` has been added to the logger, logging is done with Python's [Logger](https://docs.python.org/2/library/logging.html#logger-objects) object. Below is a simple example including the basic setup code used above:

diff --git a/docs/docs/References/Platform_Libraries.md b/docs/docs/References/Platform_Libraries.md
@@ -56,7 +56,7 @@ Argument | Type | Description
 -------- | ---- | -----------
 remote_connection | [RemoteConnection](Classes.md#remoteconnection) | Connection associated with the remote host to run the command on.
 command | String | Command to run on the host.
-variables | dict[String, String] | **Optional**. Environement variables to set when running the command.
+variables | dict[String, String] | **Optional**. Environment variables to set when running the command.
 use_login_shell | boolean | **Optional**. Whether to use a login shell.
 check | boolean | **Optional**. Whether or not to raise an exception if the `exit_code` in the `RunBashResponse` is non-zero.
 
@@ -126,7 +126,7 @@ Argument | Type | Description
 -------- | ---- | -----------
 remote_connection | [RemoteConnection](Classes.md#remoteconnection) | Connection associated with the remote host to run the command on.
 command | String | Expect(Tcl) command to run.
-variables | dict[String, String] | **Optional**. Environement variables to set when running the command.
+variables | dict[String, String] | **Optional**. Environment variables to set when running the command.
 
 ### Returns
 An object of `RunExpectResponse`
@@ -168,7 +168,7 @@ Argument | Type | Description
 -------- | ---- | -----------
 remote_connection | [RemoteConnection](Classes.md#remoteconnection) | Connection associated with the remote host to run the command on.
 command | String | Command to run to the remote host.
-variables | dict[String, String] | **Optional**. Environement variables to set when running the command.
+variables | dict[String, String] | **Optional**. Environment variables to set when running the command.
 check | boolean | **Optional**. Whether or not to raise an exception if the `exit_code` in the `RunPowershellResponse` is non-zero.
 
 ### Returns

diff --git a/docs/docs/References/Plugin_Operations.md b/docs/docs/References/Plugin_Operations.md
@@ -8,7 +8,6 @@
     For each operation, the argument names must match exactly. For example, the Repository Discovery
     operation must have a single argument named `source_connection`.
 
-
 Plugin Operation | **Required** | Decorator | Delphix Engine Operations
 ---------------- | -------- | --------- | -------------------------
 [Repository<br/>Discovery](#repository-discovery) | **Yes** |`discovery.repository()` | [Environment Discovery](Workflows.md#environment-discovery-refresh)<br/>[Environment Refresh](Workflows.md#environment-discovery-refresh)

diff --git a/docs/docs/References/Schemas.md b/docs/docs/References/Schemas.md
@@ -24,6 +24,8 @@ Delphix Object | Schema | Autogenerated Class
 [Snapshot](Glossary.md#linked-source) | [SnapshotDefinition](Schemas_and_Autogenerated_Classes.md#snapshotdefinition-schema) | [SnapshotDefinition](Schemas_and_Autogenerated_Classes.md#snapshotdefinition-class)
 [Snapshot Parameters](Glossary.md#snapshot-parameters) | [SnapshotParametersDefinition](Schemas_and_Autogenerated_Classes.md#snapshotparametersdefinition-schema) | [SnapshotParametersDefinition](Schemas_and_Autogenerated_Classes.md#snapshotparametersdefinition-class)
 
+!!! warning
+    There are limits to how much data can be stored within a plugin defined schema. See [Message Limits](/Best_Practices/Message_Limits.md) for details.
 
 ## JSON Schemas
 
@@ -116,7 +118,6 @@ For much more detail on JSON schemas, including which keywords are available, wh
 !!! info
     Be careful when using the JSON schema keyword `default`. This keyword is commonly misunderstood. Specifically, it does not mean "If the user does not provide a value, then this default value will be auto-substituted instead", as you might expect. In fact, in JSON schemas, `default` has no semantic meaning at all! Currently, the only thing the Delphix Engine will use this for is to pre-fill widgets on the UI.
 
-
 ### Delphix-specific Extensions to JSON Schema
 
 The JSON schema vocabulary is designed to be extensible for special uses, and Delphix has taken advantage of this to add some new Delphix-specific keywords.