From 62a6bdfa99f77143cad20f17cb29ad3bc46bc094 Mon Sep 17 00:00:00 2001 From: Benson Lee Date: Wed, 6 Nov 2024 17:54:55 -0800 Subject: [PATCH 1/3] document stable time and check interval --- docs/app/agents/Agent.mdx | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/docs/app/agents/Agent.mdx b/docs/app/agents/Agent.mdx index f9899fa2..3f53abd8 100644 --- a/docs/app/agents/Agent.mdx +++ b/docs/app/agents/Agent.mdx @@ -122,8 +122,6 @@ The _Configuration_ input specifies the action performed by the Agent, with opti A Connection watches a directory (and associated subdirectories) for new files and updates to existing files. The Flow configured with the Agent is triggered when all expected files are found. -The _input_path_ variable, configured for each Connection [at installation](./AgentInstall#windows-installation), specifies the directory to watch. - #### Input Parameters - `Flow Name`: The flow to run upon observing new files matching the specified pattern. @@ -141,6 +139,16 @@ The _input_path_ variable, configured for each Connection [at installation](./Ag - `Image`: An image to associate with agent in the Ganymede UI. - `Auto deploy code and configuration changes to Live Connections`: If checked, updates to Agent code will be reflected on current Connections. +#### Reserved Variables + +Reserved variables are used to configure Connection behavior. These can be set [at installation](./AgentInstall#windows-installation) or in the Connections panel post-installation. + +- `input_path`: specifies the directory to watch. +- `check_interval`: for Agents v5.1.28+, specifies frequency (in seconds) with which a Connection will poll the local directory for new files. If left blank, the default value is 5 seconds. +- `stable_time`: for Agents v5.1.28+, specifies the time (in seconds) that a file must remain unchanged before being considered stable. If left blank, the default value is 5 seconds. + +#### User-Defined Code + The user-defined code will be generated to map those file patterns to parameters. Three functions may require modification during configuration: - `get_param_mapping`: This function is called whenever a file is added or modified in the watch directory. Modify this function to specify the files you want to look for prior to triggering a flow. @@ -373,8 +381,6 @@ Multiple flow cytometers are used to observe cell populations for a related set A Connection that monitors Flow file outputs, saving them to local directories as specified by the Agent logic. -The _output_path_ variable, configured for each Connection [at installation](./AgentInstall#windows-installation), specifies the directory to store files to. - #### Input Parameters - `Flow Name`: The flow from which to download output files. This will autopopulate the glob pattern matching field correspondingly. @@ -393,6 +399,12 @@ The _output_path_ variable, configured for each Connection [at installation](./A Additional examples can be found on the [Agent Templates page](./AgentTemplates.mdx#watch-for-flow-outputs-then-save-locally) +#### Reserved Variables + +For this configuration, the following reserved variables are available: + +- `output_path`: specifies the directory to store files to. + #### Example use case Instructions for lab execution are generated in Ganymede Cloud and downloaded to the instrument PC for execution. From 422701b59362dbcc6e7d9e7730e9ed04285addb6 Mon Sep 17 00:00:00 2001 From: Benson Lee Date: Wed, 13 Nov 2024 17:07:25 -0800 Subject: [PATCH 2/3] clarify connections --- docs/app/agents/Agent.mdx | 1 + docs/app/agents/AgentMonitoring.mdx | 297 ++++++++++++++-------------- docs/app/agents/AgentTemplates.mdx | 2 +- sidebars.js | 15 +- 4 files changed, 159 insertions(+), 156 deletions(-) diff --git a/docs/app/agents/Agent.mdx b/docs/app/agents/Agent.mdx index 3f53abd8..d18dcb9b 100644 --- a/docs/app/agents/Agent.mdx +++ b/docs/app/agents/Agent.mdx @@ -2,6 +2,7 @@ id: Agent title: Agent Overview & Configuration displayed_sidebar: webUiSidebar +toc_max_heading_level: 4 --- import { DiffOutlined, PlusOutlined, SelectOutlined } from "@ant-design/icons"; diff --git a/docs/app/agents/AgentMonitoring.mdx b/docs/app/agents/AgentMonitoring.mdx index d219e18a..939de233 100644 --- a/docs/app/agents/AgentMonitoring.mdx +++ b/docs/app/agents/AgentMonitoring.mdx @@ -1,125 +1,29 @@ --- id: AgentMonitoring -title: Monitoring Agents +title: Monitoring and Configuring Connections displayed_sidebar: webUiSidebar +toc_max_heading_level: 4 --- import { DiffOutlined, EditOutlined, EyeOutlined, PlusOutlined, SelectOutlined } from "@ant-design/icons"; ## Maintaining Agents -
- Maintaining Agents - configuration -
- -### Viewing Logs - -Logs can be found on the Logs tab for each Agent and Connection, and filtered by both log severity and date. For Agents v4.9+, user-defined log messages can be added by referencing the corresponding [`agent_sdk` method for logging](../../sdk/markdowns/AgentSDK#logging-methods). - -Descriptions of log messages can be found on the [Agent Logs page](./AgentLogs). - -
- Agent logs -
- -### Monitoring Agent Connections - -Connection status updates and logs can be accessed in the Ganymede web app UI. - -
- Connection cards -
-  - -Connection data can be viewed by clicking on the appropriate connection card in the Connections tab. +Clicking on an Agent in the Ganymede web app UI opens up a page displaying the Agent's configuration. This page allows you to view the Agent's configuration, logs, and build history, as well as update the configuration of Connections associated with the Agent.
Connection Detail UI
-  - -This opens up a page displaying parameters associated with the Connection: - -| Field Name | Description | -|------------|-------------| -| Status| Current running status of the installed Connection | -| Agent | Agent associated with the Connection | -| Connection Version | The version of the Connection configuration currently running, updated when the Connection configuration is modified | -| Agent Version | The version of the Agent configuration from which the Connection is running, updated when the Agent configuration or processor code changes | -| Core Version | The version of the core Agent infrastructure from which the Connection is running; requires a reinstall to update | -| Started At | UTC Time when the Connection was first registered in Ganymede | -| Last Status Ping | Time when last status ping was received from the Connection by Ganymede | -| PC Name | Hostname of the machine hosting the connection | -| MAC Address | MAC address of the machine hosting the connection | -| Machine IP | IP address that Ganymede sees from the machine hosting the connection | -| Install Path | Directory path on the hosting machine where the executable was installed on | -| Service Name | Windows service name for the Connection on the hosting machine | -| Agent Tags | Tags applied to all file captured by any Connection running from the Agent, configured in the Agent UI | - - -#### Connection Status - -Agents send a heartbeat message to Ganymede Cloud every 30 seconds to inform their status. The status of an Agent can be one of the following: - -- **Live**: The Agent is currently running and communicating with Ganymede Cloud -- **Disconnected**: Ganymede Cloud has not received a status update from the Agent in the last 65 seconds -- **Shutdown**: The Agent was intentionally shut down, possibly due to a user shutting down the computer, manually stopping the Agent service, or encountering a known exception. -- **Deprecated**: The Agent has been disabled in Ganymede UI as described in the [Updating Agents](#updating-agents) section. - -#### Custom Logging - -You can create custom logs that will be displayed in the Ganymede UI and local log files by using the `logger` keyword argument in your user-defined processor code. For example: - -```python -def execute(**kwargs) -> Optional[TriggerFlowParams]: # type: ignore - # if there is not logger, the default logger is print - logger = kwargs.get("logger", print) - logger(f"Local files: {os.listdir('./')}") -``` - -#### Agent Notifications - -If the agent is running at least version 5.1.7, you can create send email notifications to all subscribers by using the `agent_sdk.notify` method in your user-defined processor code. For example: -```python -import agent_sdk - -def execute(**kwargs) -> Optional[TriggerFlowParams]: # type: ignore - # ... code here that checks for some condition - if(some_condition): - agent_sdk.notify(f"The file {file_name} is missing an experiment ID") -``` ### Updating Agents -To modify Agent settings after creation, select the desired Agent in the Ganymede web app UI to update the Agent. Upon clicking
Update
, the Agent will be built and then made available for download. - -
- Agent update configuration -
-  +To modify Agent settings after creation, select the desired Agent in the Ganymede web app UI to update the Agent. Upon clicking
Save
, the Agent will be built and then made available for download. -Agents can also be archived or disabled. + Agents can also be archived or disabled. Archived Agents cannot be updated, but the Agent's connections can still communicate with Ganymede Cloud. All associated connections for these Agents will continue to run, but the Agent can no longer be modified. Archived agents can be restored to an active state by selecting the desired Agent and clicking on the Restore Agent link. @@ -136,20 +40,15 @@ To update a Connection, select the desired Connection in the Connections tab and width="600" /> +  -- **Name**: Display name for the Connection -- **Image**: An image to associate with the Connection in the Ganymede UI. -- **Tags**: [File tags](../files/Tags) to apply to files uploaded by the Connection -- **Variables**: Key-value pairs that can be referenced in user-defined code -- **Labels**: Strings that can be used to group Connections in the Ganymede UI - -### Remote Update of Live Connections +#### Updating existing Connections Every time the Connection contacts Ganymede with a health check-in, Ganymede responds with any changes that the Connection needs to incorporate. There are three level of changes that can occur. -- Core Version Update: The core version corresponds to the underlying Agent code developed by Ganymede. These changes are new features or patches applied across all Agents, but each Agent manages which core version it is pinned to. -- Agent Configuration Update: Updated through the Agent page or the Agent's processor notebook. This configuration applies to all Connections for that Agent. For example, if an Agent that runs flow upon file capture is updated to run a new flow, then all compatible Connections will begin to run the new flow. -- Connection Configuration Update: Updates made to the Connection's configuration will only update a single Connection, rather than the Agent that it was generated from. Different Connection-level variables and labels can be configured for Connections associated with the same Agent. +- **Core Version Update**: The core version corresponds to the underlying Agent code developed by Ganymede. These changes are new features or patches applied across all Agents, but each Agent manages which core version it is pinned to. +- **Agent Configuration Update**: Updated through the Agent page or the Agent's processor notebook. This configuration applies to all Connections for that Agent. For example, if an Agent that runs flow upon file capture is updated to run a new flow, then all compatible Connections will begin to run the new flow. +- **Connection Configuration Update**: Updates made to the Connection's configuration will only update a single Connection, rather than the Agent that it was generated from. Different Connection-level variables and labels can be configured for Connections associated with the same Agent. There are two types of remote updates that occur based on the type of update - full update or runtime update. @@ -204,75 +103,173 @@ Changes in the Agent configuration, including the Agent user-defined code, or Co A runtime update does not require any process restart or executable reinstallation. As a result, the Connection can accept changes to configuration while continuing normal operations. The update is validated for integrity and the information is stored locally for access if the Connection restarts for any reason. -### Uninstalling Agent Connections +### Viewing Agent History -#### Windows +#### Viewing Build History -The Agent can be uninstalled and the associated service removed through the “Add or Remove Programs” panel from the Control Panel. +Each iteration of Agent build can be viewed in the History tab of the Agent. This view provides context for each change, either as a log of the configuration change or through a custom commit message from the Agent notebook.
Agent uninstall
  -After uninstalling the Agent, the Ganymede folder will remain in the Program Files directory. You can also run the `unins000.exe` file in the installation folder to remove the Connection and all other previously installed versions. - -:::warning - -By default, the installation directory of the Connection is retained, so that logs associated with Connection are kept on the instrument PC. Removing the Ganymede folder will also remove these logs. +Configuration differences between two Agent builds can be viewed for audit or debugging purposes by clicking
button. -::: +
+ Agent build history detail +
+  -#### Linux +#### Viewing Historical Code -To uninstall the Agent, stop the systemd service associated with the Agent and remove the service file from /etc/systemd/system/. +The code corresponding to historical commits can be viewed by clicking the
**View Code**
button. This opens up a modal which allows you to copy the code from the historical commit to the clipboard if desired. -```bash -# to stop the Agent Connection service -sudo systemctl stop ganymede_example_agent.service +
+ Agent build history detail +
-# remove the service file -sudo rm /etc/systemd/system/ganymede_example_agent.service -``` +## Maintaining Agent Connections -## Agent History +Connection status updates and logs can be accessed in the Ganymede web app UI. -### Viewing Build History +
+ Connection cards +
+  -Each iteration of Agent build can be viewed in the History tab of the Agent. This view provides context for each change, either as a log of the configuration change or through a custom commit message from the Agent notebook. +Connection data can be viewed by clicking on the appropriate connection card in the Connections tab.
Agent build history
  -Configuration differences between two Agent builds can be viewed for audit or debugging purposes by clicking
button. +This opens up a page displaying parameters associated with the Connection: + +### Connection Details + +| Field Name | Description | +|------------|-------------| +| Status | Current running status of the installed Connection | +| Recent Activity | Provides links to associated Flows and Files | +| Agent | Agent associated with the Connection | +| Connection Version | The version of the Connection configuration currently running, updated when the Connection configuration is modified | +| Agent Version | The version of the Agent configuration from which the Connection is running, updated when the Agent configuration or processor code changes | +| Core Version | The version of the core Agent infrastructure from which the Connection is running; requires a reinstall to update | +| Started At | UTC Time when the Connection was first registered in Ganymede | +| Last Status Ping | Time when last status ping was received from the Connection by Ganymede | +| PC Name | Hostname of the machine hosting the connection | +| MAC Address | MAC address of the machine hosting the connection | +| Machine IP | IP address that Ganymede sees from the machine hosting the connection | +| Install Path | Directory path on the hosting machine where the executable was installed on | +| Service Name | Windows service name for the Connection on the hosting machine | + +Below the Connection Details box, you'll find Connection-specific configurations: + +- **Name**: Name of the Agent associated with the Connection +- **Connection File Tags**: [Tags](../files/Tags.mdx) placed on files uploaded by the Connection +- **Variables**: [Key-value pairs](./Agent#variables-and-labels-in-windows-agent-connections) that can be referenced in user-defined code. These are also used to specify [reserved keywords](Agent#reserved-variables) for the Connection, if applicable. +- **Labels**: [Strings](./Agent#variables-and-labels-in-windows-agent-connections) that can be used to group Connections in the Ganymede UI +- **Emails**: Email addresses to notify when the Connection encounters an error or has an update in its [Connection status](#connection-status). These email addresses are also used by the [notify function found in the `agent_sdk`](../../sdk/markdowns/AgentSDK#notifications). + +### Connection Status + +Agents send a heartbeat message to Ganymede Cloud every 30 seconds to inform their status. The status of an Agent can be one of the following: + +- **Live**: The Agent is currently running and communicating with Ganymede Cloud +- **Disconnected**: Ganymede Cloud has not received a status update from the Agent in the last 65 seconds +- **Shutdown**: The Agent was intentionally shut down, possibly due to a user shutting down the computer, manually stopping the Agent service, or encountering a known exception. +- **Deprecated**: The Agent has been disabled in Ganymede UI as described in the [Updating Agents](#updating-agents) section. + +### Connection Notifications + +For Agents v5.1.7+, you can create send email notifications to all subscribers by using the `agent_sdk.notify` method in your user-defined processor code. For example: + +```python +import agent_sdk + +def execute(**kwargs) -> Optional[TriggerFlowParams]: # type: ignore + # ... code here that checks for some condition + if(some_condition): + agent_sdk.notify(f"The file {file_name} is missing an experiment ID") +``` + +### Viewing Logs + +Logs can be found on the Logs tab for each Agent and Connection, and filtered on by both log severity and date. For Agents v4.9+, user-defined log messages can be added by referencing the corresponding [`agent_sdk` method for logging](../../sdk/markdowns/AgentSDK#logging-methods). Descriptions of log messages can be found on the [Agent Logs page](./AgentLogs).
Agent build history detail
-  -### Viewing Historical Code +The Instrument tab within the Connection logs page contains downloadable text files for the Connection's logs, enabling you to search over logs locally if desired. -The code corresponding to historical commits can be viewed by clicking the
**View Code**
button. This opens up a modal which allows you to copy the code from the historical commit to the clipboard if desired. +## Uninstalling Agent Connections + +### Windows + +A Connection can be uninstalled and its associated service removed through the “Add or Remove Programs” panel from the Control Panel.
Agent build history detail -
\ No newline at end of file + +  + +After uninstalling the Agent, the Ganymede folder will remain in the Program Files directory. You can also run the `unins000.exe` file in the installation folder to remove the Connection and all other previously installed versions. + +:::warning + +By default, the installation directory of the Connection is retained, so that logs associated with Connection are kept on the instrument PC. Removing the Ganymede folder will also remove these logs. + +::: + +:::info + +If the Connection service continues to appear in the Services panel after uninstalling the Connection, you can remove it by running the following command in the Command Prompt: + +```bash +sc delete +``` + +::: + +### Linux + +To uninstall the Agent, stop the systemd service associated with the Agent and remove the service file from /etc/systemd/system/. + +```bash +# to stop the Agent Connection service +sudo systemctl stop ganymede_example_agent.service + +# remove the service file +sudo rm /etc/systemd/system/ganymede_example_agent.service +``` diff --git a/docs/app/agents/AgentTemplates.mdx b/docs/app/agents/AgentTemplates.mdx index 627ed635..6427dc79 100644 --- a/docs/app/agents/AgentTemplates.mdx +++ b/docs/app/agents/AgentTemplates.mdx @@ -2,7 +2,7 @@ id: AgentTemplates title: Agent Templates displayed_sidebar: webUiSidebar -sidebar_label: Agent Templates +sidebar_label: Example Agent Templates --- import NodeChip from '@site/src/components/NodeChip.js' diff --git a/sidebars.js b/sidebars.js index 3a5247eb..0e3f0c94 100644 --- a/sidebars.js +++ b/sidebars.js @@ -130,11 +130,6 @@ module.exports = { id: 'app/agents/AgentInstall', label: 'Installing Agent Connections' }, - { - type: 'doc', - id: 'app/agents/AgentTemplates', - label: 'Example Agent Templates' - }, { type: 'doc', id: 'app/agents/AgentMonitoring', @@ -145,6 +140,11 @@ module.exports = { id: 'app/agents/DebuggingAgents', label: 'Troubleshooting Agents & FAQ' }, + { + type: 'doc', + id: 'app/agents/AgentTemplates', + label: 'Example Agent Templates' + }, { type: 'doc', id: 'app/agents/AgentLogs', @@ -157,6 +157,11 @@ module.exports = { }, ] }, + { + type: 'doc', + id: 'app/instruments/Instruments', + label: 'Instruments', + }, { type: 'category', label: 'Files', From 5dfa7ae308abe6dc09f37c131c6a25a583b58cbe Mon Sep 17 00:00:00 2001 From: Benson Lee Date: Wed, 13 Nov 2024 17:10:12 -0800 Subject: [PATCH 3/3] clarify connections --- docs/app/agents/Agent.mdx | 2 +- docs/app/agents/AgentTemplates.mdx | 2 +- docs/app/configuration/AgentSystemRequirements.mdx | 2 +- docs/releases/202407.mdx | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/app/agents/Agent.mdx b/docs/app/agents/Agent.mdx index d18dcb9b..84c2036a 100644 --- a/docs/app/agents/Agent.mdx +++ b/docs/app/agents/Agent.mdx @@ -473,7 +473,7 @@ More generally, Variables and Labels can be flexibly used to control Connection -v "abc=1" -v "xyz=2" -l "UPLC" -l "Analytical Dev" ``` -And [modified in the corresponding Connections panel in the Ganymede web app after installation, for Agent versions 4.5+](AgentMonitoring#monitoring-agent-connections). +And [modified in the corresponding Connections panel in the Ganymede web app after installation, for Agent versions 4.5+](AgentMonitoring#maintaining-agent-connections). Variables and labels can be referenced in your user-defined code by extracting the values from `kwargs` in the _execute_ function. diff --git a/docs/app/agents/AgentTemplates.mdx b/docs/app/agents/AgentTemplates.mdx index 6427dc79..47b116ee 100644 --- a/docs/app/agents/AgentTemplates.mdx +++ b/docs/app/agents/AgentTemplates.mdx @@ -321,7 +321,7 @@ Variables (e.g. - _parent_dir_ and _recency_min_) can be configured during the i -v parent_dir=C:\Users\dev\experiment_results\output\ -v file_to_watch="output.log" -v recency_min=60 -v recency_max=1800 ``` -Post-installation, for Agents v4.8+, the parameters for Windows Connections can be [updated in the Connection UI](../../app/agents/AgentMonitoring#monitoring-agent-connections). +Post-installation, for Agents v4.8+, the parameters for Windows Connections can be [updated in the Connection UI](../../app/agents/AgentMonitoring#maintaining-agent-connections). ### Querying OPC server diff --git a/docs/app/configuration/AgentSystemRequirements.mdx b/docs/app/configuration/AgentSystemRequirements.mdx index eec2ea36..18187adc 100644 --- a/docs/app/configuration/AgentSystemRequirements.mdx +++ b/docs/app/configuration/AgentSystemRequirements.mdx @@ -76,7 +76,7 @@ Frequently, the data producer is not the same person who is configuring Agents. ### Instrument Run Metadata -[Tagging files](../files/Tags) with relevant characteristics is critical for ensuring that data remains appropriately organized and accessible once captured in Ganymede. By default, Ganymede captures PC information as shown in the [Monitoring Agent Connections section of the Agent documentation](../agents/AgentMonitoring#monitoring-agent-connections), but instrument data often can be shaped to a degree by end-user needs. Some questions to consider when tagging files are: +[Tagging files](../files/Tags) with relevant characteristics is critical for ensuring that data remains appropriately organized and accessible once captured in Ganymede. By default, Ganymede captures PC information as shown in the [Monitoring Agent Connections section of the Agent documentation](../agents/AgentMonitoring#maintaining-agent-connections), but instrument data often can be shaped to a degree by end-user needs. Some questions to consider when tagging files are: - What are the metadata fields that are relevant to the instrument? Some examples include: - Instrument name / vendor diff --git a/docs/releases/202407.mdx b/docs/releases/202407.mdx index ff4c8bc7..9186d4dc 100644 --- a/docs/releases/202407.mdx +++ b/docs/releases/202407.mdx @@ -7,7 +7,7 @@ displayed_sidebar: releaseSideBar ### July 26 2024 **Features** -- **UI**: [Connection details](../app/agents/AgentMonitoring#monitoring-agent-connections) are now visible and editable in the UI +- **UI**: [Connection details](../app/agents/AgentMonitoring#maintaining-agent-connections) are now visible and editable in the UI - Review details about your connection by clicking on a connection card - The image for the connection can be customized - See the connection logs and configuration