Skip to content

Commit

Permalink
Correct spelling errors
Browse files Browse the repository at this point in the history
  • Loading branch information
nfs0619 committed Jan 2, 2025
1 parent b4aec85 commit b6f45fc
Showing 1 changed file with 30 additions and 35 deletions.
65 changes: 30 additions & 35 deletions ...cher/Software_Framework_Overview/sensecap_watcher_software_service_framework.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,37 +16,33 @@ last_update:

This is the software service framework for the Watcher, illustrating the interactions and task flow between the user, the SenseCraft Mate app, and the Watcher device. We offer multiple configuration options that allow users to customize their service deployment based on their data security and service quality requirements.

The framework integrates cloud AI services and local AI deployments, combined with data services, device communication services, task orchestration, image analysis services, alert channel agents, and model training services.
The framework integrates cloud AI services and local AI deployments, combined with data services, device communication services, task orchestration, image analysis services, alert channel agents, and model training services.
you can choose to deploy your LLM in the cloud or on-premises infrastructure.the original Data and result can also be chosen to be storage or transmit in cloud or on-premise device.
Cloud compared to on-premise storage solutions as well as the need for higher-performance storage within GPU-enabled virtual machines.

These components form three deployment options for users to choose from:

1. **Cloud Efficient Processing Flow**
2. **Hybrid Smart Processing Flow**
3. **Local Secure Processing Flow**
1. **Cloud Efficient Processing Flow**
2. **Hybrid Smart Processing Flow**
3. **Local Secure Processing Flow**

The following sections provide a detailed explanation of the framework and guide you through configuring each of the three deployment options.

**Data Service**

Watcher offers flexible data services, allowing you to connect to the SenseCraft data platform to upload alert data, device status, and preview images. The SenseCraft platform provides an MQTT broker address and token through an HTTP request. Alternatively, you can choose to upload data to third-party platforms via Bluetooth configuration on the app, bypassing the SenseCraft platform entirely.


**Device Communication Service**

For device communication, Watcher supports remote task flows and firmware OTA updates through MQTT. You can use the SenseCraft Mate app to create tasks and send them to the device through the SenseCraft platform. The app also checks for lastest firmware version, and when available, it will notify you. After confirming, the platform will send the update to the device.

**Task Orchestration Service**


Watcher’s task orchestration service allows for direct voice interactions, where your dialogue is sent to the service via HTTP, processed, and the task flow is returned to the device. The SenseCraft Mate app can also retrieve task flows and send them remotely to the device.


**Vision Analysis Service**

In terms of vision analysis, Watcher offers a unified image analysis service. You can choose between SenseCraft, OpenAI or third-party AI agent through the Bluetooth configuration and input the relevant API keys. When the device sends an image, it will utilize the selected service, either the local llava or a third-party service like OpenAI.

In terms of vision analysis, Watcher offers a unified image analysis service. You can choose between SenseCraft, OpenAI or third-party AI agent through the Bluetooth configuration and input the relevant API keys. When the device sends an image, it will utilize the selected service, either the local llava or a third-party service like OpenAI.

**Alert Notification Service**

Expand All @@ -62,55 +58,54 @@ This diagram provides an overview of the Cloud Efficient Processing Flow, illust

The following sections explain each part of the Cloud Efficient Processing Flow and provide guidance on configuring it:

**Step 1**. Initial Setup:
**Step 1**. Initial Setup:
After completing the [setup](https://www.seeedstudio.com/getting_started_with_watcher/#device-binding-and-ota-upgrade) and selecting SenseCraft AI Services, you can begin using the Watcher’s cloud services.


**Step 2**. User Interaction:
There are two ways to activate SenseCraft services, and allocate task to Watcher, detailed as follows (refer to section XXX):
* Option 1: Via the App: Users can enable SenseCraft AI services through the mobile application. Then sending task message in chat.
* Option 2: Via Voice Commands: Users can assign task through voice chat directly with the Watcher.

* Option 1: Via the App: Users can enable SenseCraft AI services through the mobile application. Then sending task message in chat.
* Option 2: Via Voice Commands: Users can assign task through voice chat directly with the Watcher.

**Step 3**. Task Orchestration:
**Step 3**. Task Orchestration:
Once SenseCraft is activated, it functions as a Platform-as-a-Service (PaaS) to provide task orchestration services. This includes three core elements:
* SST (Speech-to-Text): Converts spoken commands into text.
* TTS (Text-to-Speech): Converts text-based responses into spoken output.
* AI Services: Responsible for understanding, processing, and orchestrating tasks. Based on the task’s content, the service selects the most suitable models from the SenseCraft model repository, including large language models (LLMs) and TinyML models.

* STT (Speech-to-Text): Converts spoken commands into text.
* TTS (Text-to-Speech): Converts text-based responses into spoken output.
* AI Services: Responsible for understanding, processing, and orchestrating tasks. Based on the task’s content, the service selects the most suitable models from the SenseCraft model repository, including large language models (LLMs) and TinyML models.

In addition to manually selecting local TinyML models in the manual configuration, tasks can also be processed automatically based on your input prompt. After the task orchestration model interprets your prompt, it can call the appropriate TinyML model from the SenseCraft library. For example, if a user requests, "Notify me when you see a bird," the task orchestration service will select a bird recognition model from the TinyML library to optimize response speed.

In the *manual configuration*, you can choose between the **Vision LLM** (for higher accuracy) or the **TinyML model** (for faster processing).


**Step 4**. Task Dispatch after Orchestration
Once task orchestration is completed, the method for dispatching tasks will depend on the status of the SenseCraft Remote Control feature:

* If remote control is enabled: The system will utilize the PaaS data service and device communication service to send task results to the Watcher via MQTT broker.
* If remote control is disabled: Tasks will be dispatched to the Watcher via Bluetooth. If the device is out of Bluetooth range, you will need to enable remote control for task transmission.
* If remote control is enabled: The system will utilize the PaaS data service and device communication service to send task results to the Watcher via MQTT broker.
* If remote control is disabled: Tasks will be dispatched to the Watcher via Bluetooth. If the device is out of Bluetooth range, you will need to enable remote control for task transmission.

**Step 5**. Vision Analysis Task Processing

After receiving the task from SenseCraft, the Watcher device will invoke the Visual Analysis Service. It will analyze the task using the AI model chosen by the user or determined by the task orchestration service.

**Step 6**. System Notifications
The Watcher can notify the user or a local system using various methods:
* App push notifications
* Serial port/UART output
* HTTP push notifications

* App push notifications
* Serial port/UART output
* HTTP push notifications

Watcher can also send data to local servers, third-party applications, or alert services like Discord, Home Assistant, or IFTTT.
For more details, refer to section HTTP Proxy Application.

## Hybrid Smart Processing Flow


<div style={{textAlign:'center'}}><img src="https://files.seeedstudio.com/wiki/watcher_getting_started/watcher_software_service_framework/3.png" style={{width:800, height:'auto'}}/></div>

This diagram illustrates the Hybrid Smart Processing Flow, which involves user interactions, some cloud services, and local applications. The hybrid approach balances data security, user experience, and efficiency by combining cloud-based task orchestration with local visual models.

**Step 1**. Initial Setup
After [getting started and device binding](https://wiki.seeedstudio.com/getting_started_with_watcher/#device-binding-and-ota-upgrade), users can select the appropriate local services based on their needs, enable the private Watcher service, and enter the Watcher server URL.
After [getting started and device binding](https://wiki.seeedstudio.com/getting_started_with_watcher/#device-binding-and-ota-upgrade), users can select the appropriate local services based on their needs, enable the private Watcher service, and enter the Watcher server URL.

<div style={{textAlign:'center'}}><img src="https://files.seeedstudio.com/wiki/watcher_getting_started/watcher_software_service_framework/1.gif" style={{width:400, height:'auto'}}/></div>

Expand All @@ -119,7 +114,8 @@ Users can send tasks via the SenseCraft app, which will initiate the task orches

**Step 3**. Task Orchestration:
Once a task is received, the SenseCraft platform, functioning as a PaaS (Platform-as-a-Service), will orchestrate the task using three core services:
* SST (Speech-to-Text service)

* STT (Speech-to-Text service)
* TTS (Text-to-Speech service)
* AI services for understanding, processing, and task orchestration.

Expand All @@ -135,15 +131,15 @@ After task orchestration, the system will choose the method of dispatch dependin
* If remote control is enabled: The system will use PaaS data service and device communication service to send task results to Watcher via MQTT broker.
* If remote control is disabled: Tasks will be dispatched to Watcher via Bluetooth. If the device is out of Bluetooth range, remote control must be enabled for continued operation.


**Step 5**. Vision Analysis Task Processing
Once the Watcher receives the task, it will invoke the local visual analysis service deployed on a local server. The task will be analyzed using the AI model selected by the user or orchestrated by the SenseCraft task service.

**Step 6**. System Notifications
The Watcher can notify the user or a local system using various methods:
* App push notifications
* Serial port/UART output
* HTTP push notifications

* App push notifications
* Serial port/UART output
* HTTP push notifications

Watcher can also send data to local servers, third-party applications, or alert services like Discord, Home Assistant, or IFTTT.
For more details, refer to section HTTP Proxy Application.
Expand All @@ -155,7 +151,7 @@ For more details, refer to section HTTP Proxy Application.
This diagram outlines the “Local Secure Processing Flow”, which focuses on complete user interaction with locally deployed applications. To ensure data privacy, task orchestration and image analysis services are deployed entirely on the local server, offering full privacy protection. However, this method limits functionality, as SenseCraft cloud services are not available. Tasks can only be initiated via in-field voice commands, and users must configure notification services to view historical data.

**Step 1**. Initial Setup
After [getting started and device binding](https://wiki.seeedstudio.com/getting_started_with_watcher/#device-binding-and-ota-upgrade), users can select the appropriate local services based on their needs, enable the private Watcher service, and enter the Watcher server URL.
After [getting started and device binding](https://wiki.seeedstudio.com/getting_started_with_watcher/#device-binding-and-ota-upgrade), users can select the appropriate local services based on their needs, enable the private Watcher service, and enter the Watcher server URL.

<div style={{textAlign:'center'}}><img src="https://files.seeedstudio.com/wiki/watcher_getting_started/watcher_software_service_framework/1.gif" style={{width:400, height:'auto'}}/></div>

Expand All @@ -166,7 +162,7 @@ Users can issue tasks by pressing the wheel button on the Watcher device and sen

The Watcher server hosted on the local server provides task orchestration services, similar to the SenseCraft task orchestration system. This local system consists of three core services:

* SST (Speech-to-Text service)
* STT (Speech-to-Text service)
* TTS (Text-to-Speech service)
* AI services for understanding, processing, and orchestrating tasks.

Expand All @@ -190,13 +186,12 @@ Thank you for choosing our products! We are here to provide you with different s

<div class="table-center">
<div class="button_tech_support_container">
<a href="https://forum.seeedstudio.com/" class="button_forum"></a>
<a href="https://forum.seeedstudio.com/" class="button_forum"></a>
<a href="https://www.seeedstudio.com/contacts" class="button_email"></a>
</div>

<div class="button_tech_support_container">
<a href="https://discord.gg/eWkprNDMU7" class="button_discord"></a>
<a href="https://discord.gg/eWkprNDMU7" class="button_discord"></a>
<a href="https://github.com/Seeed-Studio/wiki-documents/discussions/69" class="button_discussion"></a>
</div>
</div>

0 comments on commit b6f45fc

Please sign in to comment.