Skip to content

Commit

Permalink
docs: refactor connector (#2219)
Browse files Browse the repository at this point in the history
Signed-off-by: LenaLenaPan <[email protected]>
  • Loading branch information
LenaLenaPan authored Sep 1, 2023
1 parent 308603b commit 0129a89
Show file tree
Hide file tree
Showing 103 changed files with 1,888 additions and 1,214 deletions.
989 changes: 532 additions & 457 deletions docs/directory.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/en_US/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ If you get stuck, check out our community support resources.
- Join our [Slack](https://slack.lfedge.org/), and then join [ekuiper](https://lfedge.slack.com/archives/C024F4P7KCK) or [ekuiper-user](https://lfedge.slack.com/archives/C024F4SMEMR) channel.
- Mail to eKuiper help [mail list](mailto:[email protected]).
- Join WeChat group, scan the below QR code and mark "eKuiper".

<img src="./wechat.png" alt="drawing" width="200"/>

## Contribute
Expand Down
6 changes: 4 additions & 2 deletions docs/en_US/api/cli/overview.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
The eKuiper CLI (command line interface) tools provides streams and rules management.
# Command Line Tool

The eKuiper CLI acts as a client to the eKuiper server. The eKuiper server runs the engine that executes the stream or rule queries. This includes processing stream or rule definitions, manage rule status and io.
The eKuiper CLI (command line interface) tool provides streams and rules management.

The eKuiper CLI acts as a client to the eKuiper server. The eKuiper server runs the engine that executes the stream or rule queries. This includes processing stream or rule definitions, managing rule status, and io.

*eKuiper CLI Architecture*
![CLI Arch](./resources/arch.png)
Expand Down
2 changes: 1 addition & 1 deletion docs/en_US/api/cli/plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ Sample:
The command is used for drop the plugin.

```shell
drop plugin $plugin_type $plugin_name -s $stop
drop plugin $plugin_type $plugin_name -s $stop
```

In which, `-s $stop` is an optional boolean parameter. If it is set to true, the eKuiper server will be stopped for the delete to take effect. The user will need to restart it manually.
Expand Down
2 changes: 1 addition & 1 deletion docs/en_US/api/cli/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ Sample:
"source_demo_0_process_latency_ms":0,
"source_demo_0_buffer_length":0,
"source_demo_0_last_invocation":"2020-01-02T11:28:33.054821",
...
...
"op_filter_0_records_in_total":5,
"op_filter_0_records_out_total":2,
"op_filter_0_exceptions_total":0,
Expand Down
4 changes: 2 additions & 2 deletions docs/en_US/api/cli/streams.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ stream my_stream dropped

## query against streams

The command is used for querying data from stream.
The command is used for querying data from stream.

```shell
query
Expand All @@ -102,7 +102,7 @@ Sample:

```shell
# bin/kuiper query
kuiper >
kuiper >
```

After typing `query` sub-command, it prompts `kuiper >`, then type SQLs (see [eKuiper SQL reference](../../sqls/overview.md) for how to use eKuiper SQL) in the command prompt and press enter.
Expand Down
2 changes: 1 addition & 1 deletion docs/en_US/api/restapi/data.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ Content-Type: application/json
"streams":{},
"tables":{},
"rules":{},
"nativePlugins":{
"nativePlugins":{
"sinks_tdengine":"fail to download file file:///root/ekuiper-jran/_plugins/ubuntu/sinks/tdengine_amd64.zip: stat /root/ekuiper-jran/_plugins/ubuntu/sinks/tdengine_amd64.zip: no such file or directory",
"sources_random":"fail to download file file:///root/ekuiper-jran/_plugins/ubuntu/sources/random_amd64.zip: stat /root/ekuiper-jran/_plugins/ubuntu/sources/random_amd64.zip: no such file or directory"},
"portablePlugins":{},
Expand Down
2 changes: 2 additions & 0 deletions docs/en_US/api/restapi/overview.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# API Reference

eKuiper provides a set of REST API for streams and rules management in addition to CLI.

By default, the REST API are running in port 9081. You can change the port in `/etc/kuiper.yaml` for the `restPort` property.
Expand Down
6 changes: 3 additions & 3 deletions docs/en_US/api/restapi/plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,9 @@ Notice that, the install.sh will be run that the system may already had the lib
#!/bin/sh
dir=/usr/local/mysdk
cur=$(dirname "$0")
echo "Base path $cur"
echo "Base path $cur"
if [ -d "$dir" ]; then
echo "SDK path $dir exists."
echo "SDK path $dir exists."
else
echo "Creating SDK path $dir"
mkdir -p $dir
Expand All @@ -70,7 +70,7 @@ fi

apt install --no-upgrade unzip
if [ -d "$dir/lib" ]; then
echo "SDK lib path $dir/lib exists."
echo "SDK lib path $dir/lib exists."
else
echo "Unzip SDK lib to path $dir"
unzip $cur/mysdk.zip -d $dir
Expand Down
2 changes: 1 addition & 1 deletion docs/en_US/api/restapi/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ Response Sample:
"source_demo_0_process_latency_ms":0,
"source_demo_0_buffer_length":0,
"source_demo_0_last_invocation":"2020-01-02T11:28:33.054821",
...
...
"op_filter_0_records_in_total":5,
"op_filter_0_records_out_total":2,
"op_filter_0_exceptions_total":0,
Expand Down
2 changes: 1 addition & 1 deletion docs/en_US/concepts/sources/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Where the definition in the schema registry is the physical schema and the data

## Stream & Table

The source defines the external system connection. When using the source with a rule, users can define them as a stream or table according to the processing mechanism. Check [stream](stream.md) and [table](table.md) for detail.
The source defines the external system connection. When using the source with a rule, users can define them as a stream or table according to the processing mechanism. Check [stream](stream.md) and [table](table.md) for details.

## More Readings

Expand Down
2 changes: 1 addition & 1 deletion docs/en_US/concepts/sources/stream.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

A stream is the runtime form of a source in eKuiper. It must specify a source type to define how to connect to the external resource.

When using as a stream, the source must be unbounded. The stream acts like a trigger for the rule. Each event will trigger a calculation in the rule.
When used as a stream, the source must be unbounded. The stream acts as a trigger for the rule. Each event will trigger a calculation in the rule.

## More Readings

Expand Down
6 changes: 3 additions & 3 deletions docs/en_US/edgex/edgex_meta.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ When data are published into EdgeX message bus, besides the actual device value,

## Events data model received in EdgeX message bus

The data structure received from EdgeX message bus is list as in below. An `Event` structure encapsulates related metadata (ID, DeviceName, ProfileName, SourceName, Origin, Tags), along with the actual data (in `Readings` field) collected from device service.
The data structure received from EdgeX message bus is list as in below. An `Event` structure encapsulates related metadata (ID, DeviceName, ProfileName, SourceName, Origin, Tags), along with the actual data (in `Readings` field) collected from device service.

Similar to `Event`, `Reading` also has some metadata (ID, DeviceName... etc).

Expand All @@ -28,7 +28,7 @@ Similar to `Event`, `Reading` also has some metadata (ID, DeviceName... etc).
- ... // The same as in reading[0]
- ...
- reading [n] ...

### Breaking changes from EdgeX v1

If upgrading from eKuiper versions v1.2.0 and before which integrates with EdgeX v1, there will be some breaking changes of the meta datas due to the refactor of EdgeX v2.
Expand Down Expand Up @@ -75,7 +75,7 @@ Below are some other samples that extract other metadata through `meta` function

Get 'origin' metadata from reading[1], key with 'humidity'

Please notice that if you want to extract metadata from readings, you need to use `reading-name -> key` operator to access the value. In previous samples, `temperature` & `humidity` are `reading-names`, and `key` is the field names in readings.
Please notice that if you want to extract metadata from readings, you need to use `reading-name -> key` operator to access the value. In previous samples, `temperature` & `humidity` are `reading-names`, and `key` is the field names in readings.

However, if you want to get data from `Events`, just need to specify the key directly. As the 1st sample in previous list.

Expand Down
10 changes: 5 additions & 5 deletions docs/en_US/edgex/edgex_rule_engine_tutorial.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ The tutorial demonstrates how to use eKuiper to process the data from EdgeX mess

EdgeX uses [message bus](https://github.com/edgexfoundry/go-mod-messaging) to exchange information between different micro services. It contains the abstract message bus interface and implementations for ZeroMQ & MQTT. The integration work for eKuiper & EdgeX includes following 3 parts.

- An EdgeX message bus source is extended to support consuming data from EdgeX message bus.
- An EdgeX message bus source is extended to support consuming data from EdgeX message bus.

- To analyze the data, eKuiper need to know data types that passed through it. Generally, user would be better to specify data schema for analysis data when a stream is created. Such as in below, a `demo` stream has a field named `temperature` field. It is very similar to create table schema in relational database system. After creating the stream definition, eKuiper can perform type checking during compilation or runtime, and invalid SQLs or data will be reported to user.

Expand Down Expand Up @@ -94,7 +94,7 @@ add these in `environment` part and make sure the image is `1.4.0` or later.
CONNECTION__EDGEX__REDISMSGBUS__TYPE: redis
EDGEX__DEFAULT__CONNECTIONSELECTOR: edgex.redisMsgBus
```
- `mqtt/zeromq` messageBus: adjust the parameters accordingly and specify the client credentials if have.
There is a `mqtt` message bus example, make sure the connection info exists in `etc/connections/connection.yaml`, for [more info](../guide/sources/builtin/edgex.md#connectionselector) please check this.
Expand Down Expand Up @@ -186,7 +186,7 @@ default:
server: localhost
port: 5566
topic: events
.....
.....
```
For more detailed information of configuration file, please refer to [this doc](../guide/sources/builtin/edgex.md).
Expand Down Expand Up @@ -226,7 +226,7 @@ curl -X POST \

#### Option 2: Use eKuiper CLI

You can create a rule file with any text editor, and copy following contents into it. Let's say the file name is `rule.txt`.
You can create a rule file with any text editor, and copy following contents into it. Let's say the file name is `rule.txt`.
```json
{
Expand Down Expand Up @@ -265,7 +265,7 @@ to see detailed info of rule.
```text
time="2021-07-08 01:03:08" level=info msg="Serving kuiper (version - 1.2.1) on port 20498, and restful api on http://0.0.0.0:59720. \n" file="server/server.go:144"
Serving kuiper (version - 1.2.1) on port 20498, and restful api on http://0.0.0.0:59720.
Serving kuiper (version - 1.2.1) on port 20498, and restful api on http://0.0.0.0:59720.
time="2021-07-08 01:08:14" level=info msg="Successfully subscribed to edgex messagebus topic rules-events." file="extensions/edgex_source.go:111" rule=rule1
time="2021-07-08 01:08:14" level=info msg="The connection to server tcp://broker.emqx.io:1883 was established successfully" file="sinks/mqtt_sink.go:182" rule=rule1
time="2021-07-08 01:08:20" level=info msg="sink result for rule rule1: [{\"Float32\":-2.4369560555943686e+38}]" file="sinks/log_sink.go:16" rule=rule1
Expand Down
10 changes: 7 additions & 3 deletions docs/en_US/example/data_merge/merge_single_stream.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
# Merge Multiple Devices' Data in Single Stream

To run the case by hand, please check [here](../howto.md).
# Merge Multiple Devices' Data in a Single Stream

## Problem

In IoT scenarios, devices such as sensors are often numerous, and usually the acquisition software combines data from all devices into one data stream. Since each sensor has different acquisition and response cycles, the data stream is interspersed with data from various devices, and the data is more fragmented, with each event containing data from only one sensor. For example, if sensor A collects temperature data once per second, sensor B collects humidity data every 5 seconds, and sensor C collects data every 10 seconds, then there will be three kinds of data in the data stream, A, B, and C. Each kind of data is collected at different frequencies, but they are all mixed together. Back-end applications where the settings of the same group of sensors are usually correlated need to merge data from the same group of sensors together for subsequent processing.

::: tip

To run the case by hand, please check [here](../howto.md).

:::

## Sample input

The temperature and humidity sensor data is mixed in the data stream, and none of the data is complete.
Expand Down
2 changes: 1 addition & 1 deletion docs/en_US/example/howto.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Run Examples
# Step-by-Step Guide: Navigating eKuiper with the Management Console UI

This page explains how to use eKuiper to run the examples in this document. Before running the examples, you need to [install eKuiper](../installation.md).

Expand Down
6 changes: 6 additions & 0 deletions docs/en_US/example/overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Cookbook

The cookbook chapter is crafted to guide users on how to leverage eKuiper's features to address particular challenges in IoT scenarios:

- **[Step-by-Step Guide: Navigating eKuiper with the Management Console UI](howto.md)**: This guide provides a concise walkthrough on utilizing eKuiper through its Management Console UI. It offers step-by-step instructions for data preparation, example execution, and result interpretation.
- **[Merge Multiple Devices' Data in a Single Stream](./data_merge/merge_single_stream.md)**: Introduces the concept of data merging in IoT scenarios, where data from multiple sensors needs to be combined for meaningful analysis. A detailed use case is provided, explaining the problem of merging data from sensors with varying data acquisition frequencies.
2 changes: 1 addition & 1 deletion docs/en_US/extension/external/external_func.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Assuming we have a service named 'sample', we can define a service definition fi
"functions": [
{
"name": "getFeature",
"serviceName": "get_feature"
"serviceName": "get_feature"
},
{
"name": "getSimilarity",
Expand Down
4 changes: 2 additions & 2 deletions docs/en_US/extension/native/develop/function.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,9 @@ function in the SQL. The argument is a slice of the values for the function para
calculation. If the calculation is successful, return the result and true; otherwise, return nil and false.

```go
//Execute the function, return the result and if execution is successful.If execution fails, return the error and false.
//Execute the function, return the result and if execution is successful.If execution fails, return the error and false.
Exec(args []interface{}) (interface{}, bool)
```
```

As the function itself is a plugin, it must be in the main package. Given the function struct name is myFunction. At last of the file, the source must be exported as a symbol as below. There are [2 types of exported symbol supported](../overview.md#plugin-development). For function extension, if there is no internal state, it is recommended to export a singleton instance.

Expand Down
26 changes: 13 additions & 13 deletions docs/en_US/extension/native/develop/plugins_tutorial.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ func (m *mysqlSink) Collect(ctx api.StreamContext, item interface{}) error {

//TODO 生产环境中需要处理item unmarshall后的各种类型。
// 默认的类型为 []map[string]interface{}
// 如果sink的`dataTemplate`属性有设置,则可能为各种其他的类型
// 如果sink的`dataTemplate`属性有设置,则可能为各种其他的类型
logger.Debugf("mysql sink receive %s", item)
//TODO 此处列名写死。生产环境中一般可从item中的键值对获取列名
sql := fmt.Sprintf("INSERT INTO %s (`name`) VALUES ('%s')", m.conf.Table, v)
Expand Down Expand Up @@ -149,7 +149,7 @@ However, users usually need to create the new project outside of the eKuiper pro
```text
samplePlugin
sinks //source code directory of the plugin sink
mysql.go
mysql.go
go.mod //file go module
```

Expand Down Expand Up @@ -190,7 +190,7 @@ If users create plugin project inside eKuiper, then he can just use the followin
```shell
# compile the eKuiper
go build -trimpath -o ./_build/$build/bin/kuiperd cmd/kuiperd/main.go
# compile the plugin that using the extensions folder within eKuiper project
go build -trimpath --buildmode=plugin -o ./_build/$build/plugins/sinks/[email protected] extensions/sinks/mysql/mysql.go
```
Expand Down Expand Up @@ -221,7 +221,7 @@ However, if developers create plugin project outside eKuiper, he needs following
```text
workspace
ekuiper
go.mod
go.mod
samplePlugin
go.mod
go.work
Expand Down Expand Up @@ -250,7 +250,7 @@ eKuiper provides different docker images for different purpose. The development
```
2. The principle of compiling plugins in docker environment is the same as the local compilation. The compiled plugin is locating in the target directory of the plugin project.
1. get into the compiling docker environment
1. get into the compiling docker environment
```shell
# In host
Expand All @@ -271,8 +271,8 @@ eKuiper provides different docker images for different purpose. The development
kuiper
go.mod
samplePlugin
sinks
mysql.go
sinks
mysql.go
go.mod
go.work
```
Expand All @@ -295,7 +295,7 @@ eKuiper provides different docker images for different purpose. The development
eKuiper offers an Alpine version of its image, but it does not come with the Go environment pre-installed. To compile plugins using the Alpine image, users will need to install the necessary dependencies themselves. Alternatively, users can opt to use the Golang image as their base environment, which includes the Go environment and simplifies the plugin compilation process(If you are using the golang 1.20 version image and want to compile eKuiper plugins, you can use the provided base image (https://github.com/lf-edge/ekuiper/pkgs/container/ekuiper%2Fbase) as the base environment. Plugins compiled using this base image will not encounter the "Error loading shared library libresolve.so.2" when deployed to the alpine version of eKuiper). Here are the specific steps to follow when using the Golang image as the base environment:
1. To use the Golang image as the base environment, you'll need to make sure that you have the correct version of the Golang image installed. Additionally, you'll need to mount the local plugin directory and eKuiper source code into a directory within Docker, so that you can access and compile the plugin project within the Docker environment.
Assuming that your plugin project is located in the local directory `/var/git`, you can map this directory to the `/go/plugins` directory within Docker using the following command:
Assuming that your plugin project is located in the local directory `/var/git`, you can map this directory to the `/go/plugins` directory within Docker using the following command:
```shell
docker run --rm -it -v /var/git:/go/plugins -w /go/plugins golang:1.20.2 /bin/sh
Expand All @@ -308,12 +308,12 @@ Assuming that your plugin project is located in the local directory `/var/git`,
kuiper
go.mod
samplePlugin
sinks
mysql.go
sinks
mysql.go
go.mod
go.work
```
Users can use the following command:
Users can use the following command:
``` shell
# In docker instance
cd /go/plugins
Expand Down Expand Up @@ -351,7 +351,7 @@ Run eKuiper in the local or **Develop** Docker, create streams and rules, set ac
## Plugin deployment
If the production environment and development environment are different, the developed plugin needs to be compiled again and deployed to the production environment. Assuming that the production environment adopts eKuiper docker to deploy, this article will describe how to deploy the plugin to the production environment.
If the production environment and development environment are different, the developed plugin needs to be compiled again and deployed to the production environment. Assuming that the production environment adopts eKuiper docker to deploy, this article will describe how to deploy the plugin to the production environment.
### Compilation
Expand All @@ -370,7 +370,7 @@ Users can use [REST API](https://github.com/lf-edge/ekuiper/blob/master/docs/en_
```shell
POST http://{$production_eKuiper_ip}:9081/plugins/sinks
Content-Type: application/json
{"name":"mysql","file":"http://{$http_server_ip}/plugins/sinks/mysqlSink.zip"}
```
Expand Down
Loading

0 comments on commit 0129a89

Please sign in to comment.