Skip to content

Commit

Permalink
[Docs][Connector-V2][Http]Reconstruct the Http connector document (#4962
Browse files Browse the repository at this point in the history 
)

Co-authored-by: chenzy15 <[email protected]>
  • Loading branch information
@MonsterChenzhuo
MonsterChenzhuo and chenzy15 committed Jun 25, 2023
1 parent 245debe commit 447d25e
Show file tree
Hide file tree
Showing 2 changed files with 134 additions and 126 deletions.
72 changes: 30 additions & 42 deletions docs/en/connector-v2/sink/Http.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,69 +2,57 @@

> Http sink connector
## Description
## Support Those Engines

Used to launch web hooks using data.
> Spark<br/>
> Flink<br/>
> SeaTunnel Zeta<br/>
> For example, if the data from upstream is [`age: 12, name: tyrantlucifer`], the body content is the following: `{"age": 12, "name": "tyrantlucifer"}`
**Tips: Http sink only support `post json` webhook and the data from source will be treated as body content in web hook.**

## Key features
## Key Features

- [ ] [exactly-once](../../concept/connector-v2-features.md)
- [ ] [cdc](../../concept/connector-v2-features.md)

## Options

| name | type | required | default value |
|-----------------------------|--------|----------|---------------|
| url | String | Yes | - |
| headers | Map | No | - |
| params | Map | No | - |
| retry | int | No | - |
| retry_backoff_multiplier_ms | int | No | 100 |
| retry_backoff_max_ms | int | No | 10000 |
| common-options | | no | - |

### url [String]

http request url

### headers [Map]

http headers

### params [Map]

http params
## Description

### retry [int]
Used to launch web hooks using data.

The max retry times if request http return to `IOException`
> For example, if the data from upstream is [`age: 12, name: tyrantlucifer`], the body content is the following: `{"age": 12, "name": "tyrantlucifer"}`
### retry_backoff_multiplier_ms [int]
**Tips: Http sink only support `post json` webhook and the data from source will be treated as body content in web hook.**

The retry-backoff times(millis) multiplier if request http failed
## Supported DataSource Info

### retry_backoff_max_ms [int]
In order to use the Http connector, the following dependencies are required.
They can be downloaded via install-plugin.sh or from the Maven central repository.

The maximum retry-backoff times(millis) if request http failed
| Datasource | Supported Versions | Dependency |
|------------|--------------------|------------------------------------------------------------------------------------------------------------|
| Http | universal | [Download](https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-connectors-v2/connector-http) |

### common options
## Sink Options

Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details
| Name | Type | Required | Default | Description |
|-----------------------------|--------|----------|---------|-----------------------------------------------------------------------------------------------------|
| url | String | Yes | - | Http request url |
| headers | Map | No | - | Http headers |
| params | Map | No | - | Http params |
| retry | Int | No | - | The max retry times if request http return to `IOException` |
| retry_backoff_multiplier_ms | Int | No | 100 | The retry-backoff times(millis) multiplier if request http failed |
| retry_backoff_max_ms | Int | No | 10000 | The maximum retry-backoff times(millis) if request http failed |
| common-options | | No | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

## Example

simple:

```hocon
Http {
url = "http://localhost/test/webhook"
headers {
token = "9e32e859ef044462a257e1fc76730066"
}
url = "http://localhost/test/webhook"
headers {
token = "9e32e859ef044462a257e1fc76730066"
}
}
```

## Changelog
Expand Down
188 changes: 104 additions & 84 deletions docs/en/connector-v2/source/Http.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,13 @@

> Http source connector
## Description
## Support Those Engines

Used to read data from Http.
> Spark<br/>
> Flink<br/>
> SeaTunnel Zeta<br/>
## Key features
## Key Features

- [x] [batch](../../concept/connector-v2-features.md)
- [x] [stream](../../concept/connector-v2-features.md)
Expand All @@ -15,66 +17,112 @@ Used to read data from Http.
- [ ] [parallelism](../../concept/connector-v2-features.md)
- [ ] [support user-defined split](../../concept/connector-v2-features.md)

## Options

| name | type | required | default value |
|-----------------------------|---------|----------|---------------|
| url | String | Yes | - |
| schema | Config | No | - |
| schema.fields | Config | No | - |
| json_field | Config | No | - |
| content_json | String | No | - |
| format | String | No | json |
| method | String | No | get |
| headers | Map | No | - |
| params | Map | No | - |
| body | String | No | - |
| poll_interval_ms | int | No | - |
| retry | int | No | - |
| retry_backoff_multiplier_ms | int | No | 100 |
| retry_backoff_max_ms | int | No | 10000 |
| enable_multi_lines | boolean | No | false |
| common-options | config | No | - |

### url [String]

http request url

### method [String]

http request method, only supports GET, POST method.

### headers [Map]

http headers

### params [Map]

http params

### body [String]

http body

### poll_interval_ms [int]
## Description

request http api interval(millis) in stream mode
Used to read data from Http.

### retry [int]
## Key features

The max retry times if request http return to `IOException`
- [x] [batch](../../concept/connector-v2-features.md)
- [x] [stream](../../concept/connector-v2-features.md)
- [ ] [exactly-once](../../concept/connector-v2-features.md)
- [ ] [column projection](../../concept/connector-v2-features.md)
- [ ] [parallelism](../../concept/connector-v2-features.md)
- [ ] [support user-defined split](../../concept/connector-v2-features.md)

### retry_backoff_multiplier_ms [int]
Supported DataSource Info
-------------------------

In order to use the Http connector, the following dependencies are required.
They can be downloaded via install-plugin.sh or from the Maven central repository.

| Datasource | Supported Versions | Dependency |
|------------|--------------------|------------------------------------------------------------------------------------------------------------|
| Http | universal | [Download](https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-connectors-v2/connector-http) |

## Source Options

| Name | Type | Required | Default | Description |
|-----------------------------|---------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------|
| url | String | Yes | - | Http request url. |
| schema | Config | No | - | Http and seatunnel data structure mapping |
| schema.fields | Config | No | - | The schema fields of upstream data |
| json_field | Config | No | - | This parameter helps you configure the schema,so this parameter must be used with schema. |
| content_json | String | No | - | This parameter can get some json data.If you only need the data in the 'book' section, configure `content_field = "$.store.book.*"`. |
| format | String | No | json | The format of upstream data, now only support `json` `text`, default `json`. |
| method | String | No | get | Http request method, only supports GET, POST method. |
| headers | Map | No | - | Http headers. |
| params | Map | No | - | Http params. |
| body | String | No | - | Http body. |
| poll_interval_ms | Int | No | - | Request http api interval(millis) in stream mode. |
| retry | Int | No | - | The max retry times if request http return to `IOException`. |
| retry_backoff_multiplier_ms | Int | No | 100 | The retry-backoff times(millis) multiplier if request http failed. |
| retry_backoff_max_ms | Int | No | 10000 | The maximum retry-backoff times(millis) if request http failed |
| enable_multi_lines | Boolean | No | false | |
| common-options | | No | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details |

## How to Create a Http Data Synchronization Jobs

The retry-backoff times(millis) multiplier if request http failed
```hocon
env {
execution.parallelism = 1
job.mode = "BATCH"
}
### retry_backoff_max_ms [int]
source {
Http {
result_table_name = "http"
url = "http://mockserver:1080/example/http"
method = "GET"
format = "json"
schema = {
fields {
c_map = "map<string, string>"
c_array = "array<int>"
c_string = string
c_boolean = boolean
c_tinyint = tinyint
c_smallint = smallint
c_int = int
c_bigint = bigint
c_float = float
c_double = double
c_bytes = bytes
c_date = date
c_decimal = "decimal(38, 18)"
c_timestamp = timestamp
c_row = {
C_MAP = "map<string, string>"
C_ARRAY = "array<int>"
C_STRING = string
C_BOOLEAN = boolean
C_TINYINT = tinyint
C_SMALLINT = smallint
C_INT = int
C_BIGINT = bigint
C_FLOAT = float
C_DOUBLE = double
C_BYTES = bytes
C_DATE = date
C_DECIMAL = "decimal(38, 18)"
C_TIMESTAMP = timestamp
}
}
}
}
}
The maximum retry-backoff times(millis) if request http failed
# Console printing of the read Http data
sink {
Console {
parallelism = 1
}
}
```

### format [String]
## Parameter Interpretation

the format of upstream data, now only support `json` `text`, default `json`.
### format

when you assign format is `json`, you should also assign schema option, for example:

Expand Down Expand Up @@ -126,13 +174,7 @@ connector will generate data as the following:
|----------------------------------------------------------|
| {"code": 200, "data": "get success", "success": true} |

### schema [Config]

#### fields [Config]

the schema fields of upstream data

### content_json [String]
### content_json

This parameter can get some json data.If you only need the data in the 'book' section, configure `content_field = "$.store.book.*"`.

Expand Down Expand Up @@ -207,7 +249,7 @@ Here is an example:
- Test data can be found at this link [mockserver-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json)
- See this link for task configuration [http_contentjson_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_contentjson_to_assert.conf).

### json_field [Config]
### json_field

This parameter helps you configure the schema,so this parameter must be used with schema.

Expand Down Expand Up @@ -268,28 +310,6 @@ source {
- Test data can be found at this link [mockserver-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json)
- See this link for task configuration [http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).

### common options

Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details

## Example

simple:

```hocon
Http {
url = "https://tyrantlucifer.com/api/getDemoData"
schema {
fields {
code = int
message = string
data = string
ok = boolean
}
}
}
```

## Changelog

### 2.2.0-beta 2022-09-26
Expand Down

0 comments on commit 447d25e

Please sign in to comment.