Skip to content

Commit

Permalink
Merge branch 'main' into add-serve-results
Browse files Browse the repository at this point in the history
  • Loading branch information
hengm3467 authored Dec 25, 2024
2 parents 10481c2 + d0cc9f1 commit 91b5b5a
Show file tree
Hide file tree
Showing 5 changed files with 113 additions and 6 deletions.
1 change: 1 addition & 0 deletions changelog/product-lifecycle.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Below is a list of all features in the public preview phase:

| Feature name | Start version |
| :-- | :-- |
| [Ingest data from MySQL table](/integrations/sources/mysql-table)| 2.2 |
| [EXPLAIN FORMAT JSON](/sql/commands/sql-explain#explain-options)| 2.2 |
| [Ingest data from Postgres table](/integrations/sources/postgresql-table) | 2.1 |
| [Ingest data from webhook](/integrations/sources/webhook) | 2.1 |
Expand Down
4 changes: 2 additions & 2 deletions get-started/quickstart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ Unlike other deployment modes, for instance [Docker Compose](/deploy/risingwave-

For state store, we will use the embedded `LocalFs` Object Store, eliminating the need for an external service like `minio` or `s3`; for meta store, we will use the embedded `SQLite` database, eliminating the need for an external service like `etcd`.

By default, the RisingWave standalone mode will store its data in `~/risingwave`, which includes both `Metadata` and `State Data`.
By default, the RisingWave standalone mode will store its data in `~/.risingwave`, which includes both `Metadata` and `State Data`.

For a batteries-included setup, with `monitoring` tools and external services like `kafka` fully included, you can use [Docker Compose](/deploy/risingwave-docker-compose) instead. If you would like to set up these external services manually, you may check out RisingWave's [Docker Compose](https://github.com/risingwavelabs/risingwave/blob/main/docker/docker-compose.yml), and run these services using the same configurations.

Expand All @@ -147,7 +147,7 @@ The instance of RisingWave standalone mode can run without any configuration. Ho

The main options which new users may require would be the state store directory (`--state-store-directory`) and in-memory mode (`--in-memory`).

`--state-store-directory` specifies the new directory where the cluster's `Metadata` and `State Data` will reside. The default is to store it in the `~/risingwave` folder.
`--state-store-directory` specifies the new directory where the cluster's `Metadata` and `State Data` will reside. The default is to store it in the `~/.risingwave` folder.

```bash
# Reconfigure RisingWave to be stored under 'projects' folder instead.
Expand Down
6 changes: 3 additions & 3 deletions ingestion/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -74,11 +74,11 @@ The statement will create a streaming job that continuously ingests data from th
4. **Stronger consistency guarantee**: When using a table with connectors, all downstream jobs will be guaranteed to have a consistent view of the data persisted in the table; while for source, different jobs may see inconsistent results due to different ingestion speed or data retention in the external system.
5. **Greater flexibility**: Like regular tables, you can use DML statements like [INSERT](/sql/commands/sql-insert), [UPDATE](/sql/commands/sql-update) and [DELETE](/sql/commands/sql-delete) to insert or modify data in tables with connectors, and use [CREATE SINK INTO TABLE](/sql/commands/sql-create-sink-into) to merge other data streams into the table.

### PostgreSQL table
### Table-valued function

RisingWave supports using the table-valued function `postgres_query` to directly query PostgreSQL databases. This function connects to a specified PostgreSQL instance, executes the provided SQL query, and returns the results as a table in RisingWave.
RisingWave supports using the table-valued function (TVF) `postgres_query` or `mysql_query` to directly query PostgreSQL or MySQL databases. This function connects to a specified instance, executes the provided SQL query, and returns the results as a table in RisingWave.

To use it, specify connection details (such as hostname, port, username, password, database name) and the desired SQL query. This makes it easier to integrate PostgreSQL data directly into RisingWave workflows without needing additional data transfer steps. For more information, see [Ingest data from Postgres tables](/integrations/sources/postgresql-table).
To use it, specify connection details (such as hostname, port, username, password, database name) and the desired SQL query. This makes it easier to integrate databases directly into RisingWave workflows without needing additional data transfer steps. For more information, see [Ingest data from Postgres tables](/integrations/sources/postgresql-table) and [Ingest data from MySQL tables](/integrations/sources/mysql-table).

## DML on tables

Expand Down
102 changes: 102 additions & 0 deletions integrations/sources/mysql-table.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
---
title: "Ingest data from MySQL table"
description: "Describes how to ingest data from MySQL table to RisingWave using table-valued function."
sidebarTitle: MySQL table
---

RisingWave allows you to query MySQL tables directly with the `mysql_query` table-valued function (TVF). It offers a simpler alternative to Change Data Capture (CDC) when working with MySQL data in RisingWave.

Unlike CDC, which continuously syncs data changes, this function lets you fetch data directly from MySQL when needed. Therefore, this approach is ideal for static or infrequently updated data, as it's more resource-efficient than maintaining a constant CDC connection.

<Note>
**PUBLIC PREVIEW**

This feature is currently in public preview, meaning it is nearing the final product but may not yet be fully stable. If you encounter any issues or have feedback, please reach out to us via our [Slack channel](https://www.risingwave.com/slack). Your input is valuable in helping us improve this feature. For more details, see our [Public Preview Feature List](/changelog/product-lifecycle#features-in-the-public-preview-stage).
</Note>

<Note>
Added in version 2.2.
</Note>

## Syntax

Define `mysql_query` as follows:

```sql
mysql_query(
hostname varchar, -- Database hostname
port varchar, -- Database port
username varchar, -- Authentication username
password varchar, -- Authentication password
database_name varchar, -- Target database name
query varchar -- SQL query to execute
)
```

## Data type mapping

The following table shows how MySQL data types are mapped to RisingWave data types:

| MySQL Type | RisingWave Type |
|:-----------|:----------------|
| `bit(1)` | `boolean` |
| `bit(>1)` | `bytea` |
| `bool`/`boolean` | `smallint` |
| `tinyint` | `smallint` |
| `smallint` | `smallint` |
| `mediumint` | `int` |
| `int` | `int` |
| `bigint` | `bigint` |
| `float` | `float32` |
| `double` | `float64` |
| `decimal` | `decimal` |
| `numeric` | `decimal` |
| `year` | `int` |
| `date` | `date` |
| `time` | `time` |
| `datetime` | `timestamp` |
| `timestamp` | `timestamptz` |
| `varchar` | `varchar` |
| `char` | `varchar` |
| `json` | `jsonb` |
| `blob` | `bytea` |
| `tinyblob` | `bytea` |
| `mediumblob` | `bytea` |
| `longblob` | `bytea` |
| `array` | *unsupported* |
| `enum` | *unsupported* |
| `set` | *unsupported* |
| `geometry` | *unsupported* |
| `null` | *unsupported* |

## Example

1. In your MySQL database, create a table and populate it with sample data of various data types.

```sql
CREATE TABLE test (
id bigint primary key, v0 bit, v1 bool, v2 tinyint(1),
v3 tinyint(2), v4 smallint, v5 mediumint, v6 integer,
v7 bigint, v8 float, v9 double, v10 numeric(4, 2),
v11 decimal(4, 2), v12 char(255), v13 varchar(255),
v14 bit(10), v15 tinyblob, v16 blob, v17 mediumblob,
v18 longblob, v19 date, v20 time, v21 timestamp,
v22 json
);

INSERT INTO test SELECT
1 as id, true as v0, true as v1, 2 as v2, 3 as v3, 4 as v4, 5 as v5,
6 as v6, 7 as v7, 1.08 as v8, 1.09 as v9, 1.10 as v10, 1.11 as v11,
'char' as v12, 'varchar' as v13, b'1010' as v14, x'16' as v15, x'17' as v16,
x'18' as v17, x'19' as v18, '2021-01-01' as v19, '12:34:56' as v20,
'2021-01-01 12:34:56' as v21, JSON_OBJECT('key1', 1, 'key2', 'abc');
```

2. In RisingWave, use `postgres_query` function to perform the query.

```sql
SELECT *
FROM mysql_query('$MYSQL_HOST', '$MYSQL_TCP_PORT', '$RISEDEV_MYSQL_USER', '$MYSQL_PWD', 'tvf', 'select * from test;');
----RESULT
1 t 1 2 3 4 5 6 7 1.08 1.09 1.10 1.11 char varchar \x000a \x16 \x17 \x18 \x19 2021-01-01 12:34:56 2021-01-01 12:34:56+00:00 {"key1": 1, "key2": "abc"}
```
6 changes: 5 additions & 1 deletion integrations/sources/postgresql-table.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,18 @@ sidebarTitle: PostgreSQL table

RisingWave allows you to query PostgreSQL tables directly with the `postgres_query` table-valued function (TVF). It offers a simpler alternative to Change Data Capture (CDC) when working with PostgreSQL data in RisingWave.

Unlike CDC, which continuously syncs data changes, this function lets you fetch data directly from PostgreSQL when needed. Therefore, this approach is ideal static or infrequently updated data, as it's more resource-efficient than maintaining a constant CDC connection.
Unlike CDC, which continuously syncs data changes, this function lets you fetch data directly from PostgreSQL when needed. Therefore, this approach is ideal for static or infrequently updated data, as it's more resource-efficient than maintaining a constant CDC connection.

<Note>
**PUBLIC PREVIEW**

This feature is currently in public preview, meaning it is nearing the final product but may not yet be fully stable. If you encounter any issues or have feedback, please reach out to us via our [Slack channel](https://www.risingwave.com/slack). Your input is valuable in helping us improve this feature. For more details, see our [Public Preview Feature List](/changelog/product-lifecycle#features-in-the-public-preview-stage).
</Note>

<Note>
Added in version 2.1.
</Note>

## Syntax

Define `postgres_query` as follows:
Expand Down

0 comments on commit 91b5b5a

Please sign in to comment.