Skip to content

Commit

Permalink
[MINOR] docs: Polish Gravitino Flink connector document (#6315)
Browse files Browse the repository at this point in the history 
### What changes were proposed in this pull request?

polish Flink document


### Why are the changes needed?

Polish the document

### Does this PR introduce _any_ user-facing change?

no

### How was this patch tested?

just document
  • Loading branch information
@FANNG1 @web-flow
FANNG1 authored and web-flow committed Jan 17, 2025
1 parent b19506c commit 510ad6e
Show file tree
Hide file tree
Showing 4 changed files with 28 additions and 25 deletions.
2 changes: 1 addition & 1 deletion docs/flink-connector/flink-catalog-hive.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Supports most DDL and DML operations in Flink SQL, except such operations:
```sql

// Suppose hive_a is the Hive catalog name managed by Gravitino
USE hive_a;
USE CATALOG hive_a;

CREATE DATABASE IF NOT EXISTS mydatabase;
USE mydatabase;
Expand Down
21 changes: 11 additions & 10 deletions docs/flink-connector/flink-catalog-iceberg.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,12 @@ To enable the Flink connector, you must download the Iceberg Flink runtime JAR a
- `CREATE TABLE LIKE` clause

## SQL example

```sql

-- Suppose iceberg_a is the Iceberg catalog name managed by Gravitino

USE iceberg_a;
USE CATALOG iceberg_a;

CREATE DATABASE IF NOT EXISTS mydatabase;
USE mydatabase;
Expand All @@ -59,15 +60,15 @@ SELECT * FROM sample WHERE data = 'B';
The Gravitino Flink connector transforms the following properties in a catalog to Flink connector configuration.


| Gravitino catalog property name | Flink Iceberg connector configuration | Description | Since Version |
|---------------------------------|---------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------|
| `catalog-backend` | `catalog-type` | Catalog backend type, currently, only `Hive` Catalog is supported, `JDBC` and `Rest` in Continuous Validation | 0.8.0-incubating |
| `uri` | `uri` | Catalog backend URI | 0.8.0-incubating |
| `warehouse` | `warehouse` | Catalog backend warehouse | 0.8.0-incubating |
| `io-impl` | `io-impl` | The IO implementation for `FileIO` in Iceberg. | 0.8.0-incubating |
| `oss-endpoint` | `oss.endpoint` | The endpoint of Aliyun OSS service. | 0.8.0-incubating |
| `oss-access-key-id` | `client.access-key-id` | The static access key ID used to access OSS data. | 0.8.0-incubating |
| `oss-secret-access-key` | `client.access-key-secret` | The static secret access key used to access OSS data. | 0.8.0-incubating |
| Gravitino catalog property name | Flink Iceberg connector configuration | Description | Since Version |
|---------------------------------|---------------------------------------|---------------------------------------------------------------------------------------------------------------|------------------|
| `catalog-backend` | `catalog-type` | Catalog backend type, currently, only `Hive` Catalog is supported, `JDBC` and `Rest` in Continuous Validation | 0.8.0-incubating |
| `uri` | `uri` | Catalog backend URI | 0.8.0-incubating |
| `warehouse` | `warehouse` | Catalog backend warehouse | 0.8.0-incubating |
| `io-impl` | `io-impl` | The IO implementation for `FileIO` in Iceberg. | 0.8.0-incubating |
| `oss-endpoint` | `oss.endpoint` | The endpoint of Aliyun OSS service. | 0.8.0-incubating |
| `oss-access-key-id` | `client.access-key-id` | The static access key ID used to access OSS data. | 0.8.0-incubating |
| `oss-secret-access-key` | `client.access-key-secret` | The static secret access key used to access OSS data. | 0.8.0-incubating |

Gravitino catalog property names with the prefix `flink.bypass.` are passed to Flink iceberg connector. For example, using `flink.bypass.clients` to pass the `clients` to the Flink iceberg connector.

Expand Down
28 changes: 14 additions & 14 deletions docs/flink-connector/flink-catalog-paimon.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ license: "This software is licensed under the Apache License version 2."
---

This document provides a comprehensive guide on configuring and using Apache Gravitino Flink connector to access the Paimon catalog managed by the Gravitino server.

## Capabilities

### Supported Paimon Table Types
Expand All @@ -32,27 +33,26 @@ Supports most DDL and DML operations in Flink SQL, except such operations:

* Paimon 0.8

Higher version like 0.9 or above may also supported but have not been tested fully.
Higher version like 0.9 or above may also support but have not been tested fully.

## Getting Started

### Prerequisites

Place the following JAR files in the lib directory of your Flink installation:

* paimon-flink-1.18-0.8.2.jar

* gravitino-flink-connector-runtime-\${flinkMajorVersion}_$scalaVersion.jar
- `paimon-flink-1.18-${paimon-version}.jar`
- `gravitino-flink-connector-runtime-1.18_2.12-${gravitino-version}.jar`

### SQL Example

```sql

-- Suppose paimon_catalog is the Paimon catalog name managed by Gravitino
use catalog paimon_catalog;
USE CATALOG paimon_catalog;
-- Execute statement succeed.

show databases;
SHOW DATABASES;
-- +---------------------+
-- | database name |
-- +---------------------+
Expand All @@ -71,23 +71,23 @@ CREATE TABLE paimon_tabla_a (
bb BIGINT
);

show tables;
SHOW TABLES;
-- +----------------+
-- | table name |
-- +----------------+
-- | paimon_table_a |
-- +----------------+


select * from paimon_table_a;
SELECT * FROM paimon_table_a;
-- Empty set

insert into paimon_table_a(aa,bb) values(1,2);
INSERT INTO paimon_table_a(aa,bb) VALUES(1,2);
-- [INFO] Submitting SQL update statement to the cluster...
-- [INFO] SQL update statement has been successfully submitted to the cluster:
-- Job ID: 74c0c678124f7b452daf08c399d0fee2

select * from paimon_table_a;
SELECT * FROM paimon_table_a;
-- +----+----+
-- | aa | bb |
-- +----+----+
Expand All @@ -100,9 +100,9 @@ select * from paimon_table_a;

Gravitino Flink connector will transform below property names which are defined in catalog properties to Flink Paimon connector configuration.

| Gravitino catalog property name | Flink Paimon connector configuration | Description | Since Version |
|---------------------------------|----------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|
| `catalog-backend` | `metastore` | Catalog backend of Gravitino Paimon catalog. Supports `filesystem`. | 0.8.0-incubating |
| `warehouse` | `warehouse` | Warehouse directory of catalog. `file:///user/hive/warehouse-paimon/` for local fs, `hdfs://namespace/hdfs/path` for HDFS , `s3://{bucket-name}/path/` for S3 or `oss://{bucket-name}/path` for Aliyun OSS | 0.8.0-incubating |
| Gravitino catalog property name | Flink Paimon connector configuration | Description | Since Version |
|---------------------------------|--------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|
| `catalog-backend` | `metastore` | Catalog backend of Gravitino Paimon catalog. Supports `filesystem`. | 0.8.0-incubating |
| `warehouse` | `warehouse` | Warehouse directory of catalog. `file:///user/hive/warehouse-paimon/` for local fs, `hdfs://namespace/hdfs/path` for HDFS , `s3://{bucket-name}/path/` for S3 or `oss://{bucket-name}/path` for Aliyun OSS | 0.8.0-incubating |

Gravitino catalog property names with the prefix `flink.bypass.` are passed to Flink Paimon connector. For example, using `flink.bypass.clients` to pass the `clients` to the Flink Paimon connector.
2 changes: 2 additions & 0 deletions docs/flink-connector/flink-connector.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ This capability allows users to perform federation queries, accessing data from
## Capabilities

1. Supports [Hive catalog](flink-catalog-hive.md)
1. Supports [Iceberg catalog](flink-catalog-iceberg.md)
1. Supports [Paimon catalog](flink-catalog-paimon.md)
2. Supports most DDL and DML SQLs.

## Requirement
Expand Down

0 comments on commit 510ad6e

Please sign in to comment.