From 74cbf76f4620c339eb0a4df9e011cf0a829ec739 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Mon, 30 Sep 2024 17:10:46 +0800 Subject: [PATCH 1/2] remove schema.registry.name.strategy --- docs/ingest/ingest-from-kafka.md | 1 - docs/ingest/supported-sources-and-formats.md | 16 ---------------- src/pages/product-lifecycle.md | 1 - 3 files changed, 18 deletions(-) diff --git a/docs/ingest/ingest-from-kafka.md b/docs/ingest/ingest-from-kafka.md index 19d702b67..8a9cdc2c4 100644 --- a/docs/ingest/ingest-from-kafka.md +++ b/docs/ingest/ingest-from-kafka.md @@ -87,7 +87,6 @@ For tables with primary key constraints, if a new data record with an existing k |*schema.registry*| Confluent Schema Registry URL. Example: `http://127.0.0.1:8081`. For Avro data, you must specify a Confluent Schema Registry or an AWS Glue Schema Registry. For Protobuf data, you must specify either a schema location or a Confluent Schema Registry but not both.| |*schema.registry.username*|Conditional. User name for the schema registry. It must be specified with `schema.registry.password`.| |*schema.registry.password*|Conditional. Password for the schema registry. It must be specified with `schema.registry.username`.| -|*schema.registry.name.strategy*|Optional. Accepts `topic_name_strategy` (default), `record_name_strategy`, `topic_record_name_strategy`. If it is set to either `record_name_strategy` or `topic_record_name_strategy`, the `message` parameter must also be set. It can only be specified with *schema.registry*. | |*access_key*|Required if loading descriptors from S3. The access key ID of AWS. | |*secret_key*|Required if loading descriptors from S3. The secret access key of AWS. | |*region*|Required if loading descriptors from S3. The AWS service region. | diff --git a/docs/ingest/supported-sources-and-formats.md b/docs/ingest/supported-sources-and-formats.md index f1f81f4b4..ea881f2d0 100644 --- a/docs/ingest/supported-sources-and-formats.md +++ b/docs/ingest/supported-sources-and-formats.md @@ -40,18 +40,12 @@ When a source is created, RisingWave does not ingest data immediately. RisingWav When creating a source, you need to specify the data and encoding formats in the `FORMAT` and `ENCODE` section of the `CREATE SOURCE` or `CREATE TABLE` statement. Below is the complete list of the supported formats in RisingWave. -:::info Public Preview -`schema.registry.name.strategy` is in the public preview stage, meaning it's nearing the final product but is not yet fully stable. If you encounter any issues or have feedback, please contact us through our [Slack channel](https://www.risingwave.com/slack). Your input is valuable in helping us improve the feature. For more information, see our [Public preview feature list](/product-lifecycle/#features-in-the-public-preview-stage). -::: - ### Avro For data in Avro format, you must specify a message and a schema registry. For Kafka data in Avro, you need to provide a Confluent Schema Registry that RisingWave can get the schema from. For more details about using Schema Registry for Kafka data, see [Read schema from Schema Registry](/ingest/ingest-from-kafka.md#read-schemas-from-schema-registry). `schema.registry` can accept multiple addresses. RisingWave will send requests to all URLs and return the first successful result. -Optionally, you can define a `schema.registry.name.strategy` if `schema.registry` is set. Accepted options include `topic_name_strategy`, `record_name_strategy`, and `topic_record_name_strategy`. If either `record_name_strategy` or `topic_record_name_strategy` is used, the `message` field must also be defined. For additional details on name strategy, see [Subject name strategy](https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/index.html#subject-name-strategy). - Please be aware that: - For Avro data, you cannot specify the schema in the `schema_definition` section of a `CREATE SOURCE` or `CREATE TABLE` statement. @@ -64,7 +58,6 @@ Syntax: FORMAT PLAIN ENCODE AVRO ( schema.registry = 'schema_registry_url [, ...]', - [schema.registry.name.strategy = 'topic_name_strategy'], [message = 'main_message'], ) ``` @@ -85,8 +78,6 @@ When creating a source from streams in with Debezium AVRO, the schema of the sou `schema.registry` can accept multiple addresses. RisingWave will send requests to all URLs and return the first successful result. -Optionally, you can define a `schema.registry.name.strategy` if `schema.registry` is set. Accepted options include `topic_name_strategy`, `record_name_strategy`, and `topic_record_name_strategy`. If either `record_name_strategy` or `topic_record_name_strategy` is used, the `key.message` field must also be defined. For additional details on name strategy, see the [Subject name strategy](https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/index.html#subject-name-strategy) in the Confluent documentation. - `ignore_key` can be used to ignore the key part of given messages. By default, it is `false`. If set to `true`, only the payload part of the message will be consumed. In this case, the payload must not be empty and tombstone messages cannot be handled. Syntax: @@ -96,7 +87,6 @@ FORMAT DEBEZIUM ENCODE AVRO ( message = 'main_message', schema.location = 'location' | schema.registry = 'schema_registry_url [, ...]', - [schema.registry.name.strategy = 'topic_name_strategy'], [key.message = 'test_key'], [ignore_key = 'true | false'] ) @@ -108,15 +98,12 @@ When consuming data in AVRO from Kafka topics, the `FORMAT` and `ENCODE` section `schema.registry` can accept multiple addresses. RisingWave will send requests to all URLs and return the first successful result. -Optionally, you can define a `schema.registry.name.strategy` if `schema.registry` is set. Accepted options include `topic_name_strategy`, `record_name_strategy`, and `topic_record_name_strategy`. If either `record_name_strategy` or `topic_record_name_strategy` is used, the `message` field must also be defined. For additional details on name strategy, see [Subject name strategy](https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/index.html#subject-name-strategy). - Syntax: ```sql FORMAT UPSERT ENCODE AVRO ( schema.location = 'location' | schema.registry = 'schema_registry_url [, ...]', - [schema.registry.name.strategy = 'topic_name_strategy'], [message = 'main_message'], ) ``` @@ -211,8 +198,6 @@ For data in protobuf format, you must specify a message (fully qualified by pack `schema.registry` can accept multiple addresses. RisingWave will send requests to all URLs and return the first successful result. -Optionally, you can define a `schema.registry.name.strategy` if `schema.registry` is set. Accepted options include `topic_name_strategy`, `record_name_strategy`, and `topic_record_name_strategy`. For additional details on name strategy, see [Subject name strategy](https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/index.html#subject-name-strategy). - :::info For protobuf data, you cannot specify the schema in the `schema_definition` section of a `CREATE SOURCE` or `CREATE TABLE` statement. @@ -232,7 +217,6 @@ FORMAT PLAIN ENCODE PROTOBUF ( message = 'com.example.MyMessage', schema.location = 'location' | schema.registry = 'schema_registry_url [, ...]', - [schema.registry.name.strategy = 'topic_name_strategy'], ) ``` diff --git a/src/pages/product-lifecycle.md b/src/pages/product-lifecycle.md index 42cb40a58..3ef0a1b7e 100644 --- a/src/pages/product-lifecycle.md +++ b/src/pages/product-lifecycle.md @@ -38,7 +38,6 @@ Below is a list of all features in the public preview phase: | [Version column](/docs/current/sql-create-table/) | 2024.6 | 1.9 | | [Snowflake sink](/docs/current/sink-to-snowflake/) | 2024.5 | 1.9 | | [Subscription](/docs/current/subscription) | 2024.5 | 1.9 | -| [`schema.registry.name.strategy`](/docs/current/supported-sources-and-formats/) | 2024.4 | 1.8 | | [RisingWave as PostgreSQL FDW](/docs/current/risingwave-as-postgres-fdw/) | 2024.4 | 1.9 | | [Iceberg source](/docs/current/ingest-from-iceberg/) | 2024.3 | 1.8 | | [Google BigQuery sink](/docs/current/sink-to-bigquery/) | 2023.11 | 1.4 | From fd51b57b11dc0e3d1c06cd9b5381268a331aa947 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Fri, 11 Oct 2024 10:45:02 +0800 Subject: [PATCH 2/2] fix --- docs/ingest/supported-sources-and-formats.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/docs/ingest/supported-sources-and-formats.md b/docs/ingest/supported-sources-and-formats.md index ea881f2d0..5afd1a1ff 100644 --- a/docs/ingest/supported-sources-and-formats.md +++ b/docs/ingest/supported-sources-and-formats.md @@ -52,13 +52,14 @@ Please be aware that: - The timestamp displayed in RisingWave may be different from the upstream system as timezone information is lost in Avro serialization. +- RisingWave takes [`TopicNameStrategy`](https://developer.confluent.io/courses/schema-registry/schema-subjects/#topicnamestrategy) as the default subject name strategy for the schema registry and looks for the schema with the subject name `{ topic name }-value`. + Syntax: ```sql FORMAT PLAIN ENCODE AVRO ( - schema.registry = 'schema_registry_url [, ...]', - [message = 'main_message'], + schema.registry = 'schema_registry_url [, ...]', ) ``` @@ -66,7 +67,7 @@ You can ingest Avro map type into RisingWave [map type](/sql/data-types/data-typ ```sql FORMAT [ DEBEZIUM | UPSERT | PLAIN ] ENCODE AVRO ( - map.handling.mode = 'map' | 'jsonb' + map.handling.mode = 'map' | 'jsonb' ) ``` @@ -85,10 +86,9 @@ Syntax: ```sql FORMAT DEBEZIUM ENCODE AVRO ( - message = 'main_message', - schema.location = 'location' | schema.registry = 'schema_registry_url [, ...]', - [key.message = 'test_key'], - [ignore_key = 'true | false'] + message = 'main_message', + schema.registry = 'schema_registry_url [, ...]', + [ignore_key = 'true | false'] ) ``` @@ -104,7 +104,6 @@ Syntax: FORMAT UPSERT ENCODE AVRO ( schema.location = 'location' | schema.registry = 'schema_registry_url [, ...]', - [message = 'main_message'], ) ```