Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: remove schema.registry.name.strategy #2660

Merged
merged 3 commits into from
Oct 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion docs/ingest/ingest-from-kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,6 @@ For tables with primary key constraints, if a new data record with an existing k
|*schema.registry*| Confluent Schema Registry URL. Example: `http://127.0.0.1:8081`. For Avro data, you must specify a Confluent Schema Registry or an AWS Glue Schema Registry. For Protobuf data, you must specify either a schema location or a Confluent Schema Registry but not both.|
|*schema.registry.username*|Conditional. User name for the schema registry. It must be specified with `schema.registry.password`.|
|*schema.registry.password*|Conditional. Password for the schema registry. It must be specified with `schema.registry.username`.|
|*schema.registry.name.strategy*|Optional. Accepts `topic_name_strategy` (default), `record_name_strategy`, `topic_record_name_strategy`. If it is set to either `record_name_strategy` or `topic_record_name_strategy`, the `message` parameter must also be set. It can only be specified with *schema.registry*. |
|*access_key*|Required if loading descriptors from S3. The access key ID of AWS. |
|*secret_key*|Required if loading descriptors from S3. The secret access key of AWS. |
|*region*|Required if loading descriptors from S3. The AWS service region. |
Expand Down
31 changes: 7 additions & 24 deletions docs/ingest/supported-sources-and-formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,40 +40,34 @@ When a source is created, RisingWave does not ingest data immediately. RisingWav

When creating a source, you need to specify the data and encoding formats in the `FORMAT` and `ENCODE` section of the `CREATE SOURCE` or `CREATE TABLE` statement. Below is the complete list of the supported formats in RisingWave.

:::info Public Preview
`schema.registry.name.strategy` is in the public preview stage, meaning it's nearing the final product but is not yet fully stable. If you encounter any issues or have feedback, please contact us through our [Slack channel](https://www.risingwave.com/slack). Your input is valuable in helping us improve the feature. For more information, see our [Public preview feature list](/product-lifecycle/#features-in-the-public-preview-stage).
:::

### Avro

For data in Avro format, you must specify a message and a schema registry. For Kafka data in Avro, you need to provide a Confluent Schema Registry that RisingWave can get the schema from. For more details about using Schema Registry for Kafka data, see [Read schema from Schema Registry](/ingest/ingest-from-kafka.md#read-schemas-from-schema-registry).

`schema.registry` can accept multiple addresses. RisingWave will send requests to all URLs and return the first successful result.

Optionally, you can define a `schema.registry.name.strategy` if `schema.registry` is set. Accepted options include `topic_name_strategy`, `record_name_strategy`, and `topic_record_name_strategy`. If either `record_name_strategy` or `topic_record_name_strategy` is used, the `message` field must also be defined. For additional details on name strategy, see [Subject name strategy](https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/index.html#subject-name-strategy).

Please be aware that:

- For Avro data, you cannot specify the schema in the `schema_definition` section of a `CREATE SOURCE` or `CREATE TABLE` statement.

- The timestamp displayed in RisingWave may be different from the upstream system as timezone information is lost in Avro serialization.

- RisingWave takes [`TopicNameStrategy`](https://developer.confluent.io/courses/schema-registry/schema-subjects/#topicnamestrategy) as the default subject name strategy for the schema registry and looks for the schema with the subject name `{ topic name }-value`.

Syntax:

```sql
FORMAT PLAIN
ENCODE AVRO (
schema.registry = 'schema_registry_url [, ...]',
[schema.registry.name.strategy = 'topic_name_strategy'],
[message = 'main_message'],
schema.registry = 'schema_registry_url [, ...]',
)
```

You can ingest Avro map type into RisingWave [map type](/sql/data-types/data-type-map.md) or jsonb:

```sql
FORMAT [ DEBEZIUM | UPSERT | PLAIN ] ENCODE AVRO (
map.handling.mode = 'map' | 'jsonb'
map.handling.mode = 'map' | 'jsonb'
)
```

Expand All @@ -85,20 +79,16 @@ When creating a source from streams in with Debezium AVRO, the schema of the sou

`schema.registry` can accept multiple addresses. RisingWave will send requests to all URLs and return the first successful result.

Optionally, you can define a `schema.registry.name.strategy` if `schema.registry` is set. Accepted options include `topic_name_strategy`, `record_name_strategy`, and `topic_record_name_strategy`. If either `record_name_strategy` or `topic_record_name_strategy` is used, the `key.message` field must also be defined. For additional details on name strategy, see the [Subject name strategy](https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/index.html#subject-name-strategy) in the Confluent documentation.

`ignore_key` can be used to ignore the key part of given messages. By default, it is `false`. If set to `true`, only the payload part of the message will be consumed. In this case, the payload must not be empty and tombstone messages cannot be handled.

Syntax:

```sql
FORMAT DEBEZIUM
ENCODE AVRO (
message = 'main_message',
schema.location = 'location' | schema.registry = 'schema_registry_url [, ...]',
[schema.registry.name.strategy = 'topic_name_strategy'],
[key.message = 'test_key'],
[ignore_key = 'true | false']
message = 'main_message',
schema.registry = 'schema_registry_url [, ...]',
[ignore_key = 'true | false']
)
```

Expand All @@ -108,16 +98,12 @@ When consuming data in AVRO from Kafka topics, the `FORMAT` and `ENCODE` section

`schema.registry` can accept multiple addresses. RisingWave will send requests to all URLs and return the first successful result.

Optionally, you can define a `schema.registry.name.strategy` if `schema.registry` is set. Accepted options include `topic_name_strategy`, `record_name_strategy`, and `topic_record_name_strategy`. If either `record_name_strategy` or `topic_record_name_strategy` is used, the `message` field must also be defined. For additional details on name strategy, see [Subject name strategy](https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/index.html#subject-name-strategy).

Syntax:

```sql
FORMAT UPSERT
ENCODE AVRO (
schema.location = 'location' | schema.registry = 'schema_registry_url [, ...]',
[schema.registry.name.strategy = 'topic_name_strategy'],
[message = 'main_message'],
)
```

Expand Down Expand Up @@ -211,8 +197,6 @@ For data in protobuf format, you must specify a message (fully qualified by pack

`schema.registry` can accept multiple addresses. RisingWave will send requests to all URLs and return the first successful result.

Optionally, you can define a `schema.registry.name.strategy` if `schema.registry` is set. Accepted options include `topic_name_strategy`, `record_name_strategy`, and `topic_record_name_strategy`. For additional details on name strategy, see [Subject name strategy](https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/index.html#subject-name-strategy).

:::info

For protobuf data, you cannot specify the schema in the `schema_definition` section of a `CREATE SOURCE` or `CREATE TABLE` statement.
Expand All @@ -232,7 +216,6 @@ FORMAT PLAIN
ENCODE PROTOBUF (
message = 'com.example.MyMessage',
schema.location = 'location' | schema.registry = 'schema_registry_url [, ...]',
[schema.registry.name.strategy = 'topic_name_strategy'],
)
```

Expand Down
1 change: 0 additions & 1 deletion src/pages/product-lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ Below is a list of all features in the public preview phase:
| [Version column](/docs/current/sql-create-table/) | 2024.6 | 1.9 |
| [Snowflake sink](/docs/current/sink-to-snowflake/) | 2024.5 | 1.9 |
| [Subscription](/docs/current/subscription) | 2024.5 | 1.9 |
| [`schema.registry.name.strategy`](/docs/current/supported-sources-and-formats/) | 2024.4 | 1.8 |
| [RisingWave as PostgreSQL FDW](/docs/current/risingwave-as-postgres-fdw/) | 2024.4 | 1.9 |
| [Iceberg source](/docs/current/ingest-from-iceberg/) | 2024.3 | 1.8 |
| [Google BigQuery sink](/docs/current/sink-to-bigquery/) | 2023.11 | 1.4 |
Expand Down
Loading