Skip to content

Commit

Permalink
add note for alter shared source
Browse files Browse the repository at this point in the history
Signed-off-by: xxchan <[email protected]>
  • Loading branch information
xxchan committed Dec 16, 2024
1 parent 049b472 commit d9741f0
Show file tree
Hide file tree
Showing 10 changed files with 50 additions and 90 deletions.
2 changes: 0 additions & 2 deletions ingestion/modify-source-or-table-schemas.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -126,8 +126,6 @@ Refresh schema
ALTER SOURCE src_user REFRESH SCHEMA;
```

For more details about this example, see our [test file](https://github.com/risingwavelabs/risingwave/blob/994a2831088c9befc71721ed6f2f2d2e35c4d0a9/e2e%5Ftest/schema%5Fregistry/alter%5Fsr.slt).

### Table

Similarly, you can use the following statement to refresh the schema of a table with connectors. For more details, see [ALTER TABLE](/sql/commands/sql-alter-table#refresh-schema).
Expand Down
2 changes: 1 addition & 1 deletion sql/commands/sql-alter-connection.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ ALTER CONNECTION connection_name
alter_option;
```

_`alteroption`_ depends on the operation you want to perform on the connection. For all supported clauses, see the sections below.
_`alter_option`_ depends on the operation you want to perform on the connection. For all supported clauses, see the sections below.

## Clause

Expand Down
2 changes: 1 addition & 1 deletion sql/commands/sql-alter-database.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ ALTER DATABASE database_name
alter_option;
```

_`alteroption`_ depends on the operation you want to perform on the database. For all supported clauses, see the sections below.
_`alter_option`_ depends on the operation you want to perform on the database. For all supported clauses, see the sections below.

## Clause
### `OWNER TO`
Expand Down
2 changes: 1 addition & 1 deletion sql/commands/sql-alter-function.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ ALTER FUNCTION function( argument_type [, ...] )
alter_option;
```

_`alteroption`_ depends on the operation you want to perform on the function. For all supported clauses, see the sections below.
_`alter_option`_ depends on the operation you want to perform on the function. For all supported clauses, see the sections below.

## Clause

Expand Down
2 changes: 1 addition & 1 deletion sql/commands/sql-alter-index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ ALTER INDEX index_name
alter_option;
```

_`alteroption`_ depends on the operation you want to perform on the index. For all supported clauses, see the sections below.
_`alter_option`_ depends on the operation you want to perform on the index. For all supported clauses, see the sections below.

## Clause

Expand Down
2 changes: 1 addition & 1 deletion sql/commands/sql-alter-materialized-view.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ ALTER MATERIALIZED VIEW materialized_view_name
alter_option;
```

_`alteroption`_ depends on the operation you want to perform on the materialized view. For all supported clauses, see the sections below.
_`alter_option`_ depends on the operation you want to perform on the materialized view. For all supported clauses, see the sections below.

## Clause

Expand Down
2 changes: 1 addition & 1 deletion sql/commands/sql-alter-schema.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ ALTER SCHEMA current_schema_name
alter_option;
```

_`alteroption`_ depends on the operation you want to perform on the schema. For all supported clauses, see the sections below.
_`alter_option`_ depends on the operation you want to perform on the schema. For all supported clauses, see the sections below.

## Clause

Expand Down
2 changes: 1 addition & 1 deletion sql/commands/sql-alter-sink.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ ALTER SINK sink_name
alter_option;
```

_`alteroption`_ depends on the operation you want to perform on the sink. For all supported clauses, see the sections below.
_`alter_option`_ depends on the operation you want to perform on the sink. For all supported clauses, see the sections below.

## Clause

Expand Down
110 changes: 37 additions & 73 deletions sql/commands/sql-alter-source.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ ALTER SOURCE current_source_name
alter_option;
```

_`alteroption`_ depends on the operation you want to perform on the source. For all supported clauses, see the sections below.
_`alter_option`_ depends on the operation you want to perform on the source. For all supported clauses, see the sections below.

## Clause

Expand All @@ -34,11 +34,46 @@ ALTER SOURCE src1
```

<Note>
* To alter columns in a source created with a schema registry, see [FORMAT and ENCODE options](/sql/commands/sql-alter-source#format-and-encode-options).
* To alter columns in a source created with a schema registry, see [REFRESH SCHEMA](/sql/commands/sql-alter-source#refresh-schema).
* You cannot add a primary key column to a source or table in RisingWave. To modify the primary key of a source or table, you need to recreate the table.
* You cannot remove a column from a source in RisingWave. If you intend to remove a column from a source, you'll need to drop the source and create the source again.
</Note>


### `REFRESH SCHEMA`

Fetch the latest schema from the schema registry and update the source schema.

```sql
ALTER SOURCE source_name REFRESH SCHEMA;
```

<Note>
Currently when refreshing the schema registry of a source, it is not allowed to drop columns or change types.
</Note>


For example, assume we have a source as follows:

```sql Create a source
CREATE SOURCE src_user WITH (
connector = 'kafka',
topic = 'sr_pb_test',
properties.bootstrap.server = 'message_queue:29092',
scan.startup.mode = 'earliest'
)
FORMAT PLAIN ENCODE PROTOBUF(
schema.registry = 'http://message_queue:8081',
message = 'test.User'
);
```

Then we can refresh its schema with the following statement:

```sql Refresh schema
ALTER SOURCE src_user REFRESH SCHEMA;
```

### `RENAME TO`

```sql
Expand Down Expand Up @@ -91,77 +126,6 @@ ALTER SOURCE current_source_name
ALTER SOURCE test_source SET SCHEMA test_schema;
```

### `FORMAT and ENCODE options`

At present, combined with the `ALTER SOURCE` command, you can refresh the schema registry of a source by refilling the FORMAT and ENCODE options. For more details about these options, see [FORMAT and ENCODE parameters](/ingestion/format-and-encode-parameters).

```sql
ALTER SOURCE source_name FORMAT data_format ENCODE data_encode [ (
message='message',
schema.location='location', ...) ];
```

Here is an example. Let's assume the original FORMAT and ENCODE options are as follows:

```sql
-- Create a source.
CREATE SOURCE src_user WITH (
connector = 'kafka',
topic = 'sr_pb_test',
properties.bootstrap.server = 'message_queue:29092',
scan.startup.mode = 'earliest'
)
FORMAT PLAIN ENCODE PROTOBUF(
schema.registry = 'http://message_queue:8081',
message = 'test.User');
```

Then you can refresh the schema registry by the following command:

```sql
ALTER SOURCE src_user FORMAT PLAIN ENCODE PROTOBUF(
schema.registry = 'http://message_queue:8081',
message = 'test.UserWithMoreFields'
);
```

<Note>
Currently, it is not supported to modify the `data_format` and `data_encode`. Furthermore, when refreshing the schema registry of a source, it is not allowed to drop columns or change types.

Another way of refreshing the schema is using the [REFRESH SCHEMA clause](#refresh-schema).
</Note>

### `REFRESH SCHEMA`

This is another way of refreshing the schema of sources when the [FORMAT and ENCODE options](#format-and-encode-options) are not changed.

```sql
ALTER SOURCE source_name REFRESH SCHEMA;
```

For example, assume we have a source as follows:

```sql Create a source
CREATE SOURCE src_user WITH (
connector = 'kafka',
topic = 'sr_pb_test',
properties.bootstrap.server = 'message_queue:29092',
scan.startup.mode = 'earliest'
)
FORMAT PLAIN ENCODE PROTOBUF(
schema.registry = 'http://message_queue:8081',
message = 'test.User'
);
```

Then we can refresh its schema with the following statement:

```sql Refresh schema
ALTER SOURCE src_user REFRESH SCHEMA;
```

For more details about this example, see our [test file](https://github.com/risingwavelabs/risingwave/blob/994a2831088c9befc71721ed6f2f2d2e35c4d0a9/e2e%5Ftest/schema%5Fregistry/alter%5Fsr.slt).

### `SET SOURCE_RATE_LIMIT`

```sql
Expand Down
14 changes: 6 additions & 8 deletions sql/commands/sql-create-source.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,12 @@ Shared source improves resource utilization and data consistency when working wi
This feature is currently in public preview, meaning it is nearing the final product but may not yet be fully stable. If you encounter any issues or have feedback, please reach out to us via our [Slack channel](https://www.risingwave.com/slack). Your input is valuable in helping us improve this feature. For more details, see our [Public Preview Feature List](/changelog/product-lifecycle#features-in-the-public-preview-stage).
</Note>

<Note>
Shared Kafka source is available since version 2.1. Other sources are unaffected. We plan to gradually upgrade other sources to the be shared as well in the future.

`ALTER SOURCE [ADD COLUMN | REFRESH SCHEMA]` for shared source is available since version 2.2.
</Note>

### Configure

Shared source is enabled by default. You can also set the session variable `streaming_use_shared_source` to control whether to enable it.
Expand Down Expand Up @@ -160,14 +166,6 @@ Tables offer other features that enhance their utility in data ingestion workflo
<img src="/images/table-with-connectors.png"/>
</Frame>

<Note>
**LIMITATION**

Currently, shared source is only applicable to Kafka sources. Other sources are unaffected. We plan to gradually upgrade other sources to the be shared as well in the future.

Shared sources do not support `ALTER SOURCE`. Use non-shared sources if you require this functionality.
</Note>

## See also

<CardGroup>
Expand Down

0 comments on commit d9741f0

Please sign in to comment.