Skip to content

Commit

Permalink
Add import examples for Kafka source tables
Browse files Browse the repository at this point in the history
  • Loading branch information
bobbyiliev committed Sep 17, 2024
1 parent 7ad7474 commit 7870d48
Show file tree
Hide file tree
Showing 9 changed files with 125 additions and 13 deletions.
63 changes: 62 additions & 1 deletion docs/resources/source_table_kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,56 @@ description: |-

A Kafka source describes a Kafka cluster you want Materialize to read data from.


## Example Usage

```terraform
resource "materialize_source_table_kafka" "kafka_source_table" {
name = "kafka_source_table"
schema_name = "public"
database_name = "materialize"
source {
name = materialize_source_kafka.test_source_kafka.name
schema_name = materialize_source_kafka.test_source_kafka.schema_name
database_name = materialize_source_kafka.test_source_kafka.database_name
}
upstream_name = "terraform" # The kafka source topic name
include_key = true
include_key_alias = "message_key"
include_headers = true
include_headers_alias = "message_headers"
include_partition = true
include_partition_alias = "message_partition"
include_offset = true
include_offset_alias = "message_offset"
include_timestamp = true
include_timestamp_alias = "message_timestamp"
key_format {
text = true
}
value_format {
json = true
}
envelope {
upsert = true
upsert_options {
value_decoding_errors {
inline {
enabled = true
alias = "decoding_error"
}
}
}
}
ownership_role = "mz_system"
comment = "This is a test Kafka source table"
}
```

<!-- schema generated by tfplugindocs -->
## Schema
Expand Down Expand Up @@ -320,3 +369,15 @@ Optional:

- `database_name` (String) The schema_registry_connection database name. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
- `schema_name` (String) The schema_registry_connection schema name. Defaults to `public`.

## Import

Import is supported using the following syntax:

```shell
# Source tables can be imported using the source table id:
terraform import materialize_source_table_kafka.example_source_table_kafka <region>:<source_table_id>

# Source id and information be found in the `mz_catalog.mz_tables` table
# The region is the region where the database is located (e.g. aws/us-east-1)
```
4 changes: 2 additions & 2 deletions docs/resources/source_table_load_generator.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,8 @@ Optional:
Import is supported using the following syntax:

```shell
# Source tables can be imported using the source id:
terraform import materialize_source_table_load_generator.example_source_table_loadgen <region>:<source_id>
# Source tables can be imported using the source table id:
terraform import materialize_source_table_load_generator.example_source_table_loadgen <region>:<source_table_id>

# Source id and information be found in the `mz_catalog.mz_tables` table
# The region is the region where the database is located (e.g. aws/us-east-1)
Expand Down
4 changes: 2 additions & 2 deletions docs/resources/source_table_mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,8 @@ Optional:
Import is supported using the following syntax:

```shell
# Source tables can be imported using the source id:
terraform import materialize_source_table_mysql.example_source_table_mysql <region>:<source_id>
# Source tables can be imported using the source table id:
terraform import materialize_source_table_mysql.example_source_table_mysql <region>:<source_table_id>

# Source id and information be found in the `mz_catalog.mz_tables` table
# The region is the region where the database is located (e.g. aws/us-east-1)
Expand Down
4 changes: 2 additions & 2 deletions docs/resources/source_table_postgres.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,8 @@ Optional:
Import is supported using the following syntax:

```shell
# Source tables can be imported using the source id:
terraform import materialize_source_table_postgres.example_source_table_postgres <region>:<source_id>
# Source tables can be imported using the source table id:
terraform import materialize_source_table_postgres.example_source_table_postgres <region>:<source_table_id>

# Source id and information be found in the `mz_catalog.mz_tables` table
# The region is the region where the database is located (e.g. aws/us-east-1)
Expand Down
5 changes: 5 additions & 0 deletions examples/resources/materialize_source_table_kafka/import.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Source tables can be imported using the source table id:
terraform import materialize_source_table_kafka.example_source_table_kafka <region>:<source_table_id>

# Source id and information be found in the `mz_catalog.mz_tables` table
# The region is the region where the database is located (e.g. aws/us-east-1)
46 changes: 46 additions & 0 deletions examples/resources/materialize_source_table_kafka/resource.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
resource "materialize_source_table_kafka" "kafka_source_table" {
name = "kafka_source_table"
schema_name = "public"
database_name = "materialize"

source {
name = materialize_source_kafka.test_source_kafka.name
schema_name = materialize_source_kafka.test_source_kafka.schema_name
database_name = materialize_source_kafka.test_source_kafka.database_name
}

upstream_name = "terraform" # The kafka source topic name
include_key = true
include_key_alias = "message_key"
include_headers = true
include_headers_alias = "message_headers"
include_partition = true
include_partition_alias = "message_partition"
include_offset = true
include_offset_alias = "message_offset"
include_timestamp = true
include_timestamp_alias = "message_timestamp"


key_format {
text = true
}
value_format {
json = true
}

envelope {
upsert = true
upsert_options {
value_decoding_errors {
inline {
enabled = true
alias = "decoding_error"
}
}
}
}

ownership_role = "mz_system"
comment = "This is a test Kafka source table"
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Source tables can be imported using the source id:
terraform import materialize_source_table_load_generator.example_source_table_loadgen <region>:<source_id>
# Source tables can be imported using the source table id:
terraform import materialize_source_table_load_generator.example_source_table_loadgen <region>:<source_table_id>

# Source id and information be found in the `mz_catalog.mz_tables` table
# The region is the region where the database is located (e.g. aws/us-east-1)
4 changes: 2 additions & 2 deletions examples/resources/materialize_source_table_mysql/import.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Source tables can be imported using the source id:
terraform import materialize_source_table_mysql.example_source_table_mysql <region>:<source_id>
# Source tables can be imported using the source table id:
terraform import materialize_source_table_mysql.example_source_table_mysql <region>:<source_table_id>

# Source id and information be found in the `mz_catalog.mz_tables` table
# The region is the region where the database is located (e.g. aws/us-east-1)
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Source tables can be imported using the source id:
terraform import materialize_source_table_postgres.example_source_table_postgres <region>:<source_id>
# Source tables can be imported using the source table id:
terraform import materialize_source_table_postgres.example_source_table_postgres <region>:<source_table_id>

# Source id and information be found in the `mz_catalog.mz_tables` table
# The region is the region where the database is located (e.g. aws/us-east-1)

0 comments on commit 7870d48

Please sign in to comment.