From 961b6f00d091eb4a10598247754fb17c1cd42fad Mon Sep 17 00:00:00 2001 From: Bobby Iliev Date: Mon, 2 Sep 2024 15:19:36 +0300 Subject: [PATCH 1/3] Add source versioning design doc --- .../developer/source-versioning-2024-08-30.md | 183 ++++++++++++++++++ 1 file changed, 183 insertions(+) create mode 100644 docs/developer/source-versioning-2024-08-30.md diff --git a/docs/developer/source-versioning-2024-08-30.md b/docs/developer/source-versioning-2024-08-30.md new file mode 100644 index 00000000..c9dfb344 --- /dev/null +++ b/docs/developer/source-versioning-2024-08-30.md @@ -0,0 +1,183 @@ +# Design Document: Implementing Source Versioning / Tables from Sources in Terraform Provider for Materialize + +## 1. Introduction + +This design document outlines the implementation plan for supporting the new "Source Versioning / Tables from Sources" feature in the Terraform provider for Materialize. + +This feature aims to simplify the user experience, create a more unified model for managing data ingested from upstream systems, and provide more flexibility in handling upstream schema changes. + +## 2. Background + +Materialize is introducing changes to its source and table model: +- The concept of a source will be unified around a single relation representing the progress of that source's ingestion pipeline. +- Subsources will be replaced with tables created from sources. +- Users will use a `CREATE TABLE .. FROM SOURCE ..` statement to ingest data from upstream systems. + +These changes require corresponding updates to the Terraform provider to maintain alignment with Materialize's data model and provide a smooth migration path for existing users. + +## 3. Objectives + +- Update the Terraform provider to support the new source and table model. +- Provide a migration path for existing Terraform configurations. +- Maintain backwards compatibility where possible. +- Ensure the provider can work with both old and new versions of Materialize during the transition period. + +## 4. Design + +### 4.1 Schema Updates + +#### 4.1.1 Existing Source Resources + +We will maintain the existing source resources (e.g., `materialize_source_postgres`) but deprecate fields related to subsources: + +```go +var sourcePostgresSchema = map[string]*schema.Schema{ + // ... existing fields ... + "table": { + Description: "Tables to be ingested from the source. This field is deprecated and will be removed in a future version.", + Type: schema.TypeSet, + Optional: true, + Deprecated: "Use the new `materialize_table_from_source` resource instead.", + Elem: &schema.Resource{ + Schema: map[string]*schema.Schema{ + // ... existing table schema ... + }, + }, + }, +} +``` + +#### 4.1.2 New Table From Source Resource + +Introduce a new `materialize_table_from_source` resource: + +```go +var tableFromSourceSchema = map[string]*schema.Schema{ + "name": ObjectNameSchema("table", true, false), + "schema_name": SchemaNameSchema("table", false), + "database_name": DatabaseNameSchema("table", false), + "source": IdentifierSchema(IdentifierSchemaParams{ + Elem: "source", + Description: "The source this table is created from.", + Required: true, + ForceNew: true, + }), + "upstream_name": { + Type: schema.TypeString, + Required: true, + ForceNew: true, + Description: "The name of the table in the upstream database.", + }, + "upstream_schema_name": { + Type: schema.TypeString, + Optional: true, + ForceNew: true, + Description: "The schema of the table in the upstream database.", + }, + "text_columns": { + Description: "Columns to be decoded as text.", + Type: schema.TypeList, + Elem: &schema.Schema{Type: schema.TypeString}, + Optional: true, + ForceNew: true, + }, + // ... other fields as needed ... +} +``` + +### 4.2 Resource Implementation + +#### 4.2.1 Table From Source Resource + +Implement CRUD operations for the new `materialize_table_from_source` resource: + +```go +func TableFromSource() *schema.Resource { + return &schema.Resource{ + CreateContext: tableFromSourceCreate, + ReadContext: tableFromSourceRead, + UpdateContext: tableFromSourceUpdate, + DeleteContext: tableFromSourceDelete, + Importer: &schema.ResourceImporter{ + StateContext: schema.ImportStatePassthroughContext, + }, + Schema: tableFromSourceSchema, + } +} +``` + +The `CreateContext` function will use the new SQL syntax: + +```sql +CREATE TABLE .. FROM SOURCE (REFERENCE = ) WITH (TEXT COLUMNS = (..), ..) +``` + +#### 4.2.2 Update Existing Source Resources + +Modify the CRUD operations for existing source resources to handle the deprecation of subsource-related fields: + +- In `Create` and `Update` operations, if the deprecated `table` field is used, log a warning message advising users to migrate to the new `materialize_table_from_source` resource. +- In `Read` operations, continue to populate the `table` field if it exists in the state, but also log a deprecation warning. + +### 4.3 Migration Strategy + +We will not create separate resources with a v2 suffix for sources. Instead, we'll use a gradual migration approach: + +1. Deprecate the `table` field in existing source resources. +2. Introduce the new `materialize_table_from_source` resource. +3. Allow both old and new configurations to coexist during a transition period. + +This approach allows users to migrate their configurations gradually: + +- Existing sources can still be created and managed. +- New tables (formerly subsources) will be created as separate `materialize_table_from_source` resources. +- Users can migrate their configurations at their own pace by replacing `table` blocks with `materialize_table_from_source` resources. + +### 4.4 Import Logic + +The import logic should work out of the box for the new `materialize_table_from_source` resource as long we have the required information (e.g., source name, upstream table name) stored in a system catalog table. + +The read operation for the `materialize_table_from_source` resource should be able to fetch the necessary details from the Materialize system catalog to populate the state. If not all information is available, some fields may need to be ignored or set to defaults during import. + +### 4.5 Versioning and Compatibility + +- These changes will be introduced in a new minor version of the provider (e.g., v0.9.0), not a major version bump. +- The provider will support both old and new Materialize versions during the transition period while this is supported by Materialize itself. +- Deprecation warnings will be logged when users interact with the deprecated `table` field. + +### 4.6 Testing + +- Update existing tests for source resources to cover the deprecation warnings and backwards compatibility. +- Add all required tests for the new `materialize_table_from_source` resource. +- Implement integration tests to ensure compatibility with both old and new Materialize versions. + +## 5. Migration Guide for Users + +Provide a migration guide for users to update their Terraform configurations: + +1. Update the provider version to v0.9.0 or later. +2. For each source with subsources: + a. Keep the existing source resource as-is. + b. Create a new `materialize_table_from_source` resource for each former subsource. + c. Set the `source` in the new resource to the fully qualified name of the source. +3. Run `terraform import` to import the state of the new resources. +4. Gradually remove the deprecated `table` blocks from source resources as you migrate to the new structure. + +## 6. Backwards Compatibility + +- The `table` field in source resources will be marked as deprecated but still functional during the transition period as long as Materialize supports it. +- Existing Terraform configurations will continue to work without immediate changes. +- Deprecation warnings will be logged when users interact with the deprecated fields. + +## 7. Documentation Updates + +- Update provider documentation to reflect the new resource and changes to existing resources. +- Create a dedicated migration guide with step-by-step instructions and examples. +- Update examples in the documentation to use the new structure. +- Add a section in the documentation explaining the rationale behind these changes and the benefits of the new model. + +## 8. Open Questions + +- How long should we maintain support for the deprecated `table` field in source resources? +- What is the expected timeline for Materialize to fully transition to the new source and table model? +- Consider webhook sources and their impact on the migration process. From 1e062b0dc1329405077fc469b3ec144567c99597 Mon Sep 17 00:00:00 2001 From: Bobby Iliev Date: Tue, 3 Sep 2024 13:27:14 +0300 Subject: [PATCH 2/3] Use source table instead of table from source --- .../developer/source-versioning-2024-08-30.md | 38 +++++++++---------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/developer/source-versioning-2024-08-30.md b/docs/developer/source-versioning-2024-08-30.md index c9dfb344..5cad9814 100644 --- a/docs/developer/source-versioning-2024-08-30.md +++ b/docs/developer/source-versioning-2024-08-30.md @@ -37,7 +37,7 @@ var sourcePostgresSchema = map[string]*schema.Schema{ Description: "Tables to be ingested from the source. This field is deprecated and will be removed in a future version.", Type: schema.TypeSet, Optional: true, - Deprecated: "Use the new `materialize_table_from_source` resource instead.", + Deprecated: "Use the new `materialize_source_table` resource instead.", Elem: &schema.Resource{ Schema: map[string]*schema.Schema{ // ... existing table schema ... @@ -49,10 +49,10 @@ var sourcePostgresSchema = map[string]*schema.Schema{ #### 4.1.2 New Table From Source Resource -Introduce a new `materialize_table_from_source` resource: +Introduce a new `materialize_source_table` resource: ```go -var tableFromSourceSchema = map[string]*schema.Schema{ +var sourceTableSchema = map[string]*schema.Schema{ "name": ObjectNameSchema("table", true, false), "schema_name": SchemaNameSchema("table", false), "database_name": DatabaseNameSchema("table", false), @@ -87,21 +87,21 @@ var tableFromSourceSchema = map[string]*schema.Schema{ ### 4.2 Resource Implementation -#### 4.2.1 Table From Source Resource +#### 4.2.1 Table Source Table Resource -Implement CRUD operations for the new `materialize_table_from_source` resource: +Implement CRUD operations for the new `materialize_source_table` resource: ```go -func TableFromSource() *schema.Resource { +func SourceTable() *schema.Resource { return &schema.Resource{ - CreateContext: tableFromSourceCreate, - ReadContext: tableFromSourceRead, - UpdateContext: tableFromSourceUpdate, - DeleteContext: tableFromSourceDelete, + CreateContext: sourceTableCreate, + ReadContext: sourceTableRead, + UpdateContext: sourceTableUpdate, + DeleteContext: sourceTableDelete, Importer: &schema.ResourceImporter{ StateContext: schema.ImportStatePassthroughContext, }, - Schema: tableFromSourceSchema, + Schema: sourceTableSchema, } } ``` @@ -116,7 +116,7 @@ CREATE TABLE .. FROM SOURCE (REF Modify the CRUD operations for existing source resources to handle the deprecation of subsource-related fields: -- In `Create` and `Update` operations, if the deprecated `table` field is used, log a warning message advising users to migrate to the new `materialize_table_from_source` resource. +- In `Create` and `Update` operations, if the deprecated `table` field is used, log a warning message advising users to migrate to the new `materialize_source_table` resource. - In `Read` operations, continue to populate the `table` field if it exists in the state, but also log a deprecation warning. ### 4.3 Migration Strategy @@ -124,20 +124,20 @@ Modify the CRUD operations for existing source resources to handle the deprecati We will not create separate resources with a v2 suffix for sources. Instead, we'll use a gradual migration approach: 1. Deprecate the `table` field in existing source resources. -2. Introduce the new `materialize_table_from_source` resource. +2. Introduce the new `materialize_source_table` resource. 3. Allow both old and new configurations to coexist during a transition period. This approach allows users to migrate their configurations gradually: - Existing sources can still be created and managed. -- New tables (formerly subsources) will be created as separate `materialize_table_from_source` resources. -- Users can migrate their configurations at their own pace by replacing `table` blocks with `materialize_table_from_source` resources. +- New tables (formerly subsources) will be created as separate `materialize_source_table` resources. +- Users can migrate their configurations at their own pace by replacing `table` blocks with `materialize_source_table` resources. ### 4.4 Import Logic -The import logic should work out of the box for the new `materialize_table_from_source` resource as long we have the required information (e.g., source name, upstream table name) stored in a system catalog table. +The import logic should work out of the box for the new `materialize_source_table` resource as long we have the required information (e.g., source name, upstream table name) stored in a system catalog table. -The read operation for the `materialize_table_from_source` resource should be able to fetch the necessary details from the Materialize system catalog to populate the state. If not all information is available, some fields may need to be ignored or set to defaults during import. +The read operation for the `materialize_source_table` resource should be able to fetch the necessary details from the Materialize system catalog to populate the state. If not all information is available, some fields may need to be ignored or set to defaults during import. ### 4.5 Versioning and Compatibility @@ -148,7 +148,7 @@ The read operation for the `materialize_table_from_source` resource should be ab ### 4.6 Testing - Update existing tests for source resources to cover the deprecation warnings and backwards compatibility. -- Add all required tests for the new `materialize_table_from_source` resource. +- Add all required tests for the new `materialize_source_table` resource. - Implement integration tests to ensure compatibility with both old and new Materialize versions. ## 5. Migration Guide for Users @@ -158,7 +158,7 @@ Provide a migration guide for users to update their Terraform configurations: 1. Update the provider version to v0.9.0 or later. 2. For each source with subsources: a. Keep the existing source resource as-is. - b. Create a new `materialize_table_from_source` resource for each former subsource. + b. Create a new `materialize_source_table` resource for each former subsource. c. Set the `source` in the new resource to the fully qualified name of the source. 3. Run `terraform import` to import the state of the new resources. 4. Gradually remove the deprecated `table` blocks from source resources as you migrate to the new structure. From d5dbba28a3fdcfa58371bd3d539b3c52939d5fe7 Mon Sep 17 00:00:00 2001 From: Bobby Iliev Date: Fri, 13 Sep 2024 11:35:30 +0300 Subject: [PATCH 3/3] Use separate source table resources for each source type --- .../developer/source-versioning-2024-08-30.md | 24 ++++++++++--------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/docs/developer/source-versioning-2024-08-30.md b/docs/developer/source-versioning-2024-08-30.md index 5cad9814..04eeb157 100644 --- a/docs/developer/source-versioning-2024-08-30.md +++ b/docs/developer/source-versioning-2024-08-30.md @@ -47,9 +47,9 @@ var sourcePostgresSchema = map[string]*schema.Schema{ } ``` -#### 4.1.2 New Table From Source Resource +#### 4.1.2 New Table From Source Resources -Introduce a new `materialize_source_table` resource: +Introduce new `materialize_source_table_{source}` resources: ```go var sourceTableSchema = map[string]*schema.Schema{ @@ -85,11 +85,13 @@ var sourceTableSchema = map[string]*schema.Schema{ } ``` +For each individual Source we will have a corresponding `materialize_source_table_{source}` resource which will allow users to create tables from that source with the specific configuration required for that source. + ### 4.2 Resource Implementation -#### 4.2.1 Table Source Table Resource +#### 4.2.1 Table Source Table Resources -Implement CRUD operations for the new `materialize_source_table` resource: +Implement CRUD operations for the new `materialize_source_table_{source}` resources: ```go func SourceTable() *schema.Resource { @@ -106,7 +108,7 @@ func SourceTable() *schema.Resource { } ``` -The `CreateContext` function will use the new SQL syntax: +The `CreateContext` function will use the new SQL syntax with the source specific configuration to create a table from the source: ```sql CREATE TABLE .. FROM SOURCE (REFERENCE = ) WITH (TEXT COLUMNS = (..), ..) @@ -116,7 +118,7 @@ CREATE TABLE .. FROM SOURCE (REF Modify the CRUD operations for existing source resources to handle the deprecation of subsource-related fields: -- In `Create` and `Update` operations, if the deprecated `table` field is used, log a warning message advising users to migrate to the new `materialize_source_table` resource. +- In `Create` and `Update` operations, if the deprecated `table` field is used, log a warning message advising users to migrate to the new `materialize_source_table_{source}` resource. - In `Read` operations, continue to populate the `table` field if it exists in the state, but also log a deprecation warning. ### 4.3 Migration Strategy @@ -124,7 +126,7 @@ Modify the CRUD operations for existing source resources to handle the deprecati We will not create separate resources with a v2 suffix for sources. Instead, we'll use a gradual migration approach: 1. Deprecate the `table` field in existing source resources. -2. Introduce the new `materialize_source_table` resource. +2. Introduce the new `materialize_source_table_{source}` resources. 3. Allow both old and new configurations to coexist during a transition period. This approach allows users to migrate their configurations gradually: @@ -135,9 +137,9 @@ This approach allows users to migrate their configurations gradually: ### 4.4 Import Logic -The import logic should work out of the box for the new `materialize_source_table` resource as long we have the required information (e.g., source name, upstream table name) stored in a system catalog table. +The import logic should work out of the box for the new `materialize_source_table_{source}` resources as long we have the required information (e.g., source name, upstream table name) stored in a system catalog table. -The read operation for the `materialize_source_table` resource should be able to fetch the necessary details from the Materialize system catalog to populate the state. If not all information is available, some fields may need to be ignored or set to defaults during import. +The read operation for the `materialize_source_table_{source}` resources should be able to fetch the necessary details from the Materialize system catalog to populate the state. If not all information is available, some fields may need to be ignored or set to defaults during import. ### 4.5 Versioning and Compatibility @@ -148,7 +150,7 @@ The read operation for the `materialize_source_table` resource should be able to ### 4.6 Testing - Update existing tests for source resources to cover the deprecation warnings and backwards compatibility. -- Add all required tests for the new `materialize_source_table` resource. +- Add all required tests for the new `materialize_source_table_{source}` resources. - Implement integration tests to ensure compatibility with both old and new Materialize versions. ## 5. Migration Guide for Users @@ -158,7 +160,7 @@ Provide a migration guide for users to update their Terraform configurations: 1. Update the provider version to v0.9.0 or later. 2. For each source with subsources: a. Keep the existing source resource as-is. - b. Create a new `materialize_source_table` resource for each former subsource. + b. Create new `materialize_source_table_{source}` resources for each former subsource. c. Set the `source` in the new resource to the fully qualified name of the source. 3. Run `terraform import` to import the state of the new resources. 4. Gradually remove the deprecated `table` blocks from source resources as you migrate to the new structure.