Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Update conduit internals to use conduit-commons #1433

Closed
simonl2002 opened this issue Mar 12, 2024 · 1 comment · Fixed by #1622
Closed

Feature: Update conduit internals to use conduit-commons #1433

simonl2002 opened this issue Mar 12, 2024 · 1 comment · Fixed by #1622
Assignees
Labels
housekeeping Small improvements or chores
Milestone

Comments

@simonl2002
Copy link
Member

simonl2002 commented Mar 12, 2024

Feature description

This work includes removing config parameters and validations in the connector SDK in favor of using the config parameters in conduit-commons (see ConduitIO/conduit-commons#18 and ConduitIO/conduit-commons#19). This allows us to reuse the same code in connectors and processors. Also paramgen can then be removed from the SDK in favor of paramgen in conduit-commons, which will then work for both connectors and processors.

Additionally we need to remove sdk.Record (connector SDK) and record.Record (Conduit) in favor of opencdc.Record in conduit-commons. This will simplify our codebase as well as have a positive impact on performance, since we won't have to do as many transformations between the types, which is currently happening in the hot path (for every record).

@lovromazgon
Copy link
Member

Migration script for connectors regarding the changes done in this issue:

sed -i '' 's/conduit-connector-sdk\/cmd\/paramgen/conduit-commons\/paramgen/g' tools.go
find . -type f -iname "*.go" -exec sed -i '' 's/sdk.Record/opencdc.Record/g' {} +
find . -type f -iname "*.go" -exec sed -i '' 's/sdk.Operation/opencdc.Operation/g' {} +
find . -type f -iname "*.go" -exec sed -i '' 's/sdk.Position/opencdc.Position/g' {} +
find . -type f -iname "*.go" -exec sed -i '' 's/sdk.Change/opencdc.Change/g' {} +
find . -type f -iname "*.go" -exec sed -i '' 's/sdk.Data/opencdc.Data/g' {} +
find . -type f -iname "*.go" -exec sed -i '' 's/sdk.RawData/opencdc.RawData/g' {} +
find . -type f -iname "*.go" -exec sed -i '' 's/sdk.StructuredData/opencdc.StructuredData/g' {} +
find . -type f -iname "*.go" -exec sed -i '' 's/sdk.Metadata/opencdc.Metadata/g' {} +
find . -type f -iname "*.go" -exec sed -i '' 's/sdk.Parameter/config.Parameter/g' {} +
find . -type f -iname "*.go" -exec sed -i '' 's/sdk.Validation/config.Validation/g' {} +
find . -type f -iname "*.go" -exec sed -i '' 's/Parameters() map\[string\]config.Parameter/Parameters() config.Parameters/g' {} +
find . -type f -iname "*.go" -exec sed -i '' '/ *Source) Configure(/ s/map\[string\]string/config.Config/g' {} +
find . -type f -iname "*.go" -exec sed -i '' '/ *Destination) Configure(/ s/map\[string\]string/config.Config/g' {} +
find . -type f -iname "*.go" -exec sed -i '' 's/sdk.Util.ParseConfig(/sdk.Util.ParseConfig(ctx, /g' {} +

goimports -w .
gci write .
go mod tidy

make generate

Afterwards, try running tests and fix any issues that still exist. Before pushing the changes, also run make lint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
housekeeping Small improvements or chores
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants