v0.11.0

Schema Support:

• Introduced comprehensive schema support, allowing schemas to be attached to records.
• Integrated a built-in schema registry, with the option to use an external schema registry.
• Enhanced standalone connectors to retain type information.

Connector Improvements:

• Introduced middleware for automatic schema handling in built-in connectors and processors.
• Postgres Connector:
  • Added support for incremental snapshots in logical replication mode.
  • Improved management of logical replication slots.

Processor Improvements:

• error
  • Introduced an error processor to send records to DLQ or fail the pipeline based on conditions.
• webhook.http
  • Added the ability to specify headers in the webhook.http processor.
• field.convert
  • Extended to support conversion to Go time.Time objects, handling unix nano timestamps and RFC3339 formatted dates.
• avro.encode and avro.decode
  • Updated to use Conduit’s built-in schema registry.

Internal Improvements:

• Fixed bug causing a panic when trying to run a partial pipeline.

Changelog

• ab0ecac: Mention storage options in README (#1730) (@hariso)
• 428e651: Database implementation for SQLite3 (#1682) (@lyuboxa)
• bf9b0c7: Include conduit-schema-registry in release instructions (#1713) (@hariso)
• 840a650: More info about Postgres connection string (#1726) (@hariso)
• 7339241: Replace panic with error message when not declaring a source node in .yml config (#1695) (@AdamHaffar)
• 1397d5d: Replace pkg/foundation/database with commons, use schema registry with DB (#1720) (@hariso)
• 930afae: Schema support (#1702) (@lovromazgon)
• 3a9d244: Tests and examples for converting time using field.convert (#1745) (@lovromazgon)
• 5c956e1: Update SDK, protocol, Kafka connector (#1731) (@hariso)
• 7601e06: Update dependencies for v0.11.0 (#1756) (@lovromazgon)
• f53d710: Update linter config (#1675) (@hariso)
• d6e83ce: Update processor SDK, add middleware (#1742) (@lovromazgon)
• 06bd6e5: Use connector protocol v2 (#1622) (@lovromazgon)
• 0d5929d: Use gofumpt (#1741) (@hariso)
• d370cd5: Use interpreter runtime in wasm tests (#1746) (@lovromazgon)
• 306defe: Wrap builtin processors in middleware in examples (#1748) (@lovromazgon)
• 76c25f1: [Schemas] Handle name conflicts (#1718) (@hariso)
• 203aeb7: [Schemas] Limit schema service usage to connectors (#1701) (@hariso)
• 817d52c: [Schemas] Use configured schema service in Avro processors (#1743) (@hariso)
• a9812b7: add getSchema and createSchema for wasm processors (#1683) (@maha-hajja)
• 76151f4: change keep-n-most-recent to cut-off (required parameter) (#1689) (@lovromazgon)
• 6b9dfa2: doc: update releases template (#1749) (@raulb)
• fd9f261: don't release win/i386 (#1703) (@lovromazgon)
• 7bb8cd8: enable container-retention-policy action (#1693) (@lovromazgon)
• a8d76dc: feat: sets environment variable based on log level (#1740) (@raulb)
• 2da3a0b: refactor: architecture documentation (#1732) (@raulb)
• b6fa5be: support empty .payload.After / .payload.Before in avro.decode (#1678) (@samirketema)
• fab3ab9: update swagger ui (#1711) (@lovromazgon)

Release Artifacts

OSX Darwin

• 📦 conduit_0.11.0_Darwin_arm64.tar.gz
• 📦 conduit_0.11.0_Darwin_x86_64.tar.gz

Linux

• 📦 conduit_0.11.0_Linux_arm64.deb
• 📦 conduit_0.11.0_Linux_arm64.rpm
• 📦 conduit_0.11.0_Linux_arm64.tar.gz
• 📦 conduit_0.11.0_Linux_i386.deb
• 📦 conduit_0.11.0_Linux_i386.rpm
• 📦 conduit_0.11.0_Linux_i386.tar.gz
• 📦 conduit_0.11.0_Linux_x86_64.deb
• 📦 conduit_0.11.0_Linux_x86_64.rpm
• 📦 conduit_0.11.0_Linux_x86_64.tar.gz

Windows

• 📦 conduit_0.11.0_Windows_arm64.zip
• 📦 conduit_0.11.0_Windows_x86_64.zip

Docker images

To pull the image, you can execute the following command:

docker pull conduit.docker.scarf.sh/conduitio/conduit:v0.11.0

v0.10.3

Changelog

• 873fb5c: Configure dependabot for Docker (#1658) (@hariso)
• b90bec7: Upgrade pg connector to v0.7.5 (#1677) (@lyuboxa)
• ad87154: Upgrade test Docker images (#1671) (@hariso)

Release Artifacts

OSX Darwin

• 📦 conduit_0.10.3_Darwin_arm64.tar.gz
• 📦 conduit_0.10.3_Darwin_x86_64.tar.gz

Linux

• 📦 conduit_0.10.3_Linux_arm64.deb
• 📦 conduit_0.10.3_Linux_arm64.rpm
• 📦 conduit_0.10.3_Linux_arm64.tar.gz
• 📦 conduit_0.10.3_Linux_i386.deb
• 📦 conduit_0.10.3_Linux_i386.rpm
• 📦 conduit_0.10.3_Linux_i386.tar.gz
• 📦 conduit_0.10.3_Linux_x86_64.deb
• 📦 conduit_0.10.3_Linux_x86_64.rpm
• 📦 conduit_0.10.3_Linux_x86_64.tar.gz

Windows

• 📦 conduit_0.10.3_Windows_arm64.zip
• 📦 conduit_0.10.3_Windows_i386.zip
• 📦 conduit_0.10.3_Windows_x86_64.zip

Docker images

To pull the image, you can execute the following command:

docker pull conduit.docker.scarf.sh/conduitio/conduit:v0.10.3

v0.10.2

Changelog

• 4d286f4: Add handling for time.Time in avro encode processor (#1650) (@samirketema)
• 8304d42: Upgrade postgres connector to v0.7.4 (#1661) (@lyuboxa)
• 019ec4c: manually set current tag in release action (#1637) (@lovromazgon)

Release Artifacts

OSX Darwin

• 📦 conduit_0.10.2_Darwin_arm64.tar.gz
• 📦 conduit_0.10.2_Darwin_x86_64.tar.gz

Linux

• 📦 conduit_0.10.2_Linux_arm64.deb
• 📦 conduit_0.10.2_Linux_arm64.rpm
• 📦 conduit_0.10.2_Linux_arm64.tar.gz
• 📦 conduit_0.10.2_Linux_i386.deb
• 📦 conduit_0.10.2_Linux_i386.rpm
• 📦 conduit_0.10.2_Linux_i386.tar.gz
• 📦 conduit_0.10.2_Linux_x86_64.deb
• 📦 conduit_0.10.2_Linux_x86_64.rpm
• 📦 conduit_0.10.2_Linux_x86_64.tar.gz

Windows

• 📦 conduit_0.10.2_Windows_arm64.zip
• 📦 conduit_0.10.2_Windows_i386.zip
• 📦 conduit_0.10.2_Windows_x86_64.zip

Docker images

To pull the image, you can execute the following command:

docker pull conduit.docker.scarf.sh/conduitio/conduit:v0.10.2

v0.10.1

Changelog

• 252efd7: --- (#1606) (@dependabot[bot])
• 8893784: --- (#1607) (@dependabot[bot])
• ad6d01c: --- (#1609) (@dependabot[bot])
• 4f94cd4: --- (#1610) (@dependabot[bot])
• a79386e: Error processor (#1598) (@lovromazgon)
• b6f3b46: Schema support (design doc) (#1532) (@hariso)
• 7ad8e66: Update release procedure (#1514) (@hariso)
• 4641fb4: Webhook processor fixes (#1597) (@lovromazgon)
• 734fa82: add headers to http processor (#1552) (@maha-hajja)
• 07be204: automate scarf package links in conduit releases body (#1538) (@maha-hajja)
• 5d51602: chore: update conduit-connector-postgres (#1615) (@raulb)
• 00e1ec0: fix: when settings is nil (#1545) (@raulb)
• 3af1dba: populate settings map if nil (#1517) (@lovromazgon)
• 504ebbc: release v0.10.1 (@lovromazgon)
• 4cb3847: update nightly releases and docs' docker image to point to our scarf package (#1534) (@maha-hajja)

Release Artifacts

OSX Darwin

• 📦 conduit_0.10.1_Darwin_arm64.tar.gz
• 📦 conduit_0.10.1_Darwin_x86_64.tar.gz

Linux

• 📦 conduit_0.10.1_Linux_arm64.deb
• 📦 conduit_0.10.1_Linux_arm64.rpm
• 📦 conduit_0.10.1_Linux_arm64.tar.gz
• 📦 conduit_0.10.1_Linux_i386.deb
• 📦 conduit_0.10.1_Linux_i386.rpm
• 📦 conduit_0.10.1_Linux_i386.tar.gz
• 📦 conduit_0.10.1_Linux_x86_64.deb
• 📦 conduit_0.10.1_Linux_x86_64.rpm
• 📦 conduit_0.10.1_Linux_x86_64.tar.gz

Windows

• 📦 conduit_0.10.1_Windows_arm64.zip
• 📦 conduit_0.10.1_Windows_i386.zip
• 📦 conduit_0.10.1_Windows_x86_64.zip

Docker images

To pull the image, you can execute the following command:

docker pull conduit.docker.scarf.sh/conduitio/conduit:v0.10.1

v0.10.0

What's new?

This release focuses on improving connector functionality regarding multiple collections and record routing. You can read more about this on our documentation page.

It also improves the webhook.http processor to allow the usage of record data in the URL. More information can be found here.

Changelog

• 55926a2: Mutliple collections design document (#1465) (@lovromazgon)
• 9ea03b5: Upgrade builtin connectors, SDK and commons (#1515) (@lovromazgon)
• cef8ae4: Upgrade go version to 1.22.2 (#1485) (@lovromazgon)
• 4de440b: add dynamic url to HTTP processor + fix RawData bug (#1498) (@maha-hajja)
• a0c743c: add scarf pixel (#1506) (@maha-hajja)
• 15da724: chore: add scarf (#1507) (@maha-hajja)
• b208a0f: chore: upgrade ui dependencies (#1470) (@raulb)
• 98ad6b3: feature(ui): update to ember 5.7 and latest dependencies (#1483) (@raulb)
• e2cd585: http processor: replace space with + (#1499) (@maha-hajja)
• af6a8b7: upgrade Go version in docker image to 1.22 (#1490) (@lovromazgon)
• 21b2f7c: upgrade Go version to 1.22 (again) (#1501) (@lovromazgon)

Release Artifacts

OSX Darwin

• 📦 conduit_0.10.0_Darwin_arm64.tar.gz
• 📦 conduit_0.10.0_Darwin_x86_64.tar.gz

Linux

• 📦 conduit_0.10.0_Linux_arm64.deb
• 📦 conduit_0.10.0_Linux_arm64.rpm
• 📦 conduit_0.10.0_Linux_arm64.tar.gz
• 📦 conduit_0.10.0_Linux_i386.deb
• 📦 conduit_0.10.0_Linux_i386.rpm
• 📦 conduit_0.10.0_Linux_i386.tar.gz
• 📦 conduit_0.10.0_Linux_x86_64.deb
• 📦 conduit_0.10.0_Linux_x86_64.rpm
• 📦 conduit_0.10.0_Linux_x86_64.tar.gz

Windows

• 📦 conduit_0.10.0_Windows_arm64.zip
• 📦 conduit_0.10.0_Windows_i386.zip
• 📦 conduit_0.10.0_Windows_x86_64.zip

Docker images

To pull the image, you can execute the following command:

docker pull conduit.docker.scarf.sh/conduitio/conduit:v0.10.0

v0.9.1

Changelog

• db24b6a: Add more example pipelines (#1447) (@hariso)
• 300a7d8: Docker image: bump Alpine image to 3.19 (#1445) (@hariso)
• 76897a6: Expose register processor plugin (#1441) (@lovromazgon)
• 51ba7a7: Make installing of tools more stable (#1423) (@lovromazgon)
• 32ebab3: Make sure plugins with non-semantic version are registered as latest (#1439) (@lovromazgon)
• c9d70b0: Remove obsolete page about processors (#1416) (@hariso)
• c7e2813: Small fixes in processor docs (#1417) (@lovromazgon)
• 61c8af3: Update links about referencing fields in processors (#1424) (@hariso)
• 8193c56: Upgrade Kafka connector to v0.7.2 (#1448) (@hariso)
• 647e462: add support for converting []byte to string (#1463) (@lovromazgon)
• 81009b5: chore: remove unused (#1429) (@raulb)
• e7e3983: chore: update specs (#1428) (@raulb)
• f60cbcb: mention standalone processors in the readme (#1451) (@maha-hajja)
• aa2f838: remove redundant actions (#1430) (@lovromazgon)
• 1ac0964: upgrade processor sdk and postgres (#1464) (@lovromazgon)

Docker images

To pull the image, you can execute the following command:

docker pull ghcr.io/conduitio/conduit:v0.9.1

v0.9.0

This release contains a major overhaul of Conduit processors. Builtin processors have been rewritten from scratch and now provide more powerful ways to transform the data and can target any field in the record. Additionally, we have added the ability to write standalone WASM processors, which allows you to manipulate records in your language of choice while keeping the processor as performant as a built-in processor. Check out the processor SDK for information about building your custom processor.

Changelog

• 2b08f0d: Replace deprecated goreleaser argument (#1415) (@hariso)
• 00dbcba: [Feature] Better processors (#1378) (@lovromazgon)

Docker images

To pull the image, you can execute the following command:

docker pull ghcr.io/conduitio/conduit:v0.9.0

v0.8.1

Changelog

• e36be0f: Better processors design document (#1279) (@lovromazgon)
• 96481ff: Block profile (#1301) (@lovromazgon)
• d2dbf4a: Bump s3 connector to v0.5.1 (#1315) (@lyuboxa)
• abaaf49: Convert provision config pipelines into model yaml (#1278) (@lyuboxa)
• f95c941: Improve builtin plugin sandbox (#1380) (@lovromazgon)
• afc939c: Increase length limit for ID and Name for pipeline (128) and connector (256) (#1331) (@lovromazgon)
• 92142fd: Take golangci-lint version from go.mod (#1300) (@lovromazgon)
• cf84e59: Update Dockerfile to remove health check (#1335) (@lyuboxa)
• 76e801e: Update connector list to point to conduit.io (#1334) (@lovromazgon)
• 6a3fe41: Use OpenCDC proto from conduit-commons (#1330) (@lovromazgon)
• b65713a: Use goccy/go-json instead of encoding/json (#1383) (@hariso)
• 01fafc5: Validate connector name and pipeline description through API (#1242) (@AdamHaffar)
• bda52d4: add periods as allowed ids + fix http log (#1307) (@maha-hajja)
• 2c3dfaf: conditional processor execution (#1349) (@maha-hajja)
• e06af99: downgrade container policy action to v2.1.2 (#1293) (@lovromazgon)
• 442f8ee: execute processor condition template (#1352) (@maha-hajja)
• dccde3f: fix: OpenCDC unwrap (#1343) (@samirketema)
• deb2907: fix: ellipsis connector text (#1394) (@jmar910)
• f3fbcc7: fix: opencdc unwrap (#1355) (@raulb)
• f71d078: fix: unwrap openCDC records on create and update (#1362) (@raulb)
• c149030: refactor: opencdc test examples (#1363) (@raulb)
• 2e2fde6: upgrade kafka connector to v0.7.1 (#1341) (@samirketema)

Docker images

To pull the image, you can execute the following command:

docker pull ghcr.io/conduitio/conduit:v0.8.1

v0.8.0

Changelog

• 4bae904: Programmatic way of knowing that Conduit has been initialized (#1224) (@hariso)
• 0b3dd54: Escape analysis (#1245) (@nassor)
• c40884a: Pipeline created from API gets overwritten by config file with duplicate ID (#1229) (@AdamHaffar)
• 784f05f: Remove Assert Package 1/3 (#1177) (@AdamHaffar)
• 8e1cdf7: Remove Assert Package 3/3 (#1205) (@AdamHaffar)
• be519c6: [Perf] Inspector refactoring and optimization (#1248) (@lovromazgon)
• ce7bb3a: add flags for profiling, hide them in help (#1246) (@lovromazgon)
• 9fd97f9: add json tags to v2 provisioning model (#1212) (@lovromazgon)
• bbdb074: clone record without reflection (#1247) (@lovromazgon)
• 3145202: encode connectors in persister only on flush (#1249) (@lovromazgon)
• dca7d58: expose delete method in provisioning service (#1230) (@lovromazgon)
• 0a5225d: fix make target install-tools (#1185) (@lovromazgon)
• 3111797: optimize fanin and fanout nodes to use less reflection (#1250) (@lovromazgon)
• db5f1e1: update http processor request (#1201) (@maha-hajja)

Docker images

To pull the image, you can execute the following command:

docker pull ghcr.io/conduitio/conduit:v0.8.0

v0.7.2

Changelog

• 7cf9c3b: Add a check in service.go to avoid panic when 2 connector ID's are the same (#1134) (@AdamHaffar)
• f9896cb: Add make lint (#1128) (@hariso)
• 5d87fc4: Make config more flexible (#1171) (@lovromazgon)
• 99d32f0: Refactor entrypoint, parse config from env vars and file (#1131) (@lovromazgon)
• 75726c7: Refactor logger to use zerolog ctx attached to event (#1169) (@lovromazgon)
• f1fd7ea: Replace github.com/golang/mock with go.uber.org/mock (#1168) (@lovromazgon)
• 99c9593: Use zip for windows releases (#1129) (@lyuboxa)
• 427db69: bump golangci-lint to v1.53.3 (#1152) (@maha-hajja)
• 0029b39: clean up DLQ connectors on connectors service init (#1144) (@lovromazgon)
• 58aac9a: fix issue #453 add nil checkers for endpoints (#1173) (@AdamHaffar)

Docker images

To pull the image, you can execute the following command:

docker pull ghcr.io/conduitio/conduit:v0.7.2