Skip to content

Commit

Permalink
Customize Conduit's built-in connectors (#105)
Browse files Browse the repository at this point in the history
* Customize Conduit's built-in connectors

* Update docs/connectors/additional-built-in-plugins.mdx

Co-authored-by: Raúl Barroso <[email protected]>

---------

Co-authored-by: Raúl Barroso <[email protected]>
  • Loading branch information
hariso and raulb authored Jul 31, 2024
1 parent e5c2031 commit aa8fa97
Show file tree
Hide file tree
Showing 4 changed files with 73 additions and 3 deletions.
70 changes: 70 additions & 0 deletions docs/connectors/additional-built-in-plugins.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
---
title: "Adding built-in Connectors"
sidebar_position: 2
---

Built-in connectors offer better performance when compared to standalone ones,
which is why in some cases it's desirable to have a custom build of Conduit that
includes additional built-in connectors.

The simplest way to achieve so is to write a small application that embeds
Conduit (i.e. uses Conduit as a library) and adds one or more connectors to its
default configuration.

In the example below we will add
the [HTTP connector](https://github.com/conduitio-labs/conduit-connector-http)
to Conduit as a built-in connector.

First, we initialize a Go module with `go mod init github.com/conduitio-labs/custom-conduit`.

Then , we need to add Conduit and the HTTP connector as dependencies:
```shell
go get github.com/conduitio/conduit
go get github.com/conduitio-labs/conduit-connector-http
go mod tidy
```

Once that is done, we need to write a `main` function that:
1. Adds the HTTP connector the default Conduit configuration
2. Runs Conduit with the custom configuration.

That's done in the code below:

```go
package main

import (
http "github.com/conduitio-labs/conduit-connector-http"
"github.com/conduitio/conduit/pkg/conduit"
)

func main() {
// Get the default configuration, including all built-in connectors
cfg := conduit.DefaultConfig()

// Add the HTTP connector to list of built-in connectors
cfg.ConnectorPlugins["http"] = http.Connector

conduit.Serve(cfg)
}
```

This custom version of Conduit can be built with `go build main.go -o custom-conduit`. If
you run the built binary, you can check that the HTTP connector has been
included in the build by listing all the connector plugins:
```shell
curl 'http://localhost:8080/v1/connectors/plugins'

[
{
"name": "builtin:http@(devel)",
"summary": "HTTP source and destination connectors for Conduit.",
"description": "Conduit HTTP source and destination connectors, they connect to an HTTP URL and send HTTP requests.",
"version": "(devel)",
"author": "",
"destinationParams": {},
"sourceParams": {}
}
// other plugins
]
```
2 changes: 1 addition & 1 deletion docs/connectors/connector-lifecycle.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Connector Lifecycle"
sidebar_position: 7
sidebar_position: 6
---

:::info
Expand Down
2 changes: 1 addition & 1 deletion docs/connectors/kafka-connect-connector.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Kafka Connect Connectors with Conduit"
sidebar_position: 8
sidebar_position: 9
---

# Using Kafka Connect Connectors with Conduit
Expand Down
2 changes: 1 addition & 1 deletion docs/connectors/output-formats.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Output Formats"
sidebar_position: 6
sidebar_position: 5
---

One of the challenges to be solved when integrating Conduit with other systems, such as Kafka Connect and Debezium, is
Expand Down

0 comments on commit aa8fa97

Please sign in to comment.