Need a list of the available downloadable connectors

[Yashaswini1998]

Hi, can someone please direct me to the list of available downloadable list of inbound and outbound connectors?

The hub link only gives the details and not the commands to download the connectors using the cdk command as shown here for the HTTP inbound connector in the quickstart "cdk hub download infinyon/[email protected]"

The config files given in the hub link ask for a complete RUST project to be built from scratch including files like Cargo.toml etc. Can someone please share a sample project I can refer to build my custom connectors as well?

[ajhunyady]

@digikata can you please share the details, and maybe the ‘hello world’ connector as well.

[Yashaswini1998]

@ajhunyady Thank you so much for responding.

Just an update. I got the list of the available connectors by doing "fluvio hub connector list". However, the duckdb-sink connector says 404 error not found.

I'm trying the sql-sink connector now where the given config in the hub link uses json-sql smart module. The json-sql smart module also seems to be missing as this also throws a 404 Not Found error.
$ cdk hub download infinyon/[email protected]

Also, when I try to deploy the connector using cdk deploy start command, it says connector started but after listing it says Stopped under status.

I observed the same with the mqtt-source connector as well. Please find the screenshot below.

So, I think @digikata and @ajhunyady it would be really helpful if you could share that "hello world" connector along with the step-by-step instructions to set it up.

[drc-infinyon]

The sql sink connector download command is

$ cdk hub list

Which gives you

benmanns/[email protected]        public     
day-os/[email protected]        public     
infinyon-labs/[email protected]  public     
infinyon/[email protected]         public     
infinyon/[email protected]          public     
infinyon/[email protected]         public     
infinyon/[email protected]   public     
infinyon/[email protected]  public     
infinyon/[email protected]          public     
infinyon/[email protected]        public     
infinyon/[email protected]         public     
infinyon/[email protected]            public     
qdrant/[email protected]           public     
sdf-test/[email protected]   public

And then the command to download is:

cdk hub download infinyon/[email protected]

Can you share your connector config.yaml for the stopped connectors?

[ajhunyady]

@Yashaswini1998, smartmodules are downloaded on your cluster and they require a different command:

$ fluvio hub smartmodule download infinyon/[email protected]

Unfortunately DuckDb connector was experimental and we did not update it. We'll move that out of the infinyon hub.
However, http-sink should work as expected.

[Yashaswini1998]

@ajhunyady Okay, Thank you so much for your response. Can you please confirm if we can expect duckdb connector to be available anytime soon?

@drc-infinyon Thank you for your response too!
I'm using the config.yaml give in the hub documentation here for the respective connectors.
I'm following the below steps: (using mqtt-source example)

1. I download the connector cdk hub download infinyon/[email protected]
2. Create the sample-config.yaml file with the given script in the link
3. Deploy the connector cdk deploy start --config sample-config.yaml --ipkg infinyon-mqtt-source-0.2.9.ipkg
   Says started connector
4. Immediately when I try to list the deployments it says stopped
   

Am I missing any step?

Note: I have checked that the fluvio cluster is resumed and running
http-source connector works fine that is given in the quickstart

[ajhunyady]

what does cdk deploy log show?

[Yashaswini1998]

timed out searching metadata Partition failed due to timeout: 60000 ms

[ajhunyady]

A few questions:

• What environment are you running?
• Can you produce & consume from the CLI?
• What is the output of fluvio partition list ?

[Yashaswini1998]

I'm running on VM with

1. OS - Ubuntu 22.04.3 (64-bit)
2. RAM - 8 GB
3. Disk Space - 50 GB

Produce and consume from CLI was working earlier but is now throwing the same timed out error

This is the output of fluvio partition list but cluster says running

[ajhunyady]

There are some communication issues between the SC and the SPU.
Could you try deleting and recreating the cluster to see if that fixes the issue.

The connectors are essentially fluvio clients from the cluster's point of view.
If the cli produce/consume does not work, the connectors won't work either.

[Yashaswini1998]

The cluster works fine for sometime. But the observation is that I start getting these timed out errors after using it for a while. And won't recover even after deleting all the current deployments and redeploying connectors.

Anyway, for the time it's working, can you please share some example program showing a source connector with a smart module -> data transferred to sink -> sink connector with a smart module?

Basically, how do I build a complete pipeline with these connectors to send and receive data across devices?

This would be really helpful!

[sehz]

Hi @Yashaswini1998 , What's version of Fluvio and CPU Arch ( x86 or Arm64?). There is log file in the `~/.fluvio/log. Hop in our discord https://discord.com/invite/bBG2dTz, for more interactive discussion.
For more complex pipeline, would recommend to look at: https://www.fluvio.io/sdf/. SDF can model complex dataflow and pipeline. It's kind of next evolution of SmartModule.

[Yashaswini1998]

Hi @sehz, please find the details below

I'm facing another problem, wherever I deploy and start the sql-sink connector, the log file grows rapidly and hogs up the disk space even if I'm not sending any data to the sink. There is nothing in the postgres SQL table as well to which the sql-sink connector is configured to write.
The size of log file for sql-sink connector reached 30.4 GB

The log file has the below messages

[sehz]

Connector is continuously repeating same operation which results in same error which is filling the log. To fix this issues, can you share Postgres version? Not sure whether this is fluvio or Postgres connection issue.

[Yashaswini1998]

Postgres version
psql (PostgreSQL) 14.13 (Ubuntu 14.13-0ubuntu0.22.04.1)

[sehz]

In order to diagnose this further, can you describe step by step instruction on how to reproduce problem or point to existing configuration?

[Yashaswini1998]

I'm following the steps given in https://www.fluvio.io/docs/hub/connectors/outbound/sql

1. Create the connector-config.yaml file using the sample with transformation given in the link.
2. Install postgres and create a database called mydatabase with the topic_message table using the CREATE table command given in the link
3. Replace the URL with my local postgres database url
4. Deploy the connector using the cdk deploy start command from the link
5. The connector is in the running state

Now, I leave it idle, no data pushed to the topic. Still the log file grows as big as 30+ GBs within 1-2 hrs.

[sehz]

I was able to get sql sink connecter to postgres running. Can you share your connector configuration so I could replicate it?

[github-actions]

Stale issue message