connect-gcp-cloud-functions-sink

Google Cloud Functions Sink connector

Objective

Quickly test Google Cloud Functions Sink connector.

• Active Google Cloud Platform (GCP) account with authorization to create resources

Google Cloud Functions Setup

• Navigate to the Google Cloud Console

• Go to the Cloud Functions tab.

• Create a new function. Use the default code that is provided.

Note down the project id, the region (example us-central1), and the function name (example function-1) as they will be used later.

How to run

Simply run:

$ ./google-cloud-functions.sh <PROJECT> <REGION> <FUNCTION>

Details of what the script is doing

Produce test data to the functions-messages topic in Kafka

$ docker exec -i broker kafka-console-producer --broker-list broker:9092 --topic functions-messages --property parse.key=true --property key.separator=, << EOF
key1,value1
key2,value2
key3,value3
EOF

Creating Google Cloud Functions Sink connector

$ curl -X PUT \
     -H "Content-Type: application/json" \
     --data '{
               "connector.class": "io.confluent.connect.gcp.functions.GoogleCloudFunctionsSinkConnector",
                    "tasks.max" : "1",
                    "topics" : "functions-messages",
                    "key.converter":"org.apache.kafka.connect.storage.StringConverter",
                    "value.converter":"org.apache.kafka.connect.storage.StringConverter",
                    "confluent.topic.bootstrap.servers": "broker:9092",
                    "confluent.topic.replication.factor":1,
                    "function.name": "'"$FUNCTION"'",
                    "project.id": "'"$PROJECT"'",
                    "region": "'"$REGION"'",
                    "reporter.bootstrap.servers": "broker:9092",
                    "reporter.error.topic.name": "test-error",
                    "reporter.error.topic.replication.factor": 1,
                    "reporter.error.topic.key.format": "string",
                    "reporter.error.topic.value.format": "string",
                    "reporter.result.topic.name": "test-result",
                    "reporter.result.topic.key.format": "string",
                    "reporter.result.topic.value.format": "string",
                    "reporter.result.topic.replication.factor": 1
          }' \
     http://localhost:8083/connectors/gcp-functions/config | jq .

Confirm that the messages were delivered to the result topic in Kafka

$ docker exec broker kafka-console-consumer --bootstrap-server broker:9092 --topic test-result --from-beginning --max-messages 3

Result:

Hello World!
Hello World!
Hello World!
Processed a total of 3 messages

N.B: Control Center is reachable at http://127.0.0.1:9021