Connector_Publish_Tool

We have two options for deploying connectors and UDFs: directly to Lambda or via Serverless Application Repository. We find that Serverless Application Repository makes configuring any required parameters that your connector or UDFs require easier so we built a script to build and publish your connector and UDFs to Serverless Application Repository in one step.

Publish To Serverless Application Repository

From any module directory in the github repository you can run ../tools/publish.sh S3_BUCKET_NAME <module-name> to publish the connector to your private AWS Serverless Application Repository. This will allow users with permission to do so, the ability to deploy instances of the connector via 1-Click form.

If the publish command gave you an error about the aws cli or sam tool not recognizing an argument, you likely forgot to source the new bash profile after updating your development environment so run source ~/.profile and try again.

Then you can navigate to Serverless Application Repository to search for your application and deploy it before using it from Athena.

(Alternatively you can publish your connector directly to Lambda but for simplicity this tutorial uses Serverless Application Repository because with Lambda you'll also have to create an IAM Role for the Lambda.)

Validate our Connector

One of the most challenging aspects of integrating systems (in this case our connector and Athena) is testing how these two things will work together. Lambda will capture logging from out connector in Cloudwatch Logs but we've also tried to provide some tools to stream line detecting and correcting common semantic and logical issues with your custom connector. By running Athena's connector validation tool you can simulate how Athena will interact with your Lambda function and get access to diagnostic information that would normally only be available within Athena or require you to add extra diagnostics to your connector.

Run ../tools/validate_connector.sh --lambda-func <function_name> --schema <schema> --table <table> --constraints <col1>=<val1>,<col2>=<val2>

Be sure to replace lambda_func with the name you gave to your function/catalog when you deployed it via Serverless Application Repository.

If everything worked as expected you should see the script generate useful debugging info and end with:

2019-11-07 20:25:08 <> INFO  ConnectorValidator:==================================================
2019-11-07 20:25:08 <> INFO  ConnectorValidator:Successfully Passed Validation!
2019-11-07 20:25:08 <> INFO  ConnectorValidator:==================================================