Skip to content

Commit

Permalink
add sample for lambda event filtering with dynamodb and sqs (#149)
Browse files Browse the repository at this point in the history
* add sample for lambda event filtering with dynamodb and sqs

* add more context in docs
  • Loading branch information
HarshCasper authored Sep 19, 2022
1 parent 4b5ab00 commit 523d21f
Show file tree
Hide file tree
Showing 4 changed files with 191 additions and 0 deletions.
73 changes: 73 additions & 0 deletions lambda-event-filtering/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# AWS Lambda event filtering with DynamoDB and SQS

Simple demo application illustrating AWS Lambda event source filtering with DynamoDB and SQS. For this demo, we will use AWS Serverless Application Model (SAM), and a thin LocalStack wrapper `samlocal` to create our infrastructure through SAM on LocalStack.

## Prerequisites

* LocalStack
* Docker
* `make`
* [`awslocal`](https://github.com/localstack/awscli-local)
* [`samlocal`](https://github.com/localstack/aws-sam-cli-local)
* NodeJS 14.x
* [`ulid`](https://www.npmjs.com/package/ulid)

## Installing

Setup [Serverless Application Model (SAM)](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) and [AWS SAM CLI Local](https://github.com/localstack/aws-sam-cli-local) on your local machine. We also recommend using NodeJS 14.x alongside a [Node Version Manager](https://github.com/nvm-sh/nvm) to manage your NodeJS versions.


Start LocalStack via:

```sh
localstack start -d
```

## Deploy the application

Let us first install the local dependencies:

```sh
npm install --save ulid
```

To setup the infrastructure on LocalStack, run:

```sh
samlocal deploy -g
```

You will be prompted to enter a name for the stack. Use the default options for the prompts and fill `Y` (`Yes`) for the confirmation prompt. The stack will be created and the output will be printed to the console.

If you have made any changes to the application, you can update the stack by running:

```sh
samlocal deploy
```

After deploying you can send a SQS message to the queue and see the Lambda function being triggered:

```sh
awslocal sqs send-message --queue-url http://localhost:4566/000000000000/MyQueue --message-body "{ "data" : "A" }" --delay-seconds 10
```

You will see a JSON output similar to the following:

```json
{
"MD5OfMessageBody": "64dfee8647a8264b25b01b7f22d72d3a",
"MessageId": "22fbddd2-5add-4a03-a850-152780d786c1"
}
```

In the `template.yaml` we have defined the DynamoDB table and the Stream function with a filtering criteria. We instruct the Stream function to trigger the Lambda function only when the filtering criteria is satisfied.

Using the SQS, we send a message body to the DynamoDB stream to match the specific filtering criteria. After the message is sent, we can see the Lambda function being triggered and you can check the logs to verify it.

## Destroy the application

To destroy the infrastructure on LocalStack, run:

```sh
samlocal delete
```
50 changes: 50 additions & 0 deletions lambda-event-filtering/handler.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
'use strict';

const AWS = require('aws-sdk');
const ULID = require('ulid');
const dynamo = new AWS.DynamoDB.DocumentClient();

const TABLE_NAME = process.env.TABLE_NAME

exports.save = async (event) => {
console.log(event);

const object = event.body;

const item = {
id: ULID.ulid(),
object,
date: Date.now()
}

console.log(item);

const savedItem = await saveItem(item);

return {
statusCode: 200,
body: JSON.stringify(savedItem),
}
}

exports.processDynamo = async (event) => {
console.log(event);
}

exports.processASQS = async(event) => {
console.log('Process A');
console.log(event);
}

async function saveItem(item) {
const params = {
TableName: TABLE_NAME,
Item: item
};

console.log(params)

return dynamo.put(params).promise().then(() => {
return item;
});
};
12 changes: 12 additions & 0 deletions lambda-event-filtering/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"name": "lambda-event-filtering",
"version": "1.0.0",
"description": "",
"main": "handler.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"dependencies": {
"ulid": "^2.3.0"
}
}
56 changes: 56 additions & 0 deletions lambda-event-filtering/template.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:

ProcessDynamoDBStreamFunction:
Type: AWS::Serverless::Function
Properties:
Handler: handler.processDynamo
CodeUri: .
Runtime: nodejs14.x
Events:
Stream:
Type: DynamoDB
Properties:
Stream: !GetAtt DynamoDBTable.StreamArn
BatchSize: 1
StartingPosition: TRIM_HORIZON
FilterCriteria:
Filters:
- Pattern: "{ \"eventName\" : [\"INSERT\"] }"

ProcessASQSFunction:
Type: AWS::Serverless::Function
Properties:
Handler: handler.processASQS
CodeUri: .
Runtime: nodejs14.x
Events:
MySQSEvent:
Type: SQS
Properties:
Queue: !GetAtt MyQueue.Arn
BatchSize: 1
MaximumBatchingWindowInSeconds: 0
FilterCriteria:
Filters:
- Pattern: "{\"body\": { \"data\" : [\"A\"] }}"

DynamoDBTable:
Type: AWS::DynamoDB::Table
Properties:
AttributeDefinitions:
- AttributeName: id
AttributeType: S
KeySchema:
- AttributeName: id
KeyType: HASH
BillingMode: PAY_PER_REQUEST
StreamSpecification:
StreamViewType: NEW_IMAGE

MyQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: MyQueue

0 comments on commit 523d21f

Please sign in to comment.