Skip to content
This repository has been archived by the owner on Aug 27, 2021. It is now read-only.

Latest commit

 

History

History
76 lines (52 loc) · 2.77 KB

README.md

File metadata and controls

76 lines (52 loc) · 2.77 KB

ALB-Logs-to-Elasticsearch

#Send ALB logs from S3 bucket to ElasticSearch using AWS Lambda. This project based on awslabs/amazon-elasticsearch-lambda-samples,blmr/aws-elb-logs-to-elasticsearch Sample code for AWS Lambda to get AWS ELB log files from S3, parse and add them to an Amazon Elasticsearch Service domain.

Deployment Package Creation

  1. On your development machine, download and install Node.js.

  2. Go to root folder of the repository and install node dependencies by running:

    npm install

    Verify that these are installed within the node_modules subdirectory.

  3. Create a zip file to package the index.js and the node_modules directory

The zip file thus created is the Lambda Deployment Package.

AWS Configuration

Set up the Lambda function and the S3 bucket. You can refer to for more details > Lambda-S3 Walkthrough.

Set up the Lambda function & IAM permissions for VPC access / ENI create permissions > Lambda-VPC docs.

Please keep in mind the following notes and configuration overrides:

  • The S3 bucket must be created in the same region as Lambda is, so that it can push events to Lambda.

  • When registering the S3 bucket as the data-source in Lambda, add a filter for files having .log.gz suffix, so that Lambda picks up only apache log files.

  • You need to set Lambda environment variables for the following:

 ES_DOCTYPE: for the `type` field in Elasticsearch
 ES_ENDPOINT: the http://FQDN:port of your Elasticsearch Service
 ES_INDEX_PREFIX: the prefix for your indices, which will be suffixed with the date
 ES_BULKSIZE: The number of log lines to bulk index into ES at once. Try 200.

  • The following authorizations are required:

    1. Lambda permits S3 to push event notification to it
    2. S3 permits Lambda to fetch the created objects from a given bucket
    3. Lambda permissions for VPC / ENI access
    4. Lambda handler is set to index.handler
    5. Don't forget the ES domain parameters in index.js

    The Lambda console provides a simple way to create an IAM role with policies for (1).
    For (2), when creating the IAM role, choose the "S3 execution role" option; this will load the role with permissions to read from the S3 bucket. For (3), make sure you enable VPC access with ec2:eni permissions.

Event source Add Event source for your lambda function

Event source type: S3

Bucket: s3-elb-access-logs

Event type: Object Created (All)

Suffix: .log.gz

#License ASL

https://aws.amazon.com/asl/