In this example, we will walk you through the installation of Fluent Bit on an Amazon Linux 2 EC2 instance along with the amazon-cloudwatch-logs-for-fluent-bit and amazon-kinesis-firehose-for-fluent-bit output plugins via an AWS Systems Manager Document.
Note: This tutorial was created in 2019 when AWS for Fluent Bit was first launched. It uses the AWS Go plugins which are still supported, however, newer higher performance core Fluent Bit plugins embedded in its binary also now exist. Furthermore, Fluent Bit now has official docs for installing on Amazon Linux which is another option.
To get started you will need:
- Docker installed on your local machine.
- An Amazon S3 bucket to store your Fluent Bit config files along with the AWS output plugins for Fluent Bit.
An AWS Sytems Manager Document (SSM Document) defines the actions that Systems Manager performs on your managed instances. Systems Manager includes more than a dozen pre-configured documents that you can use by specifying parameters at runtime. Documents use JavaScript Object Notation (JSON) or YAML, and they include steps and parameters that you specify.
SSM Documents have a concept of plugins that allow you to perform actions to perform on a managed instance. One of those plugins is aws:DownloadContent which allows you to download documents and scripts from remote locations including Amazon S3. This is where we will store the Fluent Bit config files along with the AWS output plugins for Fluent Bit. SSM will then be able to download these files onto managed instances when the document is run.
The first step is to get the AWS output plugins for Fluent Bit onto your local machine. You can either run make to build each plugin (instructions on are their respective GitHub repositories) or simply just copy them out of the official AWS for Fluent Bit Docker Image.
docker create -ti --name ssmdemo amazon/aws-for-fluent-bit:latest
docker cp ssmdemo:/fluent-bit/cloudwatch.so .
docker cp ssmdemo:/fluent-bit/firehose.so .
docker cp ssmdemo:/fluent-bit/kinesis.so .
docker stop ssmdemo
docker rm ssmdemo
Now you can simply upload the AWS output plugins along with the plugins.conf and td-agent-bit.conf files to your S3 bucket. You can reference the aws:downloadContent sections within the SSM Document for ConfigureFluentBitEC2.yaml to see how they are referenced. The example td-agent-bit.conf for Fluent Bit is configured with the Forward Input Plugin with the Output configured to send to Cloudwatch.
Once that's done, you can then create the SSM Document for ConfigureFluentBitEC2.yaml to run against your managed instances.
Any other configurations for Fluent Bit would need to be configured and uploaded to the S3 bucket along with a section in the SSM Document to reference it.