Skip to content

Commit

Permalink
Merge pull request #19285 from gunnalahema/fluent-bit-doc-update
Browse files Browse the repository at this point in the history 
update flunet bit output plugin docs
  • Loading branch information
@cbehera-newrelic
cbehera-newrelic authored Nov 13, 2024
2 parents 23e6bd7 + 5cdfd77 commit 9d2e586
Showing 1 changed file with 86 additions and 79 deletions.
165 changes: 86 additions & 79 deletions src/content/docs/logs/forward-logs/fluent-bit-plugin-log-forwarding.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ tags:
- Logs
- Enable log management in New Relic
- Enable log monitoring in New Relic
- Fluent Bit
translate:
- kr
metaDescription: 'Install and configure the New Relic logging plugin for Fluent Bit, so you can use enhanced log management capabilities.'
Expand All @@ -13,117 +14,120 @@ redirects:
- /docs/logs/enable-logs/enable-logs/fluent-bit-plugin-logs
- /docs/logs/enable-log-monitoring-new-relic/enable-log-monitoring-new-relic/fluent-bit-plugin-log-forwarding
- /docs/logs/enable-log-management-new-relic/enable-log-monitoring-new-relic/fluent-bit-plugin-log-forwarding
freshnessValidatedDate: never
freshnessValidatedDate: 2024-10-30
---

If your log data is already being monitored by [Fluent Bit](https://fluentbit.io/), you can use our Fluent Bit output plugin to forward and enrich your log data in New Relic.

Forwarding your Fluent Bit logs to New Relic will give you enhanced <InlinePopover type="logs"/> capabilities to collect, process, explore, query, and alert on your log data.
Forwarding your Fluent Bit logs to New Relic will give you enhanced <InlinePopover type="logs"/> capabilities to collect, process, explore, query, and alert on your log data. You have these options to install the Fluent Bit:

## Basic process [#compatibility-requirements]
* [Kubernetes installation](#k8s-installation)
* [Using a Docker image](#helm-docker-image)
* [On-host installation](#on-host-installation)

We have published a container with the plugin installed. It serves as a base image to be used by our Kubernetes integration. We recommend you use this base image and layer your own custom configuration files.
## Kubernetes installation [#k8s-installation]

To forward your logs from Fluent Bit to New Relic:
New Relic has a [Fluent Bit output plugin](https://github.com/newrelic/newrelic-fluent-bit-output) to forward your logs to New Relic log management. You can install this plugin as a standalone Docker image in a Kubernetes cluster, where it functions as a DaemonSet, also known as the Kubernetes plugin.

1. Make sure you have:
* A New Relic <InlinePopover type="licenseKey"/>
* Fluent Bit 1.0 or higher (recommended), although v0.12 or higher is supported
* Fluent Bit Windows install directions can be found [here](https://docs.fluentbit.io/manual/installation/windows)
* Fluent Bit Linux install directions can be found [here](https://docs.fluentbit.io/manual/installation/linux)
2. [Install](#fluentbit-plugin) the Fluent Bit plugin.
3. [Configure](#configure-plugin) the Fluent Bit plugin.
4. [Test](#test-plugin) the Fluent Bit plugin.
5. Generate some traffic and wait a few minutes, then [check your account](#find-data) for data.
You can install it in your cluster using our [Helm chart](https://github.com/newrelic/helm-charts/tree/master) in two ways:

## Install the Fluent Bit plugin [#fluentbit-plugin]
* Through our guided install
* Manual installation

To install the Fluent Bit plugin:
### Using our guided install [#helm-guided-install]

1. Navigate to New Relic's [Fluent Bit plugin repository on GitHub](https://github.com/newrelic/newrelic-fluent-bit-output).
2\. From the repository page, [clone or download the repository](https://help.github.com/en/articles/cloning-a-repository).
3\. Run the following command to build your plugin:
Although the [`newrelic-logging`](https://github.com/newrelic/helm-charts/tree/master/charts/newrelic-logging) chart works as a standalone, we recommend installing it as part of the [`nri-bundle`](https://github.com/newrelic/helm-charts/tree/master/charts/nri-bundle) chart.

```sh
cd newrelic-fluent-bit-output && make all
```
The best way to install this is through our [guided installation](/install/kubernetes/) process. This guided install generates the Helm commands required to install it.

2. Store `out_newrelic.so` or `out_newrelic_winXX.dll` at a location that can be accessed by the `fluent-bit` daemon.
### Manual installation

<Callout variant="tip">
If you'd rather not compile the plugin yourself, you can download pre-compiled versions from our [GitHub repository's releases page](https://github.com/newrelic/newrelic-fluent-bit-output/releases).
</Callout>
You can install it manually using Helm, although this option is not recommended. Run this command to install the repo:

## Upgrade the Fluent Bit plugin [#upgrade-plugin]
```bash
helm repo add newrelic https://helm-charts.newrelic.com
```

Before you upgrade your Fluent Bit plugin, run the following NRQL query to find the current versions of the output plugin being used in your system:
To update the repo, run this command:

```sql
FROM K8sContainerSample
SELECT latest(containerImage)
WHERE podName like '%newrelic-logging%'
FACET clusterName
```bash
helm repo update newrelic
```

<Callout variant="caution">
Fluent Bit output plugin versions 1.16.0 through 1.19.2 are affected by a [security vulnerability (CVE-2024-4323)](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-4323). If you're using one of these versions, upgrade to version 2.0.0 or higher. For more information on this, see our [security bulletin NR24-01 - Fluent Bit](/docs/security/new-relic-security/security-bulletins/security-bulletin-nr24-01/).
<Callout variant="tip">
See our [supported configuration parameters](https://github.com/newrelic/helm-charts/tree/master/charts/newrelic-logging#supported-configuration-parameters) in our `newrelic-logging` Helm chart.
</Callout>

To upgrade, follow the [installation instructions](#fluentbit-plugin) or grab the latest pre-compiled version from our [GitHub repository](https://github.com/newrelic/newrelic-fluent-bit-output/releases).
See [Uninstall Kubernetes integration](/docs/kubernetes-pixie/kubernetes-integration/installation/uninstall-kubernetes/) if you want to uninstall it.

## Install Fluent Bit output plugin [#fluent-bit]
## Using a Docker image [#helm-docker-image]

New Relic has a [Fluent Bit](https://fluentbit.io/) [output plugin](https://github.com/newrelic/newrelic-fluent-bit-output) to forward your logs to New Relic log management. This plugin is also provided in a standalone Docker image that can be installed in a Kubernetes cluster in the form of a DaemonSet, also known as the Kubernetes plugin.
If you've your own custom kubernetes integration, we recommend using our [Docker image](https://hub.docker.com/r/newrelic/newrelic-fluentbit-output) that comes with the [`newrelic-fluent-bit-output`](https://github.com/newrelic/newrelic-fluent-bit-output) plugin. Or you can use the Docker image as a base image and layer your own custom configuration files.

You can install it in your cluster using our Helm chart in two ways.
## On-host installation [#on-host-installation]

### Use our guided install [#helm-guided-install]
For an on-host installation of the Fluent Bit plugin, follow these steps:

Although the [newrelic-logging](https://github.com/newrelic/helm-charts/tree/master/charts/newrelic-logging) chart works as a standalone, we recommend installing it as part of the [nri-bundle](https://github.com/newrelic/helm-charts/tree/master/charts/nri-bundle) chart.
1. Open the New Relic's [Fluent Bit plugin repository](https://github.com/newrelic/newrelic-fluent-bit-output) on GitHub.

The best way to install this is through our [guided installation](/docs/kubernetes-pixie/kubernetes-integration/installation/kubernetes-integration-install-configure/) process. This guided install can generate the Helm 3 commands required to install it ([see "Helm 3"](/docs/kubernetes-pixie/kubernetes-integration/installation/kubernetes-integration-install-configure/#finish-your-install)).
2. From the repository page, [clone or download the repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository).

### Manual installation [#manual-helm-install]
3. Run this command to build your plugin:

Alternately, you can install it manually using Helm, by running this command to install the repo:
```sh
cd newrelic-fluent-bit-output && make all
```

```bash
helm repo add newrelic https://helm-charts.newrelic.com
```
4. Store `out_newrelic.so` or `out_newrelic_winXX.dll` at a location where `fluent-bit` daemon can access them.

To update the repo you can run:
<Callout variant="tip">
If you'd rather not compile the plugin yourself, download pre-compiled versions from our [GitHub repository's releases page](https://github.com/newrelic/newrelic-fluent-bit-output/releases).
</Callout>

```bash
helm repo update newrelic
### Upgrade the Fluent Bit plugin [#upgrade-plugin]

Before you upgrade your Fluent Bit plugin, run the following NRQL query to find the current versions of the output plugin you're using in your system:

```sql
FROM K8sContainerSample
SELECT latest(containerImage)
WHERE podName like '%newrelic-logging%'
FACET clusterName
```

Go here for [uninstallation instructions](/docs/kubernetes-pixie/kubernetes-integration/uninstall-kubernetes/).
<Callout variant="caution">
The [security vulnerability (CVE-2024-4323)](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-4323) affects Fluent Bit output plugin versions 1.16.0-1.19.2. If you're using one of these versions, update to version 2.0.0 or higher. For more information on this, see our security bulletin [NR24-01 - Fluent Bit](/docs/security/new-relic-security/security-bulletins/security-bulletin-nr24-01/).
</Callout>

To update, follow the installation instructions or grab the latest pre-compiled version from our [GitHub repository](https://github.com/newrelic/newrelic-fluent-bit-output/releases).

## Configure the Fluent Bit plugin [#configure-plugin]

Fluent Bit needs to know the location of the New Relic plugin and the New Relic <InlinePopover type="licenseKey"/> to output data to New Relic. To configure your Fluent Bit plugin:
Fluent Bit needs to know the location of the New Relic plugin and the New Relic <InlinePopover type="licenseKey"/> to output data to New Relic.

<Callout variant="important">
Pay attention to white space when editing your config files. Be sure to use four spaces to indent and one space between keys and values.
</Callout>

1. Locate or create a `plugins.conf` file in your plugins directory.
To configure your Fluent Bit plugin, follow these steps:

1. Locate or create the `plugins.conf` file in your plugins directory.

2. In the `plugins.conf` file, add a reference to `out_newrelic.so`, adjacent to your `fluent-bit.conf` file:

```ini
[PLUGINS]
Path /PATH/TO/newrelic-fluent-bit-output/out_newrelic.so
```
```ini
[PLUGINS]
Path /PATH/TO/newrelic-fluent-bit-output/out_newrelic.so
```

3. In the `fluent-bit.conf` file, add the following line under the `service` block:

```ini
[SERVICE]
# This is the main configuration block for fluent bit.
# Ensure the follow line exists somewhere in the SERVICE block
Plugins_File plugins.conf
```
```ini
[SERVICE]
# This is the main configuration block for fluent bit.
# Ensure the follow line exists somewhere in the SERVICE block
Plugins_File plugins.conf
```

4. At the bottom of the `fluent-bit.conf` file, add the following to set up the input, filter, and output sections. Replace the placeholder text with your <InlinePopover type="licenseKey"/>:

Expand Down Expand Up @@ -162,30 +166,29 @@ Fluent Bit needs to know the location of the New Relic plugin and the New Relic
licenseKey YOUR_LICENSE_KEY
```

5. Restart your Fluent Bit instance with the following command:
5. Restart your Fluent Bit instance. Run this command:

```sh
fluent-bit -c /PATH/TO/fluent-bit.conf
```

<InstallFeedback/>

## Test the Fluent Bit plugin [#test-plugin]

To test if your Fluent Bit plugin is receiving input from a log file:

1. Run the following command to append a test log message to your log file:
1. Run this command to append a test log message to your log file:

```sh
echo "test message" >> /PATH/TO/YOUR/LOG/FILE
```
2. Search our [logs UI](https://one.newrelic.com/launcher/logger.log-launcher) for `test message`.

For more options, see the [Fluent Bit modify filter documentation](https://docs.fluentbit.io/manual/pipeline/filters/modify) and our documentation to [forward your logs using the infrastructure agent](/docs/logs/enable-log-management-new-relic/enable-log-monitoring-new-relic/forward-your-logs-using-infrastructure-agent/#automatically-inserted-attributes).
2. Search for `test message` in our [logs UI](https://one.newrelic.com/launcher/logger.log-launcher) .

For more options, see [the `modify` filter on the Fluent Bit documentation](https://docs.fluentbit.io/manual/pipeline/filters/modify). See also our documentation to [forward your logs using the infrastructure agent](/docs/logs/forward-logs/forward-your-logs-using-infrastructure-agent/#automatically-inserted-attributes).

## Optional: Configure plugin attributes [#instrument-plugin]

Once you have [installed](#fluentbit-plugin) and [configured](#configure-plugin) the Fluent Bit plugin, you can use the following attributes to configure how the plugin sends data to New Relic:
Once you've installed and [configured](#configure-plugin) the Fluent Bit plugin, you can use the following attributes to configure how the plugin sends data to New Relic:

<table>
<thead>
Expand Down Expand Up @@ -237,7 +240,7 @@ Once you have [installed](#fluentbit-plugin) and [configured](#configure-plugin)
</td>

<td>
<DNT>**Deprecated.**</DNT> Takes a New Relic [Insights insert key](/docs/insights/insights-data-sources/custom-data/send-custom-events-event-api#register), but using the `licenseKey` attribute is preferred. Use either `licenseKey` or `apiKey`, not both.
<DNT>**Deprecated.**</DNT> Takes a New Relic [insights insert key](/docs/insights/insights-data-sources/custom-data/send-custom-events-event-api#register), but using the `licenseKey` attribute is preferred. Use either `licenseKey` or `apiKey`, not both.
</td>
</tr>

Expand All @@ -247,33 +250,37 @@ Once you have [installed](#fluentbit-plugin) and [configured](#configure-plugin)
</td>

<td>
Defaults to `https://log-api.newrelic.com/log/v1` - If using an EU key needs to be set to `https://log-api.eu.newrelic.com/log/v1`
Defaults to `https://log-api.newrelic.com/log/v1`. If you're using an EU key, you need set it to `https://log-api.eu.newrelic.com/log/v1`
</td>
</tr>
</tbody>
</table>

## View log data [#find-data]

If everything is configured correctly and your data is being collected, you should see log data in both of these places:
If you configured everything correctly and New Relic collects your data, you should see log data in both of these places:

* Our [logs UI](https://one.newrelic.com/launcher/logger.log-launcher)
* Our tools for running [NRQL queries](/docs/chart-builder/use-chart-builder/choose-data/use-advanced-nrql-mode-specify-data). For example, you can execute a query like this:

* Our tools for running [NRQL queries](/docs/query-your-data/explore-query-data/query-builder/use-advanced-nrql-mode-query-data/). For example, you can run a query like this:

```sql
SELECT * FROM Log
SELECT *
FROM Log
```

If no data appears after you enable our log management capabilities, follow our [standard log troubleshooting procedures](/docs/logs/log-management/troubleshooting/no-log-data-appears-ui/).
If you don't get any data after you enable our log management capabilities, follow our standard log [troubleshooting procedures](/docs/logs/troubleshooting/no-log-data-appears-ui/).

## What's next? [#what-next]

Explore logging data across your platform with our [logs UI](/docs/logs/log-management/ui-data/use-logs-ui).
Explore logging data across your platform with our [logs UI](/docs/logs/ui-data/use-logs-ui/).

* Get deeper visibility into both your application and your platform performance data by forwarding your logs with our [logs in context](/docs/logs/enable-log-management-new-relic/configure-logs-context/configure-logs-context-apm-agents/) capabilities.
* Set up [alerts](/docs/alerts-applied-intelligence/new-relic-alerts/alert-conditions/create-alert-conditions/).

* Set up [alerts](/docs/alerts/create-alert/create-alert-condition/alert-conditions/).

* [Query your data](/docs/query-your-data/explore-query-data/get-started/introduction-querying-new-relic-data/) and [create dashboards](/docs/query-your-data/explore-query-data/dashboards/introduction-dashboards/).

## Disable log forwarding [#disable]

To disable log forwarding capabilities, follow standard procedures in [Fluent Bit documentation](https://fluentbit.io/). You do not need to do anything else in New Relic.
To disable log forwarding capabilities, follow standard procedures in [Fluent Bit documentation](https://fluentbit.io/). You don't need to do anything else in New Relic.

0 comments on commit 9d2e586

Please sign in to comment.