Skip to content

Commit

Permalink
Merge pull request #223 from ritza-co/prometheus-loki-language-edit
Browse files Browse the repository at this point in the history 
Prometheus after Loki addition language edit
  • Loading branch information
@sixhobbits
sixhobbits authored Oct 31, 2024
2 parents e467af6 + 67fcabb commit 56a8b86
Showing 1 changed file with 20 additions and 19 deletions.
39 changes: 20 additions & 19 deletions astro/src/content/docs/operate/secure-and-monitor/prometheus.mdx
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Monitor With Prometheus, Loki, And Grafana
description: Learn how to monitor FusionAuth with Prometheus, Grafana, and ntfy.
description: Learn how to monitor FusionAuth with Prometheus, Loki Grafana, and ntfy.
navcategory: admin
section: operate
subcategory: secure and monitor
Expand Down Expand Up @@ -328,7 +328,7 @@ Log in to Grafana at http://localhost:9091 with username and password `admin`.

![Grafana](/img/docs/operate/secure-and-monitor/prometheus/prometheusGrafana.png)

If you wanted to change the login settings in production, you could create the local file `prometheusGrafanaConfig.ini` with the example content below.
If you want to change the login settings in production, you can create the local file `prometheusGrafanaConfig.ini` with the example content below.

```ini
[security]
Expand Down Expand Up @@ -360,20 +360,20 @@ You can also create a new dashboard by importing a standard template from the Gr

## Store Logs In Loki

The final monitoring component you might want to use is [Grafana Loki](https://grafana.com/docs/loki) for storing logs. Loki indexes only the metadata of a log line (its time and attributes such as the server that sent it) and not its content. This is unlike Elasticsearch or OpenSearch, which indexes the log content too. Loki therefore uses far less disk space than OpenSearch but is not quickly searchable. The no-indexing choice Loki made is better for most applications, where you need only to monitor logs for errors and to store logs for auditing purposes, and don't need to run frequent queries against old logs.
The final monitoring component you might want to use is [Grafana Loki](https://grafana.com/docs/loki) for storing logs. Loki indexes only the metadata of a log line (its time, and attributes such as the server that sent it) and not its content. This is unlike Elasticsearch or OpenSearch, which index the log content, too. Loki therefore uses far less disk space than OpenSearch but is not quickly searchable. The no-indexing choice Loki made is better for most applications, where you need only to monitor logs for errors and store logs for auditing purposes, and don't need to run frequent queries against old logs.

Loki can run as a single app in a single Docker container, or as separate components in multiple containers. In [monolithic mode](https://grafana.com/docs/loki/latest/get-started/deployment-modes) Loki can handle up to 20 GB per day. This is enough for FusionAuth and is what you'll use in this guide.
Loki can run as a single app in a single Docker container or as separate components in multiple containers. In [monolithic mode](https://grafana.com/docs/loki/latest/get-started/deployment-modes), Loki can handle up to 20 GB per day. This is enough for FusionAuth and is what you'll use in this guide.

Below is a diagram showing all the [components](https://grafana.com/docs/loki/latest/get-started/components) Loki runs in a single container.

![Loki architecture](/img/docs/operate/secure-and-monitor/prometheus/prometheusLokiArchitecture.svg)

You can query logs in Grafana, or in the terminal with the Loki API or [LogCLI](https://grafana.com/docs/loki/latest/query/logcli).

Loki is only a log store, and will not fetch logs itself. Tools to send logs to Loki include: Promtail (the original sending tool), OpenTelemetry, and Alloy (a new OpenTelemetry-compliant tool from Grafana). For more options, see the [documentation](https://grafana.com/docs/loki/latest/send-data). In this guide, you use Promtail for simplicity and stability.
Loki is primarily a log store, and will not fetch logs itself. Tools to send logs to Loki include Promtail (the original sending tool), OpenTelemetry, and Alloy (a new OpenTelemetry-compliant tool from Grafana). For more options, see the [documentation](https://grafana.com/docs/loki/latest/send-data). In this guide, you use Promtail for simplicity and stability.

Check failure on line 373 in astro/src/content/docs/operate/secure-and-monitor/prometheus.mdx

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Promtail'?

Raw Output:
{"message": "[Vale.Spelling] Did you really mean 'Promtail'?", "location": {"path": "astro/src/content/docs/operate/secure-and-monitor/prometheus.mdx", "range": {"start": {"line": 373, "column": 99}}}, "severity": "ERROR"}

Check failure on line 373 in astro/src/content/docs/operate/secure-and-monitor/prometheus.mdx

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Promtail'?

Raw Output:
{"message": "[Vale.Spelling] Did you really mean 'Promtail'?", "location": {"path": "astro/src/content/docs/operate/secure-and-monitor/prometheus.mdx", "range": {"start": {"line": 373, "column": 327}}}, "severity": "ERROR"}

<Aside type="note">
When FusionAuth runs in Docker it writes logs to the terminal, and does not save them to a file or provide them via the [API](/docs/apis/system#export-system-logs). This means that the logs are also not available in the [web interface](/docs/operate/troubleshooting/troubleshooting#logs).
When FusionAuth runs in Docker, it writes logs to the terminal and does not save them to a file or provide them via the [API](/docs/apis/system#export-system-logs). This means that the logs are not available in the [web interface](/docs/operate/troubleshooting/troubleshooting#logs).
</Aside>

To use Loki, add the services below to your `docker-compose.yml` file. You are now using Grafana images because Ubuntu has no images for Promtail.

Check failure on line 379 in astro/src/content/docs/operate/secure-and-monitor/prometheus.mdx

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Promtail'?

Raw Output:
{"message": "[Vale.Spelling] Did you really mean 'Promtail'?", "location": {"path": "astro/src/content/docs/operate/secure-and-monitor/prometheus.mdx", "range": {"start": {"line": 379, "column": 138}}}, "severity": "ERROR"}
Expand Down Expand Up @@ -407,11 +407,11 @@ To use Loki, add the services below to your `docker-compose.yml` file. You are n
- db_net
```

The `faLoki` port 3100 is open so that Grafana can query it. The `prometheusLoki` volume persists log storage across container restarts. The `prometheusLokiConfig.yml` volume allows you to adjust Loki settings. Unlike the Ubuntu images, Grafana images don't use the root user. This means that the user in the container won't have permissions to create files on the Docker host machine. In production, you can inspect the running container to see what user it has, then create the `prometheusLoki` directory, and assign the directory owner as the container user. But for this prototype it's faster to set the container user to `user: root` instead, so the container can directly write to the shared volume. `target=all` runs the Loki container in monolithic mode.
The `faLoki` port 3100 is open so that Grafana can query it. The `prometheusLoki` volume persists log storage across container restarts. The `prometheusLokiConfig.yml` volume allows you to adjust Loki settings. Unlike the Ubuntu images, Grafana images don't use the root user. This means that the user in the container won't have permissions to create files on the Docker host machine. In production, you can inspect the running container to see what user it has, then create the `prometheusLoki` directory, and assign the directory owner as the container user. But for this prototype, it's faster to set the container user to `user: root` instead, so the container can directly write to the shared volume. The `target=all` configuration runs the Loki container in monolithic mode.

The `faPromtail` service waits for Loki to start, with `depends_on: faLoki`. It has volumes for a configuration file, and access to the log files saved by Docker and the Docker socket file.
The `faPromtail` service waits for Loki to start by using `depends_on: faLoki`. The service has volumes for a configuration file and for access to the log files saved by Docker and the Docker socket file.

Check failure on line 412 in astro/src/content/docs/operate/secure-and-monitor/prometheus.mdx

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Promtail'?

Raw Output:
{"message": "[Vale.Spelling] Did you really mean 'Promtail'?", "location": {"path": "astro/src/content/docs/operate/secure-and-monitor/prometheus.mdx", "range": {"start": {"line": 412, "column": 8}}}, "severity": "ERROR"}

You should also change the `fa` service to make FusionAuth wait for `Promtail` to run before FusionAuth starts. Otherwise, Loki will not record potential FusionAuth starting errors. Use the code below.
Use the code below to change the `fa` service to make FusionAuth wait for `Promtail` to run before FusionAuth starts. If FusionAuth isn't configured to wait, Loki will not record potential FusionAuth starting errors.

```yml
depends_on:
Expand Down Expand Up @@ -455,7 +455,7 @@ ruler:
alertmanager_url: http://alertmanager:9093
```

The `prometheusPromtailConfig.yml` file controls which containers Promtail will get logs from. It is documented [here](https://grafana.com/docs/loki/latest/send-data/promtail/configuration). Create the file and add the content below.
The `prometheusPromtailConfig.yml` file controls which containers Promtail will get logs from. It is documented [here](https://grafana.com/docs/loki/latest/send-data/promtail/configuration). Create the `prometheusPromtailConfig.yml` file and add the content below.

Check failure on line 458 in astro/src/content/docs/operate/secure-and-monitor/prometheus.mdx

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Promtail'?

Raw Output:
{"message": "[Vale.Spelling] Did you really mean 'Promtail'?", "location": {"path": "astro/src/content/docs/operate/secure-and-monitor/prometheus.mdx", "range": {"start": {"line": 458, "column": 67}}}, "severity": "ERROR"}

```yml
server:
Expand All @@ -481,35 +481,36 @@ scrape_configs:

The `clients` URL points to the Loki Docker service where Promtail will send logs. The `scrape_configs` section describes how Promtail will get logs.

Check failure on line 482 in astro/src/content/docs/operate/secure-and-monitor/prometheus.mdx

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Promtail'?

Raw Output:
{"message": "[Vale.Spelling] Did you really mean 'Promtail'?", "location": {"path": "astro/src/content/docs/operate/secure-and-monitor/prometheus.mdx", "range": {"start": {"line": 482, "column": 59}}}, "severity": "ERROR"}

[`docker_sd_configs`](https://grafana.com/docs/loki/latest/send-data/promtail/configuration/#docker_sd_configs) is one way for Promtail to get logs (along with local file logs and Kubernetes). It follows the Prometheus [configuration format](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#docker_sd_config), which uses the Docker [container reference format](https://docs.docker.com/reference/api/engine/version/v1.40/#operation/ContainerList).
The [`docker_sd_configs`](https://grafana.com/docs/loki/latest/send-data/promtail/configuration/#docker_sd_configs) configuration option is one way for Promtail to get logs (along with local file logs and Kubernetes). It follows the Prometheus [configuration format](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#docker_sd_config), which uses the Docker [container reference format](https://docs.docker.com/reference/api/engine/version/v1.40/#operation/ContainerList).

Check failure on line 484 in astro/src/content/docs/operate/secure-and-monitor/prometheus.mdx

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Promtail'?

Raw Output:
{"message": "[Vale.Spelling] Did you really mean 'Promtail'?", "location": {"path": "astro/src/content/docs/operate/secure-and-monitor/prometheus.mdx", "range": {"start": {"line": 484, "column": 153}}}, "severity": "ERROR"}

The `filters` section excludes all containers from having their logs stored other than FusionAuth, which has the regular expression container name `^fa$` (start, fa, end). There is no `/` in this name. If you instead used a filter of `fa`, the logs of `fa_db` would also be stored.

The `relabel_configs` section maps the Docker container name to the logs `container` metadata, so you can search for it when querying the logs. Note that while your container and service name in the Docker process list is `fa`, the name exposed in the Docker API is actually `/fa`. You can see the `/` used in the `regex` above. To see this is true in Docker, run `docker inspect fa`. You'll see the container name is actually `"Name": "/fa"`.
Log monitoring is ready. Run `docker compose up` to start all monitoring components. Browse to http://localhost:3100/ready to check Loki is up.
Log monitoring is ready. Run `docker compose up` to start all monitoring components. Browse to http://localhost:3100/ready to check that Loki is up.

Now let's view the logs in Grafana:
To view the logs in Grafana:
- Browse to Grafana and choose <Breadcrumb>Connections -> Data sources</Breadcrumb> in the sidebar.
- Choose <InlineUIElement>Add new data source</InlineUIElement> and select <InlineUIElement>Loki</InlineUIElement>.
- The only setting to change is <InlineField>URL</InlineField>. Enter `http://faLoki:3100`.
- Click <InlineUIElement>Save and test</InlineUIElement>. If Grafana cannot detect Loki, check that your URL matches your Docker compose file and there are no errors in the Docker terminal.
- Enter `http://faLoki:3100` in the <InlineField>URL</InlineField> field - this is the only setting to change.
- Click <InlineUIElement>Save and test</InlineUIElement>. If Grafana cannot detect Loki, check that your URL matches the one in your Docker Compose file and that there are no errors in the Docker terminal.
- Click <InlineUIElement>Explore</InlineUIElement> in the sidebar to start browsing your Loki logs.
- Choose Loki as your data source and enter a query value of `{container="fa"}`.
- Press <InlineUIElement>Run query</InlineUIElement> to view the logs.
- You can also filter logs and make complex queries. For example, try `{container="fa"} |~ "(ERROR|WARN)"`.

You can filter logs and make complex queries. For example, try `{container="fa"} |~ "(ERROR|WARN)"`.

![Prometheus metrics](/img/docs/operate/secure-and-monitor/prometheus/prometheusLoki.png)

Now that Loki stores FusionAuth logs, you can add log widgets to your Grafana dashboard, and use either Grafana or Loki directly to send alerts to Alertmanager.

## Next Steps

In addition to monitoring the Prometheus metrics provided by FusionAuth, you also might want to know various custom metrics, such as user login rates and successes. To do this, read the FusionAuth guide to [OpenTelemetry](./opentelemetry) and how to use it to create a bash script to collect any metric the FusionAuth API offers.
In addition to monitoring the Prometheus metrics provided by FusionAuth, you might want to know various custom metrics, such as user login rates and successes. To do this, read the FusionAuth guide to [OpenTelemetry](./opentelemetry) and how to use it to create a bash script to collect any metric the FusionAuth API offers.

## Final System Architecture

If you combine the Prometheus, AlertManager, Grafana, Loki, and Ntfy infrastructure shown in this guide, your architecture will be as follows.
If you combine the Prometheus, Alertmanager, Grafana, Loki, and ntfy infrastructure shown in this guide, your architecture will be as follows.

<Diagram5></Diagram5>

Expand All @@ -527,4 +528,4 @@ If you combine the Prometheus, AlertManager, Grafana, Loki, and Ntfy infrastruct
- [Grafana](https://grafana.com/grafana)
- [Ubuntu Alertmanager image](https://hub.docker.com/r/ubuntu/alertmanager)
- [Ubuntu Grafana image](https://hub.docker.com/r/ubuntu/grafana)
- [Ubuntu Prometheus image](https://hub.docker.com/r/ubuntu/prometheus)
- [Ubuntu Prometheus image](https://hub.docker.com/r/ubuntu/prometheus)

0 comments on commit 56a8b86

Please sign in to comment.