From cc9515324c7dff8c86b610f42ede6b77050e78c5 Mon Sep 17 00:00:00 2001 From: worktheclock <85885287+worktheclock@users.noreply.github.com> Date: Thu, 25 Jul 2024 12:20:18 +0200 Subject: [PATCH] Elastic language edit --- .../operate/secure-and-monitor/elastic.mdx | 172 ++++++++++-------- 1 file changed, 92 insertions(+), 80 deletions(-) diff --git a/astro/src/content/docs/operate/secure-and-monitor/elastic.mdx b/astro/src/content/docs/operate/secure-and-monitor/elastic.mdx index a412560c57..fce5cefc37 100644 --- a/astro/src/content/docs/operate/secure-and-monitor/elastic.mdx +++ b/astro/src/content/docs/operate/secure-and-monitor/elastic.mdx @@ -1,6 +1,6 @@ --- title: Monitor With Elastic -description: Learn How To Monitor FusionAuth With Elastic. +description: Learn how to monitor FusionAuth with Elastic. navcategory: admin section: operate subcategory: secure and monitor @@ -21,82 +21,85 @@ import Diagram6 from 'src/components/docs/operate/secure-and-monitor/elasticDiag ## Introduction -[Elastic](https://www.elastic.co) is a company known for managing the ELK stack: Elasticsearch, Logstash, and Kibana. These products allow you to import (ingest) data, including observability data like logs and server metrics, organize them, and monitor them. +[Elastic](https://www.elastic.co) is a company known for managing the ELK stack: Elasticsearch, Logstash, and Kibana. These products allow you to import (ingest) data, including observability data like logs and server metrics, organize it, and monitor it. -This guide explains how to connect FusionAuth to Elastic in various ways, as well as which FusionAuth metrics are useful in Elastic, and how to create a simple dashboard to show them. +This guide explains how to connect FusionAuth to Elastic in various ways, identifies FusionAuth metrics that are useful in Elastic, and shows how to create a simple dashboard to show them. Before continuing with this guide, please go through the [FusionAuth guide to monitoring](/docs/operate/secure-and-monitor/monitor) to get an overview of the available metrics and choose the ones relevant to your needs. -To learn more about the Elastic Observability platform, refer to the documentation [here](https://www.elastic.co/guide/en/observability/current/index.html). Pricing of the platform is available [here](https://www.elastic.co/pricing). The Standard (lowest) pricing tier includes sufficient observability features, including handling logs and metrics. +Refer to the [Elastic documentation](https://www.elastic.co/guide/en/observability/current/index.html) to learn more about the Elastic Observability platform, and compare Elastic subscriptions on the [pricing page](https://www.elastic.co/pricing). The Standard (lowest) pricing tier includes sufficient observability features, including handling logs and metrics. -## Cloud-Hosted Or Self-Hosted? +Elastic can import and visualize different types of data, including server performance, application (FusionAuth) performance, and logs. -Before 2021, Elastic products were open source software. In 2021, Elastic changed their licenses to [Elastic License 2.0](https://www.elastic.co/licensing/elastic-license/faq) to prevent cloud hosts (AWS, Google, Azure) from freely selling managed Elastic services that compete with [Elastic Cloud](https://www.elastic.co/cloud/shared-responsibility). +As well as monitoring FusionAuth with Elastic, you can use Elastic to monitor your custom application by following steps similar to the ones in this guide. -While not officially open source, the Elastic License 2.0 still allows free and open (gratis and libre) use of most Elastic products, as long as you are not selling them as your own service. So you can host the Elastic stack on your own server instead of buying Elastic Cloud. However, using [certain premium features](https://www.elastic.co/subscriptions), like AI, will still need payment to Elastic. +## Cloud-Hosted Or Self-Hosted? -This guide assumes you are using the paid Elastic Cloud for monitoring. If you are instead self-hosting Elastic, you should still be able to follow the guide, but you will need to change connections that point to Elastic Cloud to point to your own server. To install Elastic yourself in Docker, follow the first half of this [guide](https://www.elastic.co/blog/getting-started-with-the-elastic-stack-and-docker-compose), up to before MetricBeat. +Before 2021, Elastic products were open-source software. In 2021, Elastic changed their licenses to [Elastic License 2.0](https://www.elastic.co/licensing/elastic-license/faq) to prevent cloud hosts (AWS, Google, Azure) from freely selling managed Elastic services that compete with [Elastic Cloud](https://www.elastic.co/cloud/shared-responsibility). -When reading Elastic documentation, such as their [getting started guide](https://www.elastic.co/guide/en/observability/current/logs-metrics-get-started.html), which tells you "you need an Elastic Stack deployment", remember that this does not mean you need to deploy the entire ELK stack locally alongside your application or FusionAuth instance. Their guide means that you need to have the Elastic Stack installed *somewhere*, which is probably going to be on their Elastic Cloud. +While not officially open source, the Elastic License 2.0 still allows free and open (gratis and libre) use of most Elastic products, as long as you are not selling them as your own service. So you can host the Elastic stack on your own server instead of buying Elastic Cloud, but you will still need to pay Elastic to use [certain premium features](https://www.elastic.co/subscriptions), like AI. -## Too Many Elastic Possibilities +This guide assumes you are using the paid Elastic Cloud for monitoring. If you are self-hosting Elastic, you should still be able to follow the guide, but you will need to change connections that point to Elastic Cloud to point to your own server. To install Elastic yourself in Docker, follow the first half of this [guide](https://www.elastic.co/blog/getting-started-with-the-elastic-stack-and-docker-compose) until you reach the MetricBeat section. -[Elastic Agent](https://www.elastic.co/guide/en/fleet/current/fleet-overview.html) sends data from your server to the Elastic Cloud for processing and monitoring. To use the Cloud, you first have to install Elastic Agent on your server. +In the Elastic documentation (for example, the [getting started guide](https://www.elastic.co/guide/en/observability/current/logs-metrics-get-started.html)), you may read that "you need an Elastic Stack deployment". Note that this does not mean you need to deploy the entire ELK stack locally alongside your application or FusionAuth instance. It simply means that you need to have the Elastic Stack installed somewhere, which will likely be on their Elastic Cloud. -There are various ways of doing so: -- Put the Elastic Agent in the same Docker container as FusionAuth. -- Put the Elastic Agent in a separate container and give it access to the private Docker management data for other containers. -- Install a local [OpenTelemetry](https://opentelemetry.io/) agent that sends data to Elastic, instead of using the Elastic Agent. -- Call the Elastic API directly from a local service, instead of using the Elastic Agent. +## Create An Elastic Account -Elastic can import and visualize different types of data, including server performance, application (FusionAuth) performance, and logs. +First, register for an Elastic account: -As well as monitoring FusionAuth with Elastic, you can use Elastic to monitor your custom application too, by following steps similar to the ones in this guide. +- Register for a trial at https://cloud.elastic.co/registration. +- Verify your email address using the link in the email Elastic sends you. +- Enter your details on the registration page and create a new deployment called `fa`. Note that your two-week free trial starts when you create the deployment. -These are an overwhelming number of ways to use Elastic. But by the end of this guide you should be able to pick the simplest combination that suits the needs of your system architecture. +## How To Send Data To Elastic Cloud -## Create An Elastic Account +There are many ways to use Elastic. This guide will help you choose the simplest combination that suits the needs of your system architecture. -First, register for an Elastic account: +[Elastic Agent](https://www.elastic.co/guide/en/fleet/current/fleet-overview.html) sends data from your server to Elastic Cloud for processing and monitoring. To use Elastic Cloud, you must first install Elastic Agent on your server. -- Register for a trial at https://cloud.elastic.co/registration. -- Verify your email address with the link in the email Elastic sends you. -- Enter your details in the registration page and create a new deployment called `fa`. Note that your two-week free trial starts when you create the deployment. +The options for sending data to Elastic Cloud include: + +- Install Elastic Agent in the same Docker container as FusionAuth. +- Install Elastic Agent in a separate container and give it access to the private Docker management data for other containers. +- Install a local [OpenTelemetry](https://opentelemetry.io/) agent to send data to Elastic instead of using Elastic Agent. +- Call the Elastic API directly from a local service instead of using Elastic Agent. -## Monitor FusionAuth With Elastic In The Same Container Or Without Docker +## Monitor FusionAuth With Elastic In The Same Container Running FusionAuth and PostgreSQL in Docker usually looks like the diagram below (you might also run OpenSearch in another Docker container). -In this section you are going to install Elastic Agent in the same container as FusionAuth. The aim is to have a design like the diagram below. +In this section, you will install Elastic Agent in the same container as FusionAuth. The aim is to have a design like the diagram below. -FusionAuth and PostgreSQL are unchanged. The Elastic Agent in the FusionAuth container monitors the container (not the FusionAuth app), and uploads the data to Elasticsearch, where it is indexed and saved. +FusionAuth and PostgreSQL are unchanged. Elastic Agent in the FusionAuth container monitors the container (not the FusionAuth app) and uploads the data to Elasticsearch, where it is indexed and saved. -Kibana is a web interface that provides dashboards of the Agent's container's health. Note that Elastic Agent is not yet monitoring FusionAuth in this design, but if the container dies or suffers CPU or RAM overuse, that will show in the Kibana dashboard. +Kibana is a web interface that provides dashboards of the health of the Elastic Agent's container. Note that Elastic Agent is not yet monitoring FusionAuth in this design, but if the container dies or suffers CPU or RAM overuse, that will show in the Kibana dashboard. -Fleet is the web app that tracks all Agents on all servers. It allows you to set Agent Policies that tell the Agents remotely what data they should and shouldn't upload. You can also use Fleet to delete or edit existing Agents. +Fleet is the web app that tracks all Elastic Agents on all servers. It allows you to set Agent Policies that tell the agents remotely what data they should and shouldn't upload. You can also use Fleet to delete or edit existing agents. -This design is useful when you don't have full control over the Docker environment, as you may not have on some cloud hosts. However, the design violates the principle of one process per Docker container. This means it is possible for the Elastic Agent process to die, while the container and FusionAuth process keep running, which will cause confusion when viewing the dashboard. +This design is useful when you don't have full control over the Docker environment, as you may not have on some cloud hosts. However, the design violates the principle of one process per Docker container. This means the Elastic Agent process can die while the container and FusionAuth process keep running, which will cause confusion when viewing the dashboard. -Running Elastic Agent on the same server as FusionAuth is also useful when you aren't using Docker — when you are running FusionAuth directly on your server. In this case, your system will look like the diagram below. +Running Elastic Agent on the same server as FusionAuth is also useful when you aren't using Docker (that is, running FusionAuth directly on your server). In this case, your system will look like the diagram below. -Here's how to implement this design: -- Browse to the home page of your Elastic Cloud web interface. -- In the sidebar, at the bottom, click Management -> Fleet. +Here's how to implement this design. + +Navigate to the home page of your Elastic Cloud web interface and do the following: +- At the bottom of the sidebar, click Management -> Fleet. - Select the Agent policies tab, then click Create agent policy. -- Give it the name `efa`. Review the advanced options if you like, but leave their defaults unchanged. +- Name the policy `efa`. Review the advanced options if you like, but leave their defaults unchanged. - Click Create agent policy. - Back on the agent policy list page, click the name of your new policy, "efa". - Under the Integrations tab, the "System" integration should show. Click the name of the integration to open a page to edit it. If you don't see the system integration, click Add integration to the right, search for `System`, and add it. Add System integration in Elastic + - Disable Collect events from the Windows event log and leave the system collection and metric collection toggles enabled. Since your Docker instance runs Ubuntu there is no need for Windows events. -- Click on Save integration bottom right. -- Return to the Agents tab of the Fleet page. And click Add Agent. +- Click on Save integration at the bottom right. +- Return to the Agents tab of the Fleet page and click Add Agent. - Choose your newly created agent policy in the dropdown that appears for step 1. - Copy the text to install the "Linux Tar" version of the agent in step 3, which will be similar to the commands below, but with different parameters. ```sh @@ -105,10 +108,10 @@ Here's how to implement this design: cd elastic-agent-8.14.2-linux-x86_64 sudo ./elastic-agent install --url=https://9025e4274.fleet.us-central1.gcp.cloud.es.io:443 --enrollment-token=a2lB5aA1dUagaTk5FUsE743dw== ``` -- The last line is the most important. Note the subdomain of the URL, which is your unique Elastic Cloud home. Note also the enrollment token, which is your secret key to use Elastic. Keep the token secure and safe, and never commit it to GitHub where it can be publicly exposed. - +- From the last line of the command sequence, note the subdomain of the URL, which is your unique Elastic Cloud home, and the enrollment token, which is your secret key to use Elastic. Keep the token secure and safe, and never commit it to GitHub where it can be publicly exposed. -- Now you are going to leave the Elastic site and work in a terminal on your own computer. Install [Docker](https://docs.docker.com/get-docker/) if you don't have it on your machine. +Now leave the Elastic site and complete the following steps in a terminal on your computer: +- Install [Docker](https://docs.docker.com/get-docker/) if you don't have it on your machine. - Save the Dockerfile from the [FusionAuth containers repository](https://github.com/FusionAuth/fusionauth-containers/blob/master/docker/fusionauth/fusionauth-app/Dockerfile) to your computer. - Rename the file to `elastic.dockerfile`. - Edit the file and insert the following lines above the comment "###### Start FusionAuth App". @@ -186,7 +189,7 @@ Here's how to implement this design: db_data: fusionauth_config: ``` -- Also create a `.env` file with the default content below. +- Create a `.env` file with the default content below. ```text DATABASE_USERNAME=fusionauth DATABASE_PASSWORD=hkaLBM3RVnyYeYeqE3WI1w2e4Avpy0Wd5O3s3 @@ -197,43 +200,48 @@ Here's how to implement this design: POSTGRES_PASSWORD=postgres ``` - Start FusionAuth with `docker compose up`. -- When you return to the Elastic web interface it should now say "Agent enrollment confirmed". -- You should also see the agent appear in the Fleet dashboard and the status should change to `Healthy`. + +Return to the Elastic web interface, and you should see the message "Agent enrollment confirmed". + +You should see the agent in the Fleet dashboard with the status showing `Healthy`. + Fleet Agent overview in Elastic -- In the sidebar, browse to Analytics -> Dashboards. Search for and display `Host overview`. + +In the sidebar, browse to Analytics -> Dashboards. Search for and display `Host overview`. + Host overview dashboard in Elastic -You have successfully installed a monitoring agent for your FusionAuth machine and can now monitor it online. +The monitoring agent for your FusionAuth machine is now installed and you can now monitor it online. ### How To Debug The Container If you get errors when running FusionAuth, or the Agent does not appear in Fleet, you will need to open a terminal in the container to find the problem. -Change the last line of the Dockerfile to the line below, to stop FusionAuth and the Agent starting, so you can test manually. +To stop FusionAuth and the Agent starting so you can test manually, change the last line of the Dockerfile to the line below. ```sh CMD tail -f /dev/null ``` -To rebuild the Dockerfile after changes, run the command below. +Rebuild the Dockerfile after changes. ```sh docker rm fa; docker rm efa; docker rmi efaimage; docker build --platform linux/amd64 -t efaimage -f elastic.dockerfile . ``` -To run the container and enter it as root, use the command below. +Run the container and enter it as root. ```sh docker run -it --user root --name efa efaimage bash ``` -Start the Agent with `/opt/Elastic/Agent/elastic-agent run` and use `cat /var/log/elastic-agent.err` to look for errors. If you see the agent trying to connect to `127.0.0.1:9200`, it is not because the URL is configured incorrectly. The 127 address is just a fallback when the Agent cannot connect to the server, usually due to it not running as root. +Start the agent with `/opt/Elastic/Agent/elastic-agent run` and use `cat /var/log/elastic-agent.err` to look for errors. If you see the agent trying to connect to `127.0.0.1:9200`, it is not because the URL is configured incorrectly. The 127 address is just a fallback when the agent cannot connect to the server, usually due to it not running as root. ## Monitor FusionAuth With Elastic In Another Container -Running Elastic Agent and FusionAuth in the same container required altering the FusionAuth Dockerfile, which means you have to alter the file and rebuild the image every time you want to use a new version of FusionAuth. +Running Elastic Agent and FusionAuth in the same container requires altering the FusionAuth Dockerfile, which means you have to edit the file and rebuild the image every time you want to use a new version of FusionAuth. -If you run Elastic Agent in a separate container you avoid this problem, but you need to have full control over the machine running Docker in order to give Elastic Agent access to restricted Docker data (the Docker socket). This may not be possible on some cloud hosts. +Running Elastic Agent in a separate container will avoid this problem, but you need to have full control over the machine running Docker to give Elastic Agent access to restricted Docker data (the Docker socket). This may not be possible on some cloud hosts. Running Elastic Agent in a separate container to FusionAuth is shown in the diagram below. @@ -252,7 +260,7 @@ Below are the instructions to implement this design. They follow from the work y fa: image: fusionauth/fusionauth-app:latest ``` - - Note that you could continue using the `efaimage` image if you wanted to. Elastic allows you to run as many agents as you want, and will collect data from all of them. + Note that you could continue using the `efaimage` image if you wanted to. Elastic allows you to run as many agents as you want, and will collect data from all of them. - Add a new service to the compose file for the Elastic Agent image. Replace your URL and enrollment token in the markup below with the values from the Elastic web page. ```yaml faelastic: @@ -271,32 +279,35 @@ Below are the instructions to implement this design. They follow from the work y ``` - `/var/run/docker.sock` allows Elastic Agent to view Docker metrics about other containers. It will work on Linux and macOS hosts, but might not work on Windows. If you encounter socket problems on Windows, please read [this article](https://tomgregory.com/aws/running-docker-in-docker-on-windows/) and try prefixing your local socket path with an extra slash. - `/var/lib/docker/containers` allows Elastic Agent to read the logs of other containers. -- Run the Elastic Agent container first, so that it will see some log information from other containers when they start. First run `docker compose up faelastic`. +- Run the Elastic Agent container first, so that it will see some log information from other containers when they start. Run `docker compose up faelastic`. - In the Elastic Cloud web interface, browse to the sidebar and then Observability -> Logs -> Stream. - Run the other Docker containers with `docker compose up db fa` - You should be able to see the FusionAuth log output appearing in the Elastic web interface. Log stream from Docker containers in Elastic -- Browse to the sidebar and then Analytics -> Dashboards -> [Metrics Docker]Overview. You should be able to see details about all your Docker containers now. + +- Browse to the sidebar and then Analytics -> Dashboards -> [Metrics Docker]Overview. You should see details about all your Docker containers. Docker metrics overview in Elastic -## Monitor the FusionAuth Application Directly With Elastic +## Monitor The FusionAuth Application Directly With Elastic + +You've learned how to monitor the FusionAuth container in two different ways. Since a Docker container will exit if the process it runs exits, this monitoring will tell you if FusionAuth dies, which might be enough for you to know. But if you want to monitor details about the FusionAuth app itself, there are various approaches: -You've learnt how to monitor the FusionAuth container now in two different ways. Since a Docker container will exit if the process it runs exits, this monitoring will tell you if FusionAuth dies, which might be enough for you to know. But if you want to monitor details about the FusionAuth app itself, there are various approaches: - Use an [Elastic APM (application performance monitoring) app for Java](https://www.elastic.co/guide/en/observability/current/_step_3_install_apm_agents.html) inside the FusionAuth container to monitor the Java Virtual Machine. Since monitoring Java won't give you any useful information that monitoring the container doesn't already, you can ignore this idea. There is a guide to running FusionAuth with a Java monitoring app by altering the Dockerfile in the [FusionAuth Splunk guide](./splunk) if you would like to try. -- Use an [OpenTelemetry Java monitoring agent](https://github.com/open-telemetry/opentelemetry-java-instrumentation) to send information about FusionAuth to Elastic. This is similar to previous point - OpenTelemetry won't provide any useful information that container monitoring doesn't already. Using OpenTelemetry is also shown in the [FusionAuth Splunk guide](./splunk). Elastic has a guide to Java [here](https://www.elastic.co/guide/en/observability/8.14/apm-open-telemetry.html) that you can use in conjunction with the Splunk guide. -- Use a specialized Elastic service, like Heartbeat, to connect to the FusionAuth HTTP server and see if it returns correctly. You can install Heartbeat in a container, like we did for Elastic Agent in the previous section, and point it to the FusionAuth URL. A guide on Heartbeat is [here](https://www.elastic.co/guide/en/beats/heartbeat/current/heartbeat-overview.html). -- Send custom metrics from FusionAuth to Elastic. This is the most complex option, as you need to write a custom service to request FusionAuth metrics, extract them from the given zip file, and upload them to Elastic through their API. However, this is the only way to get precise information about FusionAuth if you want that level of detail. This option will be discussed in the next section. +- Use an [OpenTelemetry Java monitoring agent](https://github.com/open-telemetry/opentelemetry-java-instrumentation) to send information about FusionAuth to Elastic. This is similar to the previous point - OpenTelemetry won't provide any useful information that container monitoring doesn't already. Using OpenTelemetry is also shown in the [FusionAuth Splunk guide](./splunk). Elastic has a guide to Java [here](https://www.elastic.co/guide/en/observability/8.14/apm-open-telemetry.html) that you can use with the Splunk guide. +- Use a specialized Elastic service like Heartbeat to connect to the FusionAuth HTTP server and see if it returns correctly. You can install Heartbeat in a container like we did for Elastic Agent in the previous section, and point it to the FusionAuth URL. A guide on Heartbeat is [here](https://www.elastic.co/guide/en/beats/heartbeat/current/heartbeat-overview.html). +- Send custom metrics from FusionAuth to Elastic. This is the most complex option, as you need to write a custom service to request FusionAuth metrics, extract them from the given zip file, and upload them to Elastic using the Elastic API. However, this is the only way to get precise information about FusionAuth if you want that level of detail. This option will be discussed in the next section. ## Send Custom FusionAuth Metrics To The Elasticsearch API There are three possible ways to send custom data to Elastic: + - Write your own service to run continuously, get data, and send it by calling the Elasticsearch API. -- Build a custom private module for [MetricBeat](https://www.elastic.co/beats/metricbeat) in Go, compile the module with MetricBeat, and deploy the output to a container. MetricBeat is similar to Elastic Agent. It is an Elastic service that runs continuously, calling whatever integration it's configured with. To write a custom module (integration), follow the [Elastic developer documentation](https://www.elastic.co/guide/en/beats/devguide/current/metricbeat-dev-overview.html). -- Deploy MetricBeat with the stock [HTTP module](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-http.html) in a container. The HTTP module can call any custom API you want. Write a little web service in another container as an interface between MetricBeat and FusionAuth. This service will get whatever metrics you want from FusionAuth and return them in the HTTP response. +- Build a custom private module in Go for [MetricBeat](https://www.elastic.co/beats/metricbeat), compile the module with MetricBeat, and deploy the output to a container. MetricBeat is similar to Elastic Agent. It is an Elastic service that runs continuously, calling whatever integration it's configured with. To write a custom module (integration), follow the [Elastic developer documentation](https://www.elastic.co/guide/en/beats/devguide/current/metricbeat-dev-overview.html). +- Deploy MetricBeat in a container with the stock [HTTP module](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-http.html). The HTTP module can call any custom API. Write a web service in another container as an interface between MetricBeat and FusionAuth. This service will get whatever metrics you want from FusionAuth and return them in the HTTP response. -Each option has disadvantages. While calling the API directly is the least code, it is low level work that moves away from using the Elastic components as intended. And it requires you to configure values to group your custom data with other data from Elastic Agent in the dashboard. Using MetricBeat is Elastic's recommended solution, but requires writing and compiling Go code for every service change, or running an additional Docker container with its own web service. +Each option has disadvantages. While calling the API directly has the least code, it is low-level work that moves away from using the Elastic components as intended and requires you to configure values to group your custom data with other data from Elastic Agent in the dashboard. Using MetricBeat is Elastic's recommended solution, but it means writing and compiling Go code for every service change or running an additional Docker container with its own web service. -For simplicity’s sake you will use option one in this guide. In this section you will create a custom service in Docker to call the FusionAuth API and send the metrics to Elastic API directly. +This guide will show you how to send FusionAuth metrics to the Elasticsearch API by creating a custom Docker service that calls the FusionAuth API and sends metrics directly to the Elastic API. The design looks like the diagram below. @@ -314,19 +325,19 @@ You can add any other metrics you want to this service. ### Create An Elastic API Key -Earlier you used an enrollment token tied to a specific Agent Policy to configure your Elastic Agent. Now to call the Elasticsearch API directly you need an API key. +Earlier you used an enrollment token tied to a specific Agent Policy to configure Elastic Agent. Now, to call the Elasticsearch API directly, you need an API key. - In the Elastic Cloud web interface, open the sidebar and browse to Search -> Elasticsearch. - Click Endpoints & API keys. -- Note your endpoint and cloud ID values. +- Note your endpoint and cloud Id values. - Click New API key. - Give it a name and enable "Security Privileges" and "Metadata". - Click Create API Key. -- Save the key securely for later use. Never to commit the key to git, which can expose it on the Internet. +- Securely save the key to use later. Never commit the key to Git, which can expose it on the internet. ### Create An Elasticsearch Index -To prepare Elasticsearch for data upload, open a terminal and run the three commands below, after setting your API key and URL for the first two commands. The API key is the value of the `encoded` field from the JSON in Elastic, not the `api_key` field. +To prepare Elasticsearch for data upload, open a terminal and run the three commands below, using your API key and URL in the first two commands. The API key is the value of the `encoded` field from the JSON in Elastic, not the `api_key` field. ```sh export elasticKey="WGdLQ==" @@ -345,7 +356,7 @@ curl -X PUT "${elasticUrl}/falogins" -H "Authorization: ApiKey ${elasticKey}" -H # {"acknowledged":true,"shards_acknowledged":true,"index":"falogins"} ``` -These commands should have created an index (think of an index as a database table) to which you can upload FusionAuth metrics. If you receive an authentication or 404 error please check your URL and API key settings. +These commands create an index (think of an index as a database table) to which you can upload FusionAuth metrics. If you receive an authentication or 404 error, please check your URL and API key settings. Check that the index exists by running the command below. @@ -358,9 +369,10 @@ curl "${elasticUrl}/falogins" -H "Authorization: ApiKey "${elasticKey}"" -H "Con Let's review the index in Elastic Cloud. -- Browse to the web interface, open the sidebar, and browse to Management -> Stack Management -> Index Management. +- In the web interface, open the sidebar, and browse to Management -> Stack Management -> Index Management. Index management in Elastic -- You should see falogins" in the list. Click it. + +- You should see "falogins" in the list. Click it. - Click the Mappings tab. Index mappings in Elastic @@ -368,13 +380,13 @@ You can now see the type of data that the index will accept. You can create inde ### Write A Custom Service To Send Data To The API -Let's get the login records every ten seconds and send them to Elastic. All the FusionAuth APIs that give you event data are documented [here](/docs/apis). The login records API is documented [here](/docs/apis/login#request-6). Note that the documentation says the date format is the standard Java type, but some constants like `ISO_LOCAL_DATE_TIME` are not supported. You need to enter the format string you want manually. +Let's get the login records every ten seconds and send them to Elastic. All the FusionAuth APIs that give you event data are documented [here](/docs/apis). The login records API is documented [here](/docs/apis/login#request-6). Note that, while the documentation says the date format is the standard Java type, some constants like `ISO_LOCAL_DATE_TIME` are not supported. You need to enter the format string you want manually. -Unfortunately, all the APIs export events as zip files — you will not get JSON or YAML data in memory. So you will need to create a script that gets the zip file, extracts it, reads it, formats the entries for Elastic, and uploads them. +Unfortunately, all the APIs export events as zip files — you will not get JSON or YAML data in memory. You will need to create a script that gets the zip file, extracts it, reads it, formats the entries for Elastic, and uploads them. Browse to FusionAuth, which is at http://localhost:9011 if you are running through the default Docker setup. Log in and look for your application Id in System -> Login Records. -Next, create an API key by navigating to Settings -> API Keys and clicking the button. Enter a Description for the API key and click on the button to save the API key. On the API Keys list page, click the red lock next to the newly generated key to reveal the key value and copy and save it. +Next, create an API key by navigating to Settings -> API Keys and clicking the button. Enter a Description for the API key and click on the button to save the API key. On the API Keys list page, click the red lock next to the newly generated key to reveal the key value. Copy and save the key. Create a file called `app.sh`. Insert the content below, replacing your FusionAuth API key, FusionAuth application Id, and your Elastic API key and Elastic URL. @@ -434,7 +446,7 @@ The file returned from FusionAuth unzips to `login_records.csv`, which looks lik The records in this file look different from those in the FusionAuth console. Only Ids are given here, not email addresses or application names. -The second half of the script reads in the CSV file, discards the header, and sends the user ID and the time of each login to Elastic. +The second half of the script reads in the CSV file, discards the header, and sends the user Id and the time of each login to Elastic. Create a file called `metricDockerfile`. Insert the content below. @@ -497,8 +509,8 @@ The final step to use your metrics is to create a dashboard to view them in Kiba - In the Elastic Cloud web interface, open the sidebar and browse to Search -> Elasticsearch -> Indices -> falogins -> Documents. You should see data in the table. If not, please run `app.sh` manually and debug where it is failing, paying attention to all API keys and URLs. Login rate data from the FusionAuth -- In the sidebar browse to Analytics -> Dashboards -> Create dashboard -> Create visualization. -- In the index combobox in the top left, change from `logs-*` to `falogins`. +- In the sidebar, browse to Analytics -> Dashboards -> Create dashboard -> Create visualization. +- In the index combo box in the top left, change from `logs-*` to `falogins`. - Click the + button next to "timestamp" to add it to the dashboard. - Change the time in the top right to "Today". - Change the visualization type to "Bar vertical". @@ -507,7 +519,7 @@ The final step to use your metrics is to create a dashboard to view them in Kiba - Click Save. - Give the dashboard a name like `FusionAuth login rate` and save. -You now have a dashboard to monitor the health of FusionAuth. Following the process in this section, you can extract any metrics you want from the FusionAuth API, create an index for them in Elastic, upload the metrics in a Docker container, and create a dashboard for them in Kibana. +You now have a dashboard to monitor the health of FusionAuth. Following the process in this section, you can extract any metrics from the FusionAuth API, create an index for them in Elastic, upload the metrics in a Docker container, and create a dashboard for them in Kibana. ## Final System Architecture @@ -515,14 +527,14 @@ A relatively simple but adequate monitoring architecture with Elastic might look -In this design, the Elastic Agent monitors all Docker infrastructure and the FusionAuth logs, while the custom metric service provides fine-grained FusionAuth data to Elastic to monitor the app itself. +In this design, Elastic Agent monitors all Docker infrastructure and the FusionAuth logs, while the custom metric service provides fine-grained FusionAuth data to Elastic to monitor the app itself. ## Next Steps -Now that you are able to monitor FusionAuth in Elastic, you should enable Elastic [alerts](https://www.elastic.co/kibana/alerting) to notify you by email or in Slack if something goes wrong, like a massive decrease in login rates, a Docker container restarting, or a log output containing `error`. +Now that you can monitor FusionAuth in Elastic, you should enable Elastic [alerts](https://www.elastic.co/kibana/alerting) to notify you by email or in Slack if something goes wrong, like a massive decrease in login rates, a Docker container restarting, or a log output containing `error`. ## Further Reading - [FusionAuth metrics](/docs/operate/secure-and-monitor/monitor#metrics) - [Getting started with Docker and Elastic](https://www.elastic.co/blog/getting-started-with-the-elastic-stack-and-docker-compose) -- [Elasticsearch REST API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html) \ No newline at end of file +- [Elasticsearch REST API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html)