From ebee3cf5032032619b43cedba81ffe5370cb0e3c Mon Sep 17 00:00:00 2001 From: Paolo Chila Date: Wed, 18 Dec 2024 18:32:34 +0100 Subject: [PATCH 1/2] Add small doc about running a standalone agent in otel mode --- .../otel-agent-transform.asciidoc | 76 +++++++++++++++++-- 1 file changed, 68 insertions(+), 8 deletions(-) diff --git a/docs/en/ingest-management/elastic-agent/otel-agent-transform.asciidoc b/docs/en/ingest-management/elastic-agent/otel-agent-transform.asciidoc index 1c70cda60..a33e9498a 100644 --- a/docs/en/ingest-management/elastic-agent/otel-agent-transform.asciidoc +++ b/docs/en/ingest-management/elastic-agent/otel-agent-transform.asciidoc @@ -3,18 +3,78 @@ preview::[] -If you have a currently installed {agent}, you change it to run as an <>. +If you have a currently installed standalone {agent}, it can be configured to run as an <>. -To transform {agent} to an OTel Collector: +This will allow to run Elastic Agent as a service running Otel Collector with a given configuration. -. Some instruction. +In order to configure an installed standalone {agent} to run as an OTel Collector it's enough to include a valid <> configuration in `elastic-agent.yml`. -. Some other instruction. Run the `elastic-agent -transform-otel` command: + +=== Example: configure {agent} to ingest host logs and metrics into Elasticsearch using Otel Collector + +Prerequisites: + +. A suitable API Key is available for authenticating on Elasticsearch +. An installed standalone {agent} +. A valid OTel Collector configuration, in this example we are going to use an included OTel sample configuration in `otel_samples/platformlogs_hostmetrics.yml` +link:https://github.com/elastic/elastic-agent/blob/main/internal/pkg/otel/samples/linux/platformlogs_hostmetrics.yml[Linux example] +link:https://github.com/elastic/elastic-agent/blob/main/internal/pkg/otel/samples/darwin/platformlogs_hostmetrics.yml[MacOS example] + +Steps: + +. Create a directory where Otel Collector can save its state (in this example we use `<{agent} install directory>/data/otelcol`) +. Open `<{agent} install directory>/otel_samples/platformlogs_hostmetrics.yml` with your favourite editor +. Set environment details to be used by OTel Collector +* [Option 1] Define environment variables for the {agent} service: +** `ELASTIC_ENDPOINT`: URL of Elasticsearch instance where data is going to be ingested +** `ELASTIC_API_KEY`: API Key for Elasticsearch authentication +** `STORAGE_DIR`: directory where Otel Collector can persist its state +* [Option 2] Replace the environment variable references in the sample configuration with the corresponding values +** `${env:ELASTIC_ENDPOINT}`: URL of Elasticsearch instance where data is going to be ingested +** `${env:ELASTIC_API_KEY}`: API Key for Elasticsearch authentication +** `${env:STORAGE_DIR}`: directory where Otel Collector can persist its state +. Save the opened OTel configuration as `elastic-agent.yml`, overwriting the default configuration of the installed agent +. Verify that the new configuration is correctly applied by agent running a `status` command: + [source,shell] ---- -elastic-agent -switch-me-to-otel-please +elastic-agent status ---- - - - +Otel Collector running configuration should appear under `elastic-agent` key (note the `extensions` and `pipeline` keys) ++ +[source,shell] +---- +┌─ fleet +│ └─ status: (STOPPED) Not enrolled into Fleet +└─ elastic-agent + ├─ status: (HEALTHY) Running + ├─ extensions + │ ├─ status: StatusOK + │ └─ extension:file_storage + │ └─ status: StatusOK + ├─ pipeline:logs/platformlogs + │ ├─ status: StatusOK + │ ├─ exporter:elasticsearch/otel + │ │ └─ status: StatusOK + │ ├─ processor:resourcedetection + │ │ └─ status: StatusOK + │ └─ receiver:filelog/platformlogs + │ └─ status: StatusOK + └─ pipeline:metrics/hostmetrics + ├─ status: StatusOK + ├─ exporter:elasticsearch/ecs + │ └─ status: StatusOK + ├─ processor:attributes/dataset + │ └─ status: StatusOK + ├─ processor:elasticinframetrics + │ └─ status: StatusOK + ├─ processor:resource/process + │ └─ status: StatusOK + ├─ processor:resourcedetection + │ └─ status: StatusOK + └─ receiver:hostmetrics/system + └─ status: StatusOK +---- ++ +. Congratulations! Host logs and metrics are now being collected and ingested by the {agent} service running an OTel Collector instance. +For further details about OpenTelemetry collector components supported by Elastic Agent, refer to link:https://github.com/elastic/elastic-agent/tree/main/internal/pkg/otel#components[Elastic Distribution for OpenTelemetry Collector README] \ No newline at end of file From ca5248e2bab6d8875b76e1d3770792ae6bc75ba9 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Wed, 18 Dec 2024 16:54:44 -0500 Subject: [PATCH 2/2] A few small edits --- .../otel-agent-transform.asciidoc | 53 ++++++++++--------- 1 file changed, 27 insertions(+), 26 deletions(-) diff --git a/docs/en/ingest-management/elastic-agent/otel-agent-transform.asciidoc b/docs/en/ingest-management/elastic-agent/otel-agent-transform.asciidoc index a33e9498a..fb1c90017 100644 --- a/docs/en/ingest-management/elastic-agent/otel-agent-transform.asciidoc +++ b/docs/en/ingest-management/elastic-agent/otel-agent-transform.asciidoc @@ -3,44 +3,45 @@ preview::[] -If you have a currently installed standalone {agent}, it can be configured to run as an <>. +If you have a currently installed standalone {agent}, it can be configured to run as an <>. This allows you to run {agent} both as a service and in an OTel Collector mode. -This will allow to run Elastic Agent as a service running Otel Collector with a given configuration. +In order to configure an installed standalone {agent} to run as an OTel Collector, it's enough to include a valid <> configuration in the `elastic-agent.yml` file, as shown in the following example. -In order to configure an installed standalone {agent} to run as an OTel Collector it's enough to include a valid <> configuration in `elastic-agent.yml`. +=== Example: configure {agent} to ingest host logs and metrics into Elasticsearch using the OTel Collector +**Prerequisites** -=== Example: configure {agent} to ingest host logs and metrics into Elasticsearch using Otel Collector +You'll need the following: -Prerequisites: - -. A suitable API Key is available for authenticating on Elasticsearch +. A suitable <> for authenticating on Elasticsearch . An installed standalone {agent} -. A valid OTel Collector configuration, in this example we are going to use an included OTel sample configuration in `otel_samples/platformlogs_hostmetrics.yml` -link:https://github.com/elastic/elastic-agent/blob/main/internal/pkg/otel/samples/linux/platformlogs_hostmetrics.yml[Linux example] -link:https://github.com/elastic/elastic-agent/blob/main/internal/pkg/otel/samples/darwin/platformlogs_hostmetrics.yml[MacOS example] +. A valid OTel Collector configuration. In this example we'll use the OTel sample configuration included in the {agent} repository: `otel_samples/platformlogs_hostmetrics.yml`. +** link:https://github.com/elastic/elastic-agent/blob/main/internal/pkg/otel/samples/linux/platformlogs_hostmetrics.yml[Linux version] +** link:https://github.com/elastic/elastic-agent/blob/main/internal/pkg/otel/samples/darwin/platformlogs_hostmetrics.yml[MacOS version] + +**Steps** -Steps: +To change a running standalone {agent} to run as an OTel Collector: -. Create a directory where Otel Collector can save its state (in this example we use `<{agent} install directory>/data/otelcol`) -. Open `<{agent} install directory>/otel_samples/platformlogs_hostmetrics.yml` with your favourite editor -. Set environment details to be used by OTel Collector -* [Option 1] Define environment variables for the {agent} service: -** `ELASTIC_ENDPOINT`: URL of Elasticsearch instance where data is going to be ingested -** `ELASTIC_API_KEY`: API Key for Elasticsearch authentication -** `STORAGE_DIR`: directory where Otel Collector can persist its state -* [Option 2] Replace the environment variable references in the sample configuration with the corresponding values -** `${env:ELASTIC_ENDPOINT}`: URL of Elasticsearch instance where data is going to be ingested -** `${env:ELASTIC_API_KEY}`: API Key for Elasticsearch authentication -** `${env:STORAGE_DIR}`: directory where Otel Collector can persist its state -. Save the opened OTel configuration as `elastic-agent.yml`, overwriting the default configuration of the installed agent -. Verify that the new configuration is correctly applied by agent running a `status` command: +. Create a directory where the OTel Collector can save its state. In this example we use `<{agent} install directory>/data/otelcol`. +. Open the `<{agent} install directory>/otel_samples/platformlogs_hostmetrics.yml` file for editing. +. Set environment details to be used by OTel Collector: +* **Option 1:** Define environment variables for the {agent} service: +** `ELASTIC_ENDPOINT`: The URL of the {es} instance where data will be sent +** `ELASTIC_API_KEY`: The API Key to use to authenticate with {es} +** `STORAGE_DIR`: The directory where the OTel Collector can persist its state +* **Option 2:** Replace the environment variable references in the sample configuration with the corresponding values: +** `${env:ELASTIC_ENDPOINT}`:The URL of the {es} instance where data will be sent +** `${env:ELASTIC_API_KEY}`: The API Key to use to authenticate with {es} +** `${env:STORAGE_DIR}`: The directory where the OTel Collector can persist its state +. Save the opened OTel configuration as `elastic-agent.yml`, overwriting the default configuration of the installed agent. +. Run the `elastic-agent status` command to verify that the new configuration has been correctly applied: + [source,shell] ---- elastic-agent status ---- -Otel Collector running configuration should appear under `elastic-agent` key (note the `extensions` and `pipeline` keys) +The OTel Collector running configuration should appear under `elastic-agent` key (note the `extensions` and `pipeline` keys): + [source,shell] ---- @@ -77,4 +78,4 @@ Otel Collector running configuration should appear under `elastic-agent` key (no ---- + . Congratulations! Host logs and metrics are now being collected and ingested by the {agent} service running an OTel Collector instance. -For further details about OpenTelemetry collector components supported by Elastic Agent, refer to link:https://github.com/elastic/elastic-agent/tree/main/internal/pkg/otel#components[Elastic Distribution for OpenTelemetry Collector README] \ No newline at end of file +For further details about OpenTelemetry collector components supported by {agent}, refer to the link:https://github.com/elastic/elastic-agent/tree/main/internal/pkg/otel#components[Elastic Distribution for OpenTelemetry Collector README]. \ No newline at end of file