[POC] Nginx otel integration with OTEL Templates

[mrodm]

Proposed commit message

PoC creating a content package for Nginx and using OTEL configuration and templates to ingest metrics and logs into Elaticsearch.

The collector distribution used to ingest data is built from this manifest distributions/elastic-components/manifest.yaml based on the contents from this PR:
elastic/opentelemetry-collector-components#96

diff --git distributions/elastic-components/manifest.yaml distributions/elastic-components/manifest.yaml
index afd9721..ac25fa6 100644
--- distributions/elastic-components/manifest.yaml
+++ distributions/elastic-components/manifest.yaml
@@ -9,6 +9,7 @@ dist:
 extensions:
   - gomod: github.com/elastic/opentelemetry-collector-components/extension/configintegrationextension v0.0.0
   - gomod: github.com/elastic/opentelemetry-collector-components/extension/fileintegrationextension v0.0.0
+  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage v0.110.0
 
 connectors:
   - gomod: github.com/elastic/opentelemetry-collector-components/connector/spanmetricsconnectorv2 v0.0.0
@@ -18,16 +19,21 @@ converters:
 receivers:
   - gomod: github.com/elastic/opentelemetry-collector-components/receiver/integrationreceiver v0.0.0
   - gomod: go.opentelemetry.io/collector/receiver/nopreceiver v0.110.0
+  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver v0.110.0
+  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/nginxreceiver v0.110.0
 
 processors:
   - gomod: github.com/elastic/opentelemetry-collector-components/processor/elasticinframetricsprocessor v0.0.0
   - gomod: github.com/elastic/opentelemetry-collector-components/processor/elastictraceprocessor v0.0.0
   - gomod: github.com/elastic/opentelemetry-collector-components/processor/lsmintervalprocessor v0.0.0
   - gomod: github.com/elastic/opentelemetry-collector-components/processor/integrationprocessor v0.0.0
+  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor v0.110.0
+  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor v0.110.0
 
 exporters:
   - gomod: go.opentelemetry.io/collector/exporter/debugexporter v0.110.0
   - gomod: go.opentelemetry.io/collector/exporter/nopexporter v0.110.0
+  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter v0.110.0
 
 # workaround known issue https://github.com/open-telemetry/opentelemetry-collector/releases/tag/v0.110.0
 providers:

Considerations:

• Created some dashboards in the package, but they should not be considered complete nor corrects.
  • As an example, current transform definitions does not take into account Operating system information form user agent.
• Using mode: otel in elasticsearch exporter-.
• Not checked if all the fields are SemConv compatible.
• Using file_storage extension to not re-read the same lines each time the collector is restarted.
• For convenience, all the resources required are under img folder but this is not the final location for those kind of files. Just there for testing purposes.

Useful documentation found

• Functions: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/ottlfuncs/README.md
• Transform processor: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md
• Elasticsearch exporter: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/README.md
• Related to regexes:
  • Regex syntax: https://github.com/google/re2/wiki/Syntax
  • Regex parser: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/regex_parser.md
    • Used initially as an operator in filelog receiver

Author's Checklist

• Build a collector distribution compatible with the requirements of the OTEL integration/template.
• Check that data is ingested in Elasticsearch
• Test that log file receiver keep the state to not re-read all lines in each restart
• Test OTEL integrations/templates setting different pipelines

How to test this PR locally

Testing locally this requires these steps:

• Start Elasticsearch and Kibana:

# you need to run elasticsearch and kibana locally
# and configure accordingly the elasticsearch exporter
# it could be used the dev environment from Kibana
cd /path/to/kibana
# one terminal
yarn es snapshot
# another therminal
yarn start

• Build your own collector with the required dependencies (receivers, processors, exporters...):

# build the distribution from https://github.com/elastic/opentelemetry-collector-components
# with the changes aboved based on https://github.com/elastic/opentelemetry-collector-components/pull/96
cd /path/to/opentelemetry-collector-components
# apply changes shown above and build the binary
make elasticcol-validate

• Test/Run the collector

# from this integrations repository
cd packages/nginx_otel/img

# required for file_storage extension
mkdir -p data
# required for docker nginx to write logs in a folder with our own user
mkdir -p logs

# start docker service
docker-compose up -d

# Run collector
export NGINX_LOGS_DIR="$(pwd)/logs"
/path/to/opentelemetry-collector-components/_build/elastic-collector-components --config nginx-otel.deploy.yml

Related issues

• Relates New integration receiver and processor opentelemetry-collector-components#96
• Relates NGINX Otel Integration #10502

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 2232cd7

Failed CI Steps

• ✅ Check go sources

History

cc @mrodm