Skip to content

Latest commit

 

History

History
 
 

discoveryreceiver

README.md

Discovery Receiver

Status
Stability [in-development]
Supported pipeline types logs
Distributions [Splunk]

The Discovery receiver is a receiver compatible with logs pipelines that allows you to test the functional status of any receiver type whose target is reported by an Observer. It provides configurable status match rules that evaluate the generated receiver's emitted metrics (if any), or component-level log statements via the instance's zap.Logger. It works similarly to the Receiver Creator (it actually wraps an internal instance of one), but the resulting dynamically-instantiated receivers don't actually report their metric content to your metrics pipelines. Instead, the metrics are intercepted by an internal metrics consumer capable of translating desired metrics to log records based on the status: metrics rules you define. All component-level log statements are similarly intercepted by a log evaluator, and can be translated to emitted log records based on the status: statements rules you define.

The receiver also allows you to emit log records for all Endpoint events from the specified watch_observers. This way you can report your environment as observed by platform-specific observers in real time, with or without discovering receiver statuses:

extensions:
  docker_observer:
receivers:
  discovery:
    watch_observers: [docker_observer]
    log_endpoints: true
exporters:
  logging:
    loglevel: debug
service:
  extensions: [docker_observer]
  pipelines:
    logs:
      receivers: [discovery]
      exporters: [logging]
2022-07-27T19:00:32.305Z	info	LogsExporter	{"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 3}
2022-07-27T19:00:32.306Z	info	ResourceLog #0
Resource SchemaURL:
Resource labels:
     -> event.type: STRING(endpoint.added)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope
LogRecord #0
ObservedTimestamp: 2022-07-27 19:00:32.30572809 +0000 UTC
Timestamp: 2022-07-27 19:00:32.305727862 +0000 UTC
Severity: info
Body: {"alternate_port":5432,"command":"postgres -c shared_preload_libraries=pg_stat_statements -c wal_level=logical -c max_replication_slots=2","container_id":"58e71612910cd3e2a89d809f1b53fee779c4c81d9bbf36ef37f4c05b04037353","endpoint":"172.17.0.4:5432","host":"172.17.0.4","id":"58e71612910cd3e2a89d809f1b53fee779c4c81d9bbf36ef37f4c05b04037353:5432","image":"postgres","labels":{},"name":"naughty_heisenberg","port":5432,"tag":"latest","transport":"TCP","type":"container"}
Attributes:
     -> type: STRING(endpoint)
     -> image: STRING(postgres)
     -> tag: STRING(latest)
     -> port: STRING(5432)
     -> alternate_port: STRING(5432)
     -> host: STRING(172.17.0.4)
     -> transport: STRING(TCP)
     -> labels: STRING(map[])
     -> endpoint: STRING(172.17.0.4:5432)
     -> name: STRING(naughty_heisenberg)
     -> command: STRING(postgres -c shared_preload_libraries=pg_stat_statements -c wal_level=logical -c max_replication_slots=2)
     -> container_id: STRING(58e71612910cd3e2a89d809f1b53fee779c4c81d9bbf36ef37f4c05b04037353)
     -> id: STRING(58e71612910cd3e2a89d809f1b53fee779c4c81d9bbf36ef37f4c05b04037353:5432)
Trace ID:
Span ID:
Flags: 0
LogRecord #1
ObservedTimestamp: 2022-07-27 19:00:32.305771365 +0000 UTC
Timestamp: 2022-07-27 19:00:32.305771316 +0000 UTC
Severity: info
Body: {"alternate_port":0,"command":"nginx -g daemon off;","container_id":"2567fdbc764706d29120b01efcc3a310d87e9e121ec9debbc977d66f5497cdda","endpoint":"172.17.0.2:80","host":"172.17.0.2","id":"2567fdbc764706d29120b01efcc3a310d87e9e121ec9debbc977d66f5497cdda:80","image":"nginx","labels":{"maintainer":"NGINX Docker Maintainers \[email protected]\u003e"},"name":"ecstatic_davinci","port":80,"tag":"latest","transport":"TCP","type":"container"}
Attributes:
     -> type: STRING(endpoint)
     -> labels: STRING(map[maintainer:NGINX Docker Maintainers <[email protected]>])
     -> id: STRING(2567fdbc764706d29120b01efcc3a310d87e9e121ec9debbc977d66f5497cdda:80)
     -> tag: STRING(latest)
     -> port: STRING(80)
     -> alternate_port: STRING(0)
     -> command: STRING(nginx -g daemon off;)
     -> container_id: STRING(2567fdbc764706d29120b01efcc3a310d87e9e121ec9debbc977d66f5497cdda)
     -> transport: STRING(TCP)
     -> name: STRING(ecstatic_davinci)
     -> image: STRING(nginx)
     -> host: STRING(172.17.0.2)
     -> endpoint: STRING(172.17.0.2:80)
Trace ID:
Span ID:
Flags: 0
LogRecord #2
ObservedTimestamp: 2022-07-27 19:00:32.305784658 +0000 UTC
Timestamp: 2022-07-27 19:00:32.305784612 +0000 UTC
Severity: info
Body: {"alternate_port":0,"command":"redis-server","container_id":"5f7f5f007f798c59a60c765e566db5d22bff59c614268db8d1b9abbc3ee70bf7","endpoint":"172.17.0.3:6379","host":"172.17.0.3","id":"5f7f5f007f798c59a60c765e566db5d22bff59c614268db8d1b9abbc3ee70bf7:6379","image":"redis","labels":{},"name":"beautiful_clarke","port":6379,"tag":"latest","transport":"TCP","type":"container"}
Attributes:
     -> type: STRING(endpoint)
     -> command: STRING(redis-server)
     -> transport: STRING(TCP)
     -> endpoint: STRING(172.17.0.3:6379)
     -> port: STRING(6379)
     -> image: STRING(redis)
     -> tag: STRING(latest)
     -> alternate_port: STRING(0)
     -> container_id: STRING(5f7f5f007f798c59a60c765e566db5d22bff59c614268db8d1b9abbc3ee70bf7)
     -> host: STRING(172.17.0.3)
     -> labels: STRING(map[])
     -> id: STRING(5f7f5f007f798c59a60c765e566db5d22bff59c614268db8d1b9abbc3ee70bf7:6379)
     -> name: STRING(beautiful_clarke)
Trace ID:
Span ID:
Flags: 0

Example Usage

The following Collector configuration will create a Discovery receiver instance that receives endpoints from a Docker Observer that reports log records denoting the status of a Smart Agent receiver configured to use the collectd/redis monitor. The status mapping comprises entries that signal the Smart Agent receiver has been instantiated with a successful, partial, or failed status, based on reported metrics or recorded application log statements.

The Redis receiver warrants the following log record with discovery.status from the Discovery receiver:

  • successful if it emits any metrics matching the regexp: .* pattern, denoting that metric gathering and the Receiver are functional.
  • partial if it internally logs a statement matching the regexp: (WRONGPASS|NOAUTH|ERR AUTH) pattern, suggesting there is a Redis server but it's receiving incorrect credentials.
  • failed if it internally logs a statement matching the regexp: ConnectionRefusedError pattern, suggesting that no Redis server is available at the endpoint.
extensions:
  docker_observer:
receivers:
   discovery:
     watch_observers: [docker_observer]
     receivers:
       smartagent/redis:
         // Determine the functionality of the smartagent/redis receiver for any detected container
         rule: type == "container"
         config:
           type: collectd/redis
           // the metric or log statement
         status:
           metrics:
             successful:
               - regexp: '.*'
                 // Only emit a single log record for this status entry instead of one for each matching received metric (`false`, the default)
                 first_only: true
                 log_record:
                   severity_text: info
                   body: Successfully able to connect to Redis container.
           statements:
             partial:
               - regexp: (WRONGPASS|NOAUTH|ERR AUTH)
                 first_only: true
                 log_record:
                   severity_text: warn
                   body: Container appears to be accepting redis connections but the default auth setting is incorrect.
             failed:
               - regexp: ConnectionRefusedError
                 first_only: true
                 log_record:
                   severity_text: debug
                   body: Container appears to not be accepting redis connections.
exporters:
  logging:
    loglevel: debug
    sampling_initial: 1
    sampling_thereafter: 1
service:
  extensions:
    - docker_observer
  pipelines:
    logs:
      receivers:
        - discovery
      exporters:
        - logging

Given this configuration, if the Discovery receiver's Docker observer instance reports an active Redis container, and the Smart Agent receiver's associated collectd/redis monitor is able to generate metrics for the container, the receiver will emit something similar to the following log records:

2022-07-27T16:35:03.575Z	info	LogsExporter	{"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 1}
2022-07-27T16:35:03.575Z	info	ResourceLog #0
Resource SchemaURL:
Resource labels:
     -> container.name: STRING(lucid_goldberg)
     -> container.image.name: STRING(redis)
     -> discovery.receiver.rule: STRING(type == "container" && image == "redis")
     -> discovery.endpoint.id: STRING(2f0d88c1d93b2aafce4a725de370005f9e7e961144551a3df37a88b27ebed48f:6379)
     -> discovery.receiver.name: STRING(smartagent/redis)
     -> event.type: STRING(metric.match)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope
LogRecord #0
ObservedTimestamp: 2022-07-27 16:35:03.575234252 +0000 UTC
Timestamp: 2022-07-27 16:35:01.522543616 +0000 UTC
Severity: info
Body: Successfully able to connect to Redis container.
Attributes:
     -> metric.name: STRING(counter.expired_keys)
     -> discovery.status: STRING(successful)
Trace ID:
Span ID:
Flags: 0

Instead, if the Docker observer reports an active Redis container but the collectd/redis authentication information is incorrect, the Discovery receiver will emit something similar to the following log record:

2022-07-27T17:11:27.271Z	info	LogsExporter	{"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 1}
2022-07-27T17:11:27.271Z	info	ResourceLog #0
Resource SchemaURL:
Resource labels:
     -> event.type: STRING(statement.match)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope
LogRecord #0
ObservedTimestamp: 2022-07-27 17:11:27.27149884 +0000 UTC
Timestamp: 2022-07-27 17:11:27.271444488 +0000 UTC
Severity: warn
Body: Container appears to be accepting redis connections but the default auth is incorrect.
Attributes:
     -> runnerPID: STRING(119386)
     -> monitorID: STRING(smartagentredisreceiver_creatordiscoveryendpoint1721702637958d3f3335daf83f7)
     -> monitorType: STRING(collectd/redis)
     -> caller: STRING(signalfx/handler.go:189)
     -> name: STRING(smartagent/redis/receiver_creator/discovery{endpoint="172.17.0.2:6379"}(58d3f3335daf83f7))
     -> createdTime: STRING(1.658941887269115e+09)
     -> lineno: STRING(201)
     -> logger: STRING(root)
     -> sourcePath: STRING(/usr/lib/splunk-otel-collector/agent-bundle/collectd-python/redis/redis_info.py)
     -> level: STRING(error)
     -> receiver.name: STRING(smartagent/redis)
     -> discovery.status: STRING(partial)
Trace ID:
Span ID:
Flags: 0

If the Docker observer reports an unrelated container that isn't running Redis, the following log record would be emitted:

2022-07-27T17:16:57.718Z	info	LogsExporter	{"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 1}
2022-07-27T17:16:57.719Z	info	ResourceLog #0
Resource SchemaURL:
Resource labels:
     -> event.type: STRING(statement.match)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope
LogRecord #0
ObservedTimestamp: 2022-07-27 17:16:57.718720008 +0000 UTC
Timestamp: 2022-07-27 17:16:57.718670678 +0000 UTC
Severity: debug
Body: Container appears to not be accepting redis connections.
Attributes:
     -> name: STRING(smartagent/redis/receiver_creator/discovery{endpoint="172.17.0.2:80"}(54141aa1da2d2ad0))
     -> monitorType: STRING(collectd/redis)
     -> logger: STRING(root)
     -> createdTime: STRING(1.6589422177182763e+09)
     -> level: STRING(error)
     -> monitorID: STRING(smartagentredisreceiver_creatordiscoveryendpoint17217028054141aa1da2d2ad0)
     -> runnerPID: STRING(119590)
     -> sourcePath: STRING(/usr/lib/splunk-otel-collector/agent-bundle/collectd-python/redis/redis_info.py)
     -> lineno: STRING(198)
     -> caller: STRING(signalfx/handler.go:189)
     -> receiver.name: STRING(smartagent/redis)
     -> discovery.status: STRING(failed)
Trace ID:
Span ID:
Flags: 0

Config

Main

Name Type Default Docs
watch_observers (required) []string The array of Observer extensions to receive Endpoint events from
log_endpoints bool false Whether to emit log records for Observer Endpoint events
embed_receiver_config bool false Whether to embed a base64-encoded, minimal Receiver Creator config for the generated receiver as a reported metrics discovery.receiver.rule resource attribute value for status log record matches
receivers map[string]ReceiverConfig The mapping of receiver names to their Receiver sub-config

ReceiverConfig

Name Type Default Docs
rule (required) string The Receiver Creator compatible discover rule
config map[string]any The receiver instance configuration, including any Receiver Creator endpoint env value expr program value expansion
resource_attributes map[string]string A mapping of string resource attributes and their (expr program compatible) values to include in reported metrics for status log record matches
status map[string]Match A mapping of metrics and/or statements to Match items for status evaluation

Match

One of regexp, strict, or expr is required.

Name Type Default Docs
regexp string The regexp pattern to evaluate reported received metric names or component log statements
strict string The string literal to compare equivalence against reported received metric names or component log statements
expr string The expr program run with the reported received metric names or component log statements (vm env TBD)
first_only bool false Whether to emit only one log record for the first matching metric or log statement, ignoring all subsequent matches
record LogRecord The emitted log record content

LogRecord

Name Type Default Docs
severity_text string Emitted log statement severity level, if any, or "info" The emitted log record's severity text
body string Emitted log statement message The emitted log record's body
attributes map[string]string Emitted log statements fields The emitted log record's attributes

Status log record content

In addition to the effects of the configured values, each emitted log record will include:

  • event.type resource attribute with either metric.match or statement.match based on context.
  • discovery.status log record attribute with successful, partial, or failed status depending on match.