[receiver/windowseventlog] Receiver stops reading from security EventLog channel

[dpaasman00]

Component(s)

receiver/windowseventlog

What happened?

Description

Running the collector with the Windows Event Log receiver reading from the security channel. After some time (varies between 15m -1hr) the receiver stops reading events from the security channel. New events are showing up in the Windows Security Event Log but the receiver isn't grabbing any new ones. There are no errors in the collector log and other components continue functioning correctly, including other Windows Event Log receivers which are collecting from different channels like application. Restarting the machine fixes the issue for a time until it begins happening again.

Steps to Reproduce

Currently unable to reproduce.

Expected Result

The receiver should get these new events and not freeze up.

Actual Result

Receiver stops collecting new security channel log events.

Collector version

v0.110.0

Environment information

Environment

OS: Windows Server 2016

OpenTelemetry Collector configuration

receivers:
    windowseventlog/application:
        attributes:
            log_type: windows_event.application
        channel: application
        max_reads: 100
        poll_interval: 1s
        raw: false
        start_at: end
    windowseventlog/custom/0:
        attributes:
            log_type: windows_event.custom
        channel: Microsoft-Windows-PowerShell/Operational
        max_reads: 100
        poll_interval: 1s
        raw: false
        start_at: end
    windowseventlog/custom/1:
        attributes:
            log_type: windows_event.custom
        channel: Microsoft-Windows-UAC/Operational
        max_reads: 100
        poll_interval: 1s
        raw: false
        start_at: end
    windowseventlog/custom/2:
        attributes:
            log_type: windows_event.custom
        channel: Microsoft-Windows-WMI-Activity/Operational
        max_reads: 100
        poll_interval: 1s
        raw: false
        start_at: end
    windowseventlog/custom/3:
        attributes:
            log_type: windows_event.custom
        channel: Microsoft-Windows-WinRM/Operational
        max_reads: 100
        poll_interval: 1s
        raw: false
        start_at: end
    windowseventlog/custom/4:
        attributes:
            log_type: windows_event.custom
        channel: Microsoft-Windows-TaskScheduler/Operational
        max_reads: 100
        poll_interval: 1s
        raw: false
        start_at: end
    windowseventlog/security:
        attributes:
            log_type: windows_event.security
        channel: security
        max_reads: 100
        poll_interval: 1s
        raw: false
        start_at: end
    windowseventlog/system:
        attributes:
            log_type: windows_event.system
        channel: system
        max_reads: 100
        poll_interval: 1s
        raw: false
        start_at: end

exporters:
    otlp:
        compression: gzip
        endpoint: <sensitive>
        retry_on_failure:
            enabled: true
            initial_interval: 5s
            max_elapsed_time: 300s
            max_interval: 30s
        sending_queue:
            enabled: true
            num_consumers: 10
            queue_size: 5000
            storage: file_storage
        tls:
            insecure: true

extensions:
    file_storage/otlp_gateway_non_prod_pci:
        compaction:
            directory: ${OIQ_OTEL_COLLECTOR_HOME}/storage
            on_rebound: true
        directory: ${OIQ_OTEL_COLLECTOR_HOME}/storage

service:
    extensions:
        - file_storage
    pipelines:
        logs:
            receivers:
                - windowseventlog/system
                - windowseventlog/application
                - windowseventlog/security
                - windowseventlog/custom/0
                - windowseventlog/custom/1
                - windowseventlog/custom/2
                - windowseventlog/custom/3
                - windowseventlog/custom/4

            exporters:
                - otlp
 
    telemetry:
        metrics:
            address: localhost:28888

Log output

No response

Additional context

This issue was reported by a customer. We are currently trying to replicate to get more information.

[github-actions]

Pinging code owners:

• receiver/windowseventlog: @armstrmi @pjanotti

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dpaasman00]

We were able to resolve this issue. The receiver did not stop reading from the security channel, rather it wasn't reading fast enough. In this particular environment there was an extremely high load on the channel, and the default max_reads value wasn't high enough to keep up. After bumping this number we saw much better and stable performance.