Invoke-IcingaCheckService missing Automatic services.

[drapiti]

We are seeing that if we set a simple filter such as Alert on Automatic services different to running, not all services on the specific machine are captured. There are services which are down and of startup Automatic.

This is quite a serious bug.

[drapiti]

Here is a fixed version of the code:

function Invoke-IcingaCheckService()
{
    param (
        [array]$Service           = @(),
        [array]$Exclude           = @(),
        [ValidateSet('Stopped', 'StartPending', 'StopPending', 'Running', 'ContinuePending', 'PausePending', 'Paused')]
        [string]$Status           = 'Running',
        [ValidateSet('Boot', 'System', 'Automatic', 'Manual', 'Disabled', 'Unknown')]
        [array]$FilterStartupType = @(),
        [switch]$MitigateUnknown  = $FALSE,
        [ValidateSet('Ok', 'Warning', 'Critical', 'Unknown')]
        [string]$OverrideNotFound = 'Unknown',
        [ValidateSet(0, 1, 2, 3)]
        [int]$Verbosity           = 0,
        [switch]$NoPerfData
    );

    $ServicesPackage       = New-IcingaCheckPackage -Name 'Services' -OperatorAnd -Verbose $Verbosity -AddSummaryHeader -IgnoreEmptyPackage:$MitigateUnknown;
    $ServicesCountPackage  = New-IcingaCheckPackage -Name 'Count Services' -OperatorAnd -Verbose $Verbosity -Hidden;
    $FetchedServices       = @{};
    $ServiceSummary        = $null;

    # Automatic load auto start services and check for errors in case no service
    # to check for is configured
    if ($Service.Count -eq 0) {
        $AutoServices = Get-IcingaServices -Exclude $Exclude;
        foreach ($autoservice in $AutoServices.Values) {

            # Skip services which are not defined to run automatically
            if ($autoservice.configuration.StartType.Raw -ne $ProviderEnums.ServiceStartupType.Automatic) {
                # Additional check for Delayed Start if needed
                if (-not ($autoservice.configuration.DelayedAutoStart -eq $TRUE)) {
                    continue;
                }
            }

            $ServiceSummary = Add-IcingaServiceSummary -ServiceStatus $autoservice.configuration.Status.Raw -ServiceData $ServiceSummary;

            # Check if the service is running
            if ($autoservice.configuration.Status.Raw -eq $ProviderEnums.ServiceStatus.Running) {
                $ServicesPackage.AddCheck(
                    (New-IcingaWindowsServiceCheckObject -Status 'Running' -Service $autoservice -NoPerfData)
                );
                continue;
            }

            # Services which should be running, but are not
            $ServicesPackage.AddCheck(
                (New-IcingaWindowsServiceCheckObject -Status 'Running' -Service $autoservice -NoPerfData)
            );
        }
    } else {
        $FetchedServices = Get-IcingaServices -Service $Service -Exclude $Exclude;
        foreach ($services in $FetchedServices.Values) {
            if ($FilterStartupType.Count -ne 0) {
                if ($FilterStartupType -NotContains $ProviderEnums.ServiceStartupTypeName[$services.configuration.StartType.Raw]) {
                    continue;
                }
            }
            $ServicesPackage.AddCheck(
                (New-IcingaWindowsServiceCheckObject -Status $Status -Service $services)
            );

            $ServiceSummary = Add-IcingaServiceSummary -ServiceStatus $services.configuration.Status.Raw -ServiceData $ServiceSummary;
        }
    }

    # Fix invalid performance data in case only one service was checked and the service does not exist
    if ($null -eq $ServiceSummary) {
        $ServiceSummary = Add-IcingaServiceSummary;
    }

    # Check our included services and add an unknown state for each service which was not found on the system
    foreach ($ServiceArg in $Service) {
        $ServiceArg = $ServiceArg.Replace('`', '');
        if ($null -eq $FetchedServices -Or $FetchedServices.ContainsKey($ServiceArg) -eq $FALSE) {
            if ($ServiceArg.Contains('*')) {
                continue;
            }

            # As we can use the DisplayName of a service inside the filter, we need to compare the DisplayName with
            # our provided filter as well
            [bool]$ServiceKnown = $FALSE;

            foreach ($fetchedService in $FetchedServices.Keys) {
                $fetchedService = $FetchedServices[$fetchedService];

                if ($fetchedService.metadata.DisplayName -eq $ServiceArg) {
                    $ServiceKnown = $TRUE;
                    break;
                }
            }

            if ($ServiceKnown) {
                continue;
            }

            $ServiceNotFound = $null;
            $UnknownName     = [string]::Format('{0}:', $ServiceArg);
            $UnknownValue    = 'Service not found';

            switch ($OverrideNotFound.ToLower()) {
                'ok' {
                    $ServiceNotFound = (New-IcingaCheck -Name $UnknownName -Value $UnknownValue -NoPerfData).SetOk();
                    break;
                };
                'warning' {
                    $ServiceNotFound = (New-IcingaCheck -Name $UnknownName -Value $UnknownValue -NoPerfData).SetWarning();
                    break;
                };
                'critical' {
                    $ServiceNotFound = (New-IcingaCheck -Name $UnknownName -Value $UnknownValue -NoPerfData).SetCritical();
                    break;
                };
                default {
                    $ServiceNotFound = (New-IcingaCheck -Name $UnknownName -Value $UnknownValue -NoPerfData).SetUnknown();
                    break;
                };
            }

            $ServicesPackage.AddCheck($ServiceNotFound);
        }
    }

    if ($ServicesPackage.HasChecks()) {
        $ServicesCountPackage.AddCheck(
            (New-IcingaCheck -Name 'stopped services' -Value $ServiceSummary.StoppedCount -MetricIndex 'summary' -MetricName 'stopped')
        );
        $ServicesCountPackage.AddCheck(
            (New-IcingaCheck -Name 'pending started services' -Value $ServiceSummary.StartPendingCount -MetricIndex 'summary' -MetricName 'pendingstarted')
        );
        $ServicesCountPackage.AddCheck(
            (New-IcingaCheck -Name 'pending stopped services' -Value $ServiceSummary.StopPendingCount -MetricIndex 'summary' -MetricName 'pendingstopped')
        );
        $ServicesCountPackage.AddCheck(
            (New-IcingaCheck -Name 'running services' -Value $ServiceSummary.RunningCount -MetricIndex 'summary' -MetricName 'running')
        );
        $ServicesCountPackage.AddCheck(
            (New-IcingaCheck -Name 'pending continued services' -Value $ServiceSummary.ContinuePendingCount -MetricIndex 'summary' -MetricName 'pendingcontinued')
        );
        $ServicesCountPackage.AddCheck(
            (New-IcingaCheck -Name 'pending paused services' -Value $ServiceSummary.PausePendingCount -MetricIndex 'summary' -MetricName 'pendingpaused')
        );
        $ServicesCountPackage.AddCheck(
            (New-IcingaCheck -Name 'paused services' -Value $ServiceSummary.PausedCount -MetricIndex 'summary' -MetricName 'paused')
        );
        $ServicesCountPackage.AddCheck(
            (New-IcingaCheck -Name 'service count' -Value $ServiceSummary.ServicesCounted -MetricIndex 'summary' -MetricName 'count')
        );

        $ServicesPackage.AddCheck($ServicesCountPackage)
    }

    return (New-IcingaCheckResult -Name 'Services' -Check $ServicesPackage -NoPerfData $NoPerfData -Compile);
}

[LordHepipud]

Hello

Thank you for the issue. This is not a bug but by design.
Services on Windows can be configured to run "Automatic", but still stop running after while, which is in some cases expected behavior.

As for example: if you have Edge or Chrome installed on a Windows machine, you will see the service for updating the browser is set to "Automatic". But after the system boots and the update procedure is completed, the service will end itself with exit code 0.

This is the behavior we used for the plugin, as a service that ends itself with exit code 0 has not crashed and was gracefully shut down.

If you need to check important services, you should always add a check to ensure the service is running, to detect if a user has shut down a service for example.

I explained the reason for this on a talk at OSMC in 2022 in detail: https://youtu.be/Y_FQjRymPBU?si=1Xh-rrMzjpKRnF3e&t=1425