Missing output pipeline failures

[mjtrangoni]

Hi @christoe,

I just realized that I am missing some Logstash Stats (Logstash v6.3.1). See,

curl -s -XGET 'localhost:9600/_node/stats/pipelines?pretty'

I see that this failures are not reported by logstash_exporter, for example,

    "main" : {
      "events" : {
        "duration_in_millis" : 2770546365,
        "in" : 1657921575,
        "out" : 1657921575,
        "filtered" : 1657921575,
        "queue_push_duration_in_millis" : 472872
      },
      "plugins" : {
        "inputs" : [ {
[...]
       "outputs" : [ {
          "id" : "48c5afaa6f71527237076eedb20c71a5f894b9b51e2b36c63917c03611361605",
          "documents" : {
            "successes" : 12682774,
            "non_retryable_failures" : 7497226
          },
          "events" : {
            "duration_in_millis" : 42568189,
            "in" : 20180000,
            "out" : 20180000
          },
          "bulk_requests" : {
            "successes" : 160313,
            "with_errors" : 97933,
            "responses" : {
              "200" : 258246
            }
          },
          "name" : "elasticsearch"
        }, {
          "id" : "b35a3cceabb4ffd1feab8e14361adafe56019bb53d853794003844d408da02a0",
          "documents" : {
            "successes" : 1591552581,
            "retryable_failures" : 15155,
            "non_retryable_failures" : 46176108
          },
          "events" : {
            "duration_in_millis" : 2650241358,
            "in" : 1637728689,
            "out" : 1637728689
          },
          "bulk_requests" : {
            "successes" : 18249270,
            "with_errors" : 408927,
            "responses" : {
              "200" : 18658197
            },
            "failures" : 30
          },
          "name" : "elasticsearch"
        }, {
          "id" : "72dde10cd418ae02cd8e9579dccd6596d73f5c71e0170ee794f7fc30343fca43",
          "documents" : {
            "successes" : 1
          },
          "events" : {
            "duration_in_millis" : 164169,
            "in" : 12886,
            "out" : 12886
          },
          },
          "bulk_requests" : {
            "successes" : 12808,
            "responses" : {
              "200" : 12808
            },
            "failures" : 1
          },
          "name" : "elasticsearch"
        }, {
          "id" : "4f78952610769660f73e0bc5e9142c79059acffcc20aa249eb7ea3c3a2a26e9d",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 0,
            "out" : 0
          },
          "name" : "elasticsearch"
        }, {
          "id" : "215b2737000764e6ca61fdbedd32e671cc9b4e8bff6b2bf2d79df367cd2ac9e5",
          "documents" : {
            "successes" : 12886
          },
          "events" : {
            "duration_in_millis" : 169051,
            "in" : 12886,
            "out" : 12886
          },
          "bulk_requests" : {
            "successes" : 1,
            "responses" : {
              "200" : 12808
            },
            "failures" : 1
          },
          "name" : "elasticsearch"
        } ]
      },
      "reloads" : {
        "last_error" : null,
        "successes" : 0,
        "last_success_timestamp" : null,
        "last_failure_timestamp" : null,
        "failures" : 0
      },
      "queue" : {
        "type" : "memory"
      }
    }
  }

I see some like this,

logstash_node_pipeline_events_out_total{pipeline=".monitoring-logstash"} 0
logstash_node_pipeline_events_out_total{pipeline="main"} 1.658111575e+09
logstash_node_plugin_events_in_total{pipeline="main",plugin="elasticsearch",plugin_id="215b2737000764e6ca61fdbedd32e671cc9b4e8bff6b2bf2d79df367cd2ac9e5",plugin_type="output"} 12886
logstash_node_plugin_events_in_total{pipeline="main",plugin="elasticsearch",plugin_id="48c5afaa6f71527237076eedb20c71a5f894b9b51e2b36c63917c03611361605",plugin_type="output"} 2.018e+07
logstash_node_plugin_events_in_total{pipeline="main",plugin="elasticsearch",plugin_id="4f78952610769660f73e0bc5e9142c79059acffcc20aa249eb7ea3c3a2a26e9d",plugin_type="output"} 0
logstash_node_plugin_events_in_total{pipeline="main",plugin="elasticsearch",plugin_id="72dde10cd418ae02cd8e9579dccd6596d73f5c71e0170ee794f7fc30343fca43",plugin_type="output"} 12886
logstash_node_plugin_events_in_total{pipeline="main",plugin="elasticsearch",plugin_id="b35a3cceabb4ffd1feab8e14361adafe56019bb53d853794003844d408da02a0",plugin_type="output"} 1.637918689e+09

But nothings related to failures, see the failure stats

# curl -s -XGET 'localhost:9600/_node/stats/pipelines?pretty' | grep fail
        "last_failure_timestamp" : null,
        "failures" : 0
            "non_retryable_failures" : 7497226
            "retryable_failures" : 15155,
            "non_retryable_failures" : 46176108
            "failures" : 30
            "failures" : 1
            "failures" : 1
        "last_failure_timestamp" : null,
        "failures" : 0

Could you please add this? Thanks!

[wasilak]

check my PR #34