Date filter exhibits weird behaviour in month parsing when DD is used erroneously for days instead of dd

[deepybee]

• Version: 6.4.1
• Operating System: CentOS 7.5 / macOS 10.13.6

The date filter exhibits very odd behaviour if a user erroneously uses DD instead of dd for the match pattern to parse the numeric day values.

DD should return an integer value of the number of the day in the year, e.g. 27 for the 27th of January, 35 for the 4th February etc.

When DD is erroneously placed in a pattern which parses day values, the value of the day is correctly shown in the output, however the month ends up hard coded to 01.

Correct filter config:

  date {
    match => [ "source_field", "yyyy-MM-dd HH:mm:ss" ]
  }

Errant filter config:

  date {
    match => [ "source_field", "yyyy-MM-DD HH:mm:ss" ]
  }

  date {
    match => [ "source_field", "yyyy-MM-DD HH:mm:ss" ]
  }

• Config File:

input {
  stdin {
    codec => "json_lines"
  }
}

filter {
  date {
    match => [ "test_date", "yyyy-MM-DD HH:mm:ss" ]
  }
}

output {
  stdout {
    codec => "rubydebug"
  }
}

• Sample Data / Steps to reproduce:

echo '{ "test_date": "2018-12-28 09:29:38" }' | ./logstash -f date_debug.rb

{
    "@timestamp" => 2018-01-28T08:29:38.000Z,
      "@version" => "1",
     "test_date" => "2018-12-28 09:29:38",
          "host" => "DPBs-MacBook-Pro.local"
}

echo '{ "test_date": "2018-2-28 09:29:38" }' | ./logstash -f date_debug.rb

{
     "test_date" => "2018-2-28 09:29:38",
    "@timestamp" => 2018-01-28T08:29:38.000Z,
          "host" => "DPBs-MacBook-Pro.local",
      "@version" => "1"
}

echo '{ "test_date": "2018-05-15 09:29:38" }' | ./logstash -f date_debug.rb

{
     "test_date" => "2018-05-15 09:29:38",
    "@timestamp" => 2018-01-15T08:29:38.000Z,
      "@version" => "1",
          "host" => "DPBs-MacBook-Pro.local"
}

[deepybee]

I need to test whether this is purely a Logstash issue or if the Joda library will affect ingest pipelines too.

[robbavey]

@deepybee Seems possible - looks like what is happening is DD is being read as the day of the year, so when the date 2018-05-15 09:29:38 is interpreted using the yyyy-MM-DD HH:mm:ss date format, it is being read as the 15th day of the year - ie the 15th of January - this stackoverflow question about the Joda Time library asks the same question:

https://stackoverflow.com/questions/23408529/joda-time-why-does-month-value-default-to-jan-regardless-of-date-input