Field parsing chain

[Antiarchitect]

Assume I collect k8s logs and internal message format differs as I have a lot of different apps. I need to get log level from the internal message into separate field. For now I have:

          msg_parsed =
                       parse_json(.message) ??
                       parse_klog(.message) ??
                       # parse_logfmt(.message) ??
                       parse_regex(.message, r'^\[(?P<level>[A-Z]+)?\]\s') ??
                       parse_regex(.message, r'^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z\s\[(?P<level>[A-Z]+)\]\s') ??
                       parse_regex(.message, r'^[A-Z],\s\[.+\]\s+(?P<level>[A-Z]+) -- :') ??
                       parse_regex(.message, r'^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z\s\d{3}\s(?P<level>[a-z]+)\s') ??
                       parse_regex(.message, r'^\d{4}/\d{2}/\d{2}\s\d{2}:\d{2}:\d{2}\s\[(?P<level>[a-z]+)?\]\s') ??
                       {}
          if msg_parsed == {} {
            maybe_nginx = parse_nginx_log(.message, "combined") ?? {}
            if maybe_nginx != {} {
              msg_parsed = { "level": "info" }
            }
            maybe_nginx_error = parse_nginx_log(.message, "error") ?? {}
            if maybe_nginx_error != {} {
              msg_parsed = { "level": "error" }
            }
          }
          .level = downcase(msg_parsed.level || "unknown")
          if .level == "warn" {
            .level = "warning"
         }

Issues:

• parse_logfmt is infallible - I can't use it because the rest of the chain not works.
• For now everywhere the field called level and only that fact allows me to .level = downcase(msg_parsed.level || "unknown") at the end

Does VRL has ||= ruby-like operator? It would be very useful in such cases?
Do you guys know how to improve the current config to make it less ugly?
I'm absolutely new to VRL and examples are very scarce :)

[spencergilbert]

Hi @Antiarchitect - still thinking through this, but wanted to note that parse_logfmt should be fallible (ref).

EDIT: re: ||= I think that would just be || in VRL?

[Antiarchitect]

In Ruby a ||= 10 means if a is null then assign a to 10

[spencergilbert]

In Ruby a ||= 10 means if a is null then assign a to 10

Gotcha - would you mind opening an issue around that? Today you should be able to do that with:

if is_nullish(a) { a = 10 }

[spencergilbert]

Can you explain the issue you're having with parse_logfmt? The error handling should be the same as json, klog, and regex.

[Antiarchitect]

It seems parse_logfmt has special behavior wihich makes it infallible. So any parsing rules after it don't work. Frankly, I do not understand how these two statements work together:

[spencergilbert]

The Notices section was incorrectly copied from the encoding function, I've PR'd its removal here: #12967

Continuing to dive into it, I opened: https://github.com/vectordotdev/vector/issues/12973

For now you could move the parse_logfmt to the end of the coalescing chain to have an improved experience.

[Antiarchitect]

https://github.com/vectordotdev/vector/issues/12949

[JeanMertz]

Closing this, as any questions have been answered, and feature requests have been moved to the relevant tracking issues.

see:

• https://github.com/vectordotdev/vector/issues/12973
• https://github.com/vectordotdev/vector/issues/12949