Working with key_value logs when one of the keys does not have a value?

[proffalken]

I'm working on extracting data from an IP Tables log using VRL, but because the fields aren't always populated, VRL seems to have issues when using the parse_key_value function.

As an example, the line IN=eth1 OUT= MAC=fc:ff:da:47:af:13:b4:2e:99:19:12:00:00:00 SRC=10.x.x.x DST=10.x.x.x LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=62910 DF PROTO=TCP SPT=47468 DPT=9100 WINDOW=64240 RES=0x00 SYN URGP=0 with the rule

    source: >-
      . |= parse_key_value!(.message)

      .dest_ip = .DST

      .dest_port = .DPT

      .src_ip = .SRC

      .src_port = .SPT

produces OUT=MAC=fc:ff:da:47:af:13:b4:2e:99:19:12:00:00:00 when it should be two distinct fields, OUT with a null value and MAC with the MAC Address. Is there anyway to tell Vector to "skip" a field if it's in a key-value pair but doesn't have a value?

[spencergilbert]

parse_key_value has accept_standalone_key defaulted to true, as well as whitespace to "lenient".

Playing in the REPL I get this:

$ parse_key_value!(.message, whitespace: "strict")
{ "DF": true, "DPT": "9100", "DST": "10.x.x.x", "ID": "62910", "IN": "eth1", "LEN": "60", "MAC": "fc:ff:da:47:af:13:b4:2e:99:19:12:00:00:00", "OUT": "", "PREC": "0x00", "PROTO": "TCP", "RES": "0x00", "SPT": "47468", "SRC": "10.x.x.x", "SYN": true, "TOS": "0x00", "TTL": "63", "URGP": "0", "WINDOW": "64240" }

$ parse_key_value!(.message, accept_standalone_key: false, whitespace: "strict")
{ "DF PROTO": "TCP", "DPT": "9100", "DST": "10.x.x.x", "ID": "62910", "IN": "eth1", "LEN": "60", "MAC": "fc:ff:da:47:af:13:b4:2e:99:19:12:00:00:00", "OUT": "", "PREC": "0x00", "RES": "0x00", "SPT": "47468", "SRC": "10.x.x.x", "SYN URGP": "0", "TOS": "0x00", "TTL": "63", "WINDOW": "64240" }

[proffalken]

Thanks, I didn't even know there was a REPL, I'll have a play, this looks good!

[spencergilbert]

Yep! vector vrl and some options to optionally load events or programs.