Functions in VRL - Community IDs

[owen5mith]

Hello, firstly, thanks for making such a good product! 😄

I have recently implemented community ID hashing in VRL, see the bellow VRL snippet (see here for a full config file with tests):

  . = parse_json!(.message)
  saddr = ip_aton!(to_string!(.sourceIP))
  daddr = ip_aton!(to_string!(.destIP))
  sport = to_int!(.sourcePort)
  dport = to_int!(.destPort)
  proto = to_string!(.proto)


  proto_padded = ""
  if proto == "TCP" {
    proto_padded = "0600"
  } else if proto == "UDP" {
    proto_padded = "1100"
  } else {
    log("Only TCP or UDP supported")
    abort
  }


  seed_padded = "0000"
  sport_padded = to_string(format_int!(sport, 16))
  dport_padded = to_string(format_int!(dport, 16))
  saddr_padded = to_string(format_int!(saddr, 16))
  daddr_padded = to_string(format_int!(daddr, 16))


  if length(sport_padded) < 4 {
    to_add = "0" *  (4 - length(sport_padded))
    sport_padded = to_add + sport_padded
  }

  if length(dport_padded) < 4 {
    to_add = "0" *  (4 - length(dport_padded))
    dport_padded = to_add + dport_padded
  }

  if length(daddr_padded) < 8 {
    to_add = "0" *  (8 - length(daddr_padded))
    daddr_padded = to_add + daddr_padded
  }

  if length(saddr_padded) < 8 {
    to_add = "0" *  (8 - length(saddr_padded))
    saddr_padded = to_add + saddr_padded
  }


  to_hash_base16 = ""
  if saddr < daddr {
    to_hash_base16 = seed_padded + saddr_padded + daddr_padded + proto_padded + sport_padded + dport_padded
  } else {
    to_hash_base16 = seed_padded + daddr_padded + saddr_padded + proto_padded + dport_padded + sport_padded
  }
  to_hash, err = decode_base16(to_hash_base16)
  if err != null {
    log(err)
  }
  .community_id = "1:" + encode_base64(decode_base16!(sha1(to_hash)))

It's worth noting, the above is not a complete implementation of Community IDs. It has the following issues:

• It only supports UDP + TCP
• It lacks support for equal IP addresses, where it should fall back to using the ports for ordering
• It only supports a seed of 0

I've raised this discussion because I was looking to understand:

1. Does VRL support user defined functions, for example, I have had to repeat the padding functionality 4 times which could be put into a function.
2. Would you be open to me making a community_id VRL function? I have found a Rust implementation.

[jszwedko]

Hi @owen5mith !

Very cool! VRL doesn't yet have user-defined functions (see vectordotdev/vrl#47), but given this is an open standard I think we'd be open to having it implemented in VRL itself as Rust. You can see vectordotdev/vrl#277 for a recent function addition that you could model your contribution after if interested.

[owen5mith]

@jszwedko Thanks! I'll look to implement this in the VRL repo.