Vector Lua Transform Question

[atibdialpad]

I was trying some of the examples from here and getting errors. Are the configs outdated or am I doing something wrong ? @jszwedko

[sources.junos_log_source]
  type = "file"
  include = ["/Users/atib/src/VECTOR/example9/junos.log"]
  data_dir = "/Users/atib/src/VECTOR/example9"
  file_key = "file"


[transforms.junos_log_processor]
  type = "remap"
  inputs = ["junos_log_source"]
  source = '''
    .timestamp_string = "2018-04-07 06:26:02.643"
  '''

[transforms.junos_lua_transform]
  type = "lua"
  inputs = ["junos_log_processor"]

  hooks.init = '''
  timestamp_pattern = "(%d%d%d%d)[-](%d%d)[-](%d%d) (%d%d):(%d%d):(%d%d).?(%d*)"
  function parse_timestamp(str)
    local year, month, day, hour, min, sec, millis = string.match(str, timestamp_pattern)
    local ms = 0
    if millis and millis ~= "" then
        ms = tonumber(millis)
    end
    return {
        year    = tonumber(year),
        month   = tonumber(month),
        day     = tonumber(day),
        hour    = tonumber(hour),
        min     = tonumber(min),
        sec     = tonumber(sec),
        nanosec = ms * 1000000
    }
  end
  '''

  hooks.process = """
  function (event, emit)
    event.log.timestamp = parse_timestamp(event.log.timestamp_string)
    emit(event)
  end
  """

[sinks.console]
  type = "console"
  inputs = ["junos_lua_transform"]
  target = "stdout"
  encoding.codec = "json"

I get the following error :

Jul 06 10:10:57.460 ERROR vector::cli: Configuration error. error=data did not match any variant of untagged enum LuaConfig for key `transforms.junos_lua_transform` at line 46 column 1

[spencergilbert]

I think that example is missing version = "2", though I'm not seeing where that config is actually being produced for the site 🤔

[jszwedko]

Indeed, the version = "2" is missing from the examples. I'll see about updating that.

[atibdialpad]

Thanks Guys. With version=2 in the config, I see the below error :

Jul 06 22:15:06.870 ERROR vector::topology: Configuration error. error=Transform "junos_lua_transform": Cannot evaluate Lua code defining "hooks.init": error converting Lua nil to function

[jszwedko]

Ah, yes, because init needs to be a function.

You should be able to rewrite this as:

[transforms.junos_lua_transform]
  type = "lua"
  inputs = ["junos_log_processor"]
  version = "2"

  source = '''
  timestamp_pattern = "(%d%d%d%d)[-](%d%d)[-](%d%d) (%d%d):(%d%d):(%d%d).?(%d*)"
  function parse_timestamp(str)
    local year, month, day, hour, min, sec, millis = string.match(str, timestamp_pattern)
    local ms = 0
    if millis and millis ~= "" then
        ms = tonumber(millis)
    end
    return {
        year    = tonumber(year),
        month   = tonumber(month),
        day     = tonumber(day),
        hour    = tonumber(hour),
        min     = tonumber(min),
        sec     = tonumber(sec),
        nanosec = ms * 1000000
    }
  end

  function process(event, emit)
    event.log.timestamp = parse_timestamp(event.log.timestamp_string)
    emit(event)
  end
  '''

  hooks.process = "process"

[jszwedko]

I'll update the example. Seemingly it never worked.