Using Caching in LUA transform

[atibdialpad]

I need to add fields to log_events by GET'ing info via a http call. I am planning to use LUA hooks.init to maintain the cache. Will something like this work ?

hooks.init = '''
  cache = {}
  function get(key)
    if not cache[key] then
      cache[key] = <make a http call here to populate the cache>
    end
    return cache[key]
  end
'''
hooks.process = '''
  function(event, emit)
    if event.log.interface_name then
      event.log.interface_description = get(interface_name)
    end
  end
'''

[atibdialpad]

Also would love if I can rebuild the cache every N hours using the timers. @jszwedko

[jszwedko]

Hey @atibdialpad !

Yeah, the runtime will persist across events so you can do caching like that. I'd suggest rewriting this a bit to something like:

[transforms.lua]
type = "lua"
inputs = []
version = "2"

hooks.process = "process"

source = '''
  cache = {}
  function get(key)
    if not cache[key] then
      cache[key] = <make a http call here to populate the cache>
    end
    return cache[key]
  end

  function process(event, emit)
    if event.log.interface_name then
      event.log.interface_description = get(interface_name)
    end
  end
'''

You could also use the timer mechanism to clear the cache on an interval.

[atibdialpad]

Thanks @jszwedko for the suggestion. Any specific reason you suggest using "source" for caching logic and not "hooks.init". Just curios :-)

[atibdialpad]

From the doc what I could understand was, "source" is executed once the transform is created while "hooks.init" needs to wait till the first event comes in. If this is true, using "source" makes more sense for my use case.

[jszwedko]

Ah, so source is intended to an entire Lua source file that is loaded on start-up. hooks.init is expected to be a function that is called once, on the first event. You could maybe define the get function there, but Vector would expect init to resolve to a Lua function that it could call; which is not currently the case in your example.

Something like:

hooks.init = '''
  cache = {}
  function get(key)
    if not cache[key] then
      cache[key] = <make a http call here to populate the cache>
    end
    return cache[key]
  end
  function()
  end
'''
hooks.process = '''
  function(event, emit)
    if event.log.interface_name then
      event.log.interface_description = get(interface_name)
    end
  end
'''

Might work, but you can see it is a bit awkward because init is expected to be a function. I think using source makes more sense in your example to define any globals and functions.