VRL Redis API for log enrichment

[rsrdesarrollo]

Probably this is something that can be done with Lua, but It will be great if you could call a redis from VRL to CRUD some data.

In my head, the API would be something like this

redis_command(conn_id: <string>, command: <string>, key: <string>, args: <array>) :: <array> , <error>

and would be used similarly to this

output, err = redis_command(conn_id: "redis_pro", command: "hset", key: .event_id, args: ["foo", .foo, "bar", .bar]) 

Where redis_pro would be some kind of configuration reference like enrichment tables.

What do you think? Is this crazy? Does it make sense? Is there a much simpler way to do this?

Feedback is appreciated

Regards!

[jszwedko]

Modeling as an enrichment table makes sense. The tricky bit is that we, to date, have avoided having enrichment tables or VRL make any sort of network calls to avoid performance footguns as mentioned in the design doc. I definitely get the need for enrichment that can make network calls, though, and so I think we'll get there; we just need to be careful with the design and make it obvious to users when they are using potentially slow calls.

There's some more discussion of this use-case over here: #17195