[WIP] Add generic log message generation basic on a custom templates for log fields #62
Conversation
…om/mingrammer/flog
…me custom functions and template data
robskillington
changed the title
robskillington
changed the title
| buffer strings.Builder | ||
| ) | ||
|
|
||
| data = TemplateData{ |
There was a problem hiding this comment.
Why using template data? I think these could be template functions to give more control.
I think it makes more sense to use a function since its generating data, instead of accessing to some predefined values.
There was a problem hiding this comment.
Gofakeit already allows be be exposed to text/template https://github.com/brianvoe/gofakeit?tab=readme-ov-file#templates
| if err := v.Execute(&buffer, data); err != nil { | ||
| return nil, fmt.Errorf("failed to execute template for field '%s': %w", k, err) | ||
| } | ||
| record[k] = buffer.String() |
There was a problem hiding this comment.
🤔
This will always produce a string. Not allowing, numbers, booleans or more complex data types.
| } | ||
| case "template": | ||
| plug.gen = genTemplate | ||
| fieldsStr := fbit.Conf.String("fields") |
There was a problem hiding this comment.
I'm worried about field order.
Since Go plugins sends the records as map[string]any it won't keep the order of these fields. But the user might expect them in the given order.
|
|
||
| plug.template = NewLogGenerator() | ||
| for _, field := range fields { | ||
| fieldValue := fbit.Conf.String(field) |
There was a problem hiding this comment.
I'm worried about this "dynamic" config in where the config schema is unknown.
I think I would just prefer if the template evaluates to a JSON string (At least for now since we don't support reading objects from the YAML config).
pipeline:
inputs:
- name: datagen
template: |-
{
"id": "{{UUID}}",
"host": "{{IPv4Address}}",
"userAgent": "{{UserAgent}}",
"version": "{{AppVersion}}",
"date": "{{PastDate}}"
}There was a problem hiding this comment.
I agree with this, for sure
There was a problem hiding this comment.
Left a few comments.
I'll move the code into datagen plugin here https://github.com/chronosphereio/calyptia-core-fluent-bit-plugin-datagen
Based loosely on
github.com/mingrammer/flogfor field generation with ability to use arbitrary templates (have includedsprigtemplate functions too).TODO: