When does adding metadata to logs get to be too much?

[portante]

Given the recent set of questions being asked about enriching logs with metadata from different sources (primarily kubernetes related metadata sources ({node, pod, namespace} labels and annotations), how can we help folks understand the impact of adding all this metadata to logs?

Perhaps we could offer a way to track the statistical distribution of message sizes being processed through vector independent of all the enrichment, and then track separately what each category of enrichment is adding to the data being processed?

If those numbers were available as observable behaviors of vector, then SREs operating vector can make informed decisions on its behavior and how to trade off the impact of a particular enrichment.

Further, it might be worth considering how to separately forward changes to the metadata sources used by the enrichment processes instead of enriching each piece of data. Then, vector can efficiently forward off host the core data sets with only the required metadata needed to perform enrichment later, while still efficiently providing that metadata stream for some other system (providing for a late binding, if you will, of the metadata association).

[jszwedko]

These are neat ideas, thanks @portante

Perhaps we could offer a way to track the statistical distribution of message sizes being processed through vector independent of all the enrichment, and then track separately what each category of enrichment is adding to the data being processed?

We do have some precedent for reporting event byte sizes as a counter, but we could also do a histogram.

Further, it might be worth considering how to separately forward changes to the metadata sources used by the enrichment processes instead of enriching each piece of data. Then, vector can efficiently forward off host the core data sets with only the required metadata needed to perform enrichment later, while still efficiently providing that metadata stream for some other system (providing for a late binding, if you will, of the metadata association).

Agreed, we have been thinking about how to separate metadata and event fields. I think this is something we'll look into addressing in the next year. We hadn't thought about separately publishing the metadata fields elsewhere though. That could be interesting.