Source internal_logs - Transform event to readable message

[bartweber]

To monitor Vector instances I want to ship internal logs. The event can have different fields from event to event. For example

{
  "error": "\"function call error for \\\"parse_json\\\" at (4:34): function call error for \\\"string\\\" at (16:33): expected string, got null\"",
  "error_type": "conversion_failed",
  "internal_log_rate_limit": true,
  "message": "Mapping failed with event.",
  "metadata": {
    "kind": "event",
    "level": "ERROR",
    "module_path": "vector::internal_events::remap",
    "target": "vector::internal_events::remap"
  },
  "stage": "processing",
  "timestamp": "2024-08-29T14:40:27.919533Z",
  "vector": {
    "component_id": "transform_opensearch",
    "component_kind": "transform",
    "component_type": "remap"
  }
}

compared to

{
  "arch": "x86_64",
  "debug": "false",
  "message": "Vector has started.",
  "metadata": {
    "kind": "event",
    "level": "INFO",
    "module_path": "vector::internal_events::process",
    "target": "vector"
  },
  "revision": "a9392b0 2024-08-26 14:35:19.223750502",
  "timestamp": "2024-08-29T14:45:50.259364Z",
  "version": "0.40.1"
}

Could you please provide configuration to transform this to readable log messages like the ones printed in the console? The documentation is incomplete and does not describe all the possible fields and combinations.

vector: Vector has started. debug="false" version="0.40.1" arch="x86_64" revision="a9392b0 2024-08-26 14:35:19.223750502"

transform{component_kind="transform" component_id=transform_opensearch component_type=remap}: vector::internal_events::remap: Mapping failed with event. error="function call error for \"parse_json\" at (4:34): function call error for \"string\" at (16:33): expected string, got null" error_type="conversion_failed" stage="processing" internal_log_rate_limit=true

[jszwedko]

I think you could hack something together with a remap transform, but that this would also be a reasonable feature request to be able to configure internal_logs to emit unstructured events that match the console output if you wanted to open a feature request (GitHub Issue).

I think you would need to do some conditional checks to see what format to use. For example:

• if vector exists as a key, pull the fields under that to create something like transform{component_kind="transform" component_id=transform_opensearch component_type=remap}, append .message, then encode the other fields using encode_key_value`
• Otherwise emit something like you first event

I think the two examples you found are representative of the two types of events Vector emits: some are component-level and others are topology-level (and thus won't have component level metadata).

[jszwedko]

Here's an example to get you started: https://playground.vrl.dev/?state=eyJwcm9ncmFtIjoiaWYgZXhpc3RzKC52ZWN0b3IpIHtcbiAgIyBjb21wb25lbnQgbGV2ZWxcblxuICAjIGV4dHJhY3Qga25vd24gZmllbGRzXG4gIG1ldGFkYXRhID0gZGVsKC5tZXRhZGF0YSlcbiAgdmVjdG9yID0gZGVsKC52ZWN0b3IpXG4gIG1lc3NhZ2UgPSBkZWwoLm1lc3NhZ2UpXG4gIHQgPSBkZWwoLnRpbWVzdGFtcClcblxuICAubWVzc2FnZSA9IHN0cmluZyEodmVjdG9yLmNvbXBvbmVudF9raW5kKSArIFwie1wiICsgZW5jb2RlX2tleV92YWx1ZSEodmVjdG9yKSArIFwifSBcIiArIHN0cmluZyEobWV0YWRhdGEudGFyZ2V0KSArIFwiIFwiICsgc3RyaW5nIShtZXNzYWdlKSArIGVuY29kZV9rZXlfdmFsdWUoLilcbn0gZWxzZSB7XG4gICMgdG9wb2xvZ3kgbGV2ZWxcbiAgYWJvcnRcbn0iLCJldmVudCI6eyJlcnJvciI6IlwiZnVuY3Rpb24gY2FsbCBlcnJvciBmb3IgXFxcInBhcnNlX2pzb25cXFwiIGF0ICg0OjM0KTogZnVuY3Rpb24gY2FsbCBlcnJvciBmb3IgXFxcInN0cmluZ1xcXCIgYXQgKDE2OjMzKTogZXhwZWN0ZWQgc3RyaW5nLCBnb3QgbnVsbFwiIiwiZXJyb3JfdHlwZSI6ImNvbnZlcnNpb25fZmFpbGVkIiwiaW50ZXJuYWxfbG9nX3JhdGVfbGltaXQiOnRydWUsIm1lc3NhZ2UiOiJNYXBwaW5nIGZhaWxlZCB3aXRoIGV2ZW50LiIsIm1ldGFkYXRhIjp7ImtpbmQiOiJldmVudCIsImxldmVsIjoiRVJST1IiLCJtb2R1bGVfcGF0aCI6InZlY3Rvcjo6aW50ZXJuYWxfZXZlbnRzOjpyZW1hcCIsInRhcmdldCI6InZlY3Rvcjo6aW50ZXJuYWxfZXZlbnRzOjpyZW1hcCJ9LCJzdGFnZSI6InByb2Nlc3NpbmciLCJ0aW1lc3RhbXAiOiIyMDI0LTA4LTI5VDE0OjQwOjI3LjkxOTUzM1oiLCJ2ZWN0b3IiOnsiY29tcG9uZW50X2lkIjoidHJhbnNmb3JtX29wZW5zZWFyY2giLCJjb21wb25lbnRfa2luZCI6InRyYW5zZm9ybSIsImNvbXBvbmVudF90eXBlIjoicmVtYXAifX0sImlzX2pzb25sIjpmYWxzZSwiZXJyb3IiOm51bGx9

Hopefully you can see where I'm going from there.