prefix for path in a transform object, how ?

[StefanSa]

Hi there,
Is it possible to give a prefix for the path during logfile parsing, e.g. parse_syslog.
A prefix of .sophos.xg. will then become .sophos.xg.hostname during parsing.
The general question is, is it possible to dynamically generate the path with a path prefix ?

Example kv filter:

# split Sophos-XG fields
- kv:
    field: log.original
    field_split: " (?=[a-zA-Z0-9\\_\\-]+=)"
    value_split: "="
    prefix: "sophos.xg."
    ignore_missing: true
    ignore_failure: false
    trim_value: "\"

[spencergilbert]

Hey @StefanSa I'm not sure I understand, do you mind sharing some example inputs and the outputs you want to get?

[StefanSa]

@spencergilbert surely
without path prefix:

parse_key_value!(
	"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)

{
  "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
  "id": "ConsumerFetcherManager-1382721708341",
  "level": "info",
  "module": "kafka.consumer.ConsumerFetcherManager",
  "msg": "Stopping all fetchers",
  "tag#production": "stopping_fetchers"
}

with a path prefix option, if exist.

parse_key_value!(
	"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
prefix: sophos.xg.
)

result with path prefix.

{
  "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
  "sophos.xg.id": "ConsumerFetcherManager-1382721708341",
  "sophos.xg.level": "info",
  "sophos.xg.module": "kafka.consumer.ConsumerFetcherManager",
  "sophos.xg.msg": "Stopping all fetchers",
  "sophos.xg.tag#production": "stopping_fetchers"
}

[spencergilbert]

You could do something like this, maybe:

. = set!(., path: ["sophos.xg"], data: parse_key_value!(.message))
."@timestamp" = del(."sophos.xg"."@timestamp")
flatten(.)

Which for:

.message = "@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"

I get the following output:

{
  "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
  "message": "@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager",
  "sophos.xg.id": "ConsumerFetcherManager-1382721708341",
  "sophos.xg.level": "info",
  "sophos.xg.module": "kafka.consumer.ConsumerFetcherManager",
  "sophos.xg.msg": "Stopping all fetchers",
  "sophos.xg.tag#production": "stopping_fetchers"
}

EDIT: noting that set's path option can take a path as well, so you could set based on a previously parsed or existing .hostname path or whatnot.

$ hostname = "sophos.xg"
"sophos.xg"

$ set!(., path: [hostname], data: parse_key_value!(.message))
{ "message": "@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager", "sophos.xg": { "@timestamp": "Sun Jan 10 16:47:39 EST 2021", "id": "ConsumerFetcherManager-1382721708341", "level": "info", "module": "kafka.consumer.ConsumerFetcherManager", "msg": "Stopping all fetchers", "tag#production": "stopping_fetchers" } }

[StefanSa]

@spencergilbert
Wow 👍 , this works great.
you only have to delete .message, because it will be transferred too.

[spencergilbert]

You should be able to do that in a single line like:

. = set!(., path: ["sophos.xg"], data: parse_key_value!(del(.message)))

I have a tendency of leaving the original message in, but that's probably just a preference thing 😄