Remove file source additional context text

[itayvolo]

Hi,
Im using file source, and im trying to collect log lines from a file as raw as possible.
Seems like the file source adds some "context keys" to the log lines by default, how can I remove them?
Here's an example of it:

• original log lines:

Traceback (most recent call last):
File "", line 198, in _test_test

• log lines after vector pipeline with file source:

2024-11-05T14:20:03.873658952Z stderr F Traceback (most recent call last):
2024-11-05T14:20:03.873688283Z stderr F
2024-11-05T14:20:03.874434068Z stderr F File "", line 198, in _test_test

[pront]

Hi @itayvolo,

If are you referring the the metadata 2024-11-05T14:20:03.873658952Z stderr F before the log payload (and assuming you didn't turn on log namespacing) then the log payload is stored at .message.

You can use the following to see the console output:

schema:
  log_namespace: false

data_dir: /path/to/your/data/dir

sources:
  my_file_source:
    type: file
    include:
      - /path/to/file_source.txt
    read_from: "beginning"

transforms:
  remove_metadata:
    type: remap
    inputs:
      - my_file_source
    source: |
      # Modification happens here
      . = .message

sinks:
  my_console_sink:
    type: console
    inputs:
      - remove_metadata
    encoding:
      codec: json

[itayvolo]

@pront, Thanks for the reply..
Isn't there a built-in way inside the file-source's config to at least remove the initial timestamp in the message itself?
(2024-11-05T14:20:03.873658952Z stderr F Traceback (most recent call last): → stderr F Traceback (most recent call last))

[pront]

No, metadata are added to all events. See here. It's an interesting topic though, I can see how event modifications can be surprising.

So you will have to use remap to control what you send to your sinks.

If you are interested you can try:

schema:
  log_namespace: true

Then the metadata will go to a dedicated namespace. Then the output will be exactly what you see in your source file. See details here.