How to handle a json array log

[satscreate]

Input:
sample.jsonl
[{"id":"1","val":"1.0"},{"id":"2","val":"2.0"}]

vector.yaml:
sources:
stdin:
type: stdin
decoding:
codec: json

transforms:
patch:
type: remap
inputs:
- stdin
drop_on_error: true
source: |-
id= string!(.id) <---------- This reads only ID. How to read whole object & each json object one by one and store like id
log("json message " + id , level: "warn") <----------- This prints ID 1 but How to get the whole JSON first line Object?
sinks:
stdout:
type: console
inputs:
- patch
target: stdout
encoding:
codec: json

Running :
jq -c '.[0]' sample.jsonl | vector -c vector.yaml

How to deal with base source file as JSON array input and how to read the whole json object (1st object) and keep it in memory?

[spencergilbert]

Hi @satscreate,

First off I think you "running" command is a bit off, you should use jq -c '.' sample.jsonl | vector -c vector.yaml instead. The json decoder on your stdin source is going to take your array of objects and parse them into two events:

jq -c '.' sample.jsonl | vector -c vector.yaml
2022-02-08T16:41:38.530333Z  INFO vector::app: Log level is enabled.
...
{"host":"COMP-C02F514NML87","id":"1","source_type":"stdin","timestamp":"2022-02-08T16:41:38.532531Z","val":"1.0"}
{"host":"COMP-C02F514NML87","id":"2","source_type":"stdin","timestamp":"2022-02-08T16:41:38.532531Z","val":"2.0"}

Today the decoding feature is always going to parse and merge the object into the root of the event, for example your object of {"id":"1","val":"1.0"} is parsed into two paths (as seen in the events above), .id and .val. You could rebuild the object yourself if you needed them joined though, the following remap config:

transforms:
  patch:
    type: remap
    inputs:
      - stdin
    source: |-
      obj = {"id": .id, "val": .val}
      log(obj, level: "warn", rate_limit_secs: 0)

... results in:

2022-02-08T16:46:56.181324Z  WARN vrl_stdlib::log: { "id": "1", "val": "1.0" } internal_log_rate_secs=0 vrl_position=31
2022-02-08T16:46:56.181352Z  WARN vrl_stdlib::log: { "id": "2", "val": "2.0" } internal_log_rate_secs=0 vrl_position=31
{"host":"COMP-C02F514NML87","id":"1","source_type":"stdin","timestamp":"2022-02-08T16:46:56.181147Z","val":"1.0"}
{"host":"COMP-C02F514NML87","id":"2","source_type":"stdin","timestamp":"2022-02-08T16:46:56.181147Z","val":"2.0"}

BTW, you can use use triple backticks to code format in Github - ```