Exploding the array of JSON objects #10391

omurov · 2021-12-10T16:00:10Z

omurov
Dec 10, 2021

How can we explode/unnest the array of JSON objects, for example:

{ "message": "[ { "file": "/var/log/nginx/API.access.log", "host": "platform-testing-api2", "json": { "agent": "Go-http-client/1.1", "body_sent": { "bytes": 35 }, "http_version": "HTTP/1.1", "method": "GET", "referrer": "", "remote_ip": "172.17.0.3", "request_time": 0.052, "response_code": 200, "time": "2021-12-10T15:27:09+02:00", "upstream_cache_status": "MISS", "url": "/api/health/dino", "user_name": "" }, "source_type": "file", "timestamp": "2021-12-10T13:27:10.456924207Z" }, { "file": "/var/log/nginx/API.access.log", "host": "platform-testing-api2", "json": { "agent": "", "body_sent": { "bytes": 0 }, "http_version": "HTTP/1.0", "method": "GET", "referrer": "", "remote_ip": "172.18.201.21", "request_time": 0.015, "response_code": 200, "time": "2021-12-10T15:27:10+02:00", "upstream_cache_status": "", "url": "/api/health", "user_name": "" }, "source_type": "file", "timestamp": "2021-12-10T13:27:10.975833700Z" } ]" }

When we use:
. = parse_json!(.message)

we get the following:
[{ "file": "/var/log/nginx/API.access.log", "host": "platform-testing-api2", "json": { "agent": "Go-http-client/1.1", "body_sent": { "bytes": 35 }, "http_version": "HTTP/1.1", "method": "GET", "referrer": "", "remote_ip": "172.17.0.3", "request_time": 0.052, "response_code": 200, "time": "2021-12-10T15:27:09+02:00", "upstream_cache_status": "MISS", "url": "/api/health/dino", "user_name": "" }, "source_type": "file", "timestamp": "2021-12-10T13:27:10.456924207Z" }, { "file": "/var/log/nginx/API.access.log", "host": "platform-testing-api2", "json": { "agent": "", "body_sent": { "bytes": 0 }, "http_version": "HTTP/1.0", "method": "GET", "referrer": "", "remote_ip": "172.18.201.21", "request_time": 0.015, "response_code": 200, "time": "2021-12-10T15:27:10+02:00", "upstream_cache_status": "", "url": "/api/health", "user_name": "" }, "source_type": "file", "timestamp": "2021-12-10T13:27:10.975833700Z" }]

How can we explode this dynamically? as we can get 1..N JSON messages inside an array.

Answered by spencergilbert

Dec 10, 2021

. = parse_json!(.message) and getting an array of objects should pass each object as a separate event to the next component, is that not the behavior you're seeing?

There is also an unnest function.

View full answer

spencergilbert · 2021-12-10T17:08:06Z

spencergilbert
Dec 10, 2021
Maintainer

. = parse_json!(.message) and getting an array of objects should pass each object as a separate event to the next component, is that not the behavior you're seeing?

There is also an unnest function.

0 replies

omurov · 2021-12-10T18:06:55Z

omurov
Dec 10, 2021
Author

Hello Spencer, Thank you for your answer, we have tried those, but it does not split this into separate events. If I test run it and output to the file it writes the whole array to one line. For unnset there is no path that I can provide, as it’s just an array of json objects, unnset asks for the path to explode based on that. Regards, Alex

…

On Fri, 10 Dec 2021 at 18:08, Spencer Gilbert ***@***.***> wrote: . = parse_json!(.message) and getting an array of objects should pass each object as a separate event to the next component, is that not the behavior you're seeing? There is also an unnest <https://vector.dev/docs/reference/vrl/functions/#unnest> function. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#10391 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AL6HI7JQPVPKFNIKF2H6KCDUQIXYDANCNFSM5JZO5OKA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

1 reply

spencergilbert Dec 10, 2021
Maintainer

Given this log and config:

❯ cat /tmp/explode.log
[ { "file": "/var/log/nginx/API.access.log", "host": "platform-testing-api2", "json": { "agent": "Go-http-client/1.1", "body_sent": { "bytes": 35 }, "http_version": "HTTP/1.1", "method": "GET", "referrer": "", "remote_ip": "172.17.0.3", "request_time": 0.052, "response_code": 200, "time": "2021-12-10T15:27:09+02:00", "upstream_cache_status": "MISS", "url": "/api/health/dino", "user_name": "" }, "source_type": "file", "timestamp": "2021-12-10T13:27:10.456924207Z" }, { "file": "/var/log/nginx/API.access.log", "host": "platform-testing-api2", "json": { "agent": "", "body_sent": { "bytes": 0 }, "http_version": "HTTP/1.0", "method": "GET", "referrer": "", "remote_ip": "172.18.201.21", "request_time": 0.015, "response_code": 200, "time": "2021-12-10T15:27:10+02:00", "upstream_cache_status": "", "url": "/api/health", "user_name": "" }, "source_type": "file", "timestamp": "2021-12-10T13:27:10.975833700Z" } ]

❯ cat /tmp/explode.yaml
api:
  enabled: true
sources:
  file:
    type: file
    include: ["/tmp/explode.log"]
    ignore_checkpoints: true
transforms:
  remap:
    type: remap
    inputs: ["file"]
    source: |
      . = parse_json!(.message)
sinks:
  console:
    type: console
    inputs: ["remap"]
    encoding:
      codec: json

I get two separate events written to the console sink:

{"file":"/var/log/nginx/API.access.log","host":"platform-testing-api2","json":{"agent":"Go-http-client/1.1","body_sent":{"bytes":35},"http_version":"HTTP/1.1","method":"GET","referrer":"","remote_ip":"172.17.0.3","request_time":0.052,"response_code":200,"time":"2021-12-10T15:27:09+02:00","upstream_cache_status":"MISS","url":"/api/health/dino","user_name":""},"source_type":"file","timestamp":"2021-12-10T13:27:10.456924207Z"}
{"file":"/var/log/nginx/API.access.log","host":"platform-testing-api2","json":{"agent":"","body_sent":{"bytes":0},"http_version":"HTTP/1.0","method":"GET","referrer":"","remote_ip":"172.18.201.21","request_time":0.015,"response_code":200,"time":"2021-12-10T15:27:10+02:00","upstream_cache_status":"","url":"/api/health","user_name":""},"source_type":"file","timestamp":"2021-12-10T13:27:10.975833700Z"}

omurov · 2021-12-10T18:31:47Z

omurov
Dec 10, 2021
Author

Thank you, will try it again. For some reason writing to file does not separate those lines.

…

On Fri, 10 Dec 2021 at 19:22, Spencer Gilbert ***@***.***> wrote: Given this log and config: ❯ cat /tmp/explode.log [ { "file": "/var/log/nginx/API.access.log", "host": "platform-testing-api2", "json": { "agent": "Go-http-client/1.1", "body_sent": { "bytes": 35 }, "http_version": "HTTP/1.1", "method": "GET", "referrer": "", "remote_ip": "172.17.0.3", "request_time": 0.052, "response_code": 200, "time": "2021-12-10T15:27:09+02:00", "upstream_cache_status": "MISS", "url": "/api/health/dino", "user_name": "" }, "source_type": "file", "timestamp": "2021-12-10T13:27:10.456924207Z" }, { "file": "/var/log/nginx/API.access.log", "host": "platform-testing-api2", "json": { "agent": "", "body_sent": { "bytes": 0 }, "http_version": "HTTP/1.0", "method": "GET", "referrer": "", "remote_ip": "172.18.201.21", "request_time": 0.015, "response_code": 200, "time": "2021-12-10T15:27:10+02:00", "upstream_cache_status": "", "url": "/api/health", "user_name": "" }, "source_type": "file", "timestamp": "2021-12-10T13:27:10.975833700Z" } ] ❯ cat /tmp/explode.yaml api: enabled: true sources: file: type: file include: ["/tmp/explode.log"] ignore_checkpoints: true transforms: remap: type: remap inputs: ["file"] source: | . = parse_json!(.message) sinks: console: type: console inputs: ["remap"] encoding: codec: json I get two separate events written to the console sink: {"file":"/var/log/nginx/API.access.log","host":"platform-testing-api2","json":{"agent":"Go-http-client/1.1","body_sent":{"bytes":35},"http_version":"HTTP/1.1","method":"GET","referrer":"","remote_ip":"172.17.0.3","request_time":0.052,"response_code":200,"time":"2021-12-10T15:27:09+02:00","upstream_cache_status":"MISS","url":"/api/health/dino","user_name":""},"source_type":"file","timestamp":"2021-12-10T13:27:10.456924207Z"} {"file":"/var/log/nginx/API.access.log","host":"platform-testing-api2","json":{"agent":"","body_sent":{"bytes":0},"http_version":"HTTP/1.0","method":"GET","referrer":"","remote_ip":"172.18.201.21","request_time":0.015,"response_code":200,"time":"2021-12-10T15:27:10+02:00","upstream_cache_status":"","url":"/api/health","user_name":""},"source_type":"file","timestamp":"2021-12-10T13:27:10.975833700Z"} — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#10391 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AL6HI7KAJ4ETT5MFBDINVGLUQJAM7ANCNFSM5JZO5OKA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

1 reply

spencergilbert Dec 10, 2021
Maintainer

Interesting, I replaced the console out with:

sinks:
  file_out:
    type: file
    inputs: ["remap"]
    path: /tmp/vector-%Y-%m-%d.log
    encoding:
      codec: ndjson

And still get the same behavior:

❯ cat /tmp/vector-2021-12-10.log
{"file":"/var/log/nginx/API.access.log","host":"platform-testing-api2","json":{"agent":"Go-http-client/1.1","body_sent":{"bytes":35},"http_version":"HTTP/1.1","method":"GET","referrer":"","remote_ip":"172.17.0.3","request_time":0.052,"response_code":200,"time":"2021-12-10T15:27:09+02:00","upstream_cache_status":"MISS","url":"/api/health/dino","user_name":""},"source_type":"file","timestamp":"2021-12-10T13:27:10.456924207Z"}
{"file":"/var/log/nginx/API.access.log","host":"platform-testing-api2","json":{"agent":"","body_sent":{"bytes":0},"http_version":"HTTP/1.0","method":"GET","referrer":"","remote_ip":"172.18.201.21","request_time":0.015,"response_code":200,"time":"2021-12-10T15:27:10+02:00","upstream_cache_status":"","url":"/api/health","user_name":""},"source_type":"file","timestamp":"2021-12-10T13:27:10.975833700Z"}

Definitely interested to see what's happening. Please share a complete config (if you can) if the issue continues.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exploding the array of JSON objects #10391

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Exploding the array of JSON objects #10391

omurov Dec 10, 2021

Replies: 3 comments · 2 replies

spencergilbert Dec 10, 2021 Maintainer

omurov Dec 10, 2021 Author

spencergilbert Dec 10, 2021 Maintainer

omurov Dec 10, 2021 Author

spencergilbert Dec 10, 2021 Maintainer

omurov
Dec 10, 2021

Replies: 3 comments 2 replies

spencergilbert
Dec 10, 2021
Maintainer

omurov
Dec 10, 2021
Author

spencergilbert Dec 10, 2021
Maintainer

omurov
Dec 10, 2021
Author

spencergilbert Dec 10, 2021
Maintainer