Log cannot format correct

[SkyCTing]

Bug Report

Describe the bug
I have more than one servers，I need to collect logs from each server by fluent bit and sent it to fluentd,then insert to mysql and save it to file as a copy.but i failed.

To Reproduce

• my original log is like this:

{"message":"visit log","context":{"user_id":70,"channel_id":"73","click_id":"202409112012ebl19934","nm_id":"23654","cm_id":"2099450","am_id":"20279","app_id":1,"ip_addr":"89.135.131.81","tracking_code":"2kelb7b78ehj5","ua":"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36","device_type":4,"url":"https://www.google.com","date":"2024-09-11","hostname":"e29df85ad33a"},"level":200,"level_name":"INFO","channel":"production","datetime":"2024-09-11 13:55:00","extra":{}}

• Steps to reproduce the problem:
  my config file of fluent bit is like this

[SERVICE]
    Flush         5
    Daemon        Off
    Log_Level     info

[INPUT]
    Name          tail
    Path          /data/visit-log*.log
    Tag           visit_log
    Parser        json

[OUTPUT]
    Name          forward
    Match         visit_log
    Host          secret
    Port          secret

my config file of fluentd is like this

<source>
  @type forward
  port secret
</source>

<filter visit_log>
  @type record_transformer
  enable_ruby
</filter>

<match visit_log>
  @type file
  path /data/log/visit_log/
  <format>
    @type json
  </format>
</match>

everything is ok now ,but the log i got in fluentd server is like this

{"log":"{\"message\":\"visit log\",\"context\":{\"user_id\":70,\"channel_id\":\"73\",\"click_id\":\"20240911254e89n1ad7a\",\"nm_id\":\"36451\",\"cm_id\":\"2107671\",\"am_id\":\"28737\",\"app_id\":1,\"ip_addr\":\"213.37.122.34\",\"tracking_code\":\"237d11s78f7i5\",\"ua\":\"Mozilla/5.0 (iPhone; CPU iPhone OS 17_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Mobile/15E148 Safari/604.1\",\"device_type\":1,\"url\":\"https://www.google.com",\"date\":\"2024-09-11\",\"hostname\":\"e29df85ad33a\"},\"level\":200,\"level_name\":\"INFO\",\"channel\":\"production\",\"datetime\":\"2024-09-11 14:53:45\",\"extra\":{}}"}

Expected behavior

I hope the result may like this

{"message":"visit log","context":{"user_id":70,"channel_id":"73","click_id":"202409112012ebl19934","nm_id":"23654","cm_id":"2099450","am_id":"20279","app_id":1,"ip_addr":"89.135.131.81","tracking_code":"2kelb7b78ehj5","ua":"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36","device_type":4,"url":"https://www.google.com","date":"2024-09-11","hostname":"e29df85ad33a"},"level":200,"level_name":"INFO","channel":"production","datetime":"2024-09-11 13:55:00","extra":{}}

or like this

{"log":{"message":"visit log","context":{"user_id":70,"channel_id":"73","click_id":"202409112012ebl19934","nm_id":"23654","cm_id":"2099450","am_id":"20279","app_id":1,"ip_addr":"89.135.131.81","tracking_code":"2kelb7b78ehj5","ua":"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36","device_type":4,"url":"https://www.google.com","date":"2024-09-11","hostname":"e29df85ad33a"},"level":200,"level_name":"INFO","channel":"production","datetime":"2024-09-11 13:55:00","extra":{}}}

I cannot read the log now ,the content inside is not a valid json,I read the document the make the Parser as json,it does not work

help me ,thank you every one.
Screenshots

Your Environment

• Version used:
  the version of flurned is :fluent-package 5.0.4 fluentd 1.16.5
  the version of fluent bit is:Fluent Bit v3.1.7
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?):
• Server type and version:
• Operating System and version:
• Filters and plugins:

Additional context

[patrick-stephens]

I simplified the original problem I think, it looks like the JSON parser is being ignored and injecting into log key instead of parsing.

test.log

{"message":"visit log","context":{"user_id":70,"channel_id":"73","click_id":"202409112012ebl19934","nm_id":"23654","cm_id":"2099450","am_id":"20279","app_id":1,"ip_addr":"89.135.131.81","tracking_code":"2kelb7b78ehj5","ua":"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36","device_type":4,"url":"https://www.google.com","date":"2024-09-11","hostname":"e29df85ad33a"},"level":200,"level_name":"INFO","channel":"production","datetime":"2024-09-11 13:55:00","extra":{}}

Config and output

$ cat fluent-bit.conf
[INPUT]
  Name Tail
  Path /logs/*
  Read_from_head true
  Parser json

[OUTPUT]
  Name stdout
  Match *

$ docker run --rm -it -v $PWD/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro -v $PWD/test.log:/logs/test.log:ro fluent/fluent-bit:3.1.7
...
[2024/09/11 09:27:15] [ info] [input:tail:tail.0] inotify_fs_add(): inode=5 watch_fd=1 name=/logs/test.log
[0] tail.0: [[1726046835.693655839, {}], {"log"=>"{"message":"visit log","context":{"user_id":70,"channel_id":"73","click_id":"202409112012ebl19934","nm_id":"23654","cm_id":"2099450","am_id":"20279","app_id":1,"ip_addr":"89.135.131.81","tracking_code":"2kelb7b78ehj5","ua":"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36","device_type":4,"url":"https://www.google.com","date":"2024-09-11","hostname":"e29df85ad33a"},"level":200,"level_name":"INFO","channel":"production","datetime":"2024-09-11 13:55:00","extra":{}}"}]
[1] tail.0: [[1726046835.693658048, {}], {"log"=>""}]

[patrick-stephens]

Sub-note the JSON seems to include numbers both as strings and numerals plus an empty structure.

[SkyCTing]

I simplified the original problem I think, it looks like the JSON parser is being ignored and injecting into log key instead of parsing.

test.log

{"message":"visit log","context":{"user_id":70,"channel_id":"73","click_id":"202409112012ebl19934","nm_id":"23654","cm_id":"2099450","am_id":"20279","app_id":1,"ip_addr":"89.135.131.81","tracking_code":"2kelb7b78ehj5","ua":"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36","device_type":4,"url":"https://www.google.com","date":"2024-09-11","hostname":"e29df85ad33a"},"level":200,"level_name":"INFO","channel":"production","datetime":"2024-09-11 13:55:00","extra":{}}

Config and output

$ cat fluent-bit.conf
[INPUT]
  Name Tail
  Path /logs/*
  Read_from_head true
  Parser json

[OUTPUT]
  Name stdout
  Match *

$ docker run --rm -it -v $PWD/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro -v $PWD/test.log:/logs/test.log:ro fluent/fluent-bit:3.1.7
...
[2024/09/11 09:27:15] [ info] [input:tail:tail.0] inotify_fs_add(): inode=5 watch_fd=1 name=/logs/test.log
[0] tail.0: [[1726046835.693655839, {}], {"log"=>"{"message":"visit log","context":{"user_id":70,"channel_id":"73","click_id":"202409112012ebl19934","nm_id":"23654","cm_id":"2099450","am_id":"20279","app_id":1,"ip_addr":"89.135.131.81","tracking_code":"2kelb7b78ehj5","ua":"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36","device_type":4,"url":"https://www.google.com","date":"2024-09-11","hostname":"e29df85ad33a"},"level":200,"level_name":"INFO","channel":"production","datetime":"2024-09-11 13:55:00","extra":{}}"}]
[1] tail.0: [[1726046835.693658048, {}], {"log"=>""}]

Yes，the key problem is JSON parser does not work,even the message is like {"message":"visit log"}