Filelog Receiver Not Handling Multiline Logs with line_start_pattern.

[Rahuly360]

I have a mysql container logs containing multiple logs like Error logs , slow query logs and server logs together.
I need to parse slow query logs but it is not working .

Example logs :
2025-01-05T20:11:50.970826-00:00 341377 [Note] [MY-010914] [Server] Aborted connection 341377 to db: 'TEST' user: 'TEST' host: 'TEST_HOST' (The client was disconnected by the server because of inactivity.).

==> /mysql_datadir/mysql_logs/slow_query.log <==

Time: 2025-01-05T20:11:54.389753-00:00

User@Host: mysql_backup[mysql_backup] @ [HOST_IP] Id: 341646

Query_time: 3.118553 Lock_time: 0.000001 Rows_sent: 191444 Rows_examined: 191444 Thread_id: 341646 Errno: 0 Killed: 0 Bytes_received: 60 Bytes_sent: 31005283 Read_first: 1 Read_last: 0 Read_key: 1 Read_next: 0 Read_prev: 0 Read_rnd: 0 Read_rnd_next: 191445 Sort_merge_passes: 0 Sort_range_count: 0 Sort_rows: 0 Sort_scan_count: 0 abc: 0 abc: 0 Start: 2025-01-05T20:11:51.271200-00:00 End: 2025-01-05T20:11:54.389753-00:00

use TEST;
SET timestamp=1736107911;
SELECT ABC FROM ABC_Table;

==> /mysql_datadir/mysql_logs/mysqld.log <==
2025-01-05T20:12:00.224997-00:00 341658 [Note] [MY-010914] [Server] Got packets out of order
2025-01-05T20:12:06.023456-00:00 341383 [Note] [MY-013730] [Server] 'wait_timeout' period of 600 seconds was exceeded for TEST_Service@%. The idle time since last command was too long.

I need to read slow query logs only
like:

Time: 2025-01-05T20:11:54.389753-00:00

User@Host: mysql_backup[mysql_backup] @ [HOST_IP] Id: 341646

Query_time: 3.118553 Lock_time: 0.000001 Rows_sent: 191444 Rows_examined: 191444 Thread_id: 341646 Errno: 0 Killed: 0 Bytes_received: 60 Bytes_sent: 31005283 Read_first: 1 Read_last: 0 Read_key: 1 Read_next: 0 Read_prev: 0 Read_rnd: 0 Read_rnd_next: 191445 Sort_merge_passes: 0 Sort_range_count: 0 Sort_rows: 0 Sort_scan_count: 0 abc: 0 abc: 0 Start: 2025-01-05T20:11:51.271200-00:00 End: 2025-01-05T20:11:54.389753-00:00

use TEST;
SET timestamp=1736107911;
SELECT ABC FROM ABC_Table;

Receiver Configuration:
filelog:
exclude:
- some_files
include:
- some_files_containing_container_logs
start_at: beginning
include_file_name: true
include_file_path: true
multiline:
line_start_pattern: '^# Time:'
operators:
- type: regex_parser
regex: '^# Time:\s*(?P[^\n]+)\n# User@Host:\s*(?P[^[]+)[(?P<user_role>[^\]]+)]\s*@\s*(?P[^\s]+)\s*[]\sId:\s(?P\d+)\n# Query_time:\s*(?P<query_time>\d+.\d+)\sLock_time:\s(?P<lock_time>\d+.\d+)\sRows_sent:\s(?P<rows_sent>\d+)\sRows_examined:\s(?P<rows_examined>\d+)\sThread_id:\s(?P<thread_id>\d+)\sErrno:\s(?P\d+)\sKilled:\s(?P\d+)\sBytes_received:\s(?P<bytes_received>\d+)\sBytes_sent:\s(?P<bytes_sent>\d+)\sRead_first:\s(?P<read_first>\d+)\sRead_last:\s(?P<read_last>\d+)\sRead_key:\s(?P<read_key>\d+)\sRead_next:\s(?P<read_next>\d+)\sRead_prev:\s(?P<read_prev>\d+)\sRead_rnd:\s(?P<read_rnd>\d+)\sRead_rnd_next:\s(?P<read_rnd_next>\d+)\sSort_merge_passes:\s(?P<sort_merge_passes>\d+)\sSort_range_count:\s(?P<sort_range_count>\d+)\sSort_rows:\s(?P<sort_rows>\d+)\sSort_scan_count:\s(?P<sort_scan_count>\d+)\sCreated_tmp_disk_tables:\s(?P<created_tmp_disk_tables>\d+)\sCreated_tmp_tables:\s(?P<created_tmp_tables>\d+)\sStart:\s(?P<start_time>[^\n]+)\sEnd:\s(?P<end_time>[^\n]+)\nSET\stimestamp=(?P\d+);\n(?P[\s\S]+)$'

[Rahuly360]

@VihasMakwana @djaglowski Can you please help here

[VihasMakwana]

@Rahuly360 Are you facing any errors?

[VihasMakwana]

I just noticed that you have

Time: 2025-01-05T20:11:54.389753-00:00

But your multiline pattern has ^# Time: and hence, it's not working.
Please use https://regex101.com/ and figure out the right regex.

[Rahuly360]

@VihasMakwana before the
Time
User
Query_time
there is # but it is not showing in description
you can check logs here
https://mysql.wisborg.dk/2019/01/31/more-statistics-for-slow-queries-log_slow_extra/

``

[github-actions]

Pinging code owners for receiver/filelog: @djaglowski. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.