system.log -> messages.log scraping can start lagging badly after connection is lost

[piodul]

Argus run that prompted this issue: https://argus.scylladb.com/tests/scylla-cluster-tests/83efb5fa-0232-4b55-a1e1-219764056cee. I have left a comment in the discussion there which explains the problem in more detail. Summarizing:

• An event happens which causes Scylla to generate a relatively large amount of logs (it was nodetool rebuild in this run),
• The system.log -> messages.log scraping mechanism (I think it's called syslog-ng) loses connection to the node, reconnects after 60s but keeps lagging badly afterwards; I saw that one line was delayed by 16 minutes in that particular run,
• The nemesis code which waits for a log line to appear times out, even though it appeared in system.log pretty quickly, leading to test flakiness.

[fruch]

I would argue it's a scylla bug, if it get to more 100K per sec log in one node

also, can you suggest a different way to identify a rebuild has start, without looking at logs ?

[roydahan]

What are these logs (the sudden burst) about?