Tracking potential bottlenecks & performance improvements

[qwahzi]

Summary

The logging framework parser has been implemented (#4261, based on #4260), so deeper analysis can be done to identify bottlenecks & improve performance. The ultimate goal is to help achieve reliably steady prioritization & TPS, even under saturation and/or spam.

Merged / V26

• Unnecessary block read on LMDB
• Unnecessary block read on RocksDB
• Block uniquer performance improvement
• Vote uniquer performance improvement
• Reduce steady_clock sampling
• Hinted scheduler improvements (activating elections for missing blocks/dependencies)
• Hinted scheduler bug related to bootstrapping missing blocks on slow nodes
• Removing separate signature checking thread classes
• Round-robin connection attempt processing for simple load-balancing
• Increasing votes per message to 255+ (server-side)
• Bulk frontier scanning for bootstrapping (server-side)

Merged / V27

• Improving block rebroadcasting & gossip
• Increasing votes per message to 255+ (client-side)
• Fair queueing for block processor
• Fair queueing for vote processor
• Fair queueing for bootstrap server
• Improving AEC vacancy tracking (removal strategies, update frequency changes, etc)
• Multi-threaded request aggregator
• Rep crawler overhaul
• Remove final vote canary code as final votes are now always enabled

Potential mid-term performance improvements

• Bulk frontier scanning for bootstrapping (client-side)
• Bounded block backlog
  • Also separating block checking from ledger insertion, so only confirmed blocks get written to disk (unconfirmed blocks would stay in memory)
• Outbound traffic shaping (V28+)
• Bounded block backlog (V28+) - "Mempool"; Only write confirmed transactions to disk

Potential long-term performance improvements

• Improving peer connection strategies (e.g. tiered peer connections by weight)
• Replace boost.PropertyTree with a json library (e.g. nlohmann) for improved logging/RPC performance
• Splitting the block table
• Improving prioritization to prevent backlog formation (congestion control / "leaky bucket" avoidance)
• Prioritizing incoming vote processing over generation (correlations between process_confirmed & confirm_ack_received)
• Self-balancing traffic algorithm to balance outgoing traffic vs incoming traffic
• Prevent bootstrapping nodes from republishing old/historical votes while bootstrapping
• Vote storage
• Improving AEC Alignment
• Improve confirm_req/request_aggregator (reduce IO lookups & prevent dropping large vote requests)
• Consensus improvements
• Additional malicious/misbehaving node rules
• Time-bounded PoW
• Removing/drastically reducing PoW
• Network/transport layer refactor / QUIC protocol
• Auto-receiving transactions (long-term, requires protocol change)
• FoundationDB / Distributed DB?
• Horizontal scaling / Sharding?
• Re-evaluating multi-threading certain aspects of the node

[qwahzi]

EDIT - Comment merged with above issue description