LedgerDB: prune on garbage collection instead of on every change

[amesgen]

This is in preparation for #1424

This PR is intended to be reviewed commit-by-commit.

Currently, we prune the LedgerDB (ie remove all but the last k+1 states) every time we adopt a longer chain. This means that we can not rely on the fact that other threads (like the copyAndSnapshot ChainDB background) actually observe all immutable ledger states, just as described in the caveats of our Watcher abstraction.

However, a predictable ledger snapshotting rule (#1424) requires this property; otherwise, when the node is under high load and/or we are adopting multiple blocks in quick succession, the node might not be able to create a snapshot for its desired block.

This PR changes this fact: Now, when adopting new blocks, the LedgerDB is not immediately pruned. Instead, the a new dedicated background thread for ledger maintenance tasks (flushing/snapshotting/garbage collection) in the ChainDB will periodically (on every new immutable block) wake up and (in particular) garbage collect the LedgerDB based on a slot number.

Also, this makes the semantics more consistent with the existing garbage collection of previously-applied blocks in the LedgerDB, and also with how the ChainDB works, where we also don't immediately delete blocks from the VolatileDB once they are buried beneath k+1 blocks.

See #1513 (comment) for benchmarks demonstrating that the peak memory usage does not increase while syncing (where we now briefly might hold more than k+1 ledger states in memory).

[jasagredo]

Looks good.

[amesgen]

Sync benchmarks are looking good (mainnet, first 1e6 slots/blocks):

LMDB benchmark (of course, this is a bit degenerate as Byron doesn't have tables, but this still serves as a regression test for the DbChangelog aspects which are touched by this PR).

Note that d1b6215 is crucial; otherwise, there is a significant (2x) regression in max heap size.

[geo2a]

The changelog entry here seems misleading, but please correct me if I'm wrong.

If I understand correctly, we are now pruning the LedgerDB in copyAndSnapshotRunner, i.e. when moving blocks from VolatileDB to ImmutableDB. In the scheduled ChainDB GCs, we will not prune the LedgerDB at all.

[amesgen]

Good catch, I removed the "ChainDB" part here (it happens as part of LedgerDB garbage collection, which we execute as part of the copyAndSnapshot background thread in the ChainDB 👍)

---

Based on this discussion and your justified confusion in #1513 (comment): Maybe we actually want to have a separate ChainDB background thread that does LedgerDB snapshotting + pruning?

• Pro: clearer separation of concerns; taking a snapshot does not block copying blocks to the ImmutableDB (this improvement is independent of this PR)

  Especially the latter point might be more relevant with Optional random delay when creating snapshots #1573

• Con: more ceremony, more threads mean more RTS contention (seems weak)

[amesgen]

FTR we discussed this in the sync today, and decided that a new ChainDB background thread makes sense.

[amesgen]

Done in d1b6215

[amesgen]

Thanks for the great reviews, I hope I addressed your comments. Interesting changes:

• d627e84 for removing LedgerDbPruneKeeping prompted by LedgerDB: prune on garbage collection instead of on every change #1513 (comment)
• Found a bug in the LedgerDB state machine test (LedgerDB.StateMachine test: actually test rollbacks #1576) which this PR now depends on as I enriched it due to LedgerDB: prune on garbage collection instead of on every change #1513 (comment)