Replication Snapshot and Compact

[yamingk]

Changes:

1. add snapshot creation to listener (HomeObject PR is in progress) and compact handling (called by raft and in repl dev it will persist compact_lsn as a barrier for truncation).
2. add home raft log store truncation.
3. add resource manager resource audit routine to trigger raft log store truncation as needed.
4. Add JournalVDev critical high watermark, in its callback it will trigger raft log store truncation immeidately so that it doesn't need to want for next timer audit.

Reviewers please focus on resource mgr, home raft repl log, raft repl dev and JournalVDev.
Not much logic change in other layer/files.

Testing:

1. snapshot creation and truncation can be triggered from the logs with below command line.
   ./bin/test_raft_repl_dev --gtest_filter=Snapshot_and_Compact --log_mods replication:debug --num_io=999999 --snapshot_distance=200 --num_raft_logs_resv=20000 --res_mgr_audit_timer_ms=120000
2. it also needs to be added to replication long duration test.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 10.43689% with 369 lines in your changes are missing coverage. Please review.

Project coverage is 57.73%. Comparing base (2c500c5) to head (ce5c3d4).
Report is 4 commits behind head on master.

Files	Patch %	Lines
src/lib/replication/repl_dev/raft_repl_dev.cpp	0.00%	221 Missing ⚠️
...rc/lib/replication/repl_dev/raft_state_machine.cpp	0.00%	53 Missing ⚠️
src/lib/replication/service/raft_repl_service.cpp	0.00%	33 Missing ⚠️
.../lib/replication/log_store/home_raft_log_store.cpp	0.00%	25 Missing ⚠️
src/lib/common/resource_mgr.cpp	62.85%	10 Missing and 3 partials ⚠️
src/lib/replication/log_store/repl_log_store.cpp	0.00%	9 Missing ⚠️
src/lib/replication/repl_dev/raft_repl_dev.h	0.00%	3 Missing ⚠️
src/lib/replication/repl_dev/solo_repl_dev.cpp	0.00%	3 Missing ⚠️
src/lib/replication/repl_dev/solo_repl_dev.h	0.00%	3 Missing ⚠️
src/include/homestore/replication/repl_dev.h	0.00%	2 Missing ⚠️
... and 3 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #364      +/-   ##
==========================================
- Coverage   61.26%   57.73%   -3.53%     
==========================================
  Files         135      107      -28     
  Lines       14361     9547    -4814     
  Branches     1727     1229     -498     
==========================================
- Hits         8798     5512    -3286     
+ Misses       4834     3499    -1335     
+ Partials      729      536     -193     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hkadayam]

Why do homestore need to maintain the last snapshot. Isn't upper layer going to maintain snapshot. It could potentially store multiple snapshots?

[yamingk]

You point is valid. I also thought about it, I did this only because raft is asking for a nuraft::snapshot and we don't want to expose to upper layer, and in this snapshot, there is only log_idx and log_term which is not mutable to upper layer (right?), so I didn't ask for upper layer to get the last snapshot.

Raft server only asks about last snapshot (to call corresponding compact), upper layer can very well store multiple "snapshots" as it needs which is transparent to lower layer.

[hkadayam]

I think we need to ask upper layer to create the snapshot and require upper layer snapshot to be derived from repl_snapshot and store required parameters like last_term and last_log_idx. This way how snapshot created and maintained is completely controlled by upper layer.