Persistence - Remove Unowned Keys

[singku]

Proposing this PR per issue discussion:
#539

For cluster, after startup loading, remove keys
that shouldn't be served by this server based on slot assignment of a cluster.

Also added stat fields in server to count the total removed keys and skipped slots from last loading.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.59%. Comparing base (aaa7834) to head (3299f99).
Report is 133 commits behind head on unstable.

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable     #568      +/-   ##
============================================
+ Coverage     70.35%   70.59%   +0.23%     
============================================
  Files           112      114       +2     
  Lines         61467    61703     +236     
============================================
+ Hits          43248    43562     +314     
+ Misses        18219    18141      -78     

Files with missing lines	Coverage Δ
src/cluster.c	88.26% <100.00%> (-0.09%)	⬇️
src/server.c	88.63% <100.00%> (+0.07%)	⬆️
src/server.h	100.00% <ø> (ø)

... and 58 files with indirect coverage changes

[madolson]

We also should implement a test for this as well. I know it will probably be a bit tricky. I'm thinking of:

1. Create an RDB file with data in all slots.
2. Start a node with it that has a nodes.conf that doesn't own all the slots.

[singku]

We also should implement a test for this as well. I know it will probably be a bit tricky. I'm thinking of:

1. Create an RDB file with data in all slots.
2. Start a node with it that has a nodes.conf that doesn't own all the slots.

If we move the deletion after verifyClusterConfigWithData we can't simply do this test because the unassigned slot with keys will be reclaimed by the node. We can only test it in cluster mode with multiple shards, as well performing slot migration in the test

[singku]

@madolson gently ping for additional input

[singku]

@hpatro Thanks for reviewing this PR, I just rebased it to resolve conflicts and applied changes per your comments, please take a look.

[hpatro]

Mostly LGTM, thanks @singku!

[hpatro]

@madolson Could you take a pass? LGTM.

[singku]

@madolson Could you take a pass? LGTM.

Gentle Ping..

[PingXie]

For a more generic solution, I think we need to shed unowned keys during loading. The current fix drops the unowned keys after the loading is complete but the node might have already OOM'd before reaching the point.

We also should implement a test for this as well.

+1

If we move the deletion after verifyClusterConfigWithData we can't simply do this test because the unassigned slot with keys will be reclaimed by the node. We can only test it in cluster mode with multiple shards, as well performing slot migration in the test

This change does need to be tested in cluster mode. Have you tried out @madolson's recommendation?