E35 prune small batches

[awskii]

Follow up to #9031 (comment)

Ordering

AggregatorV3Context pruning is happening in following order:

1. Index pruning started from lowest txNum such that txFrom <= txn <= txTo. Progress is going towards bigger txNumbers.
2. Therefore, History pruning goes in same direction and happens along with key pruning via callback.
3. Domain pruning starts from Latest() key which is the biggest key available. We use inverted steps (^step) as a suffix for domain keys which gives us an opportunity to prune smallest steps first. So, from largest available key and smallest available step going backwards to bigger steps and smaller keys. If for given key we met savedStep > pruneStep we safely going to PrevNoDup() key without scanning and skipping steps.

Limiting

Pruning progress obviously changes state therefore affects execution - invalid reads of obsolete values could happen if pruning is broken.
Pruning indices and histories is coupled, since history table is bounded to index key and txn entries. Since index is a mapping txNum -> {key, key', ...}, looks easier to limit their pruning by txNums at once instead of going through whole list selecting by limit keys.
AggregatorV3Context.PruneSmallBatches() always set txFrom=0 since it's purpose to keep db clean but one step at a time.

domain pruning is limited by amount of keys removed at once. For slow disks and big db (>150G) domain pruning could be very slow: Database keep growing, slowing down pruning as well to 100.000 kv's per 10min session which is not enough to keep db of a constant size. So, using smaller values for --batchSize could solve the problem due to more frequent call of Prune and small changes put into db.

Domain can be pruned if savedPruneProgress key is not for this table nil, or smallest domain key has values of savedStep < pruneStep in domain files. The downside of looking up onto smallest step is that smallest key not guaranteed to be changed in each step which could give us invalid estimate on smallest available key. Saved prune progress indicates that we did not finished latest cleanup but does not give us step number. Could be used meta tables which would contain such an info (smallest step in table?).

takeouts, keep in mind

• --batchSize should be smaller on slower disks (even of size 16-64M) to keep db small. Balanced batchSize could increase throughput preserving db size.
• We have some internal functions which relies on this ordering like define available steps in db
• When --batchSize is reached, commitment evaluated and puts update into that batch which becomes x1.4-x2 of size

[AskAlexSharov]

Need check that prune with limit is “consistent” - for example Index.Prune doesn ‘t exit before collector.Load (if limit reached).

[awskii]

thanks for pointing on it

[awskii]

this pulled up few problems with context management and limits as well.

[awskii]

Since limit is decreased after each hit of prunable key, we sometimes end up with no prune progress over history at all (when limit is a small number). For now i split variables and both domains and history prunes limit keys each.

Did not decided yet if prune continuation still make sense or it's better to be removed right now, need to make a few more comparisons.

Also this PR raises the question: do we want to prune domains index tables as well (which looks meaningful) or just prune it during Unwind process only.

[AskAlexSharov]

1. limit - i think just need change meaning of this variable. avoid it's mutation - and pass same limit to all domains/history/indices.
2. do we want to prune domains index tables - don't understand this question. we wanna prune everything I guess.

[AskAlexSharov]

don't understand - does this code lost [from, to) semantic or not