[bridge-indexer] Progress saving policy #19243
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
3 Skipped Deployments
|
longbowlu
requested review from
patrickkuo,
Bridgerz,
dariorussi and
0xaslan
There was a problem hiding this comment.
It looks good to me.
Do you think we need all the customization provided?
I feel an in memory story with all DB updates done with some interval (interval in checkpoints and not time) would be best.
Maybe that is what we do, but I am getting a bit lost in the level of abstraction provided by our indexer.
One question: is this affecting the deepbook indexer?
These two basic policies are what we need for Sui ingestion and Eth ingestion:
|
@dariorussi I was wrong about what i commented earlier, in fact this PR's impact to deepbook indexer is very small: it changes the interface of |
longbowlu
force-pushed
the
progress-saving-policy
branch
from
36be154
to
27b5cc8
Compare
## Description This PR introduces `ProgressSavingPolicy` to deal with two problems: 1. The current implementation has a bug on Sui side - checkpoints data arrive out-of-order (e.g. checkpoint 10 may be processed earlier than checkpoint 9), so existing `save_process` may cause us to miss blocks. 2. In current implementation we need to write progress to DB for every call to `save_process`. This can be optimized to cache progresses in memory and flush them periodically or conditionally. We add two types of `ProgressSavingPolicy`, `SaveAfterDuration` and `OutOfOrderSaveAfterDuration`: * `SaveAfterDuration` only flushes the progress to DB after a period of time * `OutOfOrderSaveAfterDuration` assumes the data is out of order, and will only write height N when it makes sure everything before N has been received. * ## Test plan unit tests and production deployment. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:
Description
This PR introduces
ProgressSavingPolicyto deal with two problems:save_processmay cause us to miss blocks.save_process. This can be optimized to cache progresses in memory and flush them periodically or conditionally.We add two types of
ProgressSavingPolicy,SaveAfterDurationandOutOfOrderSaveAfterDuration:SaveAfterDurationonly flushes the progress to DB after a period of timeOutOfOrderSaveAfterDurationassumes the data is out of order, and will only write height N when it makes sure everything before N has been received.Test plan
unit tests and production deployment.
Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.