Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
indexer-alt: committer handles empty batches, checkpoint stream
## Description This change handles two edge cases related to out-of-order commits that were uncovered through the work on event indices. In both cases the committer could get stuck, because none of the conditions guarding select arms were met, but the scenario was different in each case: - In the first case, the committer could get stuck because there were no more pending rows to write, but there were still pre-committed checkpoints to handle, and the logic to update watermarks based on the pre-committed checkpoints was guarded by a testing that `pending_rows > 0`. - In the second case, the committer could get stuck because the pipeline shutdown before any checkpoints came through, meaning it had no work to do. The fix was to allow the main `poll.tick()` arm to run if the receiving channel was closed, or there were pending precommits left. A short circuit was also added to move empty batches directly into the precommit list because we can treat them as already written out. ## Test plan Ran the following pipeline twice: The first time, it should exit without writing any data (except the watermark), and the second time it should just exit because there is no data between its watermark and the given last checkpoint: ``` sui$ cargo run -p sui-indexer-alt -- \ --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \ --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 1000 --pipeline ev_struct_inst ```
- Loading branch information