in-memory child address tracking #1092
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* extract realtime data concurrently * cleanup * nits * more accurate matched filters * cleanup
* remove extra config * ready * sqlite database * postgres database * fix test type errors * fix tests * remove stale tests * cleanup db * schema => table_names * start docs * some docs * cleanup * database migration * remove maxHealthcheckDuration docs * create schema if not exists * drop finalized rows from reorg tables * fix docs link * nits * merge * use json operators
* remove maxBlockRange * cleanup * add isSuggested property to getLogsRetryRanges return * merge * dynamically increaing log ranges * remove maxBlockRange docs * fix test * nits * cleanup * nits * more cleanup * chore: changeset * chore: changeset
* build log events in memory * order events by checkpoint * . * realtime indexing cache * wip multichain * more work * start multichain tests * merge async generators * lol * typesafe * omnichain * prune rpc request by block * start * fix tests * cleanup * ponder_realtime_is_connected metric * status * fix accuracy of historical to realtime handoff * better generator * nits * skip refetching finalized blocks for completed networks * revert * ignore
* add debug log to sync * new sync duration metric + update tui * sum all sync metrics per network * protects against extraneous logs being thrown * chain specific metrics * cleanup * deprecate old sync metrics * fix historical metrics and tui * fix test * last refactor ever * changeset --------- Co-authored-by: typedarray <[email protected]>
kyscott18
commented
Sep 13, 2024
| @@ -624,7 +628,7 @@ export const createRealtimeSync = ( | |||
|
|
|||
| // New block is exactly one block ahead of the local chain. | |||
| // Attempt to ingest it. | |||
| await handleBlock({ block, ...rest }); | |||
| handleBlock({ block, ...rest }); | |||
There was a problem hiding this comment.
Happy path worker for the realtime queue (bottleneck for handling a new block) is now synchronous
0xOlias
reviewed
Sep 17, 2024
| "address" in filter && | ||
| isAddressFactory(filter.address) | ||
| ) { | ||
| const addresses = await args.syncStore.getChildAddresses({ |
There was a problem hiding this comment.
Any reason not to use Promise.all here?
| : factory.childAddressLocation === "topic2" | ||
| ? 2 | ||
| : 3; | ||
| log.topics; |
There was a problem hiding this comment.
Typo?
Looks good to me, but while we're here I think it's worthwhile to add a couple unit tests for this, e.g. around casing
Comment on lines
+178
to
+179
| ? finalizedChildAddresses.get(filter.address)!.has(log.address) || | ||
| unfinalizedChildAddresses.get(filter.address)!.has(log.address) |
There was a problem hiding this comment.
Are we certain that the casing is consistent when doing these comparisons?
| localChain = localChain.filter( | ||
| (lb) => | ||
| hexToNumber(lb.number) > hexToNumber(pendingFinalizedBlock.number), | ||
| ); | ||
|
|
||
| // add child address from newly finalized blocks to `finalizedChildAddresses` | ||
| for (const filter of factories) { |
There was a problem hiding this comment.
Nit, but I'd find this more readable if the factories loop came after the if (factoryLogs !== undefined) check
|
|
||
| for (const filter of factories) { | ||
| unfinalizedChildAddresses.set(filter, new Set()); | ||
| for (const { hash } of localChain) { |
There was a problem hiding this comment.
I wonder if we should rename localChain to unfinalizedBlocks
Comment on lines
+394
to
+397
| // delete reorged blocks from `factoryLogsPerBlock` | ||
| for (const { hash } of reorgedBlocks) { | ||
| factoryLogsPerBlock.delete(hash); | ||
| } |
There was a problem hiding this comment.
Should this happen before we recompute unfinalizedChildAddresses ? It shouldn't matter, just double checking.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #1091
getChildAddressRealtime child address reorg handling strategy:
unfinalizedChildAddressesandfinalizedChildAddressesfactoryLogsPerBlock, so thatunfinalizedChildAddressescan always be recomputedfactoryLogsPerBlockand recomputeunfinalizedChildAddressesfinalizedChildAddresses, evict finalized blocks fromfactoryLogsPerBlockand recomputeunfinalizedChildAddresses