pindexer: flesh out dex-explorer indexer #4917
Open
+761
−424
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The main addition are all the metrics needed for the summary page, and the supporting infrastructure therein. Arbitrary windows are now supported for summaries too. Performance was also greatly improved by making use of batch processing, making the supply app view the bottleneck now.
|
with these changes seems like it takes 24 minutes or so to sync to block 1.86 million, or about 1300 blocks a second or so. The dex explorer app view is still a bottleneck for the process though, which we can see more easily now that the app views run in parallel. |
conorsch
approved these changes
Nov 13, 2024
There was a problem hiding this comment.
Overall looks good. The performance improvements are significant! I've got a small concern about chain-specific logic in the indexing code, in the form of hardcoded asset ids, but willing to follow up on that when we've updated the interfaces like dex-explorer to work with the new schema.
This was referenced Nov 13, 2024
Open
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Describe your changes
Closes #4914.
The internal architecture of the app view tries to make use of batch processing to the extent possible, which simplifies a lot of the logic.
The price charts remain unchanged, but I collapsed the two tables into one for performance and simplicity.
I also did not implement insertion of empty candles ; if there are gaps in the events, there will be gaps in the resulting database as well.
The main addition and where I spent most of my time on this is the addition of summaries of information over arbitrary windows. The idea behind the architecture here is that any time a change to liquidity, trade count, or a candle for a directed pair happens in a block, that block then gets a "snapshot" inserted, with the current price, liquidity, volume in that block, etc. At the end of this batch, the current summary is then updated, for each window, using those timed snapshots. And then an aggregate summary, across all pairs, is created from these summaries, for each window.
In order to price values under a common denom, assets are filtered based on having a current USDC price, backed by enough liquidity (the denom and liquidity amount are parameters to the component).
For testing, I'd recommend trying to run the app view against mainnet and testnet, and checking some sanity items like the price not seeming crazy, and matching in the summary across all windows, etc.
I think for testing we'll notice potential issues relatively quickly when dogfooding the explorer.
Checklist before requesting a review
I have added guiding text to explain how a reviewer should test these changes.
If this code contains consensus-breaking changes, I have added the "consensus-breaking" label. Otherwise, I declare my belief that there are not consensus-breaking changes, for the following reason: