-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve fetch_historic_logs_stream logic #108
Comments
So i thought all RPC never return you a cross block response aka if block 7 has 10 logs they all come back in the request, are you saying that's not the case for hypersync? just to be clear: i fetch 1000 event logs the last log is block number 10 is there ever a chance on logs 1001 > 1500 has more event for block number 10? |
No, it's a little different case |
ok but in this case Case 1 are you expecting it to be Case 1 so your saying we doing wasted calls in this example here - as we know nothing is in 0-100 blocks as we just done the request |
if logs_empty {
info!(
"{} - No events found between blocks {} - {}",
info_log_name, from_block, to_block
);
let next_from_block = to_block + 1;
return if next_from_block > snapshot_to_block {
None
} else {
let new_to_block = calculate_process_historic_log_to_block(
&next_from_block,
&snapshot_to_block,
&max_block_range_limitation,
);
debug!(
"{} - {} - new_from_block {:?} new_to_block {:?}",
info_log_name,
IndexingEventProgressStatus::Syncing.log(),
next_from_block,
new_to_block
);
Some(ProcessHistoricLogsStreamResult {
next: current_filter
.set_from_block(next_from_block)
.set_to_block(new_to_block),
max_block_range_limitation,
})
};
}
if let Some(last_log) = last_log {
let next_from_block = last_log
.block_number
.expect("block number should always be present in a log") +
U64::from(1);
debug!(
"{} - {} - next_block {:?}",
info_log_name,
IndexingEventProgressStatus::Syncing.log(),
next_from_block
);
return if next_from_block > snapshot_to_block {
None
} else {
let new_to_block = calculate_process_historic_log_to_block(
&next_from_block,
&snapshot_to_block,
&max_block_range_limitation,
);
debug!(
"{} - {} - new_from_block {:?} new_to_block {:?}",
info_log_name,
IndexingEventProgressStatus::Syncing.log(),
next_from_block,
new_to_block
);
Some(ProcessHistoricLogsStreamResult {
next: current_filter
.set_from_block(next_from_block)
.set_to_block(new_to_block),
max_block_range_limitation,
})
};
} i wonder why i did this 🤔 in turn this would mean we could remove |
I had the same idea but it would require some thorough testing |
il have a think why i did this and try to see if any reason why we cant do the above thanks for raising |
I found a small problem in from/to intervals during fetch_historic_logs_stream.
To determine the next block in case when we had any logs returned in this range we are taking the block number of the last block. This causes the following problem: we are ignoring the fact that we are already processed some blocks at the end of the range (if they were empty).
Example:
Case 1
Range 0-100 returned only one log with block number 50
The current implementation will query the next batch with block range 50-...
Case 2
Range 0-100 returned no logs
The current implementation will query the next batch with block range 100-...
The text was updated successfully, but these errors were encountered: