Skip to content

Report download progress #155

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Apr 30, 2025
Merged

Report download progress #155

merged 19 commits into from
Apr 30, 2025

Conversation

simolus3
Copy link
Contributor

@simolus3 simolus3 commented Mar 27, 2025

This adds interfaces for tracking download progress during a sync operation. Following the design doc:

  1. The ProgressWithOperations class provides information about how many rows have been downloaded already and how many need to be downloaded in total to complete the current sync (and uses that to provide a fraction getter).
  2. The SyncDownloadProgress class keeps track of downloaded operations for each bucket. It can compute aggregate ProgressWithOperations results for the next complete sync (across all buckets) or up until a given priority (includes all buckets with that or a higher priority).
  3. An instance of SyncDownloadProgress is made available from SyncStatus.

The implementation uses new columns added to ps_buckets tracking the operation count at the last checkpoint and the amount of downloaded operations since then:

  1. When we receive a new checkpoint (or a diff), we fetch local progress counters from ps_buckets and use that to populate the initial progress status.
  2. When receiving sync lines, the core extension will update the local counters. We update the SyncDownloadProgress instance in memory.
  3. For a completed checkpoint, we reset the progress and update the "operation count at last checkpoint" column in the database.

I've also refactored some things around copying sync statutes internally (we don't have to write our own copy methods with special args to reset the upload + download error, optional arguments in Kotlin don't have to be constant so passingdownloadError = null will actually reset the value).

@simolus3 simolus3 changed the title WIP: Report download progress Report download progress Apr 29, 2025
@simolus3 simolus3 marked this pull request as ready for review April 29, 2025 14:33
stevensJourney
stevensJourney previously approved these changes Apr 29, 2025
Copy link
Contributor

@stevensJourney stevensJourney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm overall quite happy with the approach taken here. This is a very cool feature.

stevensJourney
stevensJourney previously approved these changes Apr 29, 2025
@simolus3 simolus3 merged commit c015326 into main Apr 30, 2025
3 checks passed
@simolus3 simolus3 deleted the sync-progress branch April 30, 2025 08:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants