-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(filecoin-api): paginated queries #1521
Conversation
InclusionRecordQueryByGroup | ||
> | ||
export type InclusionStore = Store<InclusionRecordKey, InclusionRecord> & | ||
QueryableStore<InclusionRecordQueryByGroup, InclusionRecord> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I flattened the store types to give us more flexibility and simplicity. They can be composed as shown and we now have the ability to express in types a read only store for querying data (for example).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 here for composable types :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Overall this looks really good. Appreciate the type changes and the paging interface looks really good. Do we have adequate integration testing to insure a safe deploy?
InclusionRecordQueryByGroup | ||
> | ||
export type InclusionStore = Store<InclusionRecordKey, InclusionRecord> & | ||
QueryableStore<InclusionRecordQueryByGroup, InclusionRecord> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 here for composable types :)
🤖 I have created a release *beep* *boop* --- ## [7.2.0](filecoin-api-v7.1.1...filecoin-api-v7.2.0) (2024-07-23) ### Features * **filecoin-api:** paginated queries ([#1521](#1521)) ([25ed7d7](25ed7d7)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This PR updates filecoin-api to add pagination to the query interfaces. Currently only the first page of results is returned.
This means in the fileocin pipeline cron jobs we never process more than 1 page of items, even though there are sometimes many more items to process and ample execution time remaining in the lambda.
If only the first page of items is processed it gives a false sense of everything operating ok, when actually things may be backing up. It essentially smothers the issue. If the lambda times out because there are too many items to process then this is a good indication that it needs some attention.
Essentially, the interface changes from:
To this:
Context: we currently have a lot of
filecoin/accept
receipts we need to issue that are being done in a cron job.