Introduce traversePaginatedSourceByHasMore()

[freiondrej]

Changes

Allow traversing a paginable resource with constant memory usage.

Checklist

• Apply one of following labels; major, minor, patch or skip-release
• I've updated the documentation, or no changes were necessary
• I've updated the tests, or no changes were necessary

[freiondrej]

As it stands, this is pretty much just a copy of getPaginatedEntriesByHasMore, with an addition of the processPage() call.

In the future, I can imagine something more parallelized, with apiCall being called in a promise pool as fast as the processPage() is able to process it (as I assume processPage will be the bottleneck), so that we wouldn't need to keep things in memory.

EDIT: https://superchargejs.com/docs/3.x/promise-pool#create-a-promise-pool-from-an-async-iterable might work nicely :) I'll try it if I get a chance, but cannot make any promises ATM.

[CarlosGamero]

🟢 Could we add JSDoc as in the other methods? just to explain clearly how the method works and how it is intended to be used

[CarlosGamero]

🟢 Could be interesting to be able to return the results of the processing step? something like

export async function traversePaginatedSourceByHasMore<
  T extends Record<string, unknown>,
  ReturnType = void,
>(
  pagination: OptionalPaginationParams,
  apiCall: (params: OptionalPaginationParams) => Promise<PaginatedResponse<T>>,
  processPage: (pageData: PaginatedResponse<T>['data']) => Promise<ReturnType> | ReturnType,
): Promise<ReturnType> {

An example usage could be to insert something on DB and return all new IDs. What do you think?