Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add paged mempool txids endpoint #74

Merged
merged 2 commits into from
Mar 6, 2024
Merged

Add paged mempool txids endpoint #74

merged 2 commits into from
Mar 6, 2024

Conversation

mononaut
Copy link
Contributor

@mononaut mononaut commented Jan 25, 2024

The /mempool/txids endpoint returns every txid in the mempool, which can be slow enough to time out when there are a very large number of transactions.

This PR is one of two options for an alternative endpoint to allow the contents of the mempool to be queried in a series of smaller non-overlapping requests.

This option is a "paged" endpoint /mempool/txids/page/<last_txid>?max_txs=n.

The other option is a "chunked" endpoint via PR #73.

Here, <last_txid> is an optional URL parameter representing the last txid of the previous page (or left blank if this is the first request), and max_txs is an optional query parameter setting the page size (defaults to 10000, which is also configurable at runtime via a new rest_max_mempool_txid_page_size argument).

This provides stable pagination even when transactions are added and removed between requests.

I'd expect this endpoint to be used something like:

let all_txids = [];
let last_txid = null;
while (true) {
  const result = await fetch(`https://mempool.space/api/mempool/txids/page/${last_txid || ''}`);
  const txids = await result.json();
  if (txids.length) {
    last_txid = txids[txids.length - 1];
    all_txids = all_txids.concat(txids);
  } else {
    break;
  }
}

src/new_index/mempool.rs Outdated Show resolved Hide resolved
@junderw
Copy link
Member

junderw commented Jan 26, 2024

I'm leaning towards paging. Both PRs look good though besides the label problem in this one.

@wiz wiz merged commit 48fe0aa into mempool Mar 6, 2024
7 checks passed
@wiz wiz deleted the mononaut/paged-txids branch March 6, 2024 07:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants