Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory Leak in query.entriesPaged #5798

Open
4 of 10 tasks
justraman opened this issue Feb 26, 2024 · 1 comment
Open
4 of 10 tasks

Memory Leak in query.entriesPaged #5798

justraman opened this issue Feb 26, 2024 · 1 comment
Labels
P4 - Needs Investigation Requires analysis to determine cause or feasibility. Not fully understood, needs research first.

Comments

@justraman
Copy link

  • I'm submitting a ...
  • Bug report
  • Feature request
  • Support request
  • Other
  • What is the current behavior and expected behavior?

I've encountered an issue related to memory consumption while using the entriesPaged method. As I attempt to retrieve entries from the API in batches within a loop, I've noticed a steady increase in memory usage over time. This gradual buildup eventually leads to high memory consumption, crashes with
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Here is the code which lead to memory leak

   const BATCH_SIZE = 500;
   let lastKey = '';

   while (true) {
   	const query = await api.query.multiTokens.tokens.entriesPaged({
       	args: [],
       	pageSize: BATCH_SIZE,
       	startKey: lastKey,
   	});
   	
   	if (query.length === 0) {
       	break;
   	}
   	for (const [key, value] of query) {
       	lastKey = key.toHex();
   	}
   }

please run it against rpc: wss://enjin-matrix-rpc-1.dwellir.com
or you can also quickly reproduce this by running this on polkadot.js web.

  • Please tell us about your environment:

    • Version: 10.11.2

    • Environment: Node, Browser

      • Node.js
      • Browser
      • Other (limited support for other environments)
    • Language:

      • JavaScript
      • TypeScript (include tsc --version)
      • Other
@TarikGul
Copy link
Member

Could be related: #5981

@TarikGul TarikGul added the P3 - Low Non-essential improvements or minor fixes. Can be scheduled flexibly as time permits. label Oct 29, 2024
@TarikGul TarikGul moved this from Issues (Bugs) to P3 - Low in Polkadot-js general project board Oct 29, 2024
@TarikGul TarikGul moved this from P3 - Low to P4 - Needs Investigation in Polkadot-js general project board Oct 30, 2024
@TarikGul TarikGul added P4 - Needs Investigation Requires analysis to determine cause or feasibility. Not fully understood, needs research first. and removed needs investigation P3 - Low Non-essential improvements or minor fixes. Can be scheduled flexibly as time permits. labels Oct 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P4 - Needs Investigation Requires analysis to determine cause or feasibility. Not fully understood, needs research first.
Projects
Status: P4 - Needs Investigation
Development

No branches or pull requests

2 participants