ord index update - killed

[rlisac]

Description of the Issue:

I am experiencing a consistent issue where the ord sync process is being killed due to insufficient memory. This occurs repeatedly, often near the completion of the indexing process. The problem affects both signet and mainnet.

System Information:

Operating System: Ubuntu 22.04
BTC Version: v27.1.0
Ord Version: 0.19.1
Memory: 62 GB physical, 100 GB swap

Problem Details:

When attempting to complete the ord sync, the process gets killed, presumably due to an "out of memory" event. The Bitcoin full node is fully synced without any issues, but ord cannot finish indexing. The issue occurs on both mainnet and signet.

Bitcoin Node Status:

Mainnet Status:

{
  "chain": "main",
  "blocks": 858933,
  "headers": 858933,
  "bestblockhash": "00000000000000000002331517eb075475a24c49821ed8e6ad4743bec56cabb1",
  "difficulty": 89471664776970.77,
  "time": 1724918101,
  "mediantime": 1724916074,
  "verificationprogress": 0.9999982923431967,
  "initialblockdownload": false,
  "chainwork": "00000000000000000000000000000000000000008b81cde0f6888c76cc4a178c",
  "size_on_disk": 678012144948,
  "pruned": false,
  "warnings": ""
}

Signet Status:

{
  "chain": "signet",
  "blocks": 210823,
  "headers": 210823,
  "bestblockhash": "000000aaf5eea8427d8b843a94bcbe3b2fa788e13f2ba2d2e199ea06b8b2b0c5",
  "difficulty": 0.002915872311889195,
  "time": 1724918286,
  "mediantime": 1724914836,
  "verificationprogress": 0.9999998083205959,
  "initialblockdownload": false,
  "chainwork": "00000000000000000000000000000000000000000000000000000263babcbb58",
  "size_on_disk": 2110416991,
  "pruned": false,
  "warnings": ""
}

Sync Command Used:
ord -s --cookie-file /root/.bitcoin/signet/.cookie --index-runes --index-addresses --index-spent-sats --index-transactions --index-sats index update

Progress Before Crash:
[indexing blocks] █████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████░ 210001/210822
Killed

Screenshot
Status of the working memory before the kill event is fired.

Steps Taken to Resolve the Issue:

• Deleted all redb files and attempted a fresh indexing.
• Updated to the latest Bitcoin Core version.
• Updated to the latest ord version.
• Rebuilt ord from source as of yesterday.
• Retried the sync process multiple times.
• Increased available memory by adding swap space.

Suspected Cause:

The issue appears to be related to insufficient memory during the ord sync process, but it’s unclear if there are underlying bugs contributing to the crash.

Expected Behavior:

The ord sync should complete without being terminated, fully indexing all blocks as commanded.

Actual Behavior:

The ord sync process is terminated due to what seems like an out-of-memory event before indexing is completed.

Logs / Screenshots:

Unfortunately, there are no additional logs available as the process simply outputs "Killed" when it fails.

Additional Information:

This issue might be related to GitHub Issue #2439. I am not sure if it is connected, but it appears similar. Is there a known solution for this issue?

If there are additional troubleshooting steps or specific configurations recommended to address this, please advise. Any insights into whether this is a configuration issue, a bug, or if further memory optimizations are required would be greatly appreciated.

[so7ow]

Lose the --index-spent-sats and --index-transactions flags. They dramatically increase the size of the index and the resources required to successfully complete it. (And be sure to start indexing from scratch if you do change your flags.)

Reference ➡️ #2999

[rlisac]

ok cool. thx for your feedback. i'll try it

[so7ow]

i'll try it

For what it's worth, if you're not familiar --index-sats also significantly increases the index size but you should be able to complete that with your system. If you don't need it though, losing that will make it even faster/easier.

[rlisac]

ok perfect. thank you. i have deleted the redb and restarted the index. will reconfirm as soon as it works. thanks for your support

[rlisac]

the flags --index-spent-sats and --index-transactions consume a lot of memory. that's it. it works. thx

[raphjaph]

We're going to get rid of the --index-spent-sats flag in the next release. It wasn't really meant to be used and no one has ever been able to build an index with it. The --index-transactions flag is also superfluous since we start bitcoind with -txindex anyways. I think originally we added it to test if indexing and serving content would be faster that way

[so7ow]

We're going to get rid of the --index-spent-sats flag in the next release.

Good call 👍🏻