UTxO-HD release

[jasagredo]

This issue is an aggregate of dangling issues here and there about UTxO-HD, to better reflect the current status of the feature, with updated information.

Current status

The UTxO-HD feature is targeted to be released in node 10.4.

Repository	Branch	PR
ouroboros-consensus	link	IntersectMBO/ouroboros-consensus#1412
cardano-api	link	IntersectMBO/cardano-api#774
cardano-cli	link	IntersectMBO/cardano-cli#1134
cardano-node	link	#6170

• Development work is done. Feature is complete.
• Testing passed on top of the 8.9 node release.
• Benchmarking is taking place on the benchmarking cluster. Should there be regressions, new issues will be opened.
  • Reported CPU > 85% slots is drastically increased. Seems like this is an artifact of the analysis. The performance team will look into this.
  • UTxO-HD heap increase in latest benchmarks ouroboros-consensus#1192
  • Nodes stuck in forging loop ouroboros-consensus#1193
• Testing will be conducted on this version. (link)
• Documentation is lacking Document feature UTxO-HD v0.1 ouroboros-consensus#126
• Git history is messy. The journey has been long with numerous rebases with hundreds of conflicts each.

High level description of UTxO-HD and features

UTxO-HD reworks the consensus codebase to decouple the LedgerState from the UTxO set, such that the latter can be stored on the disk therefore lowering the memory consumption of the node. It has two modes of operation: in-memory and on-disk, which use the same API so should be indistinguishable from the node's point of view.

The in-memory mode should work in a very similar way to the current node, with similar performance metrics. The on-disk mode will considerably lower the memory usage of the node at the cost of some performance. This is a tradeoff users will have to consider.

Using the on-disk mode, even a node running on a Raspberry Pi 5 (8GB of RAM) was able to follow the chain.

Currently on-disk uses LMDB underneath. In the long run, the LSM-tree library is being developed to provide a more performant on-disk backend.

In terms of functionality, a UTxO-HD node does not have more functionalities than a normal node.

You can try the feature already by building cardano-node from the utxo-hd-10.4 branch. It should work both with cabal and with nix.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

[syndeton]

Should we be able to build utxo-hd-9.1.1? Or should we hold off for now?
I'm getting the following error when building .... Thanks

.fatal: Could not parse object 'c09534a2cbaeacf1d49a4a49478ccc61cb49799d'.
HEAD is now at 1751eea4e UTXO-HD 9.1
HEAD is now at bcf0e26e1 UTXO-HD 9.1
fatal: Could not parse object 'c09534a2cbaeacf1d49a4a49478ccc61cb49799d'.

Note: I'm able to build utxo-hd-9.1

[jasagredo]

I'm sorry, I moved the tips of the branches a bit and maybe I moved the tip of the ouroboros-consensus' utxo-hd-9.0 branch (although I though I had only added new commits on top). The source-repository-package stanzas in [email protected] branch should point to the tip of the utxo-hd-9.1 branches in cardano-cli and cardano-api and to the utxo-hd-9.0 branch in ouroboros-consensus.

[syndeton]

Just wanted to say great job with utxo-hd. It's looking like my server costs could be cut by half using it. A big help for those smaller struggling pools. You guys are heroes.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

[jasagredo]

UTxO-HD branch on ouroboros-consensus (IntersectMBO/ouroboros-consensus#1267 + IntersectMBO/ouroboros-consensus#1304) is ready to be merged into main.

Before doing so, we will conduct a validation benchmarking run. We need to rebase the UTxO-HD branch on top of the latest node release (10.2.0).

The process will be:

1. Keep the utxo-hd-main branch on top of main, for eventual merge.
2. Backport utxo-hd to the version of consensus used in 10.2.
3. Use an s-r-p in cardano-api and port the utxo-hd branch on top of that one.
4. Use an s-r-p in cardano-cli and port the utxo-hd branch on top of that one.
5. Use an s-r-p in cardano-node and port the utxo-hd branch on top of that one.
6. Wait for cardano-node 10.2 release and their benchmarks.
7. Run the benchmarks and compare with 10.2.
8. If successful, merge the consensus branch into main.
9. Prepare a branch in cardano-api, cardano-cli and cardano-node main branches that integrate the changes for that consensus branch, for the next release engineer to use when integrating the next node version.

[jasagredo]

Branches have been updated to utxo-hd-main and PR links have been added

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

[jasagredo]

UTxO-HD has been set to be released in node 10.4. Merging of the branches will commence once 10.3 is released.

[syndeton]

I've compiled and am running cardano-node utxo-hd-10.2 on Preview. I see "V2InMemory = true" in the logs when running v10.2.
I cannot find how to implement OnDisk or V1LMDB (if those are the right terms).

The "utxo-hd-in-depth.md" says: "The UTXO-HD feature provides two backends (V2InMemory and V1LMDB) which can be chosen in cardano-node's configuration file." But I can find nothing on how to do that.

The runtime options: --utxos-on-disk, --lmdb-path, both availble in utxo-hd-9.1 seem to have been removed from the current versions.
Is it currently possible to implement V1LMDB (OnDisk) in utxo-hd-10.2?
Thank you

[jasagredo]

You are probably looking for the configuration option to be set in the config file:

{
  "LedgerDB": {
     "Backend": "V1LMDB"
  }
}

In this document you will find more information on how to migrate from a pre-utxo-hd node, and in particular what config options are there.

There have been some changes since 10.2 though (the checksum is now a metadata file instead) but the most part of the document technically still applies to the 10.2+utxohd (unofficial) version.

Note that 10.3 is bound to be released soon, and soon after that 10.4 will come, with UTxO-HD officially released this time. A cardano-node should be runnable already from branch utxo-hd-10.4 but it has not been released yet.

EDIT: forgot to add the document, here it is: https://github.com/IntersectMBO/ouroboros-consensus/blob/utxo-hd-10.4/docs/website/contents/for-developers/utxo-hd/migrating.md

EDIT2: do read the guide! It tells you how to convert snapshots between backends. Otherwise you will have to replay from Genesis!

[syndeton]

Cannot figure out how to build utxo-hd-10.4. Keep getting the following error. utxo-hd-10.2 builds fine and is running very nicely.

HEAD is now at 2471621 UTXO-HD
Cloning into '/home/build/src/cardano-node/dist-newstyle/src/cardano-a_-83a2d30174f489f'...
remote: Enumerating objects: 107316, done.
remote: Counting objects: 100% (461/461), done.
remote: Compressing objects: 100% (210/210), done.
remote: Total 107316 (delta 379), reused 251 (delta 251), pack-reused 106855 (from 2)
Receiving objects: 100% (107316/107316), 53.73 MiB | 29.28 MiB/s, done.
Resolving deltas: 100% (68805/68805), done.
fatal: Could not parse object 'fb871f1f9244d19e3c74a6ed54e1e138ac09a845'.
fatal: Could not parse object 'fb871f1f9244d19e3c74a6ed54e1e138ac09a845'.

I'm fine running utxo-hd-10.2 until final 10.4 is released. Just letting you know about this in case it's an issue.

[jasagredo]

So the UTxO-HD branch has some references to commits in other repositories. As we have been merging some of that work, rebasing those branches, re-signing the commits, etc, those commits might have gone lost. I'm in the process of updating the dependencies for the utxo-hd release.

I hope to have released cardano-cli and cardano-api by the end of my working day today, and then I can update the node branch to use these new versions.

In the meantime you can track the progress towards the release of 10.4 in here #6183, which has sub-issues for the different components.

[jasagredo]

UTxO-HD has been merged into master and cardano-node 10.4.0 has been tagged at 91e8d93, pending testing and benchmarking, so do not put it into production yet.