Performance degrading a lot with high number of keys

[crystalin]

Running Strorage Benchmark on 3 different networks with significant state size/content results in incoherent results.
We have beeen using Moonbeam v0.32.1 which is based on substrate 0.9.40.
The network Alphanet and Moonriver have similar state/usage overall, but Moonbeam had a project that generate a huge amount of storage (all of the same size, 42 bytes IIRC).

As you can see, the Moonbeam read and write using paritydb are way off the expected result that we see in alphanet and moonriver.

Configuration of the disk is AWS gp3 | 1000 GiB | 3000 IOPS and each network/db has its own disk (total of 6 disks).
The blocks and state are pruned to avoid having a huge disk space.

Running the storage benchmark (on c6i.4xlarge AWS):

/home/ubuntu/projects/moonbeam/target/release/moonbeam
   benchmark
   storage
   --db=${DB}
   --state-version=0
   --mul=1.1
   --weight-path  /home/ubuntu/projects/moonbeam/weights-${DB}-${NETWORK}.rs
   --chain ${NETWORK}
   --base-path /var/lib/${DB}-${NETWORK}-data

for each chain

Alphanet (~20M keys):

pub const RocksDbWeight: RuntimeDbWeight = RuntimeDbWeight {
  read: 65_167 * constants::WEIGHT_REF_TIME_PER_NANOS,
  write: 114_721 * constants::WEIGHT_REF_TIME_PER_NANOS,
};
pub const ParityDbWeight: RuntimeDbWeight = RuntimeDbWeight {
  read: 16_290 * constants::WEIGHT_REF_TIME_PER_NANOS,
  write: 65_374 * constants::WEIGHT_REF_TIME_PER_NANOS,
};

Moonriver (~30M keys state):

pub const RocksDbWeight: RuntimeDbWeight = RuntimeDbWeight {
  read: 66_865 * constants::WEIGHT_REF_TIME_PER_NANOS,
  write: 114_947 * constants::WEIGHT_REF_TIME_PER_NANOS,
};
pub const ParityDbWeight: RuntimeDbWeight = RuntimeDbWeight {
  read: 14_483 * constants::WEIGHT_REF_TIME_PER_NANOS,
  write: 64_545 * constants::WEIGHT_REF_TIME_PER_NANOS,
};

Moonbeam (~110M keys state):

pub const RocksDbWeight: RuntimeDbWeight = RuntimeDbWeight {
  read: 33_439 * constants::WEIGHT_REF_TIME_PER_NANOS,
  write: 86_828 * constants::WEIGHT_REF_TIME_PER_NANOS,
};
pub const ParityDbWeight: RuntimeDbWeight = RuntimeDbWeight {
  read: 177_320 * constants::WEIGHT_REF_TIME_PER_NANOS,
  write: 69_450 * constants::WEIGHT_REF_TIME_PER_NANOS,
};

Additionally to the paritydb numbers, we can also see that RocksDB average read is 50% on Moonbeam (110M keys) than Moonriver (30M keys), which might be related to the size of the data on Moonbeam being on average smaller than on Moonriver.

Details about the Benchmark output can be found there:
https://gist.github.com/crystalin/8e790a554b246e077c83ad04c04f330c

[crystalin]

Additionally, it took something like 40h to generate the moonbeam storage benchmark

[ggwpez]

@cheme do you have an idea why the ParityDB time for read on the 110M keys DB is so much slower than Rocks when it is normally faster?

[crystalin]

You can download a recent Moonbeam state: https://s3.console.aws.amazon.com/s3/object/alan-stuff?region=us-east-1&prefix=moonbeam-state-3631095.json.lz4 (10GB) if you want to check it

[cheme]

That is definitely not expected. Could imagine worst access on big mmap memory, but not something in these proportions. Can also think of the data not being correctly build (there is a reindexing running in background every N values, but this is flushed on exit/start).

" based on substrate 0.9.40." : is it substrate version (looks old)?
Would be interesting to have the parity-db version listed in the Cargo.lock (a version from a few month ago did have an issue that could explan some bad behavior cc\ @arkpar ).

Edit: https://github.com/PureStake/moonbeam/blob/6ed87ceeb65db27a9b2ce7ff32b90d062540bd67/Cargo.lock#L8942 parity db version is 0.4.6 which do not have #206 but I don't expect it to be related.

[crystalin]

I'm happy to cherry-pick some changes on top of it if you want to test few things. You can also probably reproduce by using the snapshot I provided

[cheme]

I'm happy to cherry-pick some changes on top of it if you want to test few things. You can also probably reproduce by using the snapshot I provided

Would be using latest version of parity db (cargo update -p parity-db), but then it only really would make sense if synching the snapshot from scratch.

Something I am thinking right now, did the memory cusumption stay correct during the process (looking at the bench code I suspect it could put many items in memory)?

Edit: just realize the snapshot is in json format so no need to resynch.

[cheme]

actually would be better patched parity-db master to include #211

[crystalin]

Ok I'll try that if I find time (also be aware that the benchmark took 40 hours so I won't get result quickly)

[ggwpez]

I cant even import the snap on a 64GB server… do you use 128?

[arkpar]

I've tried using warp sync on moonbeam. The sync went fine, although peak RAM usage was over 130GB. However the parachain is not finalizing blocks. Final block is still at zero. Is this a known issue? Unfinalized blocks are stored differently in the DB and this may affect performance.

[arkpar]

As for possible performance issues, it could be affected by how the benchmark is implemented. RocksDB uses its own caching, while ParityDb relies on the OS cache. IIRC the benchmark warmup touches a few of the keys, and for RocksDB this causes a lot more data to be pre-cached.

[crystalin]

@arkpar warp is not fully supported yet. We are still working on it.
I also suspect the benchmark implementation is the reason for those unexpected values, but it is hard/long to verify

[crystalin]

@arkpar were you able to reproduce? Let me know if I can help otherwise

[arkpar]

I could not access the snapshot linked above. It requires AWS registration and asks for my credit card number. I've started regular sync instead and it looks like it will take 3-4 days.