High memory usage on default allocator

[i1i1]

After switching from rocksdb to parity db we discovered high memory usage on system allocator:

	system	jemalloc
btree	5.93G	4.34G
kv	7.27G	5.49G

Here is the code for reproducing the issue:

#[global_allocator]
static ALLOC: jemallocator::Jemalloc = jemallocator::Jemalloc;

fn main() {
    let batch_size = 1024 * 1024;
    let handles = (0..4)
        .map(|i| {
            let p = format!("some-path/{i}");
            let _ = std::fs::remove_dir_all(&p);
            let opts = parity_db::Options {
                path: p.into(),
                columns: vec![parity_db::ColumnOptions {
                    preimage: false,
                    btree_index: true,
                    uniform: false,
                    ref_counted: false,
                    compression: parity_db::CompressionType::NoCompression,
                    compression_threshold: 4096,
                }],
                sync_wal: true,
                sync_data: true,
                stats: false,
                salt: None,
            };
            std::thread::spawn(move || {
                let db = parity_db::Db::open_or_create(&opts).unwrap();

                for range in (0..100u64).map(|i| i * batch_size..(i + 1) * batch_size) {
                    db.commit(range.map(|i| (0, i.to_be_bytes(), Some(i.to_le_bytes().to_vec()))))
                        .unwrap();
                }
            })
        })
        .collect::<Vec<_>>();
    for handle in handles {
        handle.join().unwrap();
    }
}

[arkpar]

The problem is with large batch size and small individual values. There's a lot of overhead for this use case.

Most of the overhead is storing modified index pages in IndexLogOverlay . Since we store full pages, each modified index page requires at least 512+24 bytes of memory until the record is flushed. This is not something we can fix easily.
I'd advise you to simply reduce batch_size

[nazar-pc]

Is there something on ParityDB side that doesn't allow allocator to later reclaim most of this memory BTW? We see our app using ~1.3G of RAM until this step, but doesn't go down to quite the same level afterwards.

[arkpar]

Yes, some of the internal overlays just stay at peak usage. We'll add a patch to release them when writes are idling.

[nazar-pc]

Hm... that explains further growth as we open more and more databases (about 10 of that kind typically).