Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(ledger): support LMDB as a blockstore database backend #352

Open
wants to merge 32 commits into
base: main
Choose a base branch
from
Open

feat(ledger): support LMDB as a blockstore database backend #352

wants to merge 32 commits into from

Commits on Nov 6, 2024

  1. feat(ledger): lmdb wip

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    17070a7 View commit details
    Browse the repository at this point in the history

  2. feat(ledger): wip lmdb

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    427fce7 View commit details
    Browse the repository at this point in the history

  3. feat(ledger): write last lmdb methods, but still doesn't compile

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    f1fd0cf View commit details
    Browse the repository at this point in the history

  4. feat(ledger): lmdb compiles. tests fail

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    3fc08c7 View commit details
    Browse the repository at this point in the history

  5. feat(ledger): get all the tests working for lmdb and add some more

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    3f0fe86 View commit details
    Browse the repository at this point in the history

  6. fix(ledger): remove unused

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    4c9a312 View commit details
    Browse the repository at this point in the history

  7. fix(ledger): misc bugs found when testing as blockstore db

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    8be72a0 View commit details
    Browse the repository at this point in the history

  8. feat(ledger): customizable blockstore database backend

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    d03475c View commit details
    Browse the repository at this point in the history

  9. fix(ledger): hashmapdb compile error when used as blockstore database

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    d4682f1 View commit details
    Browse the repository at this point in the history

  10. test(ledger): migrate new database tests to normal zig tests

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    e1a261c View commit details
    Browse the repository at this point in the history

  11. refactor(build.zig): use switch instead of ifs for database dependency

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    6f3bb7b View commit details
    Browse the repository at this point in the history

  12. refactor(ledger): extract out import to top

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    876eb2d View commit details
    Browse the repository at this point in the history

  13. refactor(ledger): remove c pointers from lmdb

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    79427fc View commit details
    Browse the repository at this point in the history

  14. refactor(ledger): use flags from c import

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    1c22591 View commit details
    Browse the repository at this point in the history

  15. fix(ledger): lmdb error conversion needs platform specific logic

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    cd49700 View commit details
    Browse the repository at this point in the history

  16. refactor(ledger): remove unnecessary ptrCast from lmdb

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    c01b74f View commit details
    Browse the repository at this point in the history

  17. refactor(ledger): rename lmdb "result" to "maybeError"

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    54975b2 View commit details
    Browse the repository at this point in the history

  18. fix(ledger): strings should be 0-terminated

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    11cd6cc View commit details
    Browse the repository at this point in the history

  19. docs(ledger): document txn-based allocator for lmdb

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    e9d84f9 View commit details
    Browse the repository at this point in the history

  20. refactor(ledger): rename ret to returnOutput and add docs

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    2ed42a0 View commit details
    Browse the repository at this point in the history

  21. fix(ledger): reusing hashmap write batch after execution is a bug, an… 

    …d leads to memory leaks
    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    038f9e5 View commit details
    Browse the repository at this point in the history

  22. fix(ledger): leak in purgeSlots

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    1808660 View commit details
    Browse the repository at this point in the history

  23. fix(ledger): serializeAlloc for raw bytes should not use bincode

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    8add233 View commit details
    Browse the repository at this point in the history

  24. fix(ledger): deinit shred bytes in test

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    dbe786a View commit details
    Browse the repository at this point in the history

  25. fix(ledger): write batch copy/pointer mismanagement 

    the problem was in insertShreds because it was copying the write batch instead of getting a pointer. so it would only insert the things that are inserted by the pending state, not insertShreds
    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    e1d5fdc View commit details
    Browse the repository at this point in the history

  26. fix(ledger): memory leaks in tests

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    4710c13 View commit details
    Browse the repository at this point in the history

  27. ci: explicitly test ledger databases in github workflow, and don't re… 

    …quire all dependencies during partial test runs
    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    25bd099 View commit details
    Browse the repository at this point in the history

  28. fix(ledger): memory bugs in the shred inserter

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    ac6ba87 View commit details
    Browse the repository at this point in the history

  29. test(ledger): improve deleteRange test

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    385aa53 View commit details
    Browse the repository at this point in the history

  30. fix(ledger): close lmdb env on deinit, and don't abort write txns 

    There is a bizarre behavior of lmdb that is not documented anywhere. LMDB always reuses the same transaction state for every write transaction. A pointer to it is stored in the env struct and recycled. Aborting a transaction frees the transaction pointer. So if you abort a write transaction, it frees the only write transaction pointer. This corrupts the memory of lmdb because it will try to use the same pointer later as if it is valid. I can't understand how this behavior of lmdb is in any way sane or reasonable, so maybe I'm missing something. Anyway, when you close the env, it tries to free the write transaction, leading to a double free if you already aborted the transaction. that's why it cropped up during this change. so I'm just having it reset write transactions now, instead of abort, which should be fine
    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    fd14806 View commit details
    Browse the repository at this point in the history

  31. test(ledger): run testDatabase(hashmap) tests no matter what

    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    42e7608 View commit details
    Browse the repository at this point in the history

  32. fix(ledger): allocator misuse in BytesRef 

    allocator was misused for generic recycling of resources. this broke with lmdb because it segfaults when Allocator.free attempts to overwrite the with `undefined`
    @dnut
    dnut committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    53a9ea9 View commit details
    Browse the repository at this point in the history