Use big-endian for serializing u64_wrapper keys in RocksDB
#443
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
neysofu
force-pushed
the
filippo/fix-rocksdb-u64-wrapper
branch
from
9b2e50b to
62da72b
Compare
citizen-stig
approved these changes
Jun 28, 2023
preston-evans98
previously requested changes
Jun 28, 2023
The big-endian fix for u64_wrapper! wasn't applied to all relevant types. I've made the following changes: * Renamed define_table_with_u64_wrapper_keys -> define_table_with_seek_key_codec to not be u64_wrapper! -specific. * I replaced the u64_wrapper! specific logic inside said macro to use * big-endian bincode instead, in preparation for using it for different and more complex types. Not yet used for anything other than u64_wrapper!. * I removed the blanked implementation of SeekKeyEncoder for all KeyEncoder's, so that implementors have to opt-in, providing a fail-safe. In the future we'll explore better type safety mechanisms.
neysofu
force-pushed
the
filippo/fix-rocksdb-u64-wrapper
branch
from
6d8e26c to
ae4c3d8
Compare
neysofu
dismissed
preston-evans98’s stale review
All comments addressed; Preston is OOO so I think it's reasonable to move forward with this 🙂
neysofu
added a commit
that referenced
this pull request
Jul 3, 2023
* Adding test rpc * Adding test rpc * Adding test rpc * Tests rpc * Tests rpc * Testing rpc * Finishing the ledger rpc tests * Reverting TODO commit * fixing rollup-config * fixing dependencies * Fixing comments * Moving data structures to mocks * Fixing pr comments * Changing curl to reqwest * Fixing tests * adding more tests * Refactor tests * Adding proptest for get_head * Adding proptest regression * Ledger getHead test * Get_batches * Finishing get batch * Get_txs * Get_events * Finishing ledger rpc fuzzing * Simple lint * Remove unused methods * Modify workflows to save the regression * Try to force temp folder deletion * Fix lint * nit * re-enable flaky test * Use Arbitrary for Event proptest gen * Try to save proptest regressions on cov * Use arbitrary for tx receipt * Rpc tests * Implement Arbitrary for BatchReceipts * Add bytes/serde dep to mocks * Migrate ledger rpc proptests to any() impl * Fix clippy warnings * Update examples/demo-rollup/Cargo.toml * Address comments after discussion * Use big-endian for serializing `u64_wrapper` keys in RocksDB (#443) * fix(sov-db): serialize u64 keys as big-endian * style(sov-db): formatting fix * test(demo-rollup): proptest regressions * test(demo-rollup): larger batches, json utils * fix(sov-db): serialize SlotByNumber, EventByNumber as big-endian The big-endian fix for u64_wrapper! wasn't applied to all relevant types. I've made the following changes: * Renamed define_table_with_u64_wrapper_keys -> define_table_with_seek_key_codec to not be u64_wrapper! -specific. * I replaced the u64_wrapper! specific logic inside said macro to use * big-endian bincode instead, in preparation for using it for different and more complex types. Not yet used for anything other than u64_wrapper!. * I removed the blanked implementation of SeekKeyEncoder for all KeyEncoder's, so that implementors have to opt-in, providing a fail-safe. In the future we'll explore better type safety mechanisms. * Update define_table_with_seek_key_codec macro comment * Linting * Replace &Vec<...> with &[...] * Revert changes to .github/workflows/rust.yml Co-authored-by: Nikolai Golub <[email protected]> * Small improvements to fuzzing.rs * Move MerkleHasher trait into fuzzing.rs --------- Co-authored-by: Preston Evans <[email protected]> Co-authored-by: Nikolai Golub <[email protected]> Co-authored-by: Filippo Neysofu Costa <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR changes the way we serialize
u64_wrapperkeys to RocksDB from Borsh (little-endian) to big-endian. I wanted to remove thederive(Borsh::BorshSerialize)from these types to make misuse harder, but they are needed in other parts of the codebase. I'm basing my PR off of #403, which is the branch where we discovered #417 in the first place.Linked Issues
Testing
I modified
regular_test_helperintest_rpc.rsto generate a second batch with > 256 transactions, which successfully reproduces the fixed bug. I also made two unrelated changes totest_rpc.rs:jsonrpc_req!andjsonrpc_result!macros to facilitate JSON string generation inside tests.UseEDIT: removed after discussion with @preston-evans98 .assert_json_diff!for more user-friendly JSON comparisons.I would imagine these changes could result in somewhat slower tests. It wasn't noticeable to me during local development but we can reevaluate if it's significant.
Docs
I've added some doc comments to the added macros.