feat(shwap): Optimize ods file allocations #3512
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is the foundation of new ~era~ sampling protocol and storage.
…org#2992) Adds in-memory file implementation
…r Share (celestiaorg#3012) Reworks `Share` method of `Store` to return shareWithProof containing axis field.
This PR implements the foundational types and functionalities for the Shwap system as defined in [CIP-19](https://github.com/celestiaorg/CIPs/blob/main/cips/cip-19.md). It introduces several key components essential for the operation of the shwap containers and their identification within the Celestia network. Here are the primary additions: **Features Implemented:** 1. **Base Shwap Container Types:** - **Entities Added:** - `Sample`: Represents a share with its corresponding Merkle proof. - `Row`: Denotes a part of a row in an Extended Data Square (EDS), including either its left or right half. - `RowNamespaceData `: Handles rows with specified namespaces. - **Functionalities:** - Protobuf encoding for each container type. - Validation functions to check the integrity and correctness of the data structures received from network. - Inclusion verification methods to validate each entity against a provided data root. 2. **Shwap ID Types:** - **Details:** - Declaration of unique identifier types for shwap entities,. - **Encoding:** - Implementation of binary encoding for each ID type. **Side Note** If you encounter any confusing elements in this PR, please highlight them. After working on this project for so long, it's possible I've overlooked what might be unclear to others. I'm eager to add comments wherever necessary to make everything as clear as possible. Your feedback is invaluable in enhancing the understandability of our work. --------- Co-authored-by: Wondertan <[email protected]>
Adds tests for shwap types and associated functions. There are no non-inclusion tests, because the test function `NamespacedRowFromShares` for creating test data does not support the creation of non-inclusion proofs. The non-inclusion logic itself exists and will be covered at the level of integration tests. `NamespacedRowFromShares` is not used in the shwap and may only be needed for more convenient testing. The Store implementation will use its own proof-building engine, which supports non-inclusion proofs. For now, I see no reason to fuss with this. To add these unit tests, either a lot of duplicative NMT code needs to be written, or access to the NMT tree inside the app wrapper must be provided. The app wrapper will need to be redesigned later for a “clean” implementation of proof caching. Based on celestiaorg#3383
…iaorg#3441) Updates file interface to latest prototype version
- Reintroduce file interface as eds interface. Change aims to allow
usage of EDS interface outside of storage package and to be high level
interface of EDS methods.
- Renames of eds interface methods to align with returned shwap types
names
- Share() -> Sample
- Data -> Row data
- Extracts New<shwap_type_name>FromEDS functions to eds file methods
- moves associated tests to eds pkg
**Additional refactoring:**
- **Change Interface Name**: Realized that 'EDS' is a terrible name for
an interface. Renamed `eds.EDS` to `eds.Accessor` to more accurately
reflect its functionality rather than its internal content.
- **Separate Closer**: Extracted `Closer` from `Accessor`. Now it is
available in a new composite interface `AccessorCloser`.
- **Rename InMem**: Renamed `InMem` to `rsmt2d` to better align with its
usage.
- **Decouple NamespacedData**: Separated `NamespacedData` from the
`rsmt2d` implementation. It is now a standalone function.
- **Update EDS Method**: Replaced the `EDS()` method with `Flattened`,
similar to `rsmt2d`. Considered introducing two separate methods,
`Flattened` and `FlattenedODS`, with the latter to be potentially added
later. Proposed to park this suggestion in an issue for future
consideration.
Adds generalised testing framework for file interface implementaions Also adds non-inclusion proofs handling in RowNamespaceDataFromShares and closes celestiaorg#3428
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds memory pooling for ods file biggest memory allocations.
Based on #3482