Improve memory footprint of columnar deserialization using relevant filter #895
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
Signed-off-by: Tony Xiang <[email protected]>
mgovers
reviewed
Feb 14, 2025
| @@ -336,6 +339,21 @@ template <dataset_type_tag dataset_type_> class Dataset { | |||
| add_component_info_impl(component, elements_per_scenario, total_elements); | |||
| } | |||
|
|
|||
| void enable_atrribute_indications(std::string_view component) | |||
There was a problem hiding this comment.
typo
Suggested change
| void enable_atrribute_indications(std::string_view component) | |
| void enable_attribute_indications(std::string_view component) |
| @@ -152,6 +152,14 @@ struct DefaultNullVisitor : msgpack::null_visitor { | |||
| } | |||
| }; | |||
|
|
|||
| struct NullVisitorCheckMap : DefaultNullVisitor { | |||
There was a problem hiding this comment.
it's not really a null-visitor anymore, right?
Suggested change
| struct NullVisitorCheckMap : DefaultNullVisitor { | |
| struct CheckHasMap : DefaultNullVisitor { |
Comment on lines
+87
to
+90
| * @brief Return if a component has attribute indications. | ||
| * | ||
| * Attribute indications are used to indicate the presence of meaningful attributes for a certain component in the | ||
| * dataset. |
There was a problem hiding this comment.
maybe it's good to also mention the behavior if it has both preamble attribute indications and in-data attributes
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background
When using PGM deserialization, the user can select a relevant filter for columnar data format. The de-serializer will return only relevant attributes for that component.
Internally, the Python routine still make buffers of all attributes. Then the attributes with only
nanare discarded. This is un-wanted memory footprint.Improvement
In general, we cannot know which attributes are relevant before parse the whole dataset. However, if the condense attribute list is given in the dataset header, and there are no map-like elements in the data, we know for sure that the relevant attributes can only be those attributes specified in the header. In this way, we can ask the Python routine to only create those buffers. This should save memory footprint of the de-serializer.
Implementation