fix: MinNamespace/MaxNamespace optimizations #1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this issue celestiaorg#216, it was raised that
More time was spent allocating than hashing in the HashLeaf function
The benchmark test described in the issue listed above no longer exists, and when I run the
BenchmarkComputeRoot
in the nmt package HERE I can't replicate memory allocation taking longer than the hashing, although this may be related to my test parameters.Below are FlameGraphs of CPU/Mem usage (I've updated the Benchmark to have 20k leaf nodes with 512 bytes transaction size)
Base Mem Flamegraph
Base CPU Flamegraph
I've instead focused on the second issue
The min and max namespace functions were taking up a large amount of RAM
Solution 1: Reduce the number of calls to MinNamespace & MaxNamespace
The HashNode function made 4 duplicate function calls to MaxNamespace/MinNamespace for the same node.
After reducing to a single call, here is the new benchmark output (running
go test -run="none" -bench=. -benchtime=100x -benchmem
)Solution 2: Update the MinNamespace/MaxNamespace to return a slice of the existing array rather than creating a new underlying array
Copying the slice stops the possibility array capacity memory leaks if for some reason the namespace ID variable's lifetime is longer than the lifetime of the original slice, but I don't think that is the case. Slicing with a max capacity also stops the potential for appends to the namespace ID slice editing the underlying
ns|ns|hash
array.Benchmark output after this change
New Mem Flamegraph
New CPU Flamegraph
The improvements made by the first solution are made redundant by the second, but I was initially reluctant to replace the slice copy in case it would lead to a memory leak. As I got more familiar with the package, I became more confident that a slicing the array may be more appropriate.
Note
Push
method, adding anUnsafeHashNode
that skips validation was also a method I considered, but I felt that this could lead to more complications & would not make any notable differences compared to either of the other solutions.ValidateSiblings
function, I also removed the associated tests, but it looks like the same test cases also exist inTestValidateNodes
HERE