Optimize memory usage and reduce GC by eliminating memory copies 3x #74
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The current shuffle service implementation creates multiple copies of the shuffle write which leads to increased GC and under heavy load reduced performance due to GC throughput limitations. These copies can be eliminating by simply passing the reference of the original Netty ByteBuf which can then be passed to the storage layer and eliminate the need for any Buf to array conversion / rewrapping.
This PR is an initial draft of these changes validated with the unit tests as well as basic long running tests for validity.
Original Flow-> network buf copied to custom buf -> shuffleDataWrapper to byte array (copy) -> wrapped to buf -> write to file as output stream (copy to byte array)
New Flow -> network buf to composite buf (no copy) -> handler passthru -> materialize to ByteBuffer -> write to file (via channel using direct buf)
This therefore eliminates the 3 copies of the data in memory.