Make --no-mmap
calls still use parallelism when filesizes are large
#361
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change uses double buffers that are each 1MiB large, while one buffer is filling from the OS, the other buffer is hashed using
update_rayon
. This is around twice as fast as just usingupdate_reader
for files of 1GiB in size on my machine (ryzen 2600), and half as fast as using mmap.The code also accounts for small files, if a file is under 1MiB it will fall back to
update_reader
, this ensures that the change is always at least neutral in performance, because we overshot the actual place whereupdate_rayon
becomes faster, we never see cases where it is slower.Currently the code uses the
read_chunks
crate, which is something I made to handle EINTR and try and fully fill the read buffer, if this is approved to merge I would want to take the function it calls and just cut it into this project somewhere, instead of adding an extra dependency.Some crude benchmarks below, hashing a gibibyte of random data;
(b3sum 1.5.0 vs 03e0949)