Performance Tracking

[sstadick]

@Shnatsel I'm moving performance tracking for crabz related things to here.

Run the same benchmarks as found here: https://github.com/zlib-ng/pigzbench with the different backends.

The zlib-ng benchmarks pretty clearly indicate that zlib-ng is the way to go as a backend, which matches what I see in benchmarks. zlib and the rust backends for flate2 perform about the same.

[sstadick]

Related thread: sstadick/gzp#11

[Shnatsel]

Yeah, that's not unexpected.

The good news is that if the change of backend makes a difference, that means we're not bottlenecked by parallelization overhead or by some single-threaded task. That's a good place to be.

[Shnatsel]

I wonder if the benchmarks in README should be re-run? crabz might perform even better now.

[sstadick]

Yes! Definitely worth updating. I'm tweaking some things in gzp to make the api nicer in cases where num_threads is 0. Basically wrapping flate2 types under one trait so that it's transparent. After that is on the nicer side I'll rerun and update.

I also want to add an MVP of decompression as well. With that in place crabz in nominally usable even it doesn't have all the nobs that pigz has just yet.

[sstadick]

Preview: using passthrough single threaded mode when 0 or 1 threads is selected (just like pigz):

❯ hyperfine './target/release/crabz -c 3 -p 1 < ../gzp/bench-data/shakespeare.txt' 'pigz -3 -p 1 < ../gzp/bench-data/shakespeare.txt' --runs 3     
Benchmark #1: ./target/release/crabz -c 3 -p 1 < ../gzp/bench-data/shakespeare.txt
  Time (mean ± σ):      6.698 s ±  0.127 s    [User: 6.623 s, System: 0.073 s]
  Range (min … max):    6.557 s …  6.802 s    3 runs
 
Benchmark #2: pigz -3 -p 1 < ../gzp/bench-data/shakespeare.txt
  Time (mean ± σ):     11.481 s ±  0.091 s    [User: 11.379 s, System: 0.099 s]
  Range (min … max):   11.381 s … 11.561 s    3 runs
 
Summary
  './target/release/crabz -c 3 -p 1 < ../gzp/bench-data/shakespeare.txt' ran
    1.71 ± 0.04 times faster than 'pigz -3 -p 1 < ../gzp/bench-data/shakespeare.txt'

On the feature/sync_mode branch

[sstadick]

README updated with exhaustive benchmarks comparing different backends as well as a summary of the findings. The surprise to me is that the rust backend is a tad faster than zlib.

[jelber2]

bgzip can decompress in parallel a bgzipped file. Not sure I caught that in the benchmarks or not.

[sstadick]

It can, and I have benchmarked it. Crabz and bgzip have nearly identical performance. I didn't include anything other than pigz just because I don't intended for this to become a comprehensive benchmark suite.

In the benchmarks script you can see how to run with bgzip.

If you have ideas on how to present all the benchmark info in a tidy way that includes bgzip I'm open to PRs :)

[jelber2]

Hmmm. Not sure I am much help on the PRs, just thinking about performance things. So if I understand you correctly, crabz can replace bgzip? If that were true, then this would potentially be a lot easier than installing htslib for some users. Maybe I am misunderstanding.

[jelber2]

There is also https://github.com/Piezoid/pugz , but it really does not seem to work well in my hands.

[sstadick]

Correct! For just compressing / decompressing bgzf format files crabz could replace bgzip, however bgzip does bring all the extras along for generating an index etc (which I would like to add to crabz)...

I guess maybe adding a benchmarks/ folder against other common tools would be useful and not blow up the README. I'll see about adding something like that.

pugz has also never worked well for me and has some serious limitations like only supporting a subset of ascii characters: https://github.com/Piezoid/pugz#roadmaptodos

[sstadick]

@jelber2 I finally added some! https://github.com/sstadick/crabz#crabz-pigz-and-bgzip

Also for what it's worth I ended up trying a bit harder to get pugz to work: https://twitter.com/ducktapeprogra1/status/1443723380436385792?s=20 tl;dr; it's not useful in its current state for bioinformatics mostly because it doesn't support bgzf formats.

[sstadick]

Note that some of the original benchmarks for the different backends are a bit out of date at this point and crabz has gotten faster across the board, but the general trend with the different backends has remained the same (i.e. use a format that uses libdefalte if possible).