Add rapidgzip support?

[y9c]

https://github.com/mxmlnkn/rapidgzip

[rhpvorderman]

This is rather interesting. Due to the overhead of creating subprocesses I do wonder if this will be faster than the python-isal backend in most use cases. Since rapidgzip uses ISA-L it shouldn't be that much slower. Maybe some of the algorithms could be ported to python-isal? Thanks for posting the link.

[rhpvorderman]

Oh yes, I forgot to mention, but there are some things to be aware of. I am a bioinformatician, so I can really only work on things that I can justify putting the time in. For Bioinformatic workloads the bottleneck is usually not the decompression if it can be offloaded to another thread using ISA-L. This option is already avalaible. If more speed is needed, most of our gzip files are actually bgzip files. They can be decompressed using multithreading in a much simpler way. The reason I haven't got around to doing that is that the priority is very low because I simply haven't run into cases were decompression speed is bottlenecking. For rapidgzip inclusion the priority would be even lower.

If somebody else is willing to do it however, I am willing to do the code review etc. It shouldn't be too hard. Pbzip2 is also integrated and this can be taken as a template.

[y9c]

Thank you much for the information. I will try to use bgzip format instead to speed up.