Change to xz compression for downloads

[scottcain]

https://en.wikipedia.org/wiki/XZ_Utils

xz is achieves much better compression. This change can take place along with #126 (splitting metadata and fasta)

Thanks to Art for the suggestion.

[b-f-chan]

Good suggestion, thanks @scottcain, we can consider this, but need you to indicate priority as there's a lot of other stuff to fix

@sifavahora --> Would be good to get @joneubank opinion on this also

[scottcain]

While it would save space in storage when we create downloads, and save users time when downloading, I'm not sure it is necessarily be something we should spend a lot of time on. If perhaps it could be done "easily" along with fixing the current download issues, that would be a nice to have. Otherwise it can wait for later (as will splitting the metadata from the fasta a la #126

[justincorrigible]

After brief research on the subject, while xz is indeed a more effective compression method (in terms of resulting file size), all my findings seem to point at it being substantially slower, and much more resource intensive.

In practice, this would mean three things:

• users would have to wait painstakingly long times for a large download to even start (time that is already quite long as it is). Compression speed reflects directly on the user experience, whereas download speed is tied to a user's own network connection. A better balance of both is more easily achieved with other compression methods.
• the algorithm changes would demand significantly higher resources (e.g.., cpu, memory) from the cluster, which may lead to more frequent download failures, which in turn may lead to increasing operation costs to mitigate that.
• assuming the reason for the suggestion is other than file size (e.g., concerns about connection stability during a lengthy downloads... longer it takes, more possibilities for connection "dropping"), there are UX based alternatives such as offering multipart downloads with smaller file sizes, etc. which may be more cost-effective in the long term.

For reference, a few comparison charts:
https://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO
From that site, of note, a compression speed chart where 1-9 are compression levels:

The smallest level (1) puts gzip at 4x faster than xz. The highest level (9) gzip is almost 10x faster.
Granted, a balance must be achieved between compression and speed, but the impact of that extra compression benefit would be far costlier in almost every other way than what's currently implemented.

For easier visualisation:

(from https://www.r-bloggers.com/2015/11/compression-benchmarks-brotli-gzip-xz-bz2/)

That said, I will continue researching in case these stats are somehow inaccurate or badly outdated.

---

Edit: On further thought, if the product owners determine switching to xz is the choice to make and the additional computational resources are allocated for this to be possible, I would advise we revisit how we currently handle downloads.

Currently, we use a modal window that needs to stay open until the download has started. If this initial time is increased without a progress indicator of some sort, it may be better to instead notify the user they'll receive an email with a link, to download the file once its bundling is completed; as well as warning them that the compression process make take several minutes/hours).
Switching to a much slower compression method may entail extensive additional work and changes, if we want to keep or even improve our users' experience.