How to compress gamedata

[Flapperkewiet]

I have been looking a bit into compressing the data coming from Export() functions.

My testing fuctions and result are here: pastebin.

First I tried to manualy compress EntityData and was able to compress from 124 bytes to 72 bytes in 0.0005ms.
Then I used Gzip on the binarywriter passed into Export(). It takes around 3-4 times longer than normal, but is able to compress to half the original size.

For my main game, a medium-large world with multiple colonized planets, the normal export took 460ms for 40MB of data and with gzip compression it took 1680ms for 18.1MB of (compressed) data.

Does anyone have other ideas for compression?

[hubastard]

You could look at LZ4, it's usually really fast.

[Therzok]

@Flapperkewiet you should try using a bufferedstream over gzipstream, with 4k/8k buffer size.

[Therzok]

See: https://faithlife.codes/blog/2012/06/always-wrap-gzipstream-with-bufferedstream/

[Flapperkewiet]

Thanks, i will add that with the rest of my benchmarks

[Flapperkewiet]

I did some more testing and here is the result:

RawBinary
  (min, max, avg) time in ms (450, 513, 474)
  Size: 40158105 bytes = 40,158.11KB = 40.16MB
BuiltInGZip
  (min, max, avg) time in ms (1637, 1664, 1647)
  Size: 18115549 bytes = 18,115.55KB = 18.12MB
BuiltInGZipBuffered4K
  (min, max, avg) time in ms (1399, 1441, 1412)
  Size: 18115549 bytes = 18,115.55KB = 18.12MB
BuiltInGZipBuffered8K
  (min, max, avg) time in ms (1405, 1433, 1414)
  Size: 18115549 bytes = 18,115.55KB = 18.12MB
LZ4Fast
  (min, max, avg) time in ms (535, 566, 542)
  Size: 21785101 bytes = 21,785.10KB = 21.79MB
LZ4HighCompression
  (min, max, avg) time in ms (2335, 2403, 2362)
  Size: 20851211 bytes = 20,851.21KB = 20.85MB
LZ4AltFast
  (min, max, avg) time in ms (429, 458, 434)
  Size: 21740842 bytes = 21,740.84KB = 21.74MB
LZ4AltHighCompression
  (min, max, avg) time in ms (1079, 1114, 1090)
  Size: 20899344 bytes = 20,899.34KB = 20.90MB
LZ4AltFastBuffered4K
  (min, max, avg) time in ms (414, 459, 427)
  Size: 21740434 bytes = 21,740.43KB = 21.74MB
LZ4AltFastBuffered8K
   (min, max, avg) time in ms (417, 438, 423)
  Size: 21740434 bytes = 21,740.43KB = 21.74MB

Explanation:

• RawBinary is the base stream the game produces (baseline)
• BuiltInGZip is the build in stream compression in System.IO.Compression
• LZ4 is using Lz4.Net by Khalid Salomão
• LZ4Alt is using lz4net by Milosz Krajewski

Conclusion:

• GZip has a little better compression, but is not worth the time cost
• Like @Therzok suggested, wrapping the stream in a buffered stream gives a noticable speedup, most noticably on GZip but also on LZ4.
• It seems LZ4AltFastBufferd has the best performance to size ratio (no noticable difference between 4K or 8K).
• LZ4 High compression is terrible.

[Therzok]

I think it would be valid to time both, compression and decompression.

Also do it in a loop and average it out after discarding the first few runs of each test.

That way you ensure no cold cache for either disk or cpu, which are really important factors for timing things.

[Mushroom]

I have made PR #37 - I will wait for the results of what is suggested above first though, before I merge it

[Flapperkewiet]

The data in the comment is done 10 times in a loop, hence the min, max and average time.

I will do deconpression tonight as wel as 15 runs and discarding the first 5

[Flapperkewiet]

Results:

RawBinary
  (min, max, avg) time in ms (432, 461, 446)
  Size: 40158101 bytes = 40,158.10KB = 40.16MB
BuiltInGZip
   (min, max, avg) time in ms (1666, 1691, 1672)
   Size: 18114539 bytes = 18,114.54KB = 18.11MB
BuiltInGZipBuffered4K
   (min, max, avg) time in ms (1413, 1438, 1422)
   Size: 18114539 bytes = 18,114.54KB = 18.11MB
BuiltInGZipBuffered8K
   (min, max, avg) time in ms (1410, 1448, 1418)
   Size: 18114539 bytes = 18,114.54KB = 18.11MB
LZ4
   (min, max, avg) time in ms (564, 587, 573)
   Size: 21783599 bytes = 21,783.60KB = 21.78MB
LZ4AltFast
   (min, max, avg) time in ms (449, 473, 454)
   Size: 21739603 bytes = 21,739.60KB = 21.74MB
LZ4AltFastBuffered4K
   (min, max, avg) time in ms (428, 458, 441)
   Size: 21739603 bytes = 21,739.60KB = 21.74MB
LZ4AltFastBuffered8K
  (min, max, avg) time in ms (432, 465, 443)
  Size: 21739603 bytes = 21,739.60KB = 21.74MB
RawBinaryDecompression
   (min, max, avg) time in ms (332, 380, 347)
BuiltInGZipDecompression
  (min, max, avg) time in ms (708, 845, 742)
BuiltInGZipBuffered4KDecompression
   (min, max, avg) time in ms (471, 635, 507)
BuiltInGZipBuffered8KDecompression
   (min, max, avg) time in ms (473, 704, 505)
LZ4Decompression
     (min, max, avg) time in ms (338, 626, 379)
LZ4AltFastDecompression
   (min, max, avg) time in ms (330, 656, 385)
LZ4AltFastBuffered4KDecompression
   (min, max, avg) time in ms (321, 724, 377)
LZ4AltFastBuffered8KDecompression
   (min, max, avg) time in ms (322, 775, 385)

Did some more testing with the suggestion from @Therzok:

• Did 15 runs, and only timed the last 10
• Added decompression benchmarking.

Conclusion:

• LZ4Alt seems to be the fastest at both compressing and decompressing.
• GZip compresses a little more but is considerably slower than LZ4.
• For some reason, the max is very high for LZ4 compared to the average, even though the avg is closer to the min value so there is probably one outlier?