binary serialization and zstd compression for Eigen types [OC-293] #451
Conversation
peddie
changed the title
| compressed.resize(compressed_bound); | ||
|
|
||
| const std::size_t compressed_size = | ||
| // I solemnly swear that I am up to no good. |
There was a problem hiding this comment.
Maybe describe the situation you think could cause a failure here and/or why it won't happen, in place of this and the below comments.
There was a problem hiding this comment.
Lol, I'll remove those.
|
|
||
| TEST(Compress, DecompressEmpty) { | ||
| std::string inputs; | ||
| ASSERT_DEATH({ const auto result = albatross::zstd::decompress(inputs); }, |
There was a problem hiding this comment.
Is this something we would want to assert on? One recent issue that has come up in the AQ, the current solution is to return an empty model instead of asserting. Could this cause issues on the receiving end? @jbangelo
https://github.com/swift-nav/orion-engine/pull/6939
There was a problem hiding this comment.
I'm not sure if we have a better way to handle failures here. These routines get used in the cereal serialisation / deserialisation routines, and I'm not aware of a way to propagate deserialisation failures via the cereal API.
The code this is replacing also asserted on any detected decompression failure, so for this change I chose to preserve that behavior.
peddie
changed the title
Previously the serialization routines used gzip. This commit modifies them to use zstd, which pareto-dominates gzip apart from breadth of software distribution.
Currently
albatrossserializes large vectors and matrices byIn a head-to-head test on systems internal to Swift, we were able to reduce serialization time to about 0.4 of the current approach by using Cereal's portable binary archive type and compressing with zstd. This patch switches Albatross to use the latter approach.