Is there any support for compression? #232
-
|
Can ukv support compression like zstd, snappy while storing data onto disk? This will help greatly for the workloads where data volume is huge. Today I use a key/value db where in my keys are 3M+. If the compression is good like zstd it would make a lot of sense to save disk space. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
It can, but it doesn't by design. General-purpose compression is much slower than modern IO's speed and less efficient than application-specific compression. It can't be implemented as a zero-cost feature. So we leave it to the user to call his favorite compression library before passing us the values if he needs to. This, however, doesn't mean that specialized Modalities won't have their compression. We are evaluating options for various compression algorithms for graphs and document collections. |
Beta Was this translation helpful? Give feedback.
It can, but it doesn't by design. General-purpose compression is much slower than modern IO's speed and less efficient than application-specific compression. It can't be implemented as a zero-cost feature. So we leave it to the user to call his favorite compression library before passing us the values if he needs to.
This, however, doesn't mean that specialized Modalities won't have their compression. We are evaluating options for various compression algorithms for graphs and document collections.