Streaming API

[oberstet]

It would be awesome to have streaming compressor/decompressor API as with zlib.compressobj and zlib.decompressobj.

Even more useful would be: being able to set any configuration on the compressor/decompressor objects, like i.e. HC mode and such.

[tsileo]

👍

[gojomo]

Yes – and specifically, to be able to create or read arbitrarily-large ".lz4" files created by the lz4 command-line. I believe in practice that would mean supporting this: https://docs.google.com/document/d/1cl8N1bmkTdIpPLtnlzbBSFAdUeyNo5fwfHbHU7VRNWY/edit

Can anyone confirm that's what's needed and that it doesn't already exist?

[dreamflasher]

So far I wasn't able to find any lz4 library that supports streaming, and especially for large files it's a requirement.

[Cyan4973]

What about : http://lz4.github.io/lz4/#interoperable-lz4 ?

[dreamflasher]

I checked out the two in that site and a couple more, but no currently none supports streaming/linewise iteration.

[Cyan4973]

Interesting,
both these projects announce streaming capabilities for large files,
though it could be different from what you are looking for exactly.
Little details matter.

[jonathanunderwood]

The project at github.com/python-lz4 continues this work. We have added frame support, and streaming support is planned for the future.

[dreamflasher]

@Cyan4973 They support the LZ4 streaming (frame?) interface, but they don't do that in a way that it's compatible with Python bytestreams (io.RawIOBase).