Pre-allocate buffer

[JoaoAparicio]

If we let transcode to its own allocation it will allocate a small vector, start filling it, resize the vector, fill it some more, resize the vector, etc.

Instead in this commit we pre-allocate a vector of the corect size and pass it to transcode().

Inspired by #399

[JoaoAparicio]

Waiting on release of
JuliaIO/TranscodingStreams.jl#136

[Moelf]

any thoughts on this?

• Feather file with compression and larger than RAM #340

I know you were aware of this issue, and thanks a lot for looking into quite a few issues here and in the TransdocingStream

[JoaoAparicio]

I have some thoughts. One solution to "my dataset is larger than memory" is partitioning. If your dataset is partitioned in such a way that each partition fits in memory, you can iterate it with

stream = Arrow.Stream(path)
for tbl in stream
    ...
end

You can do this right now without requiring any additional features from this package.

In contrast what is discussed in #340 (which is: don't decompress if you don't have to) is a different approach, but doesn't yet exist.

Currently I have some commits that add the feature to multi-thread decompression at the buffer level. I will be trying to upstream what I have so far. The difficulty is that these commits touch a lot of code, so this won't happen overnight. I imagine couple of weeks? On top of that it should be straightforward to implement what is discussed in #340.

[Moelf]

stream = Arrow.Stream(path)
for tbl in stream

the problem of this approach is it's decompressing every column. Consider examples such as:

• [Discussion] Need for early-returning friendly iteration interface #417

Decompressing every column would be super slower if I'm only using a small % of columns

[baumgold]

LGTM

[baumgold]

@quinnj / @ericphanson - any comments before we merge?

[ericphanson]

nope, LGTM!