scan_parquet from io.BytesIO()

[s-b90]

Problem description

Add ability to accept io.BytesIO() as source parameter for scan_parquet. As for now, it accepts only a path to file/s.
This feature may be useful in cases when your program receives parquet through rest API or socket, directly into memory.

[ritchie46]

I am pretty sure that this is a duplicate. 🤔

[s-b90]

True, I'm sorry. I've found some related issues #4950 #9511. They all are about scan_csv but you definitely can close this as a duplicate. Just don't forget about parquet also :)

[Object905]

It would also be great if scan_* and read_* functions had unified input "type" for files\bytes\etc..
Also it will be nice so that they accepted list of BytesIO or path-like, to process them in parallel like with glob pattern.

[adamgreg]

My application has Parquet embedded as BLOBs in SQL tables, and processes and combines them lazily. I would love to see support for this - at the moment I have to use read_parquet() and miss out on pushdown optimisations.

[aberres]

A similar use case here. We have a bunch of Parquet files in memory I want to work with, without having all of them in memory at the same time.

[shoz]

I would be very happy with this improvement. I have about a million parquet files stored as binaries in Redis and I want to read them as LazyFrame to save memory space.

[HWiese1980]

This is still open. Has here been progress? I need this functionality too.

[cmdlineluser]

@HWiese1980 Coincidentally, it was added on main a few hours ago #18532

[HWiese1980]

Hah! That's quite the timing! :-D Thanks!

[s-b90]

Can confirm that everything is working. Thanks @coastalwhite!