Add support for arrow stream

[djouallah]

first congratulation on the progress you made, chDB is substantially better than just 6 months ago, I am trying to read a folder of csv and export it to delta, current I am using df = sess.sql(sql,"ArrowTable") to transfer the data to deltalake Python, the problem is I am getting OOM errors, would be nice if you can add support for arrow recordbatch so the transfer is done in smaller batch

thanks

[djouallah]

@auxten how do you get a schema when using this

df = sess.sql(sql,"ArrowStream")
write_deltalake(f"/lakehouse/default/Tables/T{total_files}/chdb",df, mode="append", partition_by=['year'], storage_options= storage_options)

[auxten]

I understand that what you’re trying to do is retrieve the output schema and then stream the data into Delta Lake.

1. Regarding the issue of retrieving the schema, I believe it can be obtained by setting the output format to JSON, ArrowTable, DataFrame, etc. However, in cases of large data volumes, a LIMIT should be applied.
2. Currently, the implementation of chDB requires loading the entire dataset into memory before proceeding with further processing, which can lead to an OOM (out of memory) issue when dealing with large data volumes. This is a point that needs improvement, and I will schedule it for future development.

[djouallah]

I added chdb to my etl benchmarks, feel free to have a look, if i am doing something terribly wrong
https://github.com/djouallah/Fabric_Notebooks_Demo/blob/main/ETL/Light_ETL_Python_Notebook.ipynb

[ViggoC]

@auxten If I understand right, clickhouse-local can load and process the data in streaming style, but chdb collect data from the clickhouse-local in batch style?

[auxten]

@auxten If I understand right, clickhouse-local can load and process the data in streaming style, but chdb collect data from the clickhouse-local in batch style?

yes, you are partially right. For input side, chDB does exactly the same as clickhouse-local. Reading data from file and http or s3 in stream and also random access mode. But for output side, the data is written in batch style. This is what we need to improve.