Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Rewrite CloudWriter to use BufWriter from object_store
This makes the "ObjectStore" choose to upload using a single request or a multi request (multi part) upload depending on the size of the chunks. This change is needed because Polars is now calling the writer without buffering, so this was breaking the upload. There are two tests failing for different reasons: - the IPC upload to a unknown bucket is failing because the new CloudWriter is not propagating the error. This is probably an easy fix. - the Lazy "parquet to cloud" is failing for the reason I wrote above. This is probably related to this issue: pola-rs/polars#17172
- Loading branch information