Read parquet files from AWS S3

[philss]

This feature enables reading Parquet files directly from services like the AWS S3.
It's also compatible with other services that implements the S3 API (like minIO, localstack, DO spaces, etc).

This is using Polars' "scan_parquet" feature, that is lazy and also works for reading multiple files at once.

Explorer.DataFrame.from_parquet("s3://test-bucket/*.parquet")

[philss]

I'm having a bad time implementing the integration test using minIO. It seams that Polars (or Reqwest) does not accept connections that are not using TLS - only HTTPs endpoints are accepted.

My first attempt to have localhost certs was to generate them with mkcert.
I exposed the certs to the minio server, but it didn't work for me :/

Then I tested using a ngrok tunnel - that uses TLS - pointing to my minIO instance on localhost and it works 🎉

Unfortunately this is not a valid solution, so I'm considering running a minIO instance in a private server, or use the AWS S3 service, but with instructions on how to reproduce the environment.

If any of you have ideas, please let me know!

[josevalim]

Did you try calling this in the object_store configuration? https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html#method.with_allow_http

The docs also mention localstack testing, so it may be worth looking at object_store own tests in case they use something similar?

[philss]

Did you try calling this in the object_store configuration? https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html#method.with_allow_http

The docs also mention localstack testing, so it may be worth looking at object_store own tests in case they use something similar?

@josevalim thank you! I didn't try this config. I cannot access the internal builder that this method is from, so I'm trying the way of using the (key, value) configs, like we are doing today. But for that, I need to update Polars to v0.31, since the key pair for allowing http is only available after that version (they updated object_store, so we can use).

[philss]

@josevalim @wojtekmach @Qqwy @jonatanklosko I think it's ready for another pass, if you can review again :)

PS: sorry for the amount of failed builds 😓

[josevalim]

Amazing work, just some minor final nits. :) Also please add to the README how to run the cloud tests. :)