PERF: read list of parquet files at once #5723
Labels
new feature/request 💬
Requests and pull requests for new features
P2
Minor bugs or low-priority feature requests
Performance 🚀
Performance related issues and pull requests.
You can provide a list of local files (but not a list of directories or s3 files) to read_parquet, as in #5698. For #5698 I will make a fix to read the files separately and concat the results. I attempted a more general solution but it required too much surgery on modin's
read_parquet
code, which assumes in many places that there's just one file or directory to read. One difficulty I had is thatParquetDirectory
can take a list of files but not a list of directories, so we can't convert to a list of directories at the beginning.The text was updated successfully, but these errors were encountered: