You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There are some inconsistencies between repo.dvcfs and DvcFileSystem, but for the purpose of fs.get(), they should work similarly (and be faster). They are still a bit inefficient, ideally we want to take the underlying filesystem of the remote (say s3fs) directly for better performance.
Bug Report
Calling get on DVCFileSystem with multiple files doesn't parallelize the download.
Description
Reproduce
This example downloads multiple files with a single get call:
It works, but it downloads 1 file at a time.
Expected
This example does the same, but accesses internal datastructures to make it faster:
This internal get function parallelizes the download, and is thus about twice as fast on my machine.
Given that this functionality is already implemented, I find it a bit sad that it isn't used.
The text was updated successfully, but these errors were encountered: