You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I would like to read a lot of small files at high speed from S3. I previously used goofys and it has an option --cheap that dramatically sped up loading small files, see kahing/goofys@941ac6a by avoiding parallel checking if the URI is a file or prefix. In my case I use this for pyTorch dataloading where all my requests are valid files, so this parallel check is unnecessary.
Could we have a similar option for mountpoint-s3?
The text was updated successfully, but these errors were encountered:
Thanks for the feature request. This might not fit in with Mountpoint semantics today because we decided that files should be shadowed by directories with the same name (see detailed semantics). We are considering what can be done in this area, but we do not have anything to share yet.
Tell us more about this new feature.
I would like to read a lot of small files at high speed from S3. I previously used goofys and it has an option
--cheap
that dramatically sped up loading small files, see kahing/goofys@941ac6a by avoiding parallel checking if the URI is a file or prefix. In my case I use this for pyTorch dataloading where all my requests are valid files, so this parallel check is unnecessary.Could we have a similar option for mountpoint-s3?
The text was updated successfully, but these errors were encountered: