Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding --cheap option #946

Open
tsdev opened this issue Jul 18, 2024 · 1 comment
Open

Adding --cheap option #946

tsdev opened this issue Jul 18, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@tsdev
Copy link

tsdev commented Jul 18, 2024

Tell us more about this new feature.

I would like to read a lot of small files at high speed from S3. I previously used goofys and it has an option --cheap that dramatically sped up loading small files, see kahing/goofys@941ac6a by avoiding parallel checking if the URI is a file or prefix. In my case I use this for pyTorch dataloading where all my requests are valid files, so this parallel check is unnecessary.

Could we have a similar option for mountpoint-s3?

@tsdev tsdev added the enhancement New feature or request label Jul 18, 2024
@monthonk
Copy link
Contributor

Thanks for the feature request. This might not fit in with Mountpoint semantics today because we decided that files should be shadowed by directories with the same name (see detailed semantics). We are considering what can be done in this area, but we do not have anything to share yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants