Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix common path in _ ls_tree #1850

Merged
merged 2 commits into from
Nov 21, 2023
Merged

Fix common path in _ ls_tree #1850

merged 2 commits into from
Nov 21, 2023

Conversation

lhoestq
Copy link
Member

@lhoestq lhoestq commented Nov 21, 2023

The common path can be one of the directories to refresh.

This was causing a bug where the common path was the parent directory instead, and therefore glob would wrongly return content from the parent directory depending on the cache content

now this code returns the correct files

import fsspec
from huggingface_hub import HfFileSystem

fs: HfFileSystem
path = "hf://datasets/severo/bug-hfh-lfs/data/*"

fs = fsspec.get_fs_token_paths(path)[0]
print(fs.glob(path, detail=True))

print("=" * 100)

fs = fsspec.get_fs_token_paths(path)[0]
print(fs.glob(path, detail=True))

this bug was introduced in #1809

Copy link
Contributor

@mariosasko mariosasko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM!

@lhoestq lhoestq merged commit 8831137 into main Nov 21, 2023
13 of 16 checks passed
@lhoestq lhoestq deleted the fix-common-path-in-_ls_tree branch November 21, 2023 21:15
marcenacp pushed a commit to marcenacp/datasets-server that referenced this pull request Nov 23, 2023
* Revert "Revert "feat: 🎸 upgrade huggingface-hub (huggingface#2134)" (huggingface#2135)"

This reverts commit c6cd8a9.

* feat: 🎸 change pinned commit

after huggingface/huggingface_hub#1850
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants