Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Faster data downloads #123

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Faster data downloads #123

wants to merge 4 commits into from

Conversation

sealad886
Copy link

I've updated the three bash scripts in the /scripts folder in an attempt to account for:

  1. Unstable internet connections (wget has better support to resume partial downloads than curl does)
  2. Deal with user-terminated downloads (adding -nc with wget takes away some of the download on repeat attempts)
  3. Use native huggingface download tools that are faster than others

@sealad886
faster downloads with wget
d421bd8 
Updated scripts in /scripts. Note that only download_checkpoints.sh and download_data.sh are included. To update download_pretrained_sd.sh, it would be better to do a version where huggingface-cli and hf-transfer are required, as this would use the native download wrappers.
@sealad886
update to pretrained data
6dfdfbf 
Okay, I went ahead and just updated that script but checked if HF_HUB_ENABLE_HF_TRANSFER was set first otherwise just left the original curl in place.
@sealad886
Revert "update to pretrained data"
0f713ac 
This reverts commit 6dfdfbf.
@sealad886
update to pretrained data
5a330c3 
Okay, I went ahead and just updated that script but checked if HF_HUB_ENABLE_HF_TRANSFER was set first otherwise just left the original curl in place.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AravinthBalakrishnan01 AravinthBalakrishnan01 AravinthBalakrishnan01 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sealad886 @AravinthBalakrishnan01