Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not download what is already downloaded #10

Open
fedorov opened this issue Oct 6, 2023 · 1 comment
Open

Do not download what is already downloaded #10

fedorov opened this issue Oct 6, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@fedorov
Copy link
Member

fedorov commented Oct 6, 2023

Content already downloaded will be downloaded again if requested by the user.

This is a bit tricky, since we cannot rely on UIDs alone to confirm whether the binary content is the same in the DICOM DB as in IDC. Maybe should keep track of what was downloaded already in a separate cache, and keep the hash? Also need to check if the hash is returned by the API.

There is also a situation where user might have downloaded via the extension, but then deleted from the DICOM DB, or (highly unlikely, but not impossible) deleted from DICOM DB, but then imported into the DB instance that has the same UIDs, but is of a different version and has a different hash...

@fedorov fedorov added the enhancement New feature or request label Oct 6, 2023
@vkt1414
Copy link
Collaborator

vkt1414 commented Mar 23, 2024

As I was working on reporting download progress in idc-index, I remembered this. An elegant way to handle not downloading what is already downloaded may be to use sync feature in s5cmd. What do you think of this? We could implement right in idc-index, so downloading (or I say syncing in the future) might be faster
https://github.com/peak/s5cmd?tab=readme-ov-file#sync

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants