Replace deprecated listing / indexing with new DataChain.from_storage()
#318
Assignees
Comments
|
I think this should be prioritized as it's sometimes hard to refactor / change codebase as this old indexing part needs to be adopted. It would been much easier if it's just refactored to "new" indexing, not to mention it can only be used for CLI operations .. .e.g when we call |
|
Agreed. Moved it to the ready column - let's keep simplifying things there. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In related PR #294 we are going to have listing logic inside of
DataChain.from_storage()itself.We should replace old listing that is being called from CLI and maybe other places with
DataChain.from_storage().There is a lot of tests around listing / indexing and we should refactor them as well if needed
As a follow up for this we should remove old legacy listing codebase (maybe it's better to do this in actual separate issue / PR). We should also remove buckets and partials
Note that we also need to replace
Catalog.ls_storagesto use new listing datasets as bucket table will be removed, as well as partialsdatachain.Dataset.is_bucket_listing()to not use old listing check and maybe remove this method altogether as low levelDatasetclass should not know aboutLISTING_PREFIXand similar higher level abstractionsThe text was updated successfully, but these errors were encountered: