Migration utility

[seanses]

Usage:

xtool dedup command will run cleaning on the specified files or files under directories if in recursive mode, and output their File Reconstruction info in JSON format to a file if specified or stdout.

If -m, --migrate option is specified, the product of cleaning (Xorbs and shards) are uploaded to CAS, and the file reconstruction info are not printed out.

xtool [--endpoint [HF_ENDPOINT]] [--token [hf_..]] --repo-type [dataset | model | space] --repo-id [..]  dedup [-r] [paths] [-o path] -c [0|1|2] [-m]

HF endpoint is read in the order of

1. --endpoint
2. env var HF_ENDPOINT
3. default to https://huggingface.co

HF token is read in the order of

1. --token
2. env var HF_TOKEN