Fix huggingface downloader Windows support (#564) #575
Conversation
|
We have an issue to rewrite the CI in Rust so that it can be run on all devices #552. |
| @@ -10,6 +10,10 @@ use thiserror::Error; | |||
|
|
|||
| const PYTHON: &str = "python3"; | |||
There was a problem hiding this comment.
I don't think this is used anymore, if this is the case, we should delete this line.
There was a problem hiding this comment.
Looking at the code, I think this installs venv. Which makes me think that python3 works on Windows as well but probably python is used under venv.
There was a problem hiding this comment.
Yes, the "python3" constant is still successfully used to create the virtual env to begin with, but then that virtual env contains a "Scripts/python.exe" (no python3.exe present there) file which is used to do stuff inside of the virtual env.
| @@ -10,6 +10,10 @@ use thiserror::Error; | |||
|
|
|||
| const PYTHON: &str = "python3"; | |||
There was a problem hiding this comment.
Looking at the code, I think this installs venv. Which makes me think that python3 works on Windows as well but probably python is used under venv.
|
If possible, please add code to initialize a "venv" environment and check if it is properly initialized to ensure the code runs smoothly. |
Pull Request Template
Checklist
run-checks.shhas been executed. (I'm not sure if this can be run on Windows. In the burn-dataset directory, I have manually tried "cargo test --all-features" which has the unrelated tests "dataset::sqlite::tests::sqlite_writer_write" and "dataset::sqlite::tests::sqlite_writer_write_multi_thread" fail, and tried "cargo doc --all-features" which works fine.)Related Issues/PRs
Fixes #564
Changes
The path that the huggingface dataset downloader runs Python from has been changed to support how Python VirtualEnv arranges its files on Windows (which is arbitrarily different).
Testing
The commands
cargo run --example mnist --release --features ndarrayandcargo run --example mnist --release --features wgpuhave been tested. They both now get past the download step and successfully train and finish with this PR's changes.