Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Fix cuda toolkit speed issue. #1055

Merged
merged 3 commits into from
Feb 14, 2024

Conversation

wkpark
Copy link
Contributor

@wkpark wkpark commented Feb 8, 2024

Currently, CI uses the Jimver/cuda-toolkit for Windows which has known speed issues when working on Windows.
Jimver/cuda-toolkit#253

specific package "visual_studio_integration" is the cause of this speed issue but it is needed to use msbuild.

here is the solution.

  • do not install "visual_studio_integration" sub-package
  • compile using MSVC + ninja, instead of using msbuild.
  • and also turn off github-cache for cuda-tools to fix some broken cases.
Cache error log for example.

https://github.com/TimDettmers/bitsandbytes/actions/runs/7804268267/job/21285853144

Run Jimver/cuda-toolkit@v0.[2](https://github.com/TimDettmers/bitsandbytes/actions/runs/7804268267/job/21285853144#step:5:2).14
  with:
    cuda: 12.1.0
    method: local
    sub-packages: []
    non-cuda-sub-packages: []
    linux-local-args: ["--toolkit", "--samples"]
    use-github-cache: true
    use-local-cache: true
    log-file-suffix: log.txt
"C:\Program Files\Git\usr\bin\tar.exe" --posix -cf cache.tzst --exclude cache.tzst -P -C D:/a/bitsandbytes/bitsandbytes --files-from manifest.txt --force-local --use-compress-program "zstd -T0"
Cache Size: ~[3](https://github.com/TimDettmers/bitsandbytes/actions/runs/7804268267/job/21285853144#step:5:3)195 MB (3350[4](https://github.com/TimDettmers/bitsandbytes/actions/runs/7804268267/job/21285853144#step:5:4)07[6](https://github.com/TimDettmers/bitsandbytes/actions/runs/7804268267/job/21285853144#step:5:6)[7](https://github.com/TimDettmers/bitsandbytes/actions/runs/7804268267/job/21285853144#step:5:7)4 B)
Cache saved successfully
Error: Error: Got no files in tool cahce

without this fix: total build time is about ~30 ~40min
with this fix: total build time will be reduced about ~25min

Copy link

github-actions bot commented Feb 8, 2024

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@wkpark wkpark mentioned this pull request Feb 8, 2024
17 tasks
Copy link
Contributor

@rickardp rickardp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice spot. I did not realize msvc worked without the VS integration of CUDA.

.github/workflows/python-package.yml Show resolved Hide resolved
.github/workflows/python-package.yml Outdated Show resolved Hide resolved
.github/workflows/python-package.yml Show resolved Hide resolved
Copy link
Collaborator

@younesbelkada younesbelkada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clean fix !

@younesbelkada younesbelkada requested review from Titus-von-Koeller and removed request for rickardp February 13, 2024 02:08
@Titus-von-Koeller Titus-von-Koeller merged commit 344e851 into bitsandbytes-foundation:main Feb 14, 2024
31 checks passed
@Titus-von-Koeller
Copy link
Collaborator

Thanks @wkpark for the clean fix and @rickardp for the review. Looks good also to me!

Thanks all for the intervention last week to get everything working again after the premature merge. Shouldn't have done that so hastily, it was based on a miscommunication.

I also sent you both an invite for the bitsandbytes Slack channel. In case you want to coordinate PRs, discuss sth important or I don't see something important in time which would make sense to unblock, please feel free to message there or contact me directly through Slack PM.

@wkpark wkpark deleted the cuda-toolkit branch February 15, 2024 05:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants