-
Notifications
You must be signed in to change notification settings - Fork 400
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cost Aware BO Tutorial #1922
Cost Aware BO Tutorial #1922
Conversation
@dme65 not sure how to request an individual review it, normally I'd add an assignee but can't do it here, so hope you see this! |
Thanks @erichanslee! I'll review this! |
This looks great, thanks for putting it up @erichanslee! My main comment is that it would be nice to also implement CArBO since that is probably the method that most people are interested in and since it seems to consistently outperform EIpu. I have a few comments, but this one is good to go otherwise. Comments:
|
Yeah Implementing this with a standard (or deterministic) model makes sense. botorch/botorch/acquisition/cost_aware.py Line 36 in 954fca3
Could you please also add a |
Codecov Report
@@ Coverage Diff @@
## main #1922 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 178 178
Lines 15721 15721
=========================================
Hits 15721 15721 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
Thank you both for taking the time to look at my PR and for the suggestions! I added smoke test coverage using much smaller matrix sizes for the noisy least squares, used Having the Please let me know if there are any other things I should fix, and otherwise feel free to merge in whenever. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for making the changes, @erichanslee. This one looks good to go from my end!
@dme65 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Motivation
To add a tutorial about cost aware bayesian optimization with cost cooling, which was a paper in the AutoML workshop at ICML in 2020. A few people had been asking me for code and I thought it would be nice to have something in Botorch.
https://arxiv.org/pdf/2003.10870.pdf
The tutorial covers how to define a cost model, how to define expected improvement per unit using this cost model (expected improvement normalized by the cost model, with an additional decay factor to prevent evaluation of too many cheap points). It then runs through a fun little problem about noisy least squares estimation and shows the benefit of this cost-aware approach. It also stresses the importance of using an accurate cost model if you have a reasonable idea of what it ought to look like (as we do with least squares estimation).
Have you read the Contributing Guidelines on pull requests?
Yes I have, and I have signed the open source agreement.
Test Plan
No new testing required.
Related PRs
No related PRs.