Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add LBFGS optimizer as an option #792

Draft
wants to merge 52 commits into
base: main
Choose a base branch
from
Draft

Add LBFGS optimizer as an option #792

wants to merge 52 commits into from

Conversation

vue1999
Copy link
Collaborator

@vue1999 vue1999 commented Jan 20, 2025

No description provided.

vue1999 and others added 30 commits December 4, 2024 00:10
using torch.optim.LBFGS
83d15e3
add LBFGS step using lbfgsnew.py
7b06eca
@ttompa
add lbfgs_config argument
2677ae2
@ttompa
add lbfgs as an option for the optimizer argument
337a607
@ttompa
remove --lbfgs argument
5945ba1
@ttompa
cleanup
5d9ff24
@ttompa
move lbfgs out of optimizer args, because checkpoint loading conflict…
fd57dae 
…s with different optimizers
@ttompa
cleanup
5002ff3
@ttompa
add CUDA memory logs for debugging
93b3e41
@ttompa
test - enable keep_graph for foundation model
34d7e26
disable retain_graph
5eae9d7
@ttompa
train only readouts with lbfgs
9cb5d65
@ttompa
make LBFGS always use full batch, but calculate gradients from mini b…
76b333e 
…atches
@ttompa
fixes for the lbfgs step
246655c
@ttompa
use pytorch's LBFGS optimizer
c684f42
@ttompa
add shampoo optimizer
af19ccc
@ttompa
initial multi GPU LBFGS test
be846c0
@ttompa
small fix
eba347a
@ttompa
cleanup
2d422a3
@ttompa
add more barriers and try to prevent LBFGS steps from running on extr…
321996d 
…a GPUs
@ttompa
Revert "add more barriers and try to prevent LBFGS steps from running…
996b0b0 
… on extra GPUs"

This reverts commit 321996d.
@ttompa
add extra barriers to keep GPUs in sync
da65079
@ttompa
Revert "add extra barriers to keep GPUs in sync"
e134289 
This reverts commit da65079.
@ttompa
add logging for debugging multi GPU LBFGS
3de91bf
@ttompa
test a different shampoo implementation
098d0c8
@ttompa
log on all GPUs
3cd1cf5
@ttompa
print batch size
84aecdd
@ttompa
evert "test a different shampoo implementation"
4e1ea2f 
This reverts commit 098d0c8.
@ttompa
tune shampoo params
5fa3a34
@ttompa
add waiting signal for multi GPU LBFGS
71b86b2
vue1999 and others added 22 commits December 29, 2024 22:31
@vue1999
Merge branch 'lbfgs-multi-gpu' into shampoo-optimizer
4393f6f
@vue1999
Merge pull request #2 from vue1999/shampoo-optimizer
81020d5 
Shampoo optimizer
@vue1999
Merge branch 'ACEsuit:main' into lbfgs-multi-gpu
3774cbb
@ttompa
workaround for shampoo's lack of checkpoint handling for testing
be70e51
@ttompa
add wolfe line search, remove debug logs, remove redundant barriers
bc02bcb
@ttompa
adjust normalisation to handle varying batch sizes
99c8c8c
@ttompa
fix single GPU LBFGS training
482a864
@ttompa
remove redundant barrier
45fee81
@ttompa
fix spelling
2f2be1d
@ttompa
remove debug logging
f252276
@ttompa
adjust shampoo params for testing
9923ba1
@ttompa
adjust shampoo params for testing
4532b12
@ttompa
Merge branch 'ACEsuit:main' into lbfgs-multi-gpu
1bf4bb7
@ttompa
remove shampoo related code
63a3c7c
@ttompa
cleanup
4b9ee30
@ttompa
simplify LBFGS option
8b0b396
@ttompa
cleanup
8e26459
@ttompa
fix restarting LBFGS checkpoints
0857727
@ttompa
reduce iteration count for spice test, always save last checkpoint
aa392a6
@ttompa
disable drop_last when using LBFGS, adjust normalisation to account f…
f2243bc 
…or this
@vue1999
Fix issue by initialising opt_start_epoch as None
5da7dff
fix formatting to make CI checks happy
1b138ce
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@vue1999 @ttompa