Code

Add LBFGS optimizer as an option#792

Code

Draft

vue1999 wants to merge 52 commits intoACEsuit:mainACEsuit/mace:mainfrom vue1999:lbfgs-multi-gpuvue1999/mace:lbfgs-multi-gpuCopy head branch name to clipboard

+174-25

Commits

Commits on Dec 4, 2024

• using torch.optim.LBFGS

  

  vue1999

  committed
  83d15e3

  Copy full SHA for 83d15e3

Commits on Dec 13, 2024

• add LBFGS step using lbfgsnew.py

  

  vue1999

  committed
  7b06eca

  Copy full SHA for 7b06eca

Commits on Dec 14, 2024

• add lbfgs_config argument

  ttompa

  committed
  2677ae2

  Copy full SHA for 2677ae2

Commits on Dec 17, 2024

• add lbfgs as an option for the optimizer argument

  ttompa

  committed
  337a607

  Copy full SHA for 337a607

• remove --lbfgs argument

  ttompa

  committed
  5945ba1

  Copy full SHA for 5945ba1

• cleanup

  ttompa

  committed
  5d9ff24

  Copy full SHA for 5d9ff24

Commits on Dec 18, 2024

• move lbfgs out of optimizer args, because checkpoint loading conflicts with different optimizers

  ttompa

  committed
  fd57dae

  Copy full SHA for fd57dae

• cleanup

  ttompa

  committed
  5002ff3

  Copy full SHA for 5002ff3

• add CUDA memory logs for debugging

  ttompa

  committed
  93b3e41

  Copy full SHA for 93b3e41

• test - enable keep_graph for foundation model

  ttompa

  committed
  34d7e26

  Copy full SHA for 34d7e26

Commits on Dec 19, 2024

• disable retain_graph

  

  vue1999

  committed
  5eae9d7

  Copy full SHA for 5eae9d7

• train only readouts with lbfgs

  ttompa

  committed
  9cb5d65

  Copy full SHA for 9cb5d65

Commits on Dec 21, 2024

• make LBFGS always use full batch, but calculate gradients from mini batches

  ttompa

  committed
  76b333e

  Copy full SHA for 76b333e

Commits on Dec 22, 2024

• fixes for the lbfgs step

  ttompa

  committed
  246655c

  Copy full SHA for 246655c

• use pytorch's LBFGS optimizer

  ttompa

  committed
  c684f42

  Copy full SHA for c684f42

Commits on Dec 23, 2024

• add shampoo optimizer

  ttompa

  committed
  af19ccc

  Copy full SHA for af19ccc

• initial multi GPU LBFGS test

  ttompa

  committed
  be846c0

  Copy full SHA for be846c0

Commits on Dec 24, 2024

• small fix

  ttompa

  committed
  eba347a

  Copy full SHA for eba347a

Commits on Dec 28, 2024

• cleanup

  ttompa

  committed
  2d422a3

  Copy full SHA for 2d422a3

Commits on Dec 29, 2024

• add more barriers and try to prevent LBFGS steps from running on extra GPUs

  ttompa

  committed
  321996d

  Copy full SHA for 321996d

• Revert "add more barriers and try to prevent LBFGS steps from running on extra GPUs"

  Show description for 996b0b0
  ttompa

  committed
  996b0b0

  Copy full SHA for 996b0b0

• add extra barriers to keep GPUs in sync

  ttompa

  committed
  da65079

  Copy full SHA for da65079

• Revert "add extra barriers to keep GPUs in sync"

  Show description for e134289
  ttompa

  committed
  e134289

  Copy full SHA for e134289

• add logging for debugging multi GPU LBFGS

  ttompa

  committed
  3de91bf

  Copy full SHA for 3de91bf

• test a different shampoo implementation

  ttompa

  committed
  098d0c8

  Copy full SHA for 098d0c8

• log on all GPUs

  ttompa

  committed
  3cd1cf5

  Copy full SHA for 3cd1cf5

• print batch size

  ttompa

  committed
  84aecdd

  Copy full SHA for 84aecdd

• evert "test a different shampoo implementation"

  Show description for 4e1ea2f
  ttompa

  committed
  4e1ea2f

  Copy full SHA for 4e1ea2f

• tune shampoo params

  ttompa

  committed
  5fa3a34

  Copy full SHA for 5fa3a34

• add waiting signal for multi GPU LBFGS

  ttompa

  committed
  71b86b2

  Copy full SHA for 71b86b2

• Merge branch 'lbfgs-multi-gpu' into shampoo-optimizer

  vue1999

  authored
  4393f6f

  Copy full SHA for 4393f6f

• Merge pull request #2 from vue1999/shampoo-optimizer

  Show description for 81020d5
  vue1999

  authored
  81020d5

  Copy full SHA for 81020d5

• Merge branch 'ACEsuit:main' into lbfgs-multi-gpu

  vue1999

  authored
  3774cbb

  Copy full SHA for 3774cbb

• workaround for shampoo's lack of checkpoint handling for testing

  ttompa

  committed
  be70e51

  Copy full SHA for be70e51

Commits on Dec 30, 2024

• add wolfe line search, remove debug logs, remove redundant barriers

  ttompa

  committed
  bc02bcb

  Copy full SHA for bc02bcb

• adjust normalisation to handle varying batch sizes

  ttompa

  committed
  99c8c8c

  Copy full SHA for 99c8c8c

• fix single GPU LBFGS training

  ttompa

  committed
  482a864

  Copy full SHA for 482a864

• remove redundant barrier

  ttompa

  committed
  45fee81

  Copy full SHA for 45fee81

• fix spelling

  ttompa

  committed
  2f2be1d

  Copy full SHA for 2f2be1d

• remove debug logging

  ttompa

  committed
  f252276

  Copy full SHA for f252276

• adjust shampoo params for testing

  ttompa

  committed
  9923ba1

  Copy full SHA for 9923ba1

• adjust shampoo params for testing

  ttompa

  committed
  4532b12

  Copy full SHA for 4532b12

Commits on Jan 12, 2025

• Merge branch 'ACEsuit:main' into lbfgs-multi-gpu

  ttompa

  authored
  1bf4bb7

  Copy full SHA for 1bf4bb7

• remove shampoo related code

  ttompa

  committed
  63a3c7c

  Copy full SHA for 63a3c7c

• cleanup

  ttompa

  committed
  4b9ee30

  Copy full SHA for 4b9ee30

• simplify LBFGS option

  ttompa

  committed
  8b0b396

  Copy full SHA for 8b0b396

• cleanup

  ttompa

  committed
  8e26459

  Copy full SHA for 8e26459

• fix restarting LBFGS checkpoints

  ttompa

  committed
  0857727

  Copy full SHA for 0857727

Commits on Jan 15, 2025

• reduce iteration count for spice test, always save last checkpoint

  ttompa

  committed
  aa392a6

  Copy full SHA for aa392a6

Commits on Jan 20, 2025

• disable drop_last when using LBFGS, adjust normalisation to account for this

  ttompa

  committed
  f2243bc

  Copy full SHA for f2243bc

Commits on Jan 21, 2025

• Fix issue by initialising opt_start_epoch as None

  vue1999

  authored
  5da7dff

  Copy full SHA for 5da7dff

• fix formatting to make CI checks happy

  

  vue1999

  committed
  1b138ce

  Copy full SHA for 1b138ce