Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bayesian Ensemble model for Llama3 (SGHMC) #93

Closed
wants to merge 63 commits into from
Closed

Bayesian Ensemble model for Llama3 (SGHMC) #93

wants to merge 63 commits into from

Conversation

johnathanchiu
Copy link
Contributor

@johnathanchiu johnathanchiu commented May 21, 2024 •
edited by KaelanDt
Loading

Experiment code for paper results on statement completion for Llama3 ensemble model.

  • Need to add README for install instruction
  • Need to add instructions for training
  • Short explanation on training code vs. evaluation code

SamDuffield and others added 30 commits April 30, 2024 19:08
@SamDuffield
Merge pull request #3 from normal-computing/sync
7f82b22 
Sync with changes from posteriors
@phoebeklett
Idea for BayesLlama class
94dfd42
@phoebeklett
Pare down code, vectorize
ea3dce7
@phoebeklett
Add back hyperparams, make loading weights functional, use native fun…
b6f6914 
…ctions
@phoebeklett
Make folder for experiment, generalize to all parameter options, add …
7b1aa60 
…demo
@phoebeklett
eliminate einops dependency
176f577
@phoebeklett
Update base classes so that bayes ensemble copies register as named p…
ce2dfd9 
…arameters, v0 load functions
@phoebeklett
skeleton for training script
b54b009
@johnathanchiu
start setting up llama3 bayesian training code
840f34d
@johnathanchiu
add transform and state
b68a5ee
@johnathanchiu
get training running, need to debug OOM issues
58a0910
@johnathanchiu
get running, need to fix issue with functional call not being done pr…
e3540d3 
…operly
@johnathanchiu
remove cropping from logits
edfd744
@johnathanchiu
get code fully running, need to add logging
6139d3c
@johnathanchiu
fix aux, log posterior outputs
684e37d
@phoebeklett
remove parameter type complexity
1d623d6
@johnathanchiu
save only trained weights
8fb80eb
@johnathanchiu
Merge branch 'llama3-training' of github.com:normal-computing/posteri…
cd7ab12 
…ors-dev into llama3-training
@johnathanchiu
save folders based on experiment config file name
dad693b
@johnathanchiu
add params for alpha, beta, other hyperparams
119f528
@phoebeklett
Update inference class to use state dict, update demo accoridingly
4b755f4
@johnathanchiu
bayesian layering with deepcopy and loading
bf81f3b
@johnathanchiu
remove unused load function
caf0217
@johnathanchiu
add logs to .gitignore
1fbeff8
@johnathanchiu
allowing setting temperature in config
f54b6a8
@johnathanchiu
add shuffling for data, add sequence length multiplier in temperature
4e29406
@phoebeklett
Update for generation step
2e7e304
@johnathanchiu
fix kv cache with ensemble layers
b511d75
@phoebeklett
Add eval script, helpers, example conf
2bbf0f1
@phoebeklett
Add runner and utils
0a1c573
johnathanchiu and others added 23 commits May 9, 2024 17:29
@johnathanchiu
fix dataloader to batch after sequences are tokenized
84b8c7a
@johnathanchiu
remove truncation param
dfeeef1
@johnathanchiu
debug issue with tokenizer, need to set padding to the left side
d9658df
@johnathanchiu
add truncation back, it is necessary
a386595
@johnathanchiu
fix eval to use instruct model weights, add eval code for the base mo…
ddd6f42 
…del as well
@johnathanchiu
add ensemble training script
9083d15
@johnathanchiu
fix seq length parameter input and grab from stride length
a80d599
@johnathanchiu
fix eval script to eval base model as well, eval on ensemble model
87aa2ad
@johnathanchiu
setup to eval with head_qa
48b5a74
@johnathanchiu
remove experiments folder, fix eval folder missing param
455d7a4
@johnathanchiu
fix uncertainties calculation, reduce number of output tokens
4c489d5
@johnathanchiu
move off gpu when calculating uncertainties
c7eef9d
add base configs
6f2fec6
@johnathanchiu
add eval for base llama3
bef6efe
@johnathanchiu
train on test data
a76d803
@johnathanchiu
include train, val, test in training for tqa
c6f08aa
@johnathanchiu
add ignore first n tokens
87fd8d5
@johnathanchiu
config for experiments, allow multiple runs of pure SGD
65e5dfb
@johnathanchiu
add saving for last checkpoint
a848010
@johnathanchiu
clean up config files
491af26
@johnathanchiu
save last checkpoint
7e39ed7
@johnathanchiu
clean up eval code for statement completion
3f218f9
@johnathanchiu
add code for plotting experiment results
d198280
@johnathanchiu johnathanchiu deleted the llama3-training branch May 22, 2024 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SamDuffield SamDuffield Awaiting requested review from SamDuffield

@phoebeklett phoebeklett Awaiting requested review from phoebeklett

@KaelanDt KaelanDt Awaiting requested review from KaelanDt

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@johnathanchiu @SamDuffield @phoebeklett