Skip to content

Commit

Permalink
Changelog: add explanation r. QLoRA mem savings
Browse files Browse the repository at this point in the history
  • Loading branch information
Titus-von-Koeller committed Jul 27, 2024
1 parent a7c08af commit 7800734
Showing 1 changed file with 8 additions and 0 deletions.
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
### 0.43.2

This release is quite significant as the QLoRA bug fix big implications for higher `seqlen` and batch sizes.

For each sequence (i.e. batch size increase of one) we expect memory savings of:
- 405B: 39GB for `seqlen=1024`, and 4888GB for `seqlen=128,00`
- 70B: 10.1GB for `seqlen=1024` and 1258GB for `seqlen=128,00`

This was due to activations being unnecessary for frozen parameters, yet the memory for them was still erroneously allocated due to the now fixed bug.

#### Improvements:

- docs: FSDP+QLoRA and CPU install guide (#1211 #1227, thanks @stevhliu)
Expand Down

0 comments on commit 7800734

Please sign in to comment.