Merge pull request #72 from homebrewltd/hotfix-info

Fixed incorrect pretrain information
janhq · Aug 26, 2024 · 82ae9d6 · 82ae9d6
2 parents 41f5742 + c97baee
commit 82ae9d6
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/src/pages/blog/llama3-just-got-ears.mdx b/src/pages/blog/llama3-just-got-ears.mdx
@@ -84,7 +84,7 @@ We found it useful to pre-train llama3.1 on continuous speech, through rough abl
 | **Warmup Steps** | 20 |
 | **Weight Decay** | 0.01 |
 | **Gradient Checkpointing** | Full |
-| **Max length** | 4096 |
+| **Max length** | 512 |
 | **Precision** | bf16 |
 
 The learning rate schedule is as follows, starting with a relatively high LR for sufficient warmup.