This will be my walk through of the book "Building Large Language Models from Scratch" by Sebastian Raschka
As of Monday, September 23, 2024, this book has not yet been released but will be October 29, 2024 on amazon.ca.
I am expecting the current repo to be updated up until that publication date.
mamba activate llmfs
Setting up of the local mamba environment for this book.
- mamba create -n llmfs
- mamba activate llmfs
- mamba install conda-forge::python
- mamba install conda-forge::jupyterlab
- mamba install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
- mamba install conda-forge::tiktoken
- mamba install conda-forge::matplotlib
- mamba install conda-forge::pandas
- mamba install conda-forge::tqdm
- mamba install conda-forge::psutil
Gotta say he really explains things well and so far does not skim over any important details! I really like that!
- mamba install conda-forge::transformers
Continue to go through this most excellent resource! It is really well written! And the original repo continues to be updated to this day!
Starting to go through 'Chapter 5 - Pretraining on unlabeled data'
- pip install thop
Looks like we need tensorflow for Chapter 5. Rather then just installing it to the environment, I will create a new environment for it by cloning 'llmfs' to 'llmfs-tf' and then installing tensorflow to that environment.
mamba create -n llmfs-tf --clone llmfs mamba activate llmfs-tf pip install tensorflow
Nice! This new environment works with the code from chapter 5.
mamba activate llmfs
- mamba install conda-forge::ipywidgets
Creating an environment.yml file for the llmfs-tf environment. Then I will create a new environment from that file on KAUWITB.
Chapter07/01_main-chapter-code/ch07.ipynb can run using the 4090, not the 2070 Super.