Open
Description
Fix for gemma-2-9b - run with blfloat16
https://huggingface.co/google/gemma-2-27b/tree/main
Times
- 6:03 for CPU only 14900K 128G 4200Mhz ram - running 120G
- for GPU+RAM NVidia A6000 Ada 48G + 13900K 128G 4200Mhz ram - running 47 + 87G = 134G
code change
#model = "google/gemma-7b"
model = "google/gemma-2-27b"
tokenizer = AutoTokenizer.from_pretrained(model, token=access_token)
# GPU
model = AutoModelForCausalLM.from_pretrained(model, device_map="auto", token=access_token)
# CPUi
#model = AutoModelForCausalLM.from_pretrained(model,token=access_token)
michael@14900c MINGW64 /c/wse_github/obrienlabsdev/machine-learning/environments/windows/src/google-gemma (main)
$ python gemma-gpu.py
Traceback (most recent call last):
File "C:\opt\Python312\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 945, in from_pretrained
config_class = CONFIG_MAPPING[config_dict["model_type"]]
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\opt\Python312\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 647, in __getitem__
raise KeyError(key)
KeyError: 'gemma2'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\wse_github\obrienlabsdev\machine-learning\environments\windows\src\google-gemma\gemma-gpu.py", line 16, in <module>
model = AutoModelForCausalLM.from_pretrained(model, device_map="auto", token=access_token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\opt\Python312\Lib\site-packages\transformers\models\auto\auto_factory.py", line 523, in from_pretrained
config, kwargs = AutoConfig.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\opt\Python312\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 947, in from_pretrained
raise ValueError(
ValueError: The checkpoint you are trying to load has model type `gemma2` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.