Is it possible to use a locally downloaded model without accessing HF?

[ccp123456]

Question

I want to try to experiment with Qwen 1.5 and I already have the parameters tuned and saved locally, how can I load the trained parameters instead of downloading them in HF when using the function HookedTransformer.from_pretrained?

[neelnanda-io]

Are the params in TransformerLens format? If so, just run HookedTransformer(cfg) on the relevant config, and then do `model.load_and_process_state_dict(saved_params)` If they're in HuggingFace format, load them into a HuggingFace model and do `HookedTransformer.from_pretrained("qwen-1.5-1b", hf_model=YOUR_SAVED_TUNED_MODEL)`

…

On Thu, 4 Jul 2024 at 13:39, ccp123456 ***@***.***> wrote: Question I want to try to experiment with Qwen 1.5 and I already have the parameters tuned and saved locally, how can I load the trained parameters instead of downloading them in HF when using the function HookedTransformer.from_pretrained? — Reply to this email directly, view it on GitHub <#655>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASRPNKLTN2TIXD4ZO7U5XL3ZKU67ZAVCNFSM6AAAAABKLPC7RCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGM4TANZXG43DANA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ccp123456]

Are the params in TransformerLens format? If so, just run HookedTransformer(cfg) on the relevant config, and then do model.load_and_process_state_dict(saved_params) If they're in HuggingFace format, load them into a HuggingFace model and do HookedTransformer.from_pretrained("qwen-1.5-1b", hf_model=YOUR_SAVED_TUNED_MODEL)
…
On Thu, 4 Jul 2024 at 13:39, ccp123456 @.> wrote: Question I want to try to experiment with Qwen 1.5 and I already have the parameters tuned and saved locally, how can I load the trained parameters instead of downloading them in HF when using the function HookedTransformer.from_pretrained? — Reply to this email directly, view it on GitHub <#655>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASRPNKLTN2TIXD4ZO7U5XL3ZKU67ZAVCNFSM6AAAAABKLPC7RCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGM4TANZXG43DANA . You are receiving this because you are subscribed to this thread.Message ID: @.>

Yes, I tried to do this, but it still tried to access“ https://huggingface.co I would like to know if there is a way to load the model without accessing hf? My model is in hf format.

[neelnanda-io]

Does it try to access it on the line where you load them into a HF model, or the line where you do HookedTransformer.from_pretrained? The former should be easy to fix based on the HF docs, they support loading from local weights I think. The latter might be because it wants the config...

…

On Fri, 5 Jul 2024 at 07:03, ccp123456 ***@***.***> wrote: Are the params in TransformerLens format? If so, just run HookedTransformer(cfg) on the relevant config, and then do model.load_and_process_state_dict(saved_params) If they're in HuggingFace format, load them into a HuggingFace model and do HookedTransformer.from_pretrained("qwen-1.5-1b", hf_model=YOUR_SAVED_TUNED_MODEL) … <#m_-2970033271189213853_> On Thu, 4 Jul 2024 at 13:39, ccp123456 *@*.*> wrote: Question I want to try to experiment with Qwen 1.5 and I already have the parameters tuned and saved locally, how can I load the trained parameters instead of downloading them in HF when using the function HookedTransformer.from_pretrained? — Reply to this email directly, view it on GitHub <#655 <#655>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASRPNKLTN2TIXD4ZO7U5XL3ZKU67ZAVCNFSM6AAAAABKLPC7RCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGM4TANZXG43DANA <https://github.com/notifications/unsubscribe-auth/ASRPNKLTN2TIXD4ZO7U5XL3ZKU67ZAVCNFSM6AAAAABKLPC7RCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGM4TANZXG43DANA> . You are receiving this because you are subscribed to this thread.Message ID: @.*> Yes, I tried to do this, but it still tried to access“ https://huggingface.co I would like to know if there is a way to load the model without accessing hf? My model is in hf format. — Reply to this email directly, view it on GitHub <#655 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASRPNKNHHF2FJJM7HTOOW7LZKYZM3AVCNFSM6AAAAABKLPC7RCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJQGIZTKOBVGA> . You are receiving this because you commented.Message ID: ***@***.***>

[mylesgoose]

`MODEL_ID = "meta-llama/Meta-Llama-3.1-8B-Instruct"
NEW_MODEL_ID = "mylesgoose/Meta-Llama-3.1-8B-Instruct"
MODEL_TYPE = "meta-llama/Meta-Llama-3-8B-Instruct"

Download and load model

!git clone https://huggingface.co/{MODEL_ID} {MODEL_TYPE}

Load model and tokenizer

model = HookedTransformer.from_pretrained_no_processing(
MODEL_TYPE,
local_files_only=True,
dtype=torch.bfloat16,
default_padding_side='left'
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_TYPE)
tokenizer.padding_side = 'left'
tokenizer.pad_token = tokenizer.eos_token` Im not sure if this helps but this downloads and loads from the lcoal folder only local_files_only=True,

[clclclaiggg]

Hello, I have the same issue. Loading other local models works successfully, but loading the local Qwen model keeps attempting to download from Hugging Face.

[Nebularaid2000]

@ccp123456 @clclclaiggg Hi guys, I think there's one way to circumvent this issue. You could set the default cache dir of huggingface hub to the path where you store your local models, before loading any models.

import os
os.environ['HF_HUB_HOME'] = root_path_where_you_store_models

You may need to ensure your models are stored in the same format as huggingface models, e.g., having blobs/refs/snapshots dirs. And also make sure the model directory name appears the same as that of huggingface models, e.g., models--gpt2.
I'm currently using this method because huggingface.co is walled from my area. So I need to use my local models downloaded previously.

[gnehcoul]

@ccp123456 @clclclaiggg Hi guys, I think there's one way to circumvent this issue. You could set the default cache dir of huggingface hub to the path where you store your local models, before loading any models.

import os
os.environ['HF_HUB_HOME'] = root_path_where_you_store_models

You may need to ensure your models are stored in the same format as huggingface models, e.g., having blobs/refs/snapshots dirs. And also make sure the model directory name appears the same as that of huggingface models, e.g., models--gpt2. I'm currently using this method because huggingface.co is walled from my area. So I need to use my local models downloaded previously.

Could you show the detail code?

[Nebularaid2000]

@ccp123456 @clclclaiggg Hi guys, I think there's one way to circumvent this issue. You could set the default cache dir of huggingface hub to the path where you store your local models, before loading any models.

import os
os.environ['HF_HUB_HOME'] = root_path_where_you_store_models

You may need to ensure your models are stored in the same format as huggingface models, e.g., having blobs/refs/snapshots dirs. And also make sure the model directory name appears the same as that of huggingface models, e.g., models--gpt2. I'm currently using this method because huggingface.co is walled from my area. So I need to use my local models downloaded previously.

Could you show the detail code?

Sure. The models are in the following structure in my local folder.

huggingface_models
├─ models--gpt2
|    ├─ blobs
|    ├─ refs
|    ├─ snapshots

Then, I load the models using the following code, without requesting huggingface.co.

import transformer_lens
import os

os.environ['HF_HUB_HOME'] = './huggingface_models'
model = transformer_lens.HookedTransformer.from_pretrained('gpt2')

[andrewnam]

@Nebularaid2000 Thanks! This workaround works!

I just had to make sure to set the os.environ before loading HookedTransformer.

[hamind]

I download the model files from Huggingface website and the local folder likes this

I set the os.environ but it doesn't work.

[Nebularaid2000]

@hamind Hi, I think there's a few things you may try.

1. Rename the folder as models--gpt2
2. Use huggingface hub (e.g., see link) to download the files in a linux environment where you can access huggingface.co. Also, make ensure that your folder are organized in the following way (this is the default file hierarchy if you let huggingface automatically download the files for you)

root_dir
├─ models--gpt2
|    ├─ blobs
|    ├─ refs
|    ├─ snapshots

[hamind]

@hamind Hi, I think there's a few things you may try.

1. Rename the folder as models--gpt2
2. Use huggingface hub (e.g., see link) to download the files in a linux environment where you can access huggingface.co. Also, make ensure that your folder are organized in the following way (this is the default file hierarchy if you let huggingface automatically download the files for you)

root_dir
├─ models--gpt2
|    ├─ blobs
|    ├─ refs
|    ├─ snapshots

Thanks a lot. But it won't work.
I change some library code on my own environment and now can load huggingface model locally and successfully.

[Nebularaid2000]

@hamind That's a good practice. I've also seen your proposal to add a function for loading local files. I believe this is a feature worth to be added.

[bryce13950]

I am going to close this in favor of #754. Both of these issues are about improving this by default, and I want to keep the issue tracked in a single place.