Error while loading Mixtral based SFT MoE model VideoLLaMA2-8x7B: SafetensorError: Error while deserializing header: InvalidHeaderDeserialization

[ApoorvFrontera]

Hi Team,

When I am loading the Mixtral-based SFT MoE model 'DAMO-NLP-SG/VideoLLaMA2-8x7B' using the same inference code provided in the README.md, the following error is raised:

Traceback (most recent call last):
  File "/home/admin/apoorv/development/VideoLLaMA2/playground.py", line 33, in <module>
    inference()
  File "/home/admin/apoorv/development/VideoLLaMA2/playground.py", line 27, in inference
    model, processor, tokenizer = model_init(model_path)
  File "/home/admin/apoorv/development/VideoLLaMA2/videollama2/__init__.py", line 17, in model_init
    tokenizer, model, processor, context_len = load_pretrained_model(model_path, None, model_name, **kwargs)
  File "/home/admin/apoorv/development/VideoLLaMA2/videollama2/model/__init__.py", line 180, in load_pretrained_model
    model = Videollama2MixtralForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, config=config, **kwargs)
  File "/home/admin/.conda/envs/vl2/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3838, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/home/admin/.conda/envs/vl2/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4278, in _load_pretrained_model
    state_dict = load_state_dict(shard_file, is_quantized=is_quantized)
  File "/home/admin/.conda/envs/vl2/lib/python3.10/site-packages/transformers/modeling_utils.py", line 516, in load_state_dict
    with safe_open(checkpoint_file, framework="pt") as f:
safetensors_rust.SafetensorError: Error while deserializing header: InvalidHeaderDeserialization

I tried to find the reason for this and came across the following issues where the main reason was that the file/weights saved were not done correctly and had empty dictionaries which the safetensors can't handle.
huggingface/transformers#27397 (comment)

To solve this, there are some changes to be implemented before saving the model checkpoint by you guys:

1. SafetensorError: Error while deserializing header: InvalidHeaderDeserialization when open .safetensor model huggingface/transformers#27397
2. https://discuss.huggingface.co/t/safetensors-rust-safetenserror-while-deserializing-header-invalidheaderdeserialization/68831
3. Fine-tuning example doesn't work for custom datasets slai-labs/get-beam#94
4. https://huggingface.co/TheBloke/deepseek-coder-33B-instruct-AWQ/discussions/1

[ApoorvFrontera]

Hi Team
Any response or help for this will be very useful.

Thanks in advance.

[HugoDelhaye]

I am having the same issus

[williamium3000]

same issues here.

[leoisufa]

same issues