How to Perform Inference with Fine-Tuned VideoLLaMA3?

[JasonYeh0111]

Thanks for your great work! I have a question: After fine-tuning VideoLLaMA3-2B, how can I use the model for inference? The fine-tuned model has different weights compared to the original VideoLLaMA3-2B, as it includes a vision encoder. Should I use the same configuration file as the original VideoLLaMA3-2B?

[LiangMeng89]

Maybe step 3 can help to finetune your own data in VideoLLaMA 3: "For finetuneing, --model_path is the path to the converted checkpoint as described in step 2."

I also fallow VideoLLaMA series work. We have a wechat group to help each other sloving issues. You can add my wechat number(19357260600) or E-mail([email protected]) to talk with us.

[JasonYeh0111]

Yes, I followed the fine-tuning instructions to convert the model checkpoint and fine-tuned the model using this checkpoint. Do I need to copy same configuration and process files from origin videollama3-2B folder?

[LiangMeng89]

Yes, I followed the fine-tuning instructions to convert the model checkpoint and fine-tuned the model using this checkpoint. Do I need to copy same configuration and process files from origin videollama3-2B folder?

I'm not sure yet, I'm deploying videollama3-7B, ready to train and inference my vertical video data.

[lkhl]

Hi @JasonYeh0111 , thanks for your interests!

For trying your local model, you can follow videollama3/infer.py and you just need to replace the model_path to the path of your checkpoint.