2024.12.20🌟 We are excited to introduce the VITA-1.5, a more powerful and more real-time version!2024.08.12🌟 We are very proud to launch VITA-1.0, the First-Ever open-source interactive omni multimodal LLM! We have submitted the open-source code, yet it is under review internally. We are moving the process forward as quickly as possible, stay tuned!
- VITA-1.5: An Open-Source Interactive Multimodal LLM
On 2024.08.12, we launched VITA-1.0, the first-ever open-source interactive omni-multimodal LLM. Now (2024.12.20), we bring a new version VITA-1.5!
We are excited to present VITA-1.5, which incorporates a series of advancements:
-
Significantly Reduced Interaction Latency. The end-to-end speech interaction latency has been reduced from about 4 seconds to 1.5 seconds, enabling near-instant interaction and greatly improving user experience.
-
Enhanced Multimodal Performance. The average performance on multimodal benchmarks such as MME, MMBench, and MathVista has been significantly increased from 59.8 to 70.8.
-
Improvement in Speech Processing. The speech processing capabilities have been refined to a new level, with ASR WER (Word Error Rate, Test Other) reduced from 18.4 to 7.5. Besides, we replace the independent TTS module of VITA-1.0 with an end-to-end TTS module, which accepts the LLM's embedding as input.
-
Progressive Training Strategy. By this manner, the adding of speech has little effect on other multi-modal performance (vision-language). The average image understanding performance only drops from 71.3 to 70.8.
- Evaluation on image and video understanding benchmarks.
- VITA-1.5 outperforms professional speech models on ASR benchmarks.
- Adding the audio modality has little effect on image and video understanding capability.
git clone https://github.com/VITA-MLLM/VITA
cd VITA
conda create -n vita python=3.10 -y
conda activate vita
pip install --upgrade pip
pip install -r requirements.txt
pip install flash-attn --no-build-isolation
- An example json file of the training data:
[
...
{
"set": "sharegpt4",
"id": "000000000164",
"conversations": [
{
"from": "human",
"value": "<image>\n<audio>\n"
},
{
"from": "gpt", // follow the setting of llave, "gpt" is only used to indicate that this is the ground truth of the model output
"value": "This is a well-organized kitchen with a clean, modern aesthetic. The kitchen features a white countertop against a white wall, creating a bright and airy atmosphere. "
}
],
"image": "coco/images/train2017/000000000164.jpg",
"audio": [
"new_value_dict_0717/output_wavs/f61cf238b7872b4903e1fc15dcb5a50c.wav"
]
},
...
]
- The
setfield is used to retrieve the image or video folder for data loading. You should add its key-value pair to theFolderDictin ./vita/config/dataset_config.py:
AudioFolder = ""
FolderDict = {
#### NaturalCap
"sharegpt4": "",
}
#### NaturalCap
ShareGPT4V = {"chat_path": ""}
- Set the JSON path for
"chat_path"in the corresponding dictionary in ./vita/config/dataset_config.py. - Set the audio folder path for
AudioFolderin ./vita/config/dataset_config.py. - Add the data class in
DataConfigin ./vita/config/init.py:
from .dataset_config import *
NaturalCap = [ShareGPT4V]
DataConfig = {
"Pretrain_video": NaturalCap,
}
-
Download the required weights: (1) VITA-1.5 checkpoint, (2) InternViT-300M-448px, and (3) Our pretrained audio encoder in Stage-2 audio-language alignment (refer to Fig. 3 in the paper).
-
Replace the paths in ./script/train/finetuneTaskNeg_qwen_nodes.sh:
...
--model_name_or_path VITA1.5_ckpt \
...
--vision_tower InternViT-300M-448px \
...
--audio_encoder audio-encoder-Qwen2-7B-1107-weight-base-11wh-tunning \
...
- Execute the following commands to start the training process:
export PYTHONPATH=./
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
OUTPUT_DIR=/mnt/cfs/lhj/videomllm_ckpt/outputs/vita_video_audio
bash script/train/finetuneTaskNeg_qwen_nodes.sh ${OUTPUT_DIR}
- Text query
CUDA_VISIBLE_DEVICES=2 python video_audio_demo.py \
--model_path [vita/path] \
--image_path asset/vita_newlog.jpg \
--model_type qwen2p5_instruct \
--conv_mode qwen2p5_instruct \
--question "Describe this images."
- Audio query
CUDA_VISIBLE_DEVICES=4 python video_audio_demo.py \
--model_path [vita/path] \
--image_path asset/vita_newlog.png \
--model_type qwen2p5_instruct \
--conv_mode qwen2p5_instruct \
--audio_path asset/q1.wav
- Noisy audio query
CUDA_VISIBLE_DEVICES=4 python video_audio_demo.py \
--model_path [vita/path] \
--image_path asset/vita_newlog.png \
--model_type qwen2p5_instruct \
--conv_mode qwen2p5_instruct \
--audio_path asset/q2.wav
We have accelerated the model using vLLM. Since VITA has not yet been integrated into vLLM, you need to make some modifications to the vLLM code to adapt it for VITA.
conda create -n vita_demo python==3.10
conda activate vita_demo
pip install -r web_demo/web_demo_requirements.txt
# Backup a new weight file
cp -rL VITA_ckpt/ demo_VITA_ckpt/
mv demo_VITA_ckpt/config.json demo_VITA_ckpt/origin_config.json
cd ./web_demo/vllm_tools
cp -rf qwen2p5_model_weight_file/* ../../demo_VITA_ckpt/
cp -rf vllm_file/* your_anaconda/envs/vita_demo/lib/python3.10/site-packages/vllm/model_executor/models/vita_demo.mp4
python -m web_demo.web_ability_demo demo_VITA_ckpt/To run the real-time interactive demo, you need to make the following preparations:
-
Prepare a VAD (Voice Activity Detection) module. You can choose to download silero_vad.onnx and silero_vad.jit, and place these files in the
./web_demo/wakeup_and_vad/resource/directory. -
For a better real-time interactive experience, you need to set
max_dynamic_patchto 1 indemo_VITA_ckpt/config.json. When you run the basic demo, you can set it to the default value of 12 to enhance the model's visual capabilities.
pip install flask==3.1.0 flask-socketio==5.5.0 cryptography==44.0.0 timm==1.0.12
python -m web_demo.server --model_path demo_VITA_ckpt --ip 0.0.0.0 --port 8081Modify the model path of vita_qwen2 in VLMEvalKit/vlmeval/config.py
vita_series = {
'vita': partial(VITA, model_path='/path/to/model'),
'vita_qwen2': partial(VITAQwen2, model_path='/path/to/model'),
}
Follow the instuctions in VLMEvalKit to set the GPT as the judge model.
If the openai api are not available, you can use a local model as the judge. In our experiments, we find that Qwen1.5-1.8B-Chat judge can work well compared to GPT-4, except in MM-Vet. To start the judge:
CUDA_VISIBLE_DEVICES=0 lmdeploy serve api_server /mnt/cfs/lhj/model_weights/Qwen1.5-1.8B-Chat --server-port 23333
Then configure the .env file in the VLMEvalKit folder:
OPENAI_API_KEY=sk-123456
OPENAI_API_BASE=http://0.0.0.0:23333/v1/chat/completions
LOCAL_LLM=/mnt/cfs/lhj/model_weights/Qwen1.5-1.8B-Chat
Evaluating on these benchmarks:
CUDA_VISIBLE_DEVICES=0 python run.py --data MMBench_TEST_EN_V11 MMBench_TEST_CN_V11 MMStar MMMU_DEV_VAL MathVista_MINI HallusionBench AI2D_TEST OCRBench MMVet MME --model vita_qwen2 --verbose
Download the Video-MME dataset and extract the frames, saving them as images to improve IO efficiency.
cd ./videomme
Run the model on Video-MME in the setting of wo/ subtitles:
VIDEO_TYPE="s,m,l"
NAMES=(lyd jyg wzh wzz zcy by dyh lfy)
for((i=0; i<${#NAMES[@]}; i++))
do
CUDA_VISIBLE_DEVICES=6 python yt_video_inference_qa_imgs.py \
--model-path [vita/path] \
--model_type qwen2p5_instruct \
--conv_mode qwen2p5_instruct \
--responsible_man ${NAMES[i]} \
--video_type $VIDEO_TYPE \
--output_dir qa_wo_sub \
--video_dir [Video-MME-imgs] | tee logs/infer.log
done
Run the model on Video-MME in the setting of w/ subtitles:
VIDEO_TYPE="s,m,l"
NAMES=(lyd jyg wzh wzz zcy by dyh lfy)
for((i=0; i<${#NAMES[@]}; i++))
do
CUDA_VISIBLE_DEVICES=7 python yt_video_inference_qa_imgs.py \
--model-path [vita/path] \
--model_type qwen2p5_instruct \
--conv_mode qwen2p5_instruct \
--responsible_man ${NAMES[i]} \
--video_type $VIDEO_TYPE \
--output_dir qa_w_sub \
--video_dir [Video-MME-imgs] \
--use_subtitles | tee logs/infer.log
done
Parse the results:
python parse_answer.py --video_types "s,m,l" --result_dir qa_wo_sub
python parse_answer.py --video_types "s,m,l" --result_dir qa_w_sub
If you find our work helpful for your research, please consider citing our work.
@article{fu2024vita,
title={Vita: Towards open-source interactive omni multimodal llm},
author={Fu, Chaoyou and Lin, Haojia and Long, Zuwei and Shen, Yunhang and Zhao, Meng and Zhang, Yifan and Wang, Xiong and Yin, Di and Ma, Long and Zheng, Xiawu and others},
journal={arXiv preprint arXiv:2408.05211},
year={2024}
}VITA is trained on large-scale open-source corpus, and its output has randomness. Any content generated by VITA does not represent the views of the model developers. We are not responsible for any problems arising from the use, misuse, and dissemination of VITA, including but not limited to public opinion risks and data security issues.
Explore our related researches:
- [VITA-1.0] VITA: Towards Open-Source Interactive Omni Multimodal LLM
- [Awesome-MLLM] A Survey on Multimodal Large Language Models
- [MME] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
- [Video-MME] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
VITA is built with reference to the following outstanding works: LLaVA-1.5, Bunny, ChatUnivi, InternVL, InternViT, Qwen-2.5, VLMEvalkit, and Mixtral 8*7B. Thanks！
