Large multi-modal models for strong performance and efficient deployment

中文 | English

MiniCPM-V 2.0 🤗 🤖 | OmniLMM-12B 🤗 🤖 | Technical Blog

MiniCPM-V and OmniLMM are a family of open-source large multimodal models (LMMs) adept at vision & language modeling. The models process images and text inputs and deliver high-quality text outputs. We release two featured versions that are targeted at strong performance and efficient deployment:

MiniCPM-V 2.8B: State-of-the-art end-side large multimodal models. Our latest MiniCPM-V 2.0 can accept 1.8 million pixels (e.g., 1344x1344) images at any aspect ratio, and is adept at OCR capability. It achieves comparable performance with Gemini Pro in understanding scene-text and matches GPT-4V in preventing hallucinations.
OmniLMM 12B: The most capable version with leading performance among comparable-sized models on multiple benchmarks. The model also achieves state-of-the-art performance in trustworthy behaviors, with even less hallucination than GPT-4V.

News

[2024.04.23] MiniCPM-V-2.0 supports vLLM now! Click here to view more details.
[2024.04.18] We create a HuggingFace Space to host the demo of MiniCPM-V 2.0 at here!
[2024.04.17] MiniCPM-V-2.0 supports deploying WebUI Demo now!
[2024.04.15] MiniCPM-V-2.0 now also supports fine-tuning with the SWIFT framework!
[2024.04.12] We open-source MiniCPM-V-2.0, which achieves comparable performance with Gemini Pro in understanding scene text and outperforms strong Qwen-VL-Chat 9.6B and Yi-VL 34B on OpenCompass, a comprehensive evaluation over 11 popular benchmarks. Click here to view the MiniCPM-V 2.0 technical blog.
[2024.03.14] MiniCPM-V now supports fine-tuning with the SWIFT framework. Thanks to Jintao for the contribution！
[2024.03.01] MiniCPM-V now can be deployed on Mac!
[2024.02.01] We open-source MiniCPM-V and OmniLMM-12B, which support efficient end-side deployment and powerful multimodal capabilities correspondingly.

MiniCPM-V 2.8B

MiniCPM-V 2.8B is an efficient version with promising performance for deployment. The model is built based on SigLip-400M and MiniCPM-2.4B, connected by a perceiver resampler. Our latest version, MiniCPM-V 2.0 has several notable features.

🔥 State-of-the-art Performance.

MiniCPM-V 2.0 achieves state-of-the-art performance on multiple benchmarks (including OCRBench, TextVQA, MME, MMB, MathVista, etc) among models under 7B parameters. It even outperforms strong Qwen-VL-Chat 9.6B, CogVLM-Chat 17.4B, and Yi-VL 34B on OpenCompass, a comprehensive evaluation over 11 popular benchmarks. Notably, MiniCPM-V 2.0 shows strong OCR capability, achieving comparable performance to Gemini Pro in scene-text understanding, and state-of-the-art performance on OCRBench among open-source models.
🏆 Trustworthy Behavior.

LMMs are known for suffering from hallucination, often generating text not factually grounded in images. MiniCPM-V 2.0 is the first end-side LMM aligned via multimodal RLHF for trustworthy behavior (using the recent RLHF-V [CVPR'24] series technique). This allows the model to match GPT-4V in preventing hallucinations on Object HalBench.
🌟 High-Resolution Images at Any Aspect Raito.

MiniCPM-V 2.0 can accept 1.8 million pixels (e.g., 1344x1344) images at any aspect ratio. This enables better perception of fine-grained visual information such as small objects and optical characters, which is achieved via a recent technique from LLaVA-UHD.
⚡️ High Efficiency.

MiniCPM-V 2.0 can be efficiently deployed on most GPU cards and personal computers, and even on end devices such as mobile phones. For visual encoding, we compress the image representations into much fewer tokens via a perceiver resampler. This allows MiniCPM-V 2.0 to operate with favorable memory cost and speed during inference even when dealing with high-resolution images.
🙌 Bilingual Support.

MiniCPM-V 2.0 supports strong bilingual multimodal capabilities in both English and Chinese. This is enabled by generalizing multimodal capabilities across languages, a technique from VisCPM [ICLR'24].

Evaluation

Click to view results on TextVQA, DocVQA, OCRBench, OpenCompass, MME, MMBench, MMMU, MathVista, LLaVA Bench, Object HalBench.

Model	Size	TextVQA val	DocVQA test	OCRBench	OpenCompass	MME	MMB dev(en)	MMB dev(zh)	MMMU val	MathVista	LLaVA Bench	Object HalBench
Proprietary models
Gemini Pro Vision	-	74.6	88.1	680	63.8	2148.9	75.2	74.0	48.9	45.8	79.9	-
GPT-4V	-	78.0	88.4	645	63.2	1771.5	75.1	75.0	53.8	47.8	93.1	86.4 / 92.7
Open-source models 6B~34B
Yi-VL-6B	6.7B	45.5*	17.1*	290	49.3	1915.1	68.6	68.3	40.3	28.8	51.9	-
Qwen-VL-Chat	9.6B	61.5	62.6	488	52.1	1860.0	60.6	56.7	37.0	33.8	67.7	56.2 / 80.0
Yi-VL-34B	34B	43.4*	16.9*	290	52.6	2050.2	71.1	71.4	45.1	30.7	62.3	-
DeepSeek-VL-7B	7.3B	64.7*	47.0*	435	55.6	1765.4	74.1	72.8	38.3	36.8	77.8	-
TextMonkey	9.7B	64.3	66.7	558	-	-	-	-	-	-	-	-
CogVLM-Chat	17.4B	70.4	33.3*	590	52.5	1736.6	63.7	53.8	37.3	34.7	73.9	73.6 / 87.4
Open-source models 1B~3B
DeepSeek-VL-1.3B	1.7B	58.4*	37.9*	413	46.0	1531.6	64.0	61.2	33.8	29.4	51.1	-
MobileVLM V2	3.1B	57.5	19.4*	-	-	1440.5(P)	63.2	-	-	-	-	-
Mini-Gemini	2.2B	56.2	34.2*	-	-	1653.0	59.8	-	31.7	-	-	-
MiniCPM-V	2.8B	60.6	38.2	366	47.6	1650.2	67.9	65.3	38.3	28.9	51.3	78.4 / 88.5
MiniCPM-V 2.0	2.8B	74.1	71.9	605	55.0	1808.6	69.6	68.1	38.2	38.7	69.2	85.5 / 92.2

* We evaluate the officially released checkpoint by ourselves.

Examples

We deploy MiniCPM-V 2.0 on end devices. The demo video is the raw screen recording on a Xiaomi 14 Pro without edition.

MiniCPM-V 1.0

Please see the info about MiniCPM-V 1.0 here.

OmniLMM-12B

OmniLMM-12B is the most capable version. The model is built based on EVA02-5B and Zephyr-7B-β, connected with a perceiver resampler layer, and trained on multimodal data in a curriculum fashion. The model has three notable features:

🔥 Strong Performance.

OmniLMM-12B achieves leading performance among models with comparable sizes, surpassing established LMMs on multiple benchmarks (including MME, MMBench, SEED-Bench, etc). The model also endows rich multi-modal world knowledge.
🏆 Trustworthy Behavior.

LMMs are known for suffering from hallucination, often generating text that is not factually grounded in images (e.g., faithfully describing non-existing objects in images). OmniLMM-12B is the first state-of-the-art open-source LMM aligned via multimodal RLHF for trustworthy behavior (using the recent RLHF-V technique). It ranks #1 among open-source models on MMHal-Bench, and outperforms GPT-4V on Object HalBench.
🕹 Real-time Multimodal Interaction.

We combine the OmniLMM-12B and GPT-3.5 (text-only) into a real-time multimodal interactive assistant. The assistant accepts video streams from the camera and speech streams from the microphone and emits speech output. While still primary, we find the model can replicate some of the fun cases shown in the Gemini Demo video, without any video edition.

Evaluation

Click to view results on MME, MMBench, MMMU, MMBench, MMHal-Bench, Object HalBench, SeedBench, LLaVA Bench, MathVista.

Model	Size	MME	MMB dev (en)	MMMU val	MMHal-Bench	Object HalBench	SeedBench-I	MathVista	LLaVA Bench
GPT-4V†	-	1771.5	75.1	56.8	3.53 / 70.8	86.4 / 92.7	71.6	47.8	93.1
Qwen-VL-Plus†	-	2183.4	66.2	45.2	-	-	65.7	36.0	73.7
Yi-VL 6B	6.7B	1915.1	68.6	40.3	-	-	67.5	28.8	51.9
Qwen-VL-Chat	9.6B	1860.0	60.6	35.9	2.93 / 59.4	56.2 / 80.0	64.8	33.8	67.7
CogVLM-Chat	17.4B	1736.6	63.7	32.1	2.68 / 52.1	73.6 / 87.4	68.8	34.7	73.9
LLaVA 1.5	13.6B	1808.4	68.2	36.4	2.71 / 51.0	53.7 / 77.4	68.1	26.4	64.6
OmniLMM-12B	11.6B	1935.8	71.6	40.7	3.45 / 68.8	90.3 / 95.5	71.1	34.9	72.0

†: Proprietary models

Examples

We combine the OmniLMM-12B and GPT-3.5 (text-only) into a real-time multimodal interactive assistant. Video frames are described in text using OmniLMM-12B, and ChatGPT 3.5 (text-only) is employed to generate response according to the descriptions and user prompts. The demo video is a raw recording without edition.

OmniLMM_video_demo_en.mp4

Online Demo

Click here to try out the Demo of MiniCPM-V 2.0 and OmniLMM-12B.

Install

Clone this repository and navigate to the source folder

git clone https://github.com/OpenBMB/MiniCPM-V.git
cd MiniCPM-V

Create conda environment

conda create -n MiniCPM-V python=3.10 -y
conda activate MiniCPM-V

Install dependencies

pip install -r requirements.txt

Inference

Model Zoo

Model	Description	Download Link
MiniCPM-V 2.0	The latest version for state-of-the-art end-side capabilities with high efficiency.	🤗
MiniCPM-V	The first version of MiniCPM-V.	🤗
OmniLMM-12B	The most capable version with leading performance.	🤗

Multi-turn Conversation

Please refer to the following codes to run MiniCPM-V and OmniLMM.

import torch
from chat import OmniLMMChat, img2base64
torch.manual_seed(0)

chat_model = OmniLMMChat('openbmb/MiniCPM-V-2') # or 'openbmb/OmniLMM-12B'

im_64 = img2base64('./assets/hk_OCR.jpg')

# First round chat 
msgs = [{"role": "user", "content": "Where should I go to buy a camera?"}]

inputs = {"image": im_64, "question": json.dumps(msgs)}
answer = chat_model.chat(inputs)
print(answer)

# Second round chat 
# pass history context of multi-turn conversation
msgs.append({"role": "assistant", "content": answer})
msgs.append({"role": "user", "content": "Where is this store in the image?"})

inputs = {"image": im_64, "question": json.dumps(msgs)}
answer = chat_model.chat(inputs)
print(answer)

We can obtain the following results:

"You should go to the Canon store for a camera."

"The Canon store is located on the right side of the image."

Inference on Mac

Click to view an example, to run MiniCPM-V 2.0 on 💻 Mac with MPS (Apple silicon or AMD GPUs).

# test.py
import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2', trust_remote_code=True, torch_dtype=torch.bfloat16)
model = model.to(device='mps', dtype=torch.float16)

tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2', trust_remote_code=True)
model.eval()

image = Image.open('./assets/hk_OCR.jpg').convert('RGB')
question = 'Where is this photo taken?'
msgs = [{'role': 'user', 'content': question}]

answer, context, _ = model.chat(
    image=image,
    msgs=msgs,
    context=None,
    tokenizer=tokenizer,
    sampling=True
)
print(answer)

Run with command:

PYTORCH_ENABLE_MPS_FALLBACK=1 python test.py

Deployment on Mobile Phone

Currently MiniCPM-V 2.0 can be deployed on mobile phones with Android and Harmony operating systems. 🚀 Try it out here.

WebUI Demo

Click to see how to deploy WebUI demo on different devices

pip install -r requirements.txt

# For Nvidia GPUs support BF16 (like A100, H100, RTX3090), run:
python web_demo.py --device cuda --dtype bf16

# For Nvidia GPUs do NOT support BF16 (like V100, T4, RTX2080), run:
python web_demo.py --device cuda --dtype fp16

# For Mac with MPS (Apple silicon or AMD GPUs), run:
PYTORCH_ENABLE_MPS_FALLBACK=1 python web_demo.py --device mps --dtype fp16

Inference with vLLM

Click to see how to inference with vLLM

Because our pull request to vLLM is still waiting for reviewing, we fork this repository to build and test our vLLM demo. Here are the steps:

Clone our version of vLLM:

git clone https://github.com/OpenBMB/vllm.git

Install vLLM:

cd vllm
pip install -e .

Install timm:

pip install timm=0.9.10

Run our demo:

python examples/minicpmv_example.py

Finetune

MiniCPM-V

We now support finetune MiniCPM-V series with the SWIFT framework. SWIFT supports training, inference, evaluation and deployment of nearly 200 LLMs and MLLMs (multimodal large models). It supports the lightweight training solutions provided by PEFT and a complete Adapters Library including techniques such as NEFTune, LoRA+ and LLaMA-PRO.

Best Practices：MiniCPM-V, MiniCPM-V-2

TODO

MiniCPM-V fine-tuning support
OmniLMM fine-tuning support
Code release for real-time interactive assistant

Model License

The code in this repo is released according to Apache-2.0

The usage of MiniCPM-V's and OmniLMM's parameters is subject to "General Model License Agreement - Source Notes - Publicity Restrictions - Commercial License"

The parameters are fully open to academic research

Please contact cpm@modelbest.cn to obtain written authorization for commercial uses. Free commercial use is also allowed after registration.

Statement

As LMMs, OmniLMMs generate contents by learning a large amount of multimodal corpora, but they cannot comprehend, express personal opinions or make value judgement. Anything generated by OmniLMMs does not represent the views and positions of the model developers

We will not be liable for any problems arising from the use of OmniLMM open source models, including but not limited to data security issues, risk of public opinion, or any risks and problems arising from the misdirection, misuse, dissemination or misuse of the model.

Institutions

This project is developed by the following institutions:

THUNLP
ModelBest
Zhihu

Other Multimodal Projects from Our Team

👏 Welcome to explore other multimodal projects of our team:

VisCPM | RLHF-V | LLaVA-UHD

Citation

If you find your model/code/paper helpful, please consider cite the following papers:

@article{yu2023rlhf,
  title={Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback},
  author={Yu, Tianyu and Yao, Yuan and Zhang, Haoye and He, Taiwen and Han, Yifeng and Cui, Ganqu and Hu, Jinyi and Liu, Zhiyuan and Zheng, Hai-Tao and Sun, Maosong and others},
  journal={arXiv preprint arXiv:2312.00849},
  year={2023}
}
@article{viscpm,
    title={Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages}, 
    author={Jinyi Hu and Yuan Yao and Chongyi Wang and Shan Wang and Yinxu Pan and Qianyu Chen and Tianyu Yu and Hanghao Wu and Yue Zhao and Haoye Zhang and Xu Han and Yankai Lin and Jiao Xue and Dahai Li and Zhiyuan Liu and Maosong Sun},
    journal={arXiv preprint arXiv:2308.12038},
    year={2023}
}
@article{xu2024llava-uhd,
  title={{LLaVA-UHD}: an LMM Perceiving Any Aspect Ratio and High-Resolution Images},
  author={Xu, Ruyi and Yao, Yuan and Guo, Zonghao and Cui, Junbo and Ni, Zanlin and Ge, Chunjiang and Chua, Tat-Seng and Liu, Zhiyuan and Huang, Gao},
  journal={arXiv preprint arXiv:2403.11703},
  year={2024}
}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README_en.md

README_en.md

News

Contents

MiniCPM-V 2.8B

Evaluation

Examples

MiniCPM-V 1.0

OmniLMM-12B

Evaluation

Examples

Online Demo

Install

Inference

Model Zoo

Multi-turn Conversation

Inference on Mac

Deployment on Mobile Phone

WebUI Demo

Inference with vLLM

Finetune

MiniCPM-V

TODO

Model License

Statement

Institutions

Other Multimodal Projects from Our Team

Citation

Files

README_en.md

Latest commit

History

README_en.md

File metadata and controls

News

Contents

MiniCPM-V 2.8B

Evaluation

Examples

MiniCPM-V 1.0

OmniLMM-12B

Evaluation

Examples

Online Demo

Install

Inference

Model Zoo

Multi-turn Conversation

Inference on Mac

Deployment on Mobile Phone

WebUI Demo

Inference with vLLM

Finetune

MiniCPM-V

TODO

Model License

Statement

Institutions

Other Multimodal Projects from Our Team

Citation