DPO performance on other models

[thusharakart]

Do you have data on the performance of DPO with models other than Qwen-VL-Chat? I found that it degrades both perception and cognition in MME when used with LLaVA-1.5.

[TobiasLee]

We did not explore the DPO with LLaVA models. Could you share your results and example outputs before/after DPO so we can dig into it?

[thusharakart]

The following are the results for MME benchmark.

MME score { perception, cognition, ocr }
LLaVA-v1.5-7B with DPO {1342, 313, 125}
LLaVA-v1.5-13B with DPO {1425, 312, 130}

[TobiasLee]

How many epochs have your trained with DPO?

[thusharakart]

Above results are from 1 epoch training for 7B model and 3 epoch training for 13B model.

[TobiasLee]

I'm sorry for not getting back to you sooner. We also recently explored performing DPO training on the LLaVA backbone and observed degraded MME performance. However, the scores on other benchmarks have consistently improved.

Model	MM-Vet	MMHal	MMBench
LLaVA-v1.5-7B	30.5	2.42	63.0
LLaVA-v1.5-7B + DPO	31.7	2.62	63.9

We attribute that the simple answer format required by MME cannot be followed by the model after DPO training, and would like to investigate it later.

[choucaicai]

may be you can add a prompt like this query = f'<img>{img_path}</img>\n{question} you can only use "Yes" or "No" as your responses without adding any extra text or explanation.

[TobiasLee]

Hi all, we found a great repo with the support/results of many other models: https://github.com/TideDra/VL-RLHF

The performance can be boosted almost consistently for LLaVA-Next series models. So my guess is that the current LLaVA-v1.5 series model is too weak to serve as a starting model for DPO ( possibly due to its lower resolution 336 v.s. Qwen-VL). LLaVA-Next series is more powerful with the image tiling mechanism.

Check it out if you want to further explore the DPO/RLHF with VLFeedback!