VLMs vs LLMs evaluation

[idan-tankel]

Hello 👋

First of all thank you for the great work and evaluation results!

I have understood that in many cases you predicted outputs for each question based on the choice that minimizes the loss of the current evaluated model. I wanted to ask - if there was any difference between the evaluation for VLMs and LLMs? and if was, how did you put these results on the same scale?

many thanks
Idan

[Bohao-Lee]

Thank you for your interest in our work, and we apologize for the delayed response. In reality, there is no difference in the evaluation of VLMs and LLMs. The only distinction is that LLMs take only the question as input, while VLMs consider both the question and its corresponding image. We simply calculate the loss for each candidate.