Add Automatic GPT4V Evaluation for VLM Originality Evaluation

[ImKeTT]

This PR aims to add automatic GPT4V evaluation for vision-language originality evaluation. I also repurposed Mementos for creative storytelling.
Basically, the evaluation process is like: we feed the input image and the model-generated story to GPT4V and ask it to judge the given textual content from 1 to 5 (higher means the story is more original and creative).

We might need to discuss several prompts used in this evaluation. Currently, they are quite simple ones:

1. Evaluation: Prompt fed into the GPT4V model to tell it the specific judging criteria. I think we need to explain different originality levels in detail.
2. Repurposing scenarios: Prompt fed into the tested VLMs for generating original and creative stories given the image.

I've run ./pre-commit.sh and passed the local tests. Here are several output files of using GPT4V evaluation on Mementos (25 instances using Qwen-VL-Chat).
stats.json
per_instance_stats.json
scenario_state.json

The conf file and running command I used is:
conf file:

entries: [
    {description: "mementos:subject=comics,model=qwen/qwen-vl-chat", priority: 1}
    ]

bash command:

helm-run --conf-paths run_mem_specs.conf --suite v1 --max-eval-instances 25

Please let me know how can I improve this PR, thanks!

Oops, there seems to be a bug, fixing it...

Seems that we cannot run the GPT4V evaluator with minimal dependencies only (we need to import openai). What do you recommend to do to pass the test?

[ImKeTT]

Friendly ping @teetone, please don't forget to take a look at this PR when you are free :)

[teetone]

Friendly ping @teetone, please don't forget to take a look at this PR when you are free :)

@ImKeTT Sorry! I was working on a big merge at the moment. I will take a look by tomorrow.

[ImKeTT]

@ImKeTT Sorry! I was working on a big merge at the moment. I will take a look by tomorrow.

That's ok, no rush. I just wanted to ask, since people sometimes forget.

[teetone]

@ImKeTT I think we want the metric to have all the knowledge about the Likert scale and criteria, and the client should not be aware of any of this. Also, we want to use this auto model evaluator evaluate on other aspects aside from originality.

Could we actually follow https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/metrics/image_generation/photorealism_critique_metrics.py and use the CritiqueClient. We would want to use. critique_type == "model" to route it to the ModelCritiqueClient. I think you just have to add multimodal support for ModelCritiqueClient (cc @yifanmai). Sorry for the trouble.

[ImKeTT]

Thanks for the pointer, will refactor the code soon.

[ImKeTT]

Hi @yifanmai, I have refactored the GPT4V-aided originality evaluation for VHELM. We may still need to discuss about the detailed prompts that are fed into the GPT4V to make it a qualified evaluator.

The command I used for a canary run is:

helm-run --conf-paths run_entries.conf --suite v1 --max-eval-instances 20

The run_entries.conf file is like:

entries: [
    {description: "mementos:subject=comics,model=qwen/qwen-vl-chat,num_respondents=1", priority: 1}
    ]

And the credentials.conf file in the prod_env folder is like:

openaiApiKey: the-openai-api-key
critiqueModelName: openai/gpt-4-vision-preview
critiqueType: model

Here're some results:
scenario_state.json
per_instance_stats.json
scenario.json

Please let me know how can I improve this version, thanks! :)

[ImKeTT]

Hi @yifanmai @teetone, could you take a peek at the refactored GPT4V-aided evaluation whenever you have time? Would love to get your feedbacks on this PR before moving forward with another for the Rika-Vibe evaluator using the multimodal ModelCritiqueClient.

[ImKeTT]

Hi @teetone, I have run the metric on 100 instances from the refactored mementos using Qwen-VL-Chat. Here's the result:

run_spec.json
scenario.json
per_instance_stats.json

There's a chance that the GPT-4-Vision generates the first word other than a capital character from 'A, B, C, D, E' (5 out of 100 cases), which will result in a parsing failure and an invalid evaluation instance (the evaluation can run smoothly but 5 evaluation instances will be ignored).