Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Automatic GPT4V Evaluation for VLM Originality Evaluation #2576

Merged
merged 11 commits into from
May 23, 2024
Prev Previous commit
Next Next commit
add gpt4v metric_specs
ImKeTT committed May 19, 2024
commit 120a453c68759af0d5a4cd8e9ce8ca687ac8766c
5 changes: 5 additions & 0 deletions src/helm/benchmark/run_specs/vlm_run_specs.py
Original file line number Diff line number Diff line change
@@ -124,6 +124,11 @@ def get_image2structure_metric_specs(
]
return metric_specs + get_basic_reference_metric_specs()

def get_gpt4v_originality_metric_specs() -> List[MetricSpec]:
return [
MetricSpec(class_name="helm.benchmark.metrics.gpt4v_originality_metrics.GPT4VOriginalityMetric", args={}),
]


############################################################
# VHELM run specs