Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BLEURT or BERTScore added to NLG datasets. #59

Closed
jordiclive opened this issue May 11, 2022 · 1 comment
Closed

BLEURT or BERTScore added to NLG datasets. #59

jordiclive opened this issue May 11, 2022 · 1 comment

Comments

@jordiclive
Copy link

As in #50, maybe having a machine-learned metric would be a good dimension for this work to have. As it is a recent trend in evaluation. The downside is that learned metrics take a while to run.

I'm not sure if COMET is just for MT, BERTScore or BLEURT supposedly can be used for all NLG tasks. Versions are in HF metrics (not sure about the implementation). I can see BLEURT is used for one EH dataset already.

@StellaAthena
Copy link
Collaborator

We are choosing to not do this as they don't tend to generalize beyond the specific context in which they were developed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants