-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Change outputs of AnswerExactMatchEvaluator #7390
Conversation
Pull Request Test Coverage Report for Build 8377370259Warning: This coverage report may be inaccurate.This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.
Details
💛 - Coveralls |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I disagree with Changes scores to return the number of matches per question. It differs from the semantics we had in the past for ExactMatch in Haystack and for the use cases we so far support.
If there are multiple answers predicted for one query, they are top k answer candidates in a specific order and we don't aim to cover multiple ground truth questions with them.
Multiple ground truth answers can come from annotations with slightly different punctuation etc. For the users, there is no additional value if the predicted answers for one query contain more than one ground truth answer. Let's leave the metric as is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! 👍
Proposed Changes:
Standardize outputs with other evaluators.
This PR also adds a new test with more complex data.
How did you test it?
I ran tests locally.
Notes for the reviewer
This decision stems from this discussion.
I ignored the release notes as this has still not been released and the PR that added this Component already has release notes.
Checklist
fix:
,feat:
,build:
,chore:
,ci:
,docs:
,style:
,refactor:
,perf:
,test:
.