Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Llm evaluation #117

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Llm evaluation #117

wants to merge 9 commits into from

Conversation

Mazenh2003
Copy link

Senselab project integrates deepeval for evaluating conversations,
using an api.py script to interface with deep_eval.py,
which includes a custom ROUGE metric for comprehensive evaluation.
The ScriptLine class standardizes input data, and unit tests ensure accurate functionality,
making Senselab a robust wrapper for deepeval and other tools.

Copy link
Collaborator

@batwood-1 batwood-1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice use of style and cleanliness, some quick points -

-Still failing checks
-We have a script_line data structure available already in utils/datastructures, is there a reason you are rewriting rather than importing this?
-write a file called metrics.py, with an abstract base class called Metric, define the abstractmethods you want to be inherited by the different implementations of Metric (see https://docs.python.org/3/library/abc.html), then define various implementations (rouge, etc)
-give some various options for calculating overall_score in evaluate_conversation, perhaps harmonic mean, etc

@Mazenh2003 Mazenh2003 marked this pull request as draft July 29, 2024 21:12
@Mazenh2003 Mazenh2003 marked this pull request as ready for review July 29, 2024 21:15
batwood-1
batwood-1 previously approved these changes Jul 29, 2024
@fabiocat93 fabiocat93 dismissed batwood-1’s stale review July 30, 2024 02:47

by the time of the review, all tests failed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants