-
Notifications
You must be signed in to change notification settings - Fork 5.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Refactor]: Refactor the evaluation directory #5222
Labels
fix-me
Attempt to fix this issue with OpenHands
Comments
Sorry, to clarify, this was me accidentally logged in as the openhands-agent account... |
A potential fix has been generated and a draft PR #5223 has been created. Please review the changes. |
enyst
added a commit
that referenced
this issue
Nov 23, 2024
…ation directory" This reverts commit 4136c53.
neubig
pushed a commit
that referenced
this issue
Nov 25, 2024
Co-authored-by: Engel Nyst <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What problem or use case are you trying to solve?
Right now in the evaluation directory, the directory structure is very flat, and it is hard to tell which subdirectories are utilities related to implementing benchmarks or doing basic tests for openhands (
utils
,integration_tests
,regression
,static
), and which are actual benchmarks from the ML literature (everything else).To make this more clear, we can move all benchmarks to live under the
evaluation/benchmarks/
directory. In addition, all other files that have to do with evaluation (including documentation, github workflows, etc.) will need to be checked and changed to maintain consistency.While we do this, we can also add some of the benchmarks that are missing from the
evaluation/README.md
documentation.The text was updated successfully, but these errors were encountered: