Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add evaluation data to the LLMs docs #4225

Closed
rbren opened this issue Oct 4, 2024 · 6 comments · Fixed by #4312
Closed

Add evaluation data to the LLMs docs #4225

rbren opened this issue Oct 4, 2024 · 6 comments · Fixed by #4312
Assignees
@neubig
Labels
enhancement New feature or request fix-me Attempt to fix this issue with OpenHands

Comments

@rbren
Copy link
Collaborator

rbren commented Oct 4, 2024 •
edited by neubig
Loading

What problem or use case are you trying to solve?

The blog article available at https://www.all-hands.dev/blog/evaluation-of-llms-as-coding-agents-on-swe-bench-at-30x-speed describes the quality and cost of various language models.

We should summarize this information, and add it to docs/modules/usage/llms/llms.md in order to make it easier for users to know which models they are recommended to use. We can also link back to the article for people who want to see more comprehensive information.
@rbren rbren added the enhancement New feature or request label Oct 4, 2024
@neubig neubig added the fix-me Attempt to fix this issue with OpenHands label Oct 9, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 9, 2024

OpenHands started fixing the issue! You can monitor the progress here.

@neubig neubig added fix-me Attempt to fix this issue with OpenHands and removed fix-me Attempt to fix this issue with OpenHands labels Oct 9, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 9, 2024

OpenHands started fixing the issue! You can monitor the progress here.

@neubig neubig added fix-me Attempt to fix this issue with OpenHands and removed fix-me Attempt to fix this issue with OpenHands labels Oct 9, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 9, 2024

OpenHands started fixing the issue! You can monitor the progress here.

@neubig neubig added fix-me Attempt to fix this issue with OpenHands and removed fix-me Attempt to fix this issue with OpenHands labels Oct 9, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 9, 2024

OpenHands started fixing the issue! You can monitor the progress here.

@neubig neubig added fix-me Attempt to fix this issue with OpenHands and removed fix-me Attempt to fix this issue with OpenHands labels Oct 9, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 9, 2024

OpenHands started fixing the issue! You can monitor the progress here.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 9, 2024

The workflow to fix this issue encountered an error. Please check the workflow logs for more information.

openhands-agent added a commit that referenced this issue Oct 9, 2024
@openhands-agent
Fix issue #4225: 'Add evaluation data to the LLMs docs'
b7f269c
@openhands-agent openhands-agent mentioned this issue Oct 9, 2024
Merged
neubig added a commit that referenced this issue Oct 13, 2024
@openhands-agent @neubig
Fix issue #4225: Add evaluation data to the LLMs docs (#4312)
ff8a9a1 
Co-authored-by: Graham Neubig <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@neubig neubig

Labels
enhancement New feature or request fix-me Attempt to fix this issue with OpenHands
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix issue #4225: Add evaluation data to the LLMs docs All-Hands-AI/OpenHands
2 participants
@neubig @rbren