Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RFC] enhanced evaluation support #184

Open
sustcsonglin opened this issue Feb 13, 2025 · 3 comments
Open

[RFC] enhanced evaluation support #184

sustcsonglin opened this issue Feb 13, 2025 · 3 comments
Labels
enhancement New feature or request

Comments

@sustcsonglin
Copy link
Collaborator

Proposal

Integrate common benchmark evaluations such as RULER, LongBench, Recall-Intensive Tasks, etc., into https://github.com/fla-org/flash-linear-attention/tree/main/evals

Rationale

No response
@sustcsonglin sustcsonglin added the enhancement New feature or request label Feb 13, 2025
@sustcsonglin
Copy link
Collaborator Author

It might be a better choice to have a new repo: fla-org/fla-eval where there could have many third-party evaluation repositories.

@sustcsonglin
Copy link
Collaborator Author

@BeeGass expressed willingness to contribute to this RFC.

@BeeGass
Copy link

BeeGass commented Feb 17, 2025 •
edited
Loading

@BeeGass expressed willingness to contribute to this RFC.

@sustcsonglin
Yes, going to attend to this within next week or so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sustcsonglin @BeeGass