Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: testbed #39

Merged
merged 59 commits into from
Nov 6, 2023
Merged

feat: testbed #39

merged 59 commits into from
Nov 6, 2023

Conversation

McPatate
Copy link
Member

testbed will be evaluating completion quality

@github-actions
Copy link

github-actions bot commented Oct 30, 2023

Repository name Source type Average hole completion time (s) Pass percentage
simple local 3.11255 93%
gcanti/io-ts github 109.3835 0%
lancedb/lance github 96.7374 69.46386%
lancedb/lancedb github 78.153206 90%
mmaitre314/picklescan github 4.4968 100%
smol-rs/async-executor github 10.5044 50%
colinhacks/zod github 52.4536 80%
helix-editor/helix github 86.479706 70%
encode/starlette github 37.8284 89.73298%
tkaitchuck/constrandom github 2.444 10%
tiangolo/fastapi github 88.779205 80%
huggingface/huggingface_hub github 20.571901 100%
jaemk/cached github 34.3235 100%
Total -- 29.694595 80.41803%

Note: The "hole completion time" represents the full process of:

  • replacing the code from the file with a completion from the model
  • building the project
  • running the tests

@huggingface huggingface deleted a comment from github-actions bot Oct 30, 2023
@huggingface huggingface deleted a comment from github-actions bot Oct 30, 2023
@huggingface huggingface deleted a comment from github-actions bot Oct 30, 2023
@McPatate McPatate merged commit c7affd0 into main Nov 6, 2023
1 check passed
@McPatate McPatate deleted the feat/test_framework branch November 6, 2023 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants