Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New model submission to CoIR Leaderboard #9

Open
yeliu918 opened this issue Oct 31, 2024 · 1 comment
Open

New model submission to CoIR Leaderboard #9

yeliu918 opened this issue Oct 31, 2024 · 1 comment

Comments

@yeliu918
Copy link

Hi @archersama,

We have a family of new embedding models, CodeXEmbed, and would like to submit its results to our leaderboard. Could you please assist us with this?

{"Model": "CodeXEmbed-7B", "Model Size (Million Parameters)": 7B, "URL": "”, "Apps": 85.22, "CosQA": 33.27, "Synthetic Text2sql":64.57, "CodeSearchNet": 78.84, "CodeSearchNet-CCR": 86.77, "CodeTrans-Contest": 90.64, "CodeTrans-DL": 32.31, "StackOverFlow QA": 94.25, "CodeFeedBack-ST": 80.93, "CodeFeedBack-MT": 57.83, "Avg": 70.46}

{"Model": "CodeXEmbed-2B", "Model Size (Million Parameters)": 2B, "URL": "”, "Apps": 74.99, "CosQA": 36.31, "Synthetic Text2sql":59.00, "CodeSearchNet": 73.50, "CodeSearchNet-CCR": 85.77, "CodeTrans-Contest": 86.63, "CodeTrans-DL": 33.17, "StackOverFlow QA": 90.54, "CodeFeedBack-ST": 81.15, "CodeFeedBack-MT": 53.08, "Avg": 67.41}

{"Model": "CodeXEmbed-400M", "Model Size (Million Parameters)": 2B, "URL": "”, "Apps": 48.57, "CosQA": 34.05, "Synthetic Text2sql":58.96, "CodeSearchNet": 72.53, "CodeSearchNet-CCR": 80.15, "CodeTrans-Contest": 75.67, "CodeTrans-DL": 34.85, "StackOverFlow QA": 89.51, "CodeFeedBack-ST": 78.87, "CodeFeedBack-MT": 45.75, "Avg": 61.89}

@archersama
Copy link
Contributor

archersama commented Nov 1, 2024

Great Work!!! But could you provide some URLs or API endpoints? This would help prevent individuals from submitting unverifiable models, which could potentially compromise the integrity of the leaderboard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants