Open-source model parser

[wildphoton]

Hi authors, thanks for the great work! The paper mentioned that there is "an open-source model parser with its stability test to ensure long-term reproducibility." But I did not find any details in the paper or the code. Is it possible to share them? Thanks!

[Psycoy]

Hi @wildphoton , we are working on the local support of this feature (insufficient bandwidth recently lol).
But if you want, you could refer to here to use the api for those open-source models first, which is cheaper than the GPT-3.5.

[wildphoton]

Hi @wildphoton , we are working on the local support of this feature (insufficient bandwidth recently lol). But if you want, you could refer to here to use the api for those open-source models first, which is cheaper than the GPT-3.5.

Hi Jinjie @Psycoy, do you mean that you have not tested any open-source model yet and this is more like a future work clam? If you have tested any model, could you share which model you tested at least?

[Psycoy]

Hi @wildphoton ,

We have integrated the OSS local judge feature.
You can now use any language model on the huggingface to judge.

You can configure the OSJudgeCloseendFreeform and OSJudgeCloseendMultichoice in judge_freeform_parser.py and judge_multichoice_parser.py to customize the judge model settings.

Note that its stability has not yet been tested, which will be done soon.

[wildphoton]

Hi @Psycoy, I have noticed that PR being merged. Thank you! I wonder have you tested any open source model as judge for MixEval? Thanks!