Differences in MATH-Lvl-5 on Llama3.1 between LM Eval Harness

[vithursant]

what the difference is between Math Lvl 5 in the ZeroEval evaluation and the Eleuther eval harness (OpenLLM leaderboard v2)?

For 3.1-70B, ZeroEval is showing 43.3 and HF leaderboard is showing 28%

[yuchenlin]

hi vithursant,

thanks for the question! We use a structured prompt template to encourage the model to first output its reasoning step and then output the final answer. I'm not sure whether Eleuther is doing the same.