reproduce llama 3 evals

[baberabb]

llama 3.{1,2,3} have released most of their eval details here in HF and also some in this repo. Would be great if we can upstream them here. I'm adding some in #2556

[baberabb]

added llama_arc_challenge (base) and mgsm_chat #2556, and (arc_challenge_chat, mmlu_llama) in #2615

[cjluo-nv]

Hi, @baberabb do you plan to add the benchmark that can reproduce the gpqa result as well?

[baberabb]

added as llama_gpqa. Still have to test it out! Note, looks like they use the main subset and not diamond

[cjluo-nv]

Also @baberabb do you know why the max_gen_toks is set to 10 for the mmlu_llama here https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/llama3/instruct/mmlu/_continuation_template_yaml? It only needs to generate max 2 tokens right? (Which will also make this test run faster)

[cjluo-nv]

Also @baberabb do you know if the mmlu_llama with --fewshot_as_multiturn --apply_chat_template also works for non-llama models? I try it on mixtral and it seems that the result is lower than the mmlu benchmark

[baberabb]

Also @baberabb do you know if the mmlu_llama with --fewshot_as_multiturn --apply_chat_template also works for non-llama models? I try it on mixtral and it seems that the result is lower than the mmlu benchmark

The prompt should work well with most instruct models, I think. Have you looked at the outputs (saved with --log_samples)? Right now it assumes the model will generate the answer letter right away, which might not be true for all models and we could make the answer extraction more flexible.

And the generation tokens are 10 as thats whats used in llama evals (they do not provide the stop tokens though, but I added . here)