add_bos_token causes very unstable results for quantized llama3-70B

[wenhuach21]

https://huggingface.co/OPEA/Llama-3.3-70B-Instruct-int3-sym-inc
arc_easy WO BOS 0.2643 vs with BOS 0.8523

https://huggingface.co/OPEA/Llama-3.3-70B-Instruct-int2-sym-inc
mmlu WO BOS 0.7142 vs with BOS 0.7606
lambada_openai WO BOS 0.7013 vs with BOS 0.7413

is it possible to align with the 16 bits model directly in lm-eval?

[baberabb]

Hi! Is this with the chat template? generally with a template, you don't have to include the bos token manually as the chat template assures it is included

[wenhuach21]

Hi! Is this with the chat template? generally with a template, you don't have to include the bos token manually as the chat template assures it is included

we used the cmd recommended in the homepage, I don't know whether it used chat template or not
With BOS
lm-eval --model hf --model_args pretrained=OPEA/Llama-3.3-70B-Instruct-int3-sym-inc,add_bos_token=True --tasks mmlu --batch_size 16

Without BOS

lm-eval --model hf --model_args pretrained=OPEA/Llama-3.3-70B-Instruct-int3-sym-inc --tasks mmlu --batch_size 16