[Feature Request] Saving Prediction Results

[guanqun-yang]

The current version of the code base only returns the final evaluation metric back to the user. However, it is not possible to see what exactly are the model's predictions. The LM-Evaluation-Harness supports this feature by specifying --write_out flag like the one below:

 python main.py \
  --model hf-causal-experimental \
  --model_args pretrained=gpt2,use_accelerate=True,dtype=half \
  --batch_size 8 \
  --tasks arc_challenge \
  --num_fewshot 25 \
  --write_out 

I hope the same feature will be supported by this code base.