IWSLT'14 DE-EN Numbers

[nikhil-iyer-97]

Hi,

I followed all the commands mentioned in https://github.com/lancopku/Prime/blob/master/examples/parallel_intersected_multi-scale_attention(Prime)/README.md#iwslt14-de-en and ran it till 20000 steps. The bleu score for the best ckpt was 35.07 and the bleu score for the avg of the last 10 ckpts was 35.78. PPL was 4.7+. The repo mentions that the bleu score for the best ckpt is around 35.7. Is there any mistake in my implementation? or do i have tune the lenpen and beam size to get the numbers mentioned? Would be helpful if you could clarify these doubts. Thanks!

[zhaoguangxiang]

Many thanks for your interest and test, we reproduce the 35.7 BLEU of best ckpt in the other version of the code, we have uploaded pretrained models for IWSLT De-En including 70-th checkpoint and the average of last 10 checkpoints at 70 epoch (70-epoch is near to 20k updates) , we are checking the code difference and will update the code later.

[nikhil-iyer-97]

Thanks for the quick response. I ran ( https://github.com/lancopku/Prime/blob/master/examples/parallel_intersected_multi-scale_attention(Prime)/README.md#iwslt14-de-en ) till 25k steps (89 epochs) and then I could see that the average of the last 25 checkpoints comes around 36.02 bleu score and the bleu score of the final ckpt was 35.71. Please let me know when the updates are done. Cheers!