Language Form Evaluation

This package includes efficient python implementation of the following metrics

• BLEU
  • reference implementation: nltk.translate.bleu_score
    • error: < 1%
    • speed: + 189%
• ROUGE (in progress)
• METEOR (in progress)
• CIDEr/CIDEr-D
  • reference implementation: https://github.com/vrama91/cider
    • with the same tokenizer
      • error: < 1 %
      • speed: + 81 %
    • with different tokenizers (FormEval use Regexp by default)
      • error: ~ 15 %
      • speed: + 332 %
• SPICE
  • placeholder wrapper of reference implementation: https://github.com/tylin/coco-caption/tree/master/pycocoevalcap/spice
    • TODO: python scene graph parser

*All stats shown above are estimations

Dependencies

• python 3.6 +
• nltk 3.5+

Setup

pip install formeval
python -m formeval.setup

Examples

• https://github.com/sudongqi/lfeval/blob/main/examples.py

Test Data

• PASCAL VOC