John Snow Labs NLP Test 1.4.0: Enhancing Support for Toxicity test and new QA benchmark datasets (NarrativeQA, TruthfulQA, QuAC, HellaSwag, MMLU and OpenbookQA)

[ArshaanNazir]

John Snow Labs NLP Test 1.4.0: Enhancing Support for Toxicity test and new QA benchmark datasets (NarrativeQA, TruthfulQA, QuAC, HellaSwag, MMLU and OpenbookQA)

---

📢 Overview

NLP Test 1.4.0 🚀 comes with brand new features, including: new capabilities for testing Large Language Models for toxicity and support for new QA benchmark datasets (NarrativeQA, TruthfulQA, QuAC, HellaSwag, MMLU and OpenbookQA) for robustness, representation, fairness and accuracy tests. It also includes addition of some new robustness tests and many other enhancements and bug fixes!

A big thank you to our early-stage community for their contributions, feedback, questions, and feature requests 🎉

Make sure to give the project a star right here ⭐

---

🔥 New Features & Enhancements

• Adding support for NarrativeQA dataset Feature/narrativeqa dataset #487
• Adding support for toxicity task Feature/create toxicity test 438 #488
• Adding support for TruthfulQA dataset Added TruthfulQA Dataset #477
• Adding support for new dyslexia swap test for robustness testing Feature/dyslexia swap test #474
• Adding support for new slangificator test for robustness testing feature/Add support for slangificator test #463
• Adding support for new abbreviation test for robustness testing Feature/add support for abbreviation test #471
• Adding support for OpenBookQA dataset Dataset/OpenBookQA datasets #479
• Adding support for MMLU dataset Datasets/MMLU Datasets #481
• Adding support for hellaswag dataset Feature/hellaswag dataset #486
• Adding new tutorial notebooks Feature/add tutorial nbs #497

---

❓ How to Use

Get started now! 👇

pip install nlptest

Create your test harness in 3 lines of code 🧪

# Set OpenAI API keys
os.environ['OPENAI_API_KEY'] = ''

# Import and create a Harness object
from nlptest import Harness
h = Harness(task='toxicity', model='text-davinci-002', hub='openai', data='toxicity-test-tiny')

# Generate test cases, run them and view a report
h.generate().run().report()

---

📖 Documentation

• NLP Test: Documentation
• NLP Test: Notebooks
• NLP Test: Test Types
• NLP Test: GitHub Repo

---

❤️ Community support

• Slack For live discussion with the NLP Test community, join the #nlptest channel
• GitHub For bug reports, feature requests, and contributions
• Discussions To engage with other community members, share ideas, and show off how you use NLP Test!

We would love to have you join the mission 👉 open an issue, a PR, or give us some feedback on features you'd like to see! 🙌

---

♻️ Changelog

What's Changed

• updated/doc by @Prikshit7766 in updated/doc #459
• docs/Update documentation of models by @RakshitKhajuria in docs/Update documentation of models #465
• refactor user prompt by @alytarik in refactor user prompt #472
• Feature/dyslexia swap feature by @ArkajyotiChakraborty in Feature/dyslexia swap feature #417
• Feature/add support for abbreviation test by @RakshitKhajuria in Feature/add support for abbreviation test #471
• Hotfix/get rid of some dependencies by @chakravarthik27 in Hotfix/get rid of some dependencies #473
• Draft: refactor/perturbations and samples to support QA. by @chakravarthik27 in Draft: refactor/perturbations and samples to support QA. #460
• feature/Add speech to text typo by @Prikshit7766 in feature/Add speech to text typo #475
• hotfix/get rid of inflect dependency and refactoring robustness by @RakshitKhajuria in hotfix/get rid of inflect dependency and refactoring robustness #478
• Added TruthfulQA Dataset by @RakshitKhajuria in Added TruthfulQA Dataset #477
• feature/Add support for slangificator test by @Prikshit7766 in feature/Add support for slangificator test #463
• Dataset/OpenBookQA datasets by @Prikshit7766 in Dataset/OpenBookQA datasets #479
• Datasets/MMLU Datasets by @Prikshit7766 in Datasets/MMLU Datasets #481
• Docs/update model hub-summarization nb-readme by @RakshitKhajuria in Docs/update model hub-summarization nb-readme #480
• Hotfix/fixed some tests and refactored number_to_word.py by @RakshitKhajuria in Hotfix/fixed some tests and refactored number_to_word.py #483
• Dataset/quac dataset by @Prikshit7766 in Dataset/quac dataset #484
• Feature/dyslexia swap test by @alytarik in Feature/dyslexia swap test #474
• Feature/hellaswag dataset by @alytarik in Feature/hellaswag dataset #486
• Feature/narrativeqa dataset by @alytarik in Feature/narrativeqa dataset #487
• Feature/create toxicity test 438 by @chakravarthik27 in Feature/create toxicity test 438 #488
• hot-fix/fix-slangify-test by @RakshitKhajuria in hot-fix/fix-slangify-test #489
• DRAFT : Docs/update nb and docs by @RakshitKhajuria in DRAFT : Docs/update nb and docs #490
• Update datasets by @RakshitKhajuria in Update datasets #493
• Fix/toxicity by @chakravarthik27 in Fix/toxicity #492
• Feature/add tutorial nbs by @ArshaanNazir in Feature/add tutorial nbs #497
• default toxicity config by @chakravarthik27 in default toxicity config #498
• docs/add dataset notebooks by @alytarik in docs/add dataset notebooks #499
• Release/1.4.0 by @ArshaanNazir in Release/1.4.0 #500

New Contributors

• @ArkajyotiChakraborty made their first contribution in Feature/dyslexia swap feature #417

Full Changelog: v1.3.0...v1.4.0

---

This discussion was created from the release John Snow Labs NLP Test 1.4.0: Enhancing Support for Toxicity test and new QA benchmark datasets (NarrativeQA, TruthfulQA, QuAC, HellaSwag, MMLU and OpenbookQA).