FEAT Add TrustAIRLab/forbidden_question_set Dataset

[nina-msft]

Name: TrustAIRLab/forbidden_question_set

Link: https://github.com/verazuo/jailbreak_llms/blob/main/data/forbidden_question/forbidden_question_set.csv

Relevant Columns: "content_policy_name","question"

Originally posted by @divyaamin9825 in #429

---

Describe the solution you'd like

This dataset should be available within PyRIT: https://huggingface.co/datasets/TrustAIRLab/forbidden_question_set
Also available here: https://github.com/verazuo/jailbreak_llms/blob/main/data/forbidden_question/forbidden_question_set.csv
Associated paper: https://arxiv.org/html/2308.03825v2

Additional context

There are examples of how PyRIT interacts with other datasets here: https://github.com/search?q=repo%3AAzure%2FPyRIT%20%23%20The%20dataset%20sources%20can%20be%20found%20at%3A&type=code

[[Content Warning: Prompts are aimed at provoking the model, and may contain offensive content.]]
Additional Disclaimer: Given the content of these prompts, keep in mind that you may want to check with your relevant legal department before trying them against LLMs.

[ritikakumar0204]

Hi @nina-msft , I am new to contributing but have been looking into this feature and can open a pull request for what I have worked on so far. Can I contribute?

[romanlutz]

Yes please! Thank you @ritikakumar0204