Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add procedural Alice in Wonderland (AIW problem) dataset #4

Closed
andreaskoepf opened this issue Jan 24, 2025 · 3 comments · Fixed by #41
Closed

Add procedural Alice in Wonderland (AIW problem) dataset #4

andreaskoepf opened this issue Jan 24, 2025 · 3 comments · Fixed by #41
Assignees
Labels

Comments

@andreaskoepf
Copy link
Contributor

The Alice in Wonderland problem has following base template (and many variations):
"Alice has N brothers and she also has M sisters. How many sisters does Alice’s brother have?"

See paper: Alice in Wonderland: Simple Tasks Reveal Severe Generalization and Basic Reasoning Deficits in State-Of-the-Art Large Language Models

A set of prompts the authors used can be found here: LAION-AI/AIW/blob/main/prompts/prompts.json

It would be great to generalize the prompt generation with templates and procedural generation and configurable ranges of numbers and distraction facts.

@rishabhranawat
Copy link
Contributor

@andreaskoepf is anyone working on this? If not, may I take a stab at it?

@andreaskoepf
Copy link
Contributor Author

@rishabhranawat this would be awesome! I assigned the issue to you.

@rishabhranawat
Copy link
Contributor

Good to close this issue?

@andreaskoepf andreaskoepf linked a pull request Feb 3, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants