Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve definition hints #122

Open
apmwebdev opened this issue Apr 1, 2024 · 0 comments
Open

Improve definition hints #122

apmwebdev opened this issue Apr 1, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request hints Anything related to hint profiles and hint panels

Comments

@apmwebdev
Copy link
Owner

apmwebdev commented Apr 1, 2024

As described in PR #121:

The definition hints are unquestionably an improvement over the dictionary definitions for use in hint panels, which is great! That being said, they could definitely be improved. The part of the instructions concerning how to generate hints is as follows:

Each hint should be one or two sentences and should not contain any form of the word you're providing a hint for, or any word or phrase that the submitted word can be an abbreviation for. For example, the hint for "allay" should not contain "allay," "allaying," "allays," etc. The hints should focus on the meaning of the word, although hints about the pronunciation are also acceptable. These hints are designed to replicate the user-provided hints that users provide on the New York Times Spelling Bee forums for words in that day's Spelling Bee puzzle.

The issues with the current hint generation are:

  1. They are still more "definition" than "hint." While they give much less away now, it would be great if they were more tailored to hinting at the word rather than just giving a definition. For example, in the spelling bee forums, someone provided a hint for the words "eave" and "eaved" for one puzzle with Night before "roof overhang day"? This combines clues about both the pronunciation (via a homonym) as well as the definition into one succinct, effective hint. The AI can't generate anything that "clever" yet, but it might be able to with proper instructions or fine-tuning. Essentially, the goal is crossword-style hints that combine clues about the definition, pronunciation, and potentially context clues like other words in the puzzle to generate more interesting and varied hints.
  2. Even when following the guidelines above, many hints still give away too much information. For example, the hint for "vale" is "A valley." Technically, "valley" and "vale" are different words, so providing a very similarly spelled and pronounced synonym isn't breaking the rules about not providing any form of the word, but... it would be better if the hint was something else. This might be fixable by tweaking the instructions.
  3. The AI sometimes just straight-up ignores the instructions, such as not to include any form of the word in the hint. For example, the hint for "uvulae" reads "Plural of 'uvula'; fleshy extensions at the back of the soft palate." The hint for "valuate" is "To determine or estimate the value of something, especially in a financial context." This might be fixable by slightly tweaking the instructions to "...should not contain any form of the word you're providing a hint for or that word's root word, ...", but it's also possible that some amount of this instruction non-adherence is just inherent to AI models, in which case the issue could likely be mitigated somewhat by switching to a newer model (e.g., GPT 4 instead of the current model of GPT 3.5).

There will likely need to be iterative improvement on the hint generation logic before the hints are consistently generated correctly.

At a high level, this iterative improvement could include any of the following elements:

  • UI functionality for marking words that have unacceptable hints and regenerating them manually (likely in batches)
  • Using GPT 4 going forward and/or regenerating all existing hints using GPT 4 instead of GPT 3.5
  • Refining the instructions for generating hints
  • Fine-tuning an AI model for hint generation by giving it examples of good and bad hints
@apmwebdev apmwebdev added enhancement New feature or request hints Anything related to hint profiles and hint panels labels Apr 1, 2024
@apmwebdev apmwebdev self-assigned this Apr 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request hints Anything related to hint profiles and hint panels
Projects
None yet
Development

No branches or pull requests

1 participant