Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retrying relevance score prompt if score was a float or response isn't valid JSON #749

Open
jamesbraza opened this issue Dec 4, 2024 · 0 comments
Labels
bug Something isn't working enhancement New feature or request

Comments

@jamesbraza
Copy link
Collaborator

jamesbraza commented Dec 4, 2024

Our LLM response --> Context logic is a bit fragile:

What currently happens is this blows up Docs.aquery.

However, this should not be a critical failure because:

  1. It's not in the caller's (e.g. an agent) control, it's a failure internal to paper-qa.
  2. We can retry the summary prompt once or twice, saying something like:
    • "We failed to parse valid JSON according to exception {exc!r}. Please respond with JSON."
    • "We failed to extract a valid score according to exception {exc!r}. Please provide an integer score in 1-10."
@jamesbraza jamesbraza added the enhancement New feature or request label Dec 4, 2024
@dosubot dosubot bot added the bug Something isn't working label Dec 4, 2024
@jamesbraza jamesbraza changed the title Retrying relevance score prompt if score was a float Retrying relevance score prompt if score was a float or response isn't valid JSON Dec 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant