You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the evaluate_relevancy block, the LLM begins assigning scores immediately after seeing the response. This can lead to suboptimal evaluations, as the model typically needs to reason first on what feedback to even give before it can provide an accurate score.
We should adjust this block to instead do the following:
[Start of Question]
How does photosynthesis work?
[End of Question]
[Start of Response]
Plants require sunlight and water to grow.
[End of Response]
[Start of Feedback]
- Subject Matter Relevance:
reasoning: The response is related to plant growth, but does not specifically address the process of photosynthesis.
score: 0
- Alignment with Query's Focus:
reasoning: The response fails to detail the photosynthesis process, missing the specific focus of the query.
score: 0
[End of Feedback]
[Start of Score]
0
[End of Score]
With this minimal change, we can condition the model to provide a more accurate score by conditioning itself on the reasoning for which score it should give.
The text was updated successfully, but these errors were encountered:
This seems like a reasonable suggestion, although we may need to get someone to quantify the impact of changing this and how much it helps or hurts the evaluations.
In the
evaluate_relevancy
block, the LLM begins assigning scores immediately after seeing the response. This can lead to suboptimal evaluations, as the model typically needs to reason first on what feedback to even give before it can provide an accurate score.We should adjust this block to instead do the following:
With this minimal change, we can condition the model to provide a more accurate score by conditioning itself on the reasoning for which score it should give.
The text was updated successfully, but these errors were encountered: