Open
Description
In the evaluate_relevancy
block, the LLM begins assigning scores immediately after seeing the response. This can lead to suboptimal evaluations, as the model typically needs to reason first on what feedback to even give before it can provide an accurate score.
We should adjust this block to instead do the following:
[Start of Question]
How does photosynthesis work?
[End of Question]
[Start of Response]
Plants require sunlight and water to grow.
[End of Response]
[Start of Feedback]
- Subject Matter Relevance:
reasoning: The response is related to plant growth, but does not specifically address the process of photosynthesis.
score: 0
- Alignment with Query's Focus:
reasoning: The response fails to detail the photosynthesis process, missing the specific focus of the query.
score: 0
[End of Feedback]
[Start of Score]
0
[End of Score]
With this minimal change, we can condition the model to provide a more accurate score by conditioning itself on the reasoning for which score it should give.