Skip to content

Improve accuracy of evaluate relevance blocks #543

Open
@RobotSail

Description

@RobotSail

In the evaluate_relevancy block, the LLM begins assigning scores immediately after seeing the response. This can lead to suboptimal evaluations, as the model typically needs to reason first on what feedback to even give before it can provide an accurate score.

We should adjust this block to instead do the following:

  [Start of Question]
  How does photosynthesis work?
  [End of Question]

  [Start of Response]
  Plants require sunlight and water to grow.
  [End of Response]

  [Start of Feedback]
  - Subject Matter Relevance:
      reasoning: The response is related to plant growth, but does not specifically address the process of photosynthesis.
      score: 0
  - Alignment with Query's Focus:
      reasoning: The response fails to detail the photosynthesis process, missing the specific focus of the query.
      score: 0  
  [End of Feedback]

  [Start of Score]
  0
  [End of Score]

With this minimal change, we can condition the model to provide a more accurate score by conditioning itself on the reasoning for which score it should give.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingstale

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions