FaithfulnessEvaluator with Mistral AI API

[miam-bonbon]

Describe the bug
Cannot use FaithfulnessEvaluator with Mistral AI API

Error message
Error: Error code: 422 - {'object': 'error', 'message': {'detail': [{'type': 'extra_forbidden', 'loc': ['body', 'seed'], 'msg': 'Extra inputs are not permitted', 'input': 42}]}, 'type': 'invalid_request_error', 'param': None, 'code': None}
httpx.HTTPStatusError: Client error '422 Unprocessable Entity' for url 'https://api.mistral.ai/v1/chat/completions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/422

During handling of the above exception, another exception occurred:

openai.UnprocessableEntityError: Error code: 422 - {'object': 'error', 'message': {'detail': [{'type': 'extra_forbidden', 'loc': ['body', 'seed'], 'msg': 'Extra inputs are not permitted', 'input': 42}]}, 'type': 'invalid_request_error', 'param': None, 'code': None}

Expected behavior
Using FaithfulnessEvaluator with Mistral AI API

To Reproduce
Here's my code:
api_endpoint = "https://api.mistral.ai/v1" evaluator = FaithfulnessEvaluator(api_params={"api_base_url": api_endpoint, "api_key": os.environ["MISTRAL_API_KEY"], "model": "mistral-large-latest"})

I tried changing
evaluator.api = "mistral"
but to no avail

And even
evaluator.api_key._env_vars to "MISTRAL_API_KEY" since it's value is "OPENAI_API_KEY"

FAQ Check

• Have you had a look at our new FAQ page?

System:

• OS:
• GPU/CPU:
• Haystack version (commit or version number):
• DocumentStore:
• Reader:
• Retriever:

Question
What am I missing? Any help or hint would be greatly appreciated!
Or is this simply not supported by FaithfulnessEvaluator or does the Mistral AI API not fully implement the openai "quasi-standard"?

Thank you.

[julian-risch]

Hello @miam-bonbon Haystack's Evaluators, including the FaithfulnessEvaluator currently don't support Mistral AI API yet. We will keep this issue as a feature request in our backlog.
If you are interested, the main part of the code that needs to be adapted is in the LLMEvaluator here:

haystack/haystack/components/evaluators/llm_evaluator.py

Line 108 in db76ae2

if api == "openai":

Once the LLMEvaluator supports the Mistral API, you could set the FaithfulnessEvaluator's init parameter called api to mistral.

[lbux]

Hi @miam-bonbon ! Unfortunately, I do not think Mistral follows the OpenAI compatible API standard. There seems to be differences in the payload causing Mistral to reject the request. The issue seems to be because we hard code a seed value to ensure consistency during testing and user end evaluating. The seed value is important, and we can not remove it, so as @julian-risch suggested, the best bet is to modify the LLMEvaluator to suit your needs. You can add an elif statement to check if the api is mistral and then pass in the values without the seed entry.

[julian-risch]

We are working on a ChatGenerator protocol as part of this issue that will enable configuring ChatGenerators for use in more complex components like LLMMetadataExtractor, Agent, Summarizer, LLMEvaluator (once we use a ChatGenerator instead of a Generator in LLMEvaluator).
I'll close this issue here and we continue tracking the more general request in:
#8918