adding tool_choice to ModelSettings

[webcoderz]

building upon : #785

i added the tool_choice to ModelSettings

[iabgcb]

pls pls see to it urgently that linting errors are solved.

[webcoderz]

pls pls see to it urgently that linting errors are solved.

yea i dont think those failures are on what i added, i think therye elsewhere :)

[webcoderz]

can we get this landed?

[silkspace]

This is a nice to have, and unblocks core functionality. Which runway will this land on?

[webcoderz]

since it's supported by the others will add to the other clients as well

[webcoderz]

looks like mostof them already supprt / have it integrated, im not sure how it should look for gemini/cohere, cc @sydney-runkle

[kernelzeroday]

+1 request for review, are additional changes needed to get this functionality merged? Thank you!

[webcoderz]

confused on what more is needed here ?

[sydney-runkle]

Thanks for the contribution - this still needs some more support across other model files. Happy to help get this across the line, but we need impl for all supported models.

[sydney-runkle]

The indication that it's supported by all of these models needs to be backed up with code changes - in all of the corresponding model files, we need to check model_settings.tool_choice like you've done for groq and openai

[sydney-runkle]

This could also use more documentation - we'll want to document what each of these specifically means.

Additionally, I'm concerned about having this on the top ModelSettings level - anthropic supports "auto", "any", or a specific tool name, which is different than the above. Thus, I think we should implement tool_choice on individual model settings (like AnthropicModelSettings, OpenAIModelSettings) with the appropriate options.

[webcoderz]

yea I agree, will rebase onto each individual one since they all seem to to something slightly different

[webcoderz]

looks like mistral already supports the tool_choice where when is creating the chat completion, it has a method self._get_tool_choice() at L251 that does the conditional check i added inline in groq and openai already, should i refactor those to match this ? seems to be a cleaner pattern

[webcoderz]

also mistral has tool_choice set already in MistralToolChoiceEnum which was generated by speakeasy , not sure i like this pattern , i feel your rationale makes the most sense to have it inside each specific providers model settings, this seems out of place in its current state

[webcoderz]

also seems like anthropic already has it supported via ToolChoiceParam which was also generated by speakeasy, not sure how to proceed on that as its already on that and mistral insofar

[webcoderz]

Thanks @sydney-runkle , will try to get these changes in tommorow; I don't have a cohere , groq, or anthropic sub so unsure how to test those ; I only have access to llama based models and openAI so I might need assist on those; anthropic looks pretty straightforward I can prob get that one in also, and groq is llama so can prob swing that one as well, never touched cohere before though

[sydney-runkle]

@webcoderz, you can look at their API/sdk docs to figure out what values are accepted + supported.

Given the demand for this, I'm also happy to pick this up tomorrow and get this into our next release.

[webcoderz]

alright, so it seems mistral and anthropic are already using their own system for tool_choice already thats outside of their respective ModelSettings classes ( anthropic: L207 of anthropic.py) & (mistral: L247 of mistral.py) , gemini and vertex the only docs i find that mention tool_choice are when theyre using the openai client, so since its already there i dont think we need to add there? cc @sydney-runkle

[sydney-runkle]

Thanks for the updates - I'll pick this up tomorrow to get things across the line.

[webcoderz]

Awesome lmk; I was also unsure where to add (if even necessary?) in the docs

[TensorTemplar]

I will test this with sglang and vllm oai apis instead for llama models, since i am getting the same issues. @sydney-runkle are there plans to add integration tests that do real api calls (behind flag) for these cases? I could add an sglang/vllm/ollama container and tiny llama for such tests, if needed

[webcoderz]

I will test this with sglang and vllm oai apis instead for llama models, since i am getting the same issues. @sydney-runkle are there plans to add integration tests that do real api calls (behind flag) for these cases? I could add an sglang/vllm/ollama container and tiny llama for such tests, if needed

Wouldn’t be a bad idea; im using the OpenAIModel with vllm already but unsure how the others perform

[sydney-runkle]

In my queue for tomorrow - sorry, spent most of my time on FallbackModel today.

[webcoderz]

Hahaha it happens! I also found myself down a rabbit hole:)

[TensorTemplar]

I will test this with sglang and vllm oai apis instead for llama models, since i am getting the same issues. @sydney-runkle are there plans to add integration tests that do real api calls (behind flag) for these cases? I could add an sglang/vllm/ollama container and tiny llama for such tests, if needed

Wouldn’t be a bad idea; im using the OpenAIModel with vllm already but unsure how the others perform

Can confirm this works with sglang and llama3.x models, apologies for the copy paste formatting:

class InformationIsNeeded(BaseModel):
    """Indicates that more information is needed to start the research."""
    justification: str

class InformationIsSufficient(BaseModel):
    """Indicates that the information is sufficient to start the research."""
    justification: str

IntentResponse = Union[InformationIsNeeded, InformationIsSufficient]

llama_32_11b_vlm = OpenAIModel(
    "meta-llama/Llama-3.2-11B-Vision-Instruct",
    base_url=os.getenv("PIPELINE_OPENAI_COMPATIBLE_PROXY_URL"),  # e.g. the sglang instance
    api_key=os.getenv("PIPELINE_PROXY_API_KEY"),
)

query_intent_agent = Agent(
    llama_32_11b_vlm,
    result_type=IntentResponse,  # type: ignore
    retries=1,
    model_settings=ModelSettings(tool_choice="auto"),  # with this i am getting a tool call now
)

result = await query_intent_agent.run(
                user_prompt="Decide if the following user resquest is sufficiently clear or if more info is required: What is 3+3?",
                result_type=IntentResponse,  # type: ignore
)

ModelResponse(parts=[ToolCallPart(tool_name='final_result_InformationIsSufficient', args='{"justification": "The user request is clear and can be answered directly without needing more information."}', tool_call_id='1', part_kind='tool-call')], model_name='Llama-3.2-11B-Vision-Instruct', timestamp='2025-02-14T13:10:50+00:00', kind='response')

According to sglang docs you need to add --tool-call-parser

docker run -p 30000:30000 docker.io/lmsysorg/sglang:v0.4.3-cu124-srt \
    python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.2-11B-Vision-Instruct \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 30000 \
    --mem-fraction-static 0.85 \
    --tool-call-parser llama3

[sydney-runkle]

Ok, my understanding of tool_choice specs:

Anthropic:

# basically, auto, any, or a specific tool
ToolChoiceParam: TypeAlias = Union[ToolChoiceAutoParam, ToolChoiceAnyParam, ToolChoiceToolParam]

Cohere:

V2ChatRequestToolChoice = typing.Union[typing.Literal["REQUIRED", "NONE"], typing.Any]

Gemini:

# reference: https://ai.google.dev/gemini-api/docs/function-calling
FunctionCallConfigMode = Literal["ANY", "NONE", "AUTO"]

Groq:

# ChatCompletionNamedToolChoiceParam indicates a specific tool
ChatCompletionToolChoiceOptionParam: TypeAlias = Union[
    Literal["none", "auto", "required"], ChatCompletionNamedToolChoiceParam
]

Mistral:

ChatCompletionStreamRequestToolChoice = Union[ToolChoice, ToolChoiceEnum]
ToolChoiceEnum = Literal["auto", "none", "any", "required"]

OpenAI:

ChatCompletionToolChoiceOptionParam: TypeAlias = Union[
    Literal["none", "auto", "required"], ChatCompletionNamedToolChoiceParam
]

Looks like we can unify these into the following options:

• 'none': the model will not call any tool
• 'auto': the model chooses to call tool(s), or not
• 'required': the model will call at least one tool
• named function: the model will call said tool - can structure this as a typed dict, similar to OpenAI

We should implement support for this across all models.

@webcoderz, given this analysis, are you ok with continuing here? We can store this tool_choice setting in ModelSettings.

[webcoderz]

Ok, my understanding of tool_choice specs:

Anthropic:

# basically, auto, any, or a specific tool
ToolChoiceParam: TypeAlias = Union[ToolChoiceAutoParam, ToolChoiceAnyParam, ToolChoiceToolParam]

Cohere:

V2ChatRequestToolChoice = typing.Union[typing.Literal["REQUIRED", "NONE"], typing.Any]

Gemini:

# reference: https://ai.google.dev/gemini-api/docs/function-calling
FunctionCallConfigMode = Literal["ANY", "NONE", "AUTO"]

Groq:

# ChatCompletionNamedToolChoiceParam indicates a specific tool
ChatCompletionToolChoiceOptionParam: TypeAlias = Union[
    Literal["none", "auto", "required"], ChatCompletionNamedToolChoiceParam
]

Mistral:

ChatCompletionStreamRequestToolChoice = Union[ToolChoice, ToolChoiceEnum]
ToolChoiceEnum = Literal["auto", "none", "any", "required"]

OpenAI:

ChatCompletionToolChoiceOptionParam: TypeAlias = Union[
    Literal["none", "auto", "required"], ChatCompletionNamedToolChoiceParam
]

Looks like we can unify these into the following options:

• 'none': the model will not call any tool
• 'auto': the model chooses to call tool(s), or not
• 'required': the model will call at least one tool
• named function: the model will call said tool - can structure this as a typed dict, similar to OpenAI

We should implement support for this across all models.

@webcoderz, given this analysis, are you ok with continuing here? We can store this tool_choice setting in ModelSettings.

yea no prob can take care of it this weekend
Cool ok going back into Modelsettings? 👍

[webcoderz]

So @sydney-runkle , a little confused here, should I pmuch just revert everything back to my PR where I did have in ModelSettings originally?

[sydney-runkle]

Yep, but you'll have to implement logic for all of the models (not just groq + openai)

[webcoderz]

gotcha, so a mixture of adding to to top level, and then pmuch your research from above at the modelsettings of the. specific model level

Yep, but you'll have to implement logic for all of the models (not just groq + openai)