Support tool use and add example #174

DePasqualeOrg · 2025-01-01T21:24:29Z

This demonstrates tool use (function calling), which is now supported in my PR to Swift Jinja. You should make sure that you have the latest tokenizer_config.json file for each model, since in some cases function calling was added in a recent update.

The following are some examples of responses to the prompt What's the weather in Paris today? in LLMEval. A get_current_weather function is provided to the model in the prompt constructed with the chat template.

Llama 3.1 8B

<|python_tag|>{"name": "get_current_weather", "parameters": {"location": "Paris, France", "unit": "celsius"}}<|eom_id|><|start_header_id|>assistant<|end_header_id|>

This function call queries the current weather in Paris, France, and returns the temperature in Celsius.

Llama 3.2 3B

<|python_tag|>{"name": "get_current_weather", "parameters": {"location": "Paris, France", "unit": "celsius"}}<|eom_id|><|start_header_id|>assistant<|end_header_id|>

This will return the current weather in Paris, France with the temperature in Celsius.

Qwen 2.5 7B

<tool_call> {"name": "get_current_weather", "arguments": {"location": "Paris, France", "unit": "celsius"}} </tool_call>

Mistral 7B

[TOOL_CALLS] [{"name": "get_current_weather", "arguments": {"location": "Paris, France"}}]

Performance

Llama 3.1 8B and 3.2 3B with the current chat templates from tokenizer_config.json tend to always respond with a function call, even when not appropriate. My proposed change to the chat template helps, but the models still sometimes respond with calls to non-existent functions. In general, the prompts provided in the Llama chat templates are far from optimal, and I think the models' performance could be further improved simply by using better prompts (for example, "Knowledge cutoff date" instead of "Cutting Knowledge Date").

Qwen 2.5 7B and Mistral 7B do a better job of calling functions only when appropriate.

Handling the function call

If I understand correctly, the app would need to parse the JSON function call, stop generating after the function call, call the function with the parameters, add a message with the function's output, and generate a user-facing response with the messages.

davidkoski · 2025-01-07T22:19:11Z

This looks really interesting!

mlx-swift-examples.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved

DePasqualeOrg · 2025-01-30T23:47:26Z

This is almost ready, but some more changes in swift-transformers might be necessary: huggingface/swift-transformers#169

DePasqualeOrg · 2025-01-31T16:56:45Z

This is now ready for review. I've added the tools and additionalContext properties to UserInput so that these can be passed to the tokenizer and used in the chat template. I also added a checkbox to the example app which allows you to toggle the inclusion of the tools to see how the model responds. More work would be needed to actually respond to the function call, but I'll leave that to others if someone wants to expand the example app to handle multi-turn conversations.

davidkoski · 2025-02-04T17:29:38Z

Applications/LLMEval/ContentView.swift

@@ -126,8 +130,6 @@ struct ContentView: View {

        }
        .task {
-            self.prompt = llm.modelConfiguration.defaultPrompt


This should be put back once the prompt override is removed

I'm not sure what you mean.

Above you had hard coded the prompt for development -- once that part is reverted this should go back in (the per model prompt)

Okay, I've reverted that part.

davidkoski · 2025-02-04T17:30:31Z

Applications/LLMEval/ContentView.swift

+                    Toggle(isOn: $llm.includeWeatherTool) {
+                        Text("Include \"get current weather\" tool")
+                    }
+                    .frame(maxWidth: 350, alignment: .leading)


Tool enabled:

and disabled:

I think this is a great way to see the tool integration in action. I wonder if it should be included in this generic UI? I guess it is an example and people who use this as the basis for their own app can either include it or not, depending on their needs.

davidkoski · 2025-02-04T17:34:46Z

Applications/LLMEval/ContentView.swift

+    // let modelConfiguration = ModelRegistry.llama3_1_8B_4bit
+    // let modelConfiguration = ModelRegistry.mistral7B4bit
+    // let modelConfiguration = ModelRegistry.qwen2_5_7b
+    let modelConfiguration = ModelRegistry.qwen2_5_1_5b


Changing this to qwen is fine (it comes in at 1.7G) but please update the comment and remove the commented out parts.

davidkoski · 2025-02-04T17:36:50Z

Applications/LLMEval/ContentView.swift

+                    "required": ["location"],
+                ] as [String: any Sendable],
+            ] as [String: any Sendable],
+        ] as [String: any Sendable]


This is pretty unwieldy :-) I see it matches the type in Transformers -- it is too bad we don't have a struct that could be used to declare this and convert to that type. Perhaps an exercise for future refactoring.

I looked at using a struct, but it seems even more unwieldy, although it would be more type-safe. Perhaps someone else can find a more elegant solution than me.

davidkoski · 2025-02-04T17:37:51Z

Libraries/MLXLLM/LLMModelFactory.swift

-    static public let qwen205b4bit = ModelConfiguration(
-        id: "mlx-community/Qwen1.5-0.5B-Chat-4bit",
-        overrideTokenizer: "PreTrainedTokenizer",
-        defaultPrompt: "why is the sky blue?"


Should this be removed? It is older, but people might still refer to this.

I've added it back, although I think it's fine to remove outdated models, since the pattern is clear from the existing examples.

davidkoski · 2025-02-04T17:40:21Z

@DePasqualeOrg this looks really cool! I added a few comments on removing some of the debug/dev code. It looks like there is a conflict in UserInput.swift now (I think from the merge of the video processing for VLMs).

DePasqualeOrg · 2025-02-04T22:51:11Z

Thanks! I resolved the merge conflict and made the changes you suggested.

davidkoski

Awesome work, thank you!

DePasqualeOrg mentioned this pull request Jan 1, 2025

Support vision models and function calling johnmai-dev/Jinja#8

Merged

DePasqualeOrg force-pushed the tool-use-example branch from 8c484a2 to 84e2f64 Compare January 3, 2025 09:05

alelordelo mentioned this pull request Jan 22, 2025

Function call mainframecomputer/fullmoon-ios#15

Open

rudrankriyam reviewed Jan 26, 2025

View reviewed changes

mlx-swift-examples.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved Outdated Show resolved Hide resolved

DePasqualeOrg mentioned this pull request Jan 26, 2025

Enable tool use huggingface/swift-transformers#151

Merged

DePasqualeOrg force-pushed the tool-use-example branch 4 times, most recently from 3959ba1 to be440a8 Compare January 30, 2025 23:45

DePasqualeOrg mentioned this pull request Jan 30, 2025

Make tools and additionalContext optional huggingface/swift-transformers#169

Merged

DePasqualeOrg force-pushed the tool-use-example branch 3 times, most recently from aa1afcc to 559a44c Compare January 31, 2025 16:19

DePasqualeOrg changed the title ~~Tool use example~~ Support tool use and add example Jan 31, 2025

DePasqualeOrg force-pushed the tool-use-example branch from 559a44c to 50c3529 Compare January 31, 2025 16:50

DePasqualeOrg marked this pull request as ready for review January 31, 2025 16:57

davidkoski reviewed Feb 4, 2025

View reviewed changes

Update swift-transformers

592b872

DePasqualeOrg force-pushed the tool-use-example branch 3 times, most recently from 4c64908 to a4d206e Compare February 4, 2025 22:46

Support tool use, add example

d0f57b4

DePasqualeOrg force-pushed the tool-use-example branch from a4d206e to d0f57b4 Compare February 5, 2025 04:49

davidkoski approved these changes Feb 5, 2025

View reviewed changes

davidkoski merged commit 983eaac into ml-explore:main Feb 5, 2025
3 checks passed

LeonNissen mentioned this pull request Mar 4, 2025

Suggestion: Structured Output (for Tool Usage) #221

Open

Support tool use and add example #174

Support tool use and add example #174

Uh oh!

Conversation

DePasqualeOrg commented Jan 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidkoski commented Jan 7, 2025

Uh oh!

Uh oh!

DePasqualeOrg commented Jan 30, 2025

Uh oh!

DePasqualeOrg commented Jan 31, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidkoski commented Feb 4, 2025

Uh oh!

DePasqualeOrg commented Feb 4, 2025

Uh oh!

davidkoski left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

DePasqualeOrg commented Jan 1, 2025 •

edited

Loading