Tool Support w/o strings

[rseymour]

In late July 2024 tool support was added to ollama: https://ollama.com/blog/tool-support . About a year ago function calling was added to chatGPT and just this month 2024-08 Structured Outputs were added. I added function calling support to a rust library but I was not happy with the stringly-typed json-schema-esque part.

I think I have worked out a way to mash together schemars, proc macros, serde and some other bits and pieces to get something like:

#[function_as_tool]
fn my_code(a: String, b: float) -> MyType {
    MyType {} 
}

and spit out the associated schema with a function like:
tool_my_code
and ideally create some sort of tool dictionary that either behind the scenes or in rust code outside of the interaction can be used to actually execute the my_code function with the appropriate args.

Ideally with no extra strings or json!() calls.

I think I could implement it entirely separate from ollama-rs, but for maximum effectiveness it might need to go in here: https://github.com/pepperoni21/ollama-rs/blob/0.2.0/src/generation/functions/tools/mod.rs#L24 but at that point the concerns of a trait might mess with the introspective nature of the proc macro (which would rewrite functions and add code auto-magically)

Just gauging interest, seeing if anyone else is interested in using functions without needing to write strings (and ideally also support when StructuredOutput comes to the open models). I think it's a point where other languages are more stringy, and rust can show just how strict we can make LLM calling (for better or worse).

Thanks again for this nice library.

[rseymour]

In the example above you can see that the tools defined inside of the repo (which are extremely useful) define the Tool trait which creates a generic interface for all tools:

One issue as I see is the call and run functions just return Result<String, Box<dyn Error>>. The String is de facto JSON, but in the duck duck go example, the function that is called is actually returning a cleanly typed Vec<SearchResults> which is then serialized to JSON.

For some functions, the user might not want to return JSON or have the JSON creation be separate from the function return. I know why this was done (ie to keep the messages array full of json responses from the tool) but I think it would be nice to be able to call any Rust function essentially regardless of what it returns.

[rseymour]

Not yet integrated or complete but the basic idea is here:
https://github.com/rseymour/func_me/tree/main

I think I could make the proc macro (now called #[json_value]) into a full fledged thing that generated the impl block. I did this mainly as a proof of concept to see if the arguments and types could be pulled from an attribute-decorated function or if that was just impossible.

It isn't pretty but it shows that tool definitions don't necessarily need to be done by hand. I would argue that stringifying the tool output is for the user to decide, not for the library to insist on.

[rseymour]

Updated with the tool calling json completely autogenerated. essentially fn -> value but with a single decorator attribute on the function and one on the struct containing it and the ability to get the entire set of tools with one call. Working now on calling the tool with just the tool_calls Vec. Essentially going from the Value -> fn(variable number of arguments). My plan is to make fn(variable number of arguments) into fn(Arguments) where Arguments is a generated struct that can to/from to Value w/ some amount of type checking, which then calls the "real" function underneath it. All with just the attribute macro.

[rseymour]

Made some updates so that now the tool can be called: https://github.com/rseymour/func_me This allows for tool calling without any 'stringly typed' work going into the JSON. I'm going to create an ollama-rs example next.

The boilerplate request/json logic shown in these lines should be reduced when I do so:
https://github.com/rseymour/func_me/blob/main/examples/ollama_fn.rs#L28-L54

But as you can see the function name is written only once and the LLM calls / tool calls update automatically.

[rseymour]

So now that I've got func_me working I think I can definitely make an attribute to automatically implement the Tool trait from essentially the run function. On one hand I think this is a bit of overkill, but it clearly adds type safety. I will implement it in func_me but I think it could be added as a feature to ollama-rs directly as well.

[rseymour]

Further updates. I created a branch where I added a trait called Toolbox that acts like the Vec<Arc<dyn Tool>>.
Branch: https://github.com/rseymour/ollama-rs/blob/add-toolbox-trait/
Example: https://github.com/rseymour/ollama-rs/blob/add-toolbox-trait/examples/chat_with_toolbox.rs

The Toolbox trait has 2 functions:

pub trait Toolbox: Send + Sync {
    fn get_impl_json(&self) -> Value;
    fn call_value_fn(&self, tool_name: &str, tool_args: Value) -> Value;
}

Both of those are (almost) autogenerated by the macros in func_me. Right now in my example I generated them by hand, but I'm going to get them to autogenerate soon.

get_impl_json just returns the json necessary to send with the system prompt

call_value_fn does the string lookup and function call, returning a Value

The neat thing here is that the tool_args Value is generated and type consistent with the functions that have the add_to_toolbox attribute on them and no boilerplate is needed.

Caveat re: autogen of the trait, when I first did this I wanted to just create static functions on a struct type. After looking at the ollama-rs code I realized I'd need to implement a trait to make it work.

This chunk of code in chat_with_toolbox.rs, is the last piece I need to autogen:

impl Toolbox for MyToolBox {
    fn get_impl_json(&self) -> Value {
        MyToolBox::get_impl_json()
    }

    fn call_value_fn(&self, tool_name: &str, tool_args: Value) -> Value {
        MyToolBox::call_value_fn(tool_name, tool_args)
    }
}

There's also a lot of error handling and cleanup that may be needed, and for now this only runs the first function called (as seen in the other issue).

Thanks for entertaining my commentary and code, hopefully this makes some sense.

[mcmah309]

@rseymour Is this still being developed and are you planning to create PR to this repo?

[rseymour]

@mcmah309 If folks were interested I could definitely put in a PR. I see some work has been done around function calling, but this is still unique. Would you want it hidden behind a feature? Right now it's 2 proc macros. Using func_me w/ ollama-rs is also totally doable now without PR-ing against ollama-rs -> https://github.com/rseymour/func_me Up to you, I'm fine adding or closing this out. Doing a proper PR may take a few weeks on my end, busy with the holidays and work.

…

On Mon, Dec 2, 2024 at 5:56 AM Henry ***@***.***> wrote: @rseymour <https://github.com/rseymour> Is this still being developed and are you planning to create PR to this repo? — Reply to this email directly, view it on GitHub <#62 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAZ2TP3IFBFDLGZET5VAUD2DQ4GTAVCNFSM6AAAAABMLDK242VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJRGIYDSMZVG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mcmah309]

Merging it into this library would be great if possible!