Avoid calling/sending data to LLM with native plugin and function calling

[EVENFLOW212]

We are using the SK version 1.13 and Azure Open AI.

We have 2 native plugins (P1 and P2) with one function/method each (M1 and M2). P1M1 gets called thru function calling and gets data from internal rest service which is json data. After P1M1 gets called thru function calling, we need to "avoid" sending data to LLM but not prevent calls to P2M2 if orchestrated as such. Is there a way to avoid sending data to LLM yet continue successfully with function calling and other calls if deemed necessary(considering that kernel chat-history context is based on tool and assistant data prior to the LLM call). This can apply to any number of chained function calls. We just want to avoid sending data to LLM based on our discretion

[dmytrostruk]

@EVENFLOW212 Could you please try to register IAutoFunctionInvocationFilter and as soon as all your plugins are executed, terminate process with this syntax?

semantic-kernel/dotnet/samples/Concepts/Filtering/AutoFunctionInvocationFiltering.cs

Lines 78 to 79 in 546ec65

	// Example: Terminate function invocation
	context.Terminate = true;

Basically, your plugins/functions will be invoked, but function result won't be sent to LLM, but instead it will be returned to the place where you started the execution. Please let us know if this helps to resolve your scenario. Thanks!

[EVENFLOW212]

Thanks for your quick response @dmytrostruk. We will try it out and definitely and promise to come back.

[EVENFLOW212]

Hi @dmytrostruk

I tried your suggestion and it did not work and then realized that its only for auto function calling.

We are using ToolCallBehavior.EnableKernelFunctions, hence manual function calling using line 63 to 107 of the sample below.
https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/AutoFunctionCalling/OpenAI_FunctionCalling.cs

Is there a solution for avoiding sending to LLM based on some logic as asked in the original question when we use ToolCallBehavior.EnableKernelFunctions ?

Thanks

[dmytrostruk]

We are using ToolCallBehavior.EnableKernelFunctions, hence manual function calling using line 63 to 107 of the sample below.

@EVENFLOW212 In this case, you send request to LLM and ask which functions to call, get a result with function list, execute it in order manually, and that should be it. Without initial request to LLM, you won't know which functions to call, so this is required.

After P1M1 gets called thru function calling, we need to "avoid" sending data to LLM but not prevent calls to P2M2 if orchestrated as such.

With manual function calling, after P1M1 gets called, there should not be any requests to LLM in between, unless your P1M1 function performs request to LLM on its own.

[EVENFLOW212]

Hi @dmytrostruk

We are using the example - https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/FunctionCalling/OpenAI_FunctionCalling.cs

We are using SK version 1.13 and using native plugins ( 2 plugins , 1 function in each plugin)

We have a prompt that we know will call the 2 plugins (2 functions back to back). But, when we run the code above the part below only returns only 1 plugin/function and does not return the other plugin function. In essence according to what you are saying we should expect 2 plugins/functions, but only see 1.

IEnumerable functionCalls = FunctionCallContent.GetFunctionCalls(result);
if (!functionCalls.Any())
{
break;
}
foreach (FunctionCallContent functionCall in functionCalls)

When the while loop is executed again after the first Plugin/function called, then the 2nd plugin is called, but that's too late for our condition logic for not sending to LLM.

We would appreciate it when the FunctionCallContent.GetFunctionCalls(result); executes the first time to know which other functions calls are going to be made to execute our conditional logic.

Thanks

[dmytrostruk]

We have a prompt that we know will call the 2 plugins (2 functions back to back). But, when we run the code above the part below only returns only 1 plugin/function and does not return the other plugin function. In essence according to what you are saying we should expect 2 plugins/functions, but only see 1.

@EVENFLOW212 Got it, thanks for providing more details. It sounds like LLM decides to call only one function but not a second one, so you need to perform second request to LLM in order to call that second function. There could be two cases:

1. Your second function relies on results of your first function. If that's a case, I'm not sure there will be a way to call a second function without calling LLM because your workflow requires to provide result from first function as input to second function. You will need to call LLM again, so it can take that result, and put it as input to second function.
2. Your functions are independent of each other and can be called in parallel. In this case, LLM should provide all necessary functions to call during first request, and no more requests will be required. But parallel function calling is not supported by all models. Here is a list of supported models: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models. You should use a model that has parallel function calling listed in Description field.

Could you please try to use one of the models listed above and see if LLM provides all functions to execute during first request? If you already use some of these models, then I suggest improving the prompt and try to be more specific what result you want to achieve. Make sure that both plugins and functions are imported in Kernel, provide more information about functions using Description attribute on function and argument levels, so LLM can understand its purpose.

I just tested this scenario using the example from our codebase which you shared, and I believe it's similar to your use-case.
We import two functions, one of them returns current datetime, and other one returns weather in some city:

semantic-kernel/dotnet/samples/Concepts/FunctionCalling/OpenAI_FunctionCalling.cs

Lines 235 to 236 in 01e826d

	kernel.CreateFunctionFromMethod(() => DateTime.UtcNow.ToString("R"), "GetCurrentUtcTime", "Retrieves the current time in UTC."),
	kernel.CreateFunctionFromMethod((string cityName) =>

Then we ask LLM a question that involves these two functions:

semantic-kernel/dotnet/samples/Concepts/FunctionCalling/OpenAI_FunctionCalling.cs

Line 137 in 01e826d

chatHistory.AddUserMessage("Given the current time of day and weather, what is the likely color of the sky in Boston?");

As you can see, LLM responds with two functions to call within the same request, so all I need is to invoke these functions and I don't need to perform second LLM request if I'm satisfied with result. This is what I get as result from FunctionCallContent.GetFunctionCalls:

Let me know if this is helpful.

[EVENFLOW212]

Hi @dmytrostruk

Again, much obliged and thanks for your response.

The example you have given are 2 independent functions. For our use case, you are right, the 2nd function in the 2nd plugin depends on the results of the first function in the first plugin. I would have thought that I could get some metadata about the number of functions that will be called based on the prompt initially given. I know that the LLM may decide more functions to be called after the fact. But would have been nice to get the initial metadata of the number of functions and their metadata before hand. In this case 2.

The issues for us is that the LLM token by token response or ChatCompletion is slow and we have our own UI to display a specific formatted data. The LLM response is not structured unless specified in the prompt.

All in all, I was looking to see if any metadata is available for the dependent functions before their actual execution. I am assuming from your response before, this cannot be achieved.

Thanks

[dmytrostruk]

The example you have given are 2 independent functions. For our use case, you are right, the 2nd function in the 2nd plugin depends on the results of the first function in the first plugin. I would have thought that I could get some metadata about the number of functions that will be called based on the prompt initially given. I know that the LLM may decide more functions to be called after the fact. But would have been nice to get the initial metadata of the number of functions and their metadata before hand. In this case 2.

@EVENFLOW212 I tried this scenario using gpt-4 and example with manual function calling from our codebase with following changes:
I made functions to depend on each other:

kernel.CreateFunctionFromMethod(() => "Boston", "GetCityName", "Retrieves the city name."),
kernel.CreateFunctionFromMethod((string cityName) =>
    cityName switch
    {
        "Boston" => "61 and rainy",
        "London" => "55 and cloudy",
        "Miami" => "80 and sunny",
        "Paris" => "60 and rainy",
        "Tokyo" => "50 and sunny",
        "Sydney" => "75 and sunny",
        "Tel Aviv" => "80 and sunny",
        _ => "31 and snowing",
    }, "Get_Weather_For_City", "Gets the current weather for the specified city"),

And I used following prompt: Get a city name and the weather for it.. In my case, it returned two functions during first request, but since cityName is required, LLM just came up with random argument value, which in my case was London, but as you can see, my function to get a city name returns Boston.

So, I can say that this scenario is supported, you just have to be ready that LLM will come up with some random argument values, and in your case maybe you need to change a prompt a little bit or try to set system prompt and say Return all functions to be called or something like that. I think your scenario is achievable. But in this case, in order to invoke second function, you will have to map your function results and arguments on your side.