Separate Function Calling LLM Framework for Complex In-Game Actions

[YetAnotherModder]

NOTE: requirements.txt has been modified to allow specific data folders and files to be read (JSONs, new GPT key).

This PR introduces separate LLM calls to a dedicated LLM dubbed "Function LLM."

LLM calls are structured differently depending on whether they are made to an OpenAI LLM (which includes a tools key in the request) or another provider (such as OpenRouter or Local, which do not define tools).

This PR introduces a function manager and a tools manager script, as well as new LLMFunction and Tooltips classes:

• The function manager handles the interaction between context, conversation, and client script requests.
• The tools manager stores and manages various components necessary for making Function LLM calls.

Function Limits

A "soft cap" exists on the number of functions that can be sent to an LLM at once to avoid confusion. This largely depends on the model’s training, parameter size, and the complexity of the functions themselves. Therefore, functions that are not relevant (e.g., those involving items not present in the context) are filtered out before being sent.

Currently Supported Actions

• Make one or more NPCs move to another actor.
• Make one or more NPCs stop.
• Make one or more NPCs loot specific item types.
• Make one or more NPCs attack a target.
• Make one or more NPCs use a specific item on another actor (currently limited to Stimpaks and RadAway).

Process Flow

1. Player responds to the LLM. Conversation scripts checks the custom context to see if function calling is enable. If it is, it sends a request to process_function_call() to the function_manager script.
2. Function_manager parses the context, checks conditions, and available parameters (only the last user response is used to avoid confusion).
3. Function_manager builds all functions and tooltips to send to the LLM based on conditions and context. The context payload is established at this step.
4. Function_manager formats and sends the request to the Function LLM using the appropriate prompt specified in the config.
5. A specific client script class makes the request to the LLM and implements tools if necessary (for OpenAI).
6. The response is sent back to output_manager.
7. Output_manager processes the response:
   • process_tool_call(): Handles OpenAI responses.
   • process_pseudo_tool_call(): Handles non-OpenAI responses that followed the Pydantic <tool_call> format.
   • process_unlabeled_function_content(): Attempts to process non-OpenAI responses without a <tool_call> flag, extracting identifiable data.
8. Function_manager parses the response data, comparing returned arguments with the context payload. Matches are retained and formatted.
9. A warning is sent to the roleplay LLM to inform it that an NPC is about to act.
10. The roleplay LLM may veto the action in the output_manager process_response() step if it deems it out of character (e.g., a farmer killing their cattle for no reason).
11. At the continue_conversation step of the conversation script, a call to function_manager post_response_actions() is made. If the function is valid and contains a context payload and is not vetoed it will be is added to the sentence instance properties.
12. Game_manager packages sources, targets, and modes found in the sentence object and sends a JSON response to the HTTP DLL.
13. The in-game engine (Papyrus) is flagged and executes the action.

NOTE: For non-OpenAI calls, </tool_call> is hardcoded as a stop value to prevent excessive generation and unnecessary LLM output.

---

Function Structure

LLMFunction JSON objects are stored in data/actions/functions/. Each object includes:

• GPT_func_name: Descriptive function name sent to ChatGPT and the in-game engine (prefixed with mantella_ when returned to the game).
• GPT_func_description: Brief description of what the function does.
• function_parameters: Dictionary of parameter objects specifying type, description, and contained item types (follows OpenAI's function-calling model: [OpenAI Function Calling](https://platform.openai.com/docs/guides/function-calling)).
• system_prompt_info: Instructions on function usage (optional but useful for guiding the LLM beyond function descriptions).
• GPT_required: Specifies required return values. Parameters containing "target," "source," or "mode" in their names are parsed and matched to the context payload before returning valid results to the in-game engine.
• allowed_games: Array of valid games. Functions will not load if there is no match.
• is_generic_npc_function, is_follower_function, is_settler_function: Boolean values determining when a function should load based on NPC type.
• is_pre_dialogue, is_post_dialogue, is_interrupting, is_radiant: Future-proofing fields (not yet in use).
• is_one_on_one, is_multi_npc: Determines function availability in one-on-one or multi-NPC conversations.
• llm_feedback: If validated, this description is sent to the roleplay LLM to provide context.
• parameter_package_key: Bundle of parameters and conditions required for function loading.
• veto_warning: Instruction to the roleplay LLM, allowing it to cancel an action if necessary.
• conditions: Specifies a JSON file containing custom checks for function loading.

---

Context and Transient Data

• context_payload: Stores transient data (targets, sources, and modes) and is cleared as needed.
• targets: Stores target-related data (IDs, names, distances), with IDs being most important for in-game execution.
• sources: Stores actor-related data (names, IDs).
• modes: Specifies how an action is executed (e.g., item usage modes).

Condition JSONs

Stored in data/actions/conditions/, condition JSONs define checks for function loading. Each contains:

• condition_name: String identifier.
• condition_type: Currently only allows boolean_check.
• operator_type: Supports and or or.
• keys_to_check: Array of context keys to verify.

---

Tooltips

Tooltips provide contextual information to the LLM without redundant function data, reducing confusion and token usage. Custom tooltips are stored in data/actions/tooltips/.

• TargetInfo: Stores names, distances, and IDs of function targets.
• ModeInfo: Stores available function modes.
• Special Formatting: {playername} can be used for personalized context strings.

A sample tooltip (placeholder_tooltip) is included in the files for reference.

---

New WebUI Menus

Function Inference Tab:

• Enable Function Veto (Free Will): Allows the roleplay LLM to veto out-of-character or inappropriate actions.
• Function LLM Service & Model: Independent from the roleplay LLM settings.
• Custom Token Count: Default 4096 (context changes per request).
• Temperature: Default 0.5 (lower values recommended).
• Top_p: Default 1.
• Frequency Penalty: Default 0.
• Max Tokens: 2000 (most responses are within 300 tokens).
• Function LLM Timeout: Default 15 seconds (prevents OpenRouter stalling before roleplay LLM is engaged).

New Prompt Categories:

• Non-OpenAI single NPC conversation
• Non-OpenAI multi-NPC conversation
• OpenAI single NPC conversation
• OpenAI multi-NPC conversation

NOTE: The flag NO_REGEX_FORMATTING_PAST_THIS_POINT prevents the Python script from modifying bracketed {} values used in Pydantic format explanations.

---

Script Updates

• message_thread now removes LLM warning class messages to prevent duplicate warnings.
• messages script introduces a new property for assistant messages to flag LLM warnings for function-roleplay LLM coordination.