[BFCL] Introducing custom handlers

[sangmandu]

Hi there,

I am a current user of bfcl and would like to submit this PR to address an improvement opportunity I've identified while working with the module.

Currently, model handlers are managed through the handler_map.py, which presents some challenges. This managed approach requires code updates from maintainers whenever new models are released. Additionally, users must wait for updates if their desired model isn't yet supported.

While technically proficient users might implement their own solutions for unsupported models, this especially becomes problematic when using bfcl as a submodule. Modifying the original code is not ideal from a maintenance perspective, as it requires additional resources for ongoing maintenance.

Therefore, I'm submitting this PR to enhance the main codebase.

The current build_handler function only allows fetching predefined model handlers. I propose implementing a HandlerLoader that enables users to input custom handlers. This approach minimizes changes to the main codebase while allowing users to manage their custom handlers through just two files: a Python file containing the handler definition and a JSON configuration file.

[sangmandu]

Previously, you could simply use dictionary mapping to get handler.
For now, HandlerLoader will controll.

You can set environment variable for BFCL_HANDLER_CONFIG.
This environ var. point to the path of json file.
and if this exists, Loader can find the path of json file.

[sangmandu]

This is example of json files

{
    "my-custom-model": {
        "module_path": "/path/to/my_handlers.py",
        "class_name": "MyCustomHandler"
    }
}

and example of setting env var.

# you can enter this on command"
export BFCL_HANDLER_CONFIG=/path/to/handler_config.json

python -m bfcl generate --model my-custom-model --test-category all

[sangmandu]

At /path/to/my_handlers.py you can make custom handlers clas

from bfcl.model_handler.base_handler import BaseHandler
from bfcl.model_handler.model_style import ModelStyle

class MyCustomHandler(BaseHandler):
    def __init__(self, model_name, temperature):
        super().__init__(model_name, temperature)
        self.model_style = ...
        ...

[sangmandu]

For detailed example, I've introduced my case.

berkeley-function-call-leaderboard/custom_handler.py

class MyCustomHandler(OpenAIHandler):
    def __init__(self, model_name, temperature):
        BaseHandler.__init__(self, model_name, temperature)
        self.model_style = ModelStyle.OpenAI
        self.client = OpenAI(
            api_key="EMPTY",
            base_url="..."
        )

berkeley-function-call-leaderboard/handler_config.json

{
    "allganize/Alpha-Ko-32B-GPTQ-Int8-202411": {
        "module_path": "custom_handler.py",
        "class_name": "MyCustomHandler"
    }
}

export BFCL_HANDLER_CONFIG=handler_config.json

[sangmandu]

The result works well when I enter below command
python3 -m bfcl generate --model allganize/Alpha-Ko-32B-GPTQ-Int8-202411 --test-category multi_turn_base --num-threads 40

[HuanzhiMao]

Thanks for the PR @sangmandu ! Could you provide an example BFCL_HANDLER_CONFIG?

[sangmandu]

Thanks for the PR @sangmandu ! Could you provide an example BFCL_HANDLER_CONFIG?

@HuanzhiMao

Wow, rapid reply! Thanks.
It looks like our timing is a little off, and I've provided an example in the code explanation section.

[sangmandu]

For dynamic load, we'd used model_metadata rather than MODEL_METADATA_MAPPING

[sangmandu]

I've implemented the same config-based handler loading functionality in the evaluate module that was previously available in generation.

export BFCL_MODEL_METADATA=model_metadata.json

python3 -m bfcl evaluate --model allganize/Alpha-Ko-32B-GPTQ-Int8-202411 --test-category multi_turn_base

[sangmandu]

This is example of model_metadata.json

{
    "metadata": {
        "allganize/Alpha-Ko-32B-GPTQ-Int8-202411": [
            "alpha-ko-32b-gptq-int8",
            "https://huggingface.co/allganize/Alpha-Ko-32B-GPTQ-Int8-202411",
            "Allganize",
            "Custom License"
        ]
    },
    "prices": {
        "allganize/Alpha-Ko-32B-GPTQ-Int8-202411": 0.0
    },
    "no_cost_models": [
        "allganize/Alpha-Ko-32B-GPTQ-Int8-202411"
    ]
} 

[sangmandu]

The result after evaluation