[BFCL] Rename Directories: proprietary_model->api_inference, `oss…

…_model`->`local_inference` for Better Clarity (#859) Address #856 The folder names `oss_model` and `proprietary_model` within the `model_handler` directory are updated to better reflect their actual purpose. The current naming is misleading, as there are open-source model handlers located in the proprietary_model folder, which causes confusion. Updating these names will improve clarity and maintain logical organization.
ShishirPatil · Jan 3, 2025 · 5fe4a87 · 5fe4a87
1 parent 0cea216
commit 5fe4a87
Show file tree

Hide file tree

Showing 40 changed files with 63 additions and 63 deletions.
diff --git a/berkeley-function-call-leaderboard/CONTRIBUTING.md b/berkeley-function-call-leaderboard/CONTRIBUTING.md
@@ -23,12 +23,12 @@ berkeley-function-call-leaderboard/
 │   │   ├── executable_eval/      # Evaluation by execution
 │   │   ├── multi_turn_eval/      # Multi-turn evaluation
 │   ├── model_handler/            # All model-specific handlers
-│   │   ├── oss_model/            # Handlers for locally-hosted models
+│   │   ├── local_inference/            # Handlers for locally-hosted models
 │   │   │   ├── base_oss_handler.py   # Base handler for OSS models
 │   │   │   ├── llama_fc.py           # Example: LLaMA (FC mode)
 │   │   │   ├── deepseek_coder.py     # Example: DeepSeek Coder
 │   │   │   ├── ...
-│   │   ├── proprietary_model/    # Handlers for API-based models
+│   │   ├── api_inference/    # Handlers for API-based models
 │   │   │   ├── openai.py             # Example: OpenAI models
 │   │   │   ├── claude.py             # Example: Claude models
 │   │   │   ├── ...
@@ -46,7 +46,7 @@ To add a new model, focus primarily on the `model_handler` directory. You do not
 ## Where to Begin
 
 - **Base Handler:** Start by reviewing `bfcl/model_handler/base_handler.py`. All model handlers inherit from this base class. The `inference_single_turn` and `inference_multi_turn` methods defined there are helpful for understanding the model response generation pipeline. The `base_handler.py` contains many useful details in the docstrings of each abstract method, so be sure to review them.
-  - If your model is hosted locally, you should also look at `bfcl/model_handler/oss_model/base_oss_handler.py`.
+  - If your model is hosted locally, you should also look at `bfcl/model_handler/local_inference/base_oss_handler.py`.
 - **Reference Handlers:** Checkout some of the existing model handlers (such as `openai.py`, `claude.py`, etc); you can likely reuse some of the existing code if your new model outputs in a similar format.
   - If your model is OpenAI-compatible, the `openai.py` handler will be helpful (and you might be able to just use it as is).
   - If your model is locally hosted, the `llama_fc.py` handler or the `deepseek_coder.py` handler can be good starting points.

diff --git a/berkeley-function-call-leaderboard/README.md b/berkeley-function-call-leaderboard/README.md
@@ -228,7 +228,7 @@ When specifying multiple models or test categories, separate them with **spaces*
 
 We welcome contributions! To add a new model:
 
-1. Review `bfcl/model_handler/base_handler.py` and/or `bfcl/model_handler/oss_model/base_oss_handler.py` (if your model is hosted locally).
+1. Review `bfcl/model_handler/base_handler.py` and/or `bfcl/model_handler/local_inference/base_oss_handler.py` (if your model is hosted locally).
 2. Implement a new handler class for your model.
 3. Update `bfcl/model_handler/handler_map.py` and `bfcl/eval_checker/model_metadata.py`.
 4. Submit a Pull Request.

diff --git a/.../bfcl/model_handler/oss_model/__init__.py → ...l/model_handler/api_inference/__init__.py b/.../bfcl/model_handler/oss_model/__init__.py → ...l/model_handler/api_inference/__init__.py
diff --git a/...model_handler/proprietary_model/claude.py → ...fcl/model_handler/api_inference/claude.py b/...model_handler/proprietary_model/claude.py → ...fcl/model_handler/api_inference/claude.py
diff --git a/...model_handler/proprietary_model/cohere.py → ...fcl/model_handler/api_inference/cohere.py b/...model_handler/proprietary_model/cohere.py → ...fcl/model_handler/api_inference/cohere.py
diff --git a/...l_handler/proprietary_model/databricks.py → ...model_handler/api_inference/databricks.py b/...l_handler/proprietary_model/databricks.py → ...model_handler/api_inference/databricks.py
@@ -2,7 +2,7 @@
 import re
 import time
 
-from bfcl.model_handler.proprietary_model.openai import OpenAIHandler
+from bfcl.model_handler.api_inference.openai import OpenAIHandler
 from bfcl.model_handler.model_style import ModelStyle
 from bfcl.model_handler.utils import (
     ast_parse,

diff --git a/...del_handler/proprietary_model/deepseek.py → ...l/model_handler/api_inference/deepseek.py b/...del_handler/proprietary_model/deepseek.py → ...l/model_handler/api_inference/deepseek.py
@@ -2,7 +2,7 @@
 import time
 
 from bfcl.model_handler.model_style import ModelStyle
-from bfcl.model_handler.proprietary_model.openai import OpenAIHandler
+from bfcl.model_handler.api_inference.openai import OpenAIHandler
 from bfcl.model_handler.utils import retry_with_backoff
 from openai import OpenAI, RateLimitError
 from overrides import override

diff --git a/...el_handler/proprietary_model/fireworks.py → .../model_handler/api_inference/fireworks.py b/...el_handler/proprietary_model/fireworks.py → .../model_handler/api_inference/fireworks.py
@@ -2,7 +2,7 @@
 import time
 
 from bfcl.model_handler.model_style import ModelStyle
-from bfcl.model_handler.proprietary_model.openai import OpenAIHandler
+from bfcl.model_handler.api_inference.openai import OpenAIHandler
 from openai import OpenAI
 
 

diff --git a/..._handler/proprietary_model/functionary.py → ...odel_handler/api_inference/functionary.py b/..._handler/proprietary_model/functionary.py → ...odel_handler/api_inference/functionary.py
@@ -1,4 +1,4 @@
-from bfcl.model_handler.proprietary_model.openai import OpenAIHandler
+from bfcl.model_handler.api_inference.openai import OpenAIHandler
 from bfcl.model_handler.model_style import ModelStyle
 from openai import OpenAI
 

diff --git a/...model_handler/proprietary_model/gemini.py → ...fcl/model_handler/api_inference/gemini.py b/...model_handler/proprietary_model/gemini.py → ...fcl/model_handler/api_inference/gemini.py
diff --git a/...el_handler/proprietary_model/gogoagent.py → .../model_handler/api_inference/gogoagent.py b/...el_handler/proprietary_model/gogoagent.py → .../model_handler/api_inference/gogoagent.py
@@ -1,6 +1,6 @@
 import os
 
-from bfcl.model_handler.proprietary_model.openai import OpenAIHandler
+from bfcl.model_handler.api_inference.openai import OpenAIHandler
 from openai import OpenAI
 
 

diff --git a/...odel_handler/proprietary_model/gorilla.py → ...cl/model_handler/api_inference/gorilla.py b/...odel_handler/proprietary_model/gorilla.py → ...cl/model_handler/api_inference/gorilla.py
diff --git a/...l/model_handler/proprietary_model/grok.py → .../bfcl/model_handler/api_inference/grok.py b/...l/model_handler/proprietary_model/grok.py → .../bfcl/model_handler/api_inference/grok.py
@@ -1,6 +1,6 @@
 import os
 
-from bfcl.model_handler.proprietary_model.openai import OpenAIHandler
+from bfcl.model_handler.api_inference.openai import OpenAIHandler
 from openai import OpenAI
 
 

diff --git a/...odel_handler/proprietary_model/mistral.py → ...cl/model_handler/api_inference/mistral.py b/...odel_handler/proprietary_model/mistral.py → ...cl/model_handler/api_inference/mistral.py
diff --git a/.../model_handler/proprietary_model/nexus.py → ...bfcl/model_handler/api_inference/nexus.py b/.../model_handler/proprietary_model/nexus.py → ...bfcl/model_handler/api_inference/nexus.py
diff --git a/...l/model_handler/proprietary_model/nova.py → .../bfcl/model_handler/api_inference/nova.py b/...l/model_handler/proprietary_model/nova.py → .../bfcl/model_handler/api_inference/nova.py
diff --git a/...model_handler/proprietary_model/nvidia.py → ...fcl/model_handler/api_inference/nvidia.py b/...model_handler/proprietary_model/nvidia.py → ...fcl/model_handler/api_inference/nvidia.py
@@ -1,7 +1,7 @@
 import os
 
 from bfcl.model_handler.model_style import ModelStyle
-from bfcl.model_handler.proprietary_model.openai import OpenAIHandler
+from bfcl.model_handler.api_inference.openai import OpenAIHandler
 from bfcl.model_handler.utils import (
     ast_parse,
     combine_consecutive_user_prompts,

diff --git a/...model_handler/proprietary_model/openai.py → ...fcl/model_handler/api_inference/openai.py b/...model_handler/proprietary_model/openai.py → ...fcl/model_handler/api_inference/openai.py
diff --git a/...model_handler/proprietary_model/writer.py → ...fcl/model_handler/api_inference/writer.py b/...model_handler/proprietary_model/writer.py → ...fcl/model_handler/api_inference/writer.py
@@ -2,7 +2,7 @@
 import time
 
 from bfcl.model_handler.model_style import ModelStyle
-from bfcl.model_handler.proprietary_model.openai import OpenAIHandler
+from bfcl.model_handler.api_inference.openai import OpenAIHandler
 from writerai import Writer
 
 

diff --git a/...fcl/model_handler/proprietary_model/yi.py → ...rd/bfcl/model_handler/api_inference/yi.py b/...fcl/model_handler/proprietary_model/yi.py → ...rd/bfcl/model_handler/api_inference/yi.py
diff --git a/berkeley-function-call-leaderboard/bfcl/model_handler/handler_map.py b/berkeley-function-call-leaderboard/bfcl/model_handler/handler_map.py
@@ -1,36 +1,36 @@
-from bfcl.model_handler.oss_model.deepseek import DeepseekHandler
-from bfcl.model_handler.oss_model.deepseek_coder import DeepseekCoderHandler
-from bfcl.model_handler.oss_model.gemma import GemmaHandler
-from bfcl.model_handler.oss_model.glaive import GlaiveHandler
-from bfcl.model_handler.oss_model.glm import GLMHandler
-from bfcl.model_handler.oss_model.granite import GraniteHandler
-from bfcl.model_handler.oss_model.hammer import HammerHandler
-from bfcl.model_handler.oss_model.hermes import HermesHandler
-from bfcl.model_handler.oss_model.llama import LlamaHandler
-from bfcl.model_handler.oss_model.llama_fc import LlamaFCHandler
-from bfcl.model_handler.oss_model.minicpm import MiniCPMHandler
-from bfcl.model_handler.oss_model.minicpm_fc import MiniCPMFCHandler
-from bfcl.model_handler.oss_model.mistral_fc import MistralFCHandler
-from bfcl.model_handler.oss_model.phi import PhiHandler
-from bfcl.model_handler.oss_model.qwen import QwenHandler
-from bfcl.model_handler.oss_model.salesforce import SalesforceHandler
-from bfcl.model_handler.proprietary_model.claude import ClaudeHandler
-from bfcl.model_handler.proprietary_model.cohere import CohereHandler
-from bfcl.model_handler.proprietary_model.databricks import DatabricksHandler
-from bfcl.model_handler.proprietary_model.deepseek import DeepSeekAPIHandler
-from bfcl.model_handler.proprietary_model.fireworks import FireworksHandler
-from bfcl.model_handler.proprietary_model.functionary import FunctionaryHandler
-from bfcl.model_handler.proprietary_model.gemini import GeminiHandler
-from bfcl.model_handler.proprietary_model.gogoagent import GoGoAgentHandler
-from bfcl.model_handler.proprietary_model.gorilla import GorillaHandler
-from bfcl.model_handler.proprietary_model.grok import GrokHandler
-from bfcl.model_handler.proprietary_model.mistral import MistralHandler
-from bfcl.model_handler.proprietary_model.nexus import NexusHandler
-from bfcl.model_handler.proprietary_model.nova import NovaHandler
-from bfcl.model_handler.proprietary_model.nvidia import NvidiaHandler
-from bfcl.model_handler.proprietary_model.openai import OpenAIHandler
-from bfcl.model_handler.proprietary_model.writer import WriterHandler
-from bfcl.model_handler.proprietary_model.yi import YiHandler
+from bfcl.model_handler.api_inference.claude import ClaudeHandler
+from bfcl.model_handler.api_inference.cohere import CohereHandler
+from bfcl.model_handler.api_inference.databricks import DatabricksHandler
+from bfcl.model_handler.api_inference.deepseek import DeepSeekAPIHandler
+from bfcl.model_handler.api_inference.fireworks import FireworksHandler
+from bfcl.model_handler.api_inference.functionary import FunctionaryHandler
+from bfcl.model_handler.api_inference.gemini import GeminiHandler
+from bfcl.model_handler.api_inference.gogoagent import GoGoAgentHandler
+from bfcl.model_handler.api_inference.gorilla import GorillaHandler
+from bfcl.model_handler.api_inference.grok import GrokHandler
+from bfcl.model_handler.api_inference.mistral import MistralHandler
+from bfcl.model_handler.api_inference.nexus import NexusHandler
+from bfcl.model_handler.api_inference.nova import NovaHandler
+from bfcl.model_handler.api_inference.nvidia import NvidiaHandler
+from bfcl.model_handler.api_inference.openai import OpenAIHandler
+from bfcl.model_handler.api_inference.writer import WriterHandler
+from bfcl.model_handler.api_inference.yi import YiHandler
+from bfcl.model_handler.local_inference.deepseek import DeepseekHandler
+from bfcl.model_handler.local_inference.deepseek_coder import DeepseekCoderHandler
+from bfcl.model_handler.local_inference.gemma import GemmaHandler
+from bfcl.model_handler.local_inference.glaive import GlaiveHandler
+from bfcl.model_handler.local_inference.glm import GLMHandler
+from bfcl.model_handler.local_inference.granite import GraniteHandler
+from bfcl.model_handler.local_inference.hammer import HammerHandler
+from bfcl.model_handler.local_inference.hermes import HermesHandler
+from bfcl.model_handler.local_inference.llama import LlamaHandler
+from bfcl.model_handler.local_inference.llama_fc import LlamaFCHandler
+from bfcl.model_handler.local_inference.minicpm import MiniCPMHandler
+from bfcl.model_handler.local_inference.minicpm_fc import MiniCPMFCHandler
+from bfcl.model_handler.local_inference.mistral_fc import MistralFCHandler
+from bfcl.model_handler.local_inference.phi import PhiHandler
+from bfcl.model_handler.local_inference.qwen import QwenHandler
+from bfcl.model_handler.local_inference.salesforce import SalesforceHandler
 
 # TODO: Add meta-llama/Llama-3.1-405B-Instruct
 

diff --git a/...del_handler/proprietary_model/__init__.py → ...model_handler/local_inference/__init__.py b/...del_handler/proprietary_model/__init__.py → ...model_handler/local_inference/__init__.py
diff --git a/...del_handler/oss_model/base_oss_handler.py → ...ndler/local_inference/base_oss_handler.py b/...del_handler/oss_model/base_oss_handler.py → ...ndler/local_inference/base_oss_handler.py
@@ -7,7 +7,7 @@
 from bfcl.constant import RESULT_PATH, VERSION_PREFIX
 from bfcl.model_handler.base_handler import BaseHandler
 from bfcl.model_handler.model_style import ModelStyle
-from bfcl.model_handler.oss_model.constant import VLLM_PORT
+from bfcl.model_handler.local_inference.constant import VLLM_PORT
 from bfcl.model_handler.utils import (
     default_decode_ast_prompting,
     default_decode_execute_prompting,

diff --git a/.../bfcl/model_handler/oss_model/constant.py → ...model_handler/local_inference/constant.py b/.../bfcl/model_handler/oss_model/constant.py → ...model_handler/local_inference/constant.py
diff --git a/.../bfcl/model_handler/oss_model/deepseek.py → ...model_handler/local_inference/deepseek.py b/.../bfcl/model_handler/oss_model/deepseek.py → ...model_handler/local_inference/deepseek.py
@@ -1,4 +1,4 @@
-from bfcl.model_handler.oss_model.base_oss_handler import OSSHandler
+from bfcl.model_handler.local_inference.base_oss_handler import OSSHandler
 from overrides import override
 
 

diff --git a/...model_handler/oss_model/deepseek_coder.py → ...handler/local_inference/deepseek_coder.py b/...model_handler/oss_model/deepseek_coder.py → ...handler/local_inference/deepseek_coder.py
@@ -1,7 +1,7 @@
 import json
 import re
 
-from bfcl.model_handler.oss_model.base_oss_handler import OSSHandler
+from bfcl.model_handler.local_inference.base_oss_handler import OSSHandler
 from bfcl.model_handler.utils import (
     combine_consecutive_user_prompts,
     convert_system_prompt_into_user_prompt,

diff --git a/...ard/bfcl/model_handler/oss_model/gemma.py → ...cl/model_handler/local_inference/gemma.py b/...ard/bfcl/model_handler/oss_model/gemma.py → ...cl/model_handler/local_inference/gemma.py
@@ -1,4 +1,4 @@
-from bfcl.model_handler.oss_model.base_oss_handler import OSSHandler
+from bfcl.model_handler.local_inference.base_oss_handler import OSSHandler
 from bfcl.model_handler.utils import (
     combine_consecutive_user_prompts,
     convert_system_prompt_into_user_prompt,

diff --git a/...rd/bfcl/model_handler/oss_model/glaive.py → ...l/model_handler/local_inference/glaive.py b/...rd/bfcl/model_handler/oss_model/glaive.py → ...l/model_handler/local_inference/glaive.py
@@ -1,6 +1,6 @@
 import json
 
-from bfcl.model_handler.oss_model.base_oss_handler import OSSHandler
+from bfcl.model_handler.local_inference.base_oss_handler import OSSHandler
 from bfcl.model_handler.utils import convert_to_function_call
 from overrides import override
 

diff --git a/...board/bfcl/model_handler/oss_model/glm.py → ...bfcl/model_handler/local_inference/glm.py b/...board/bfcl/model_handler/oss_model/glm.py → ...bfcl/model_handler/local_inference/glm.py
@@ -2,7 +2,7 @@
 
 from bfcl.model_handler.constant import GORILLA_TO_OPENAPI
 from bfcl.model_handler.model_style import ModelStyle
-from bfcl.model_handler.oss_model.base_oss_handler import OSSHandler
+from bfcl.model_handler.local_inference.base_oss_handler import OSSHandler
 from bfcl.model_handler.utils import (
     convert_to_function_call,
     convert_to_tool,

diff --git a/...d/bfcl/model_handler/oss_model/granite.py → .../model_handler/local_inference/granite.py b/...d/bfcl/model_handler/oss_model/granite.py → .../model_handler/local_inference/granite.py
@@ -2,7 +2,7 @@
 
 from bfcl.model_handler.constant import GORILLA_TO_OPENAPI
 from bfcl.model_handler.model_style import ModelStyle
-from bfcl.model_handler.oss_model.base_oss_handler import OSSHandler
+from bfcl.model_handler.local_inference.base_oss_handler import OSSHandler
 from bfcl.model_handler.utils import (
     convert_to_tool,
     func_doc_language_specific_pre_processing,

diff --git a/...rd/bfcl/model_handler/oss_model/hammer.py → ...l/model_handler/local_inference/hammer.py b/...rd/bfcl/model_handler/oss_model/hammer.py → ...l/model_handler/local_inference/hammer.py
@@ -1,6 +1,6 @@
 import json
 
-from bfcl.model_handler.oss_model.base_oss_handler import OSSHandler
+from bfcl.model_handler.local_inference.base_oss_handler import OSSHandler
 from bfcl.model_handler.utils import (
     convert_system_prompt_into_user_prompt,
     func_doc_language_specific_pre_processing,

diff --git a/...rd/bfcl/model_handler/oss_model/hermes.py → ...l/model_handler/local_inference/hermes.py b/...rd/bfcl/model_handler/oss_model/hermes.py → ...l/model_handler/local_inference/hermes.py
@@ -3,7 +3,7 @@
 
 from bfcl.model_handler.constant import GORILLA_TO_OPENAPI
 from bfcl.model_handler.model_style import ModelStyle
-from bfcl.model_handler.oss_model.base_oss_handler import OSSHandler
+from bfcl.model_handler.local_inference.base_oss_handler import OSSHandler
 from bfcl.model_handler.utils import (
     convert_to_tool,
     func_doc_language_specific_pre_processing,

diff --git a/...ard/bfcl/model_handler/oss_model/llama.py → ...cl/model_handler/local_inference/llama.py b/...ard/bfcl/model_handler/oss_model/llama.py → ...cl/model_handler/local_inference/llama.py
@@ -1,4 +1,4 @@
-from bfcl.model_handler.oss_model.base_oss_handler import OSSHandler
+from bfcl.model_handler.local_inference.base_oss_handler import OSSHandler
 from overrides import override
 
 

diff --git a/.../bfcl/model_handler/oss_model/llama_fc.py → ...model_handler/local_inference/llama_fc.py b/.../bfcl/model_handler/oss_model/llama_fc.py → ...model_handler/local_inference/llama_fc.py
@@ -1,6 +1,6 @@
 import json
 
-from bfcl.model_handler.oss_model.base_oss_handler import OSSHandler
+from bfcl.model_handler.local_inference.base_oss_handler import OSSHandler
 from bfcl.model_handler.utils import func_doc_language_specific_pre_processing
 from overrides import override
 

diff --git a/...d/bfcl/model_handler/oss_model/minicpm.py → .../model_handler/local_inference/minicpm.py b/...d/bfcl/model_handler/oss_model/minicpm.py → .../model_handler/local_inference/minicpm.py
@@ -1,4 +1,4 @@
-from bfcl.model_handler.oss_model.base_oss_handler import OSSHandler
+from bfcl.model_handler.local_inference.base_oss_handler import OSSHandler
 from overrides import override
 
 

diff --git a/...fcl/model_handler/oss_model/minicpm_fc.py → ...del_handler/local_inference/minicpm_fc.py b/...fcl/model_handler/oss_model/minicpm_fc.py → ...del_handler/local_inference/minicpm_fc.py
@@ -4,7 +4,7 @@
 
 import datamodel_code_generator
 from bfcl.model_handler.constant import GORILLA_TO_OPENAPI
-from bfcl.model_handler.oss_model.base_oss_handler import OSSHandler
+from bfcl.model_handler.local_inference.base_oss_handler import OSSHandler
 from bfcl.model_handler.utils import (
     convert_to_tool,
     func_doc_language_specific_pre_processing,

diff --git a/...fcl/model_handler/oss_model/mistral_fc.py → ...del_handler/local_inference/mistral_fc.py b/...fcl/model_handler/oss_model/mistral_fc.py → ...del_handler/local_inference/mistral_fc.py
@@ -2,7 +2,7 @@
 import random
 import string
 
-from bfcl.model_handler.oss_model.base_oss_handler import OSSHandler
+from bfcl.model_handler.local_inference.base_oss_handler import OSSHandler
 from bfcl.model_handler.utils import (
     convert_to_function_call,
     func_doc_language_specific_pre_processing,

diff --git a/...board/bfcl/model_handler/oss_model/phi.py → ...bfcl/model_handler/local_inference/phi.py b/...board/bfcl/model_handler/oss_model/phi.py → ...bfcl/model_handler/local_inference/phi.py
@@ -1,4 +1,4 @@
-from bfcl.model_handler.oss_model.base_oss_handler import OSSHandler
+from bfcl.model_handler.local_inference.base_oss_handler import OSSHandler
 from bfcl.model_handler.utils import (
     combine_consecutive_user_prompts,
     convert_system_prompt_into_user_prompt,

diff --git a/...oard/bfcl/model_handler/oss_model/qwen.py → ...fcl/model_handler/local_inference/qwen.py b/...oard/bfcl/model_handler/oss_model/qwen.py → ...fcl/model_handler/local_inference/qwen.py
@@ -1,4 +1,4 @@
-from bfcl.model_handler.oss_model.base_oss_handler import OSSHandler
+from bfcl.model_handler.local_inference.base_oss_handler import OSSHandler
 from overrides import override
 
 

diff --git a/...fcl/model_handler/oss_model/salesforce.py → ...del_handler/local_inference/salesforce.py b/...fcl/model_handler/oss_model/salesforce.py → ...del_handler/local_inference/salesforce.py
@@ -2,8 +2,8 @@
 import time
 
 from bfcl.model_handler.model_style import ModelStyle
-from bfcl.model_handler.oss_model.base_oss_handler import OSSHandler
-from bfcl.model_handler.oss_model.constant import VLLM_PORT
+from bfcl.model_handler.local_inference.base_oss_handler import OSSHandler
+from bfcl.model_handler.local_inference.constant import VLLM_PORT
 from openai import OpenAI
 from overrides import override