How to use LLM deployed locally？

[deeper-coder]

Your current code uses OpenAI’s API key to access the LLM service by default. I’d like to switch it to use a local LLM, which I have deployed through LLaMA-Factory and is accessible via a local API, for example, at http://localhost:7788/v1/. Could you guide me on how to make this adjustment? Thank you!

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[kyegomez]

@deeper-coder we need function calling, if you can get a function calling model to work reliably then it will work. But you need a class with a run(task: str) or __call__(task: str) method to integrate into the ToTAgent class

[deeper-coder]

I plan to use llama3 70B and I noticed that in the OpenAIFunctionCaller class, you’ve implemented the run method as shown in the image. So, can I achieve my desired functionality by passing base_url = "http://localhost:7788/v1/" in **kwargs?

@deeper-coder we need function calling, if you can get a function calling model to work reliably then it will work. But you need a class with a run(task: str) or __call__(task: str) method to integrate into the ToTAgent class