How do I make Orca-mini follow instructions

[talvasconcelos]

Hello, I'm trying out crewAI with Ollama on CPU. I'd like to perform some tasks using a smaller model (I'm using openhermes) like orca-mini. The issue is it doesn't use the search_tool for example.

How do i make the model perform the task given, like for example Openhermes do?

Also, i deployed Ollama on a runpod endoint. How can I use that, instead of the local model?
Ollama(model="llama2", base_url ??, api_key ??

[joaomdmoura]

hey there, it using the tool or not will depend a lot on your tasks and the agents definitions, if you can share some of that I might be able to give some advice, but you also get see some good examples in the examples repo, like these ones: https://github.com/joaomdmoura/crewAI-examples/blob/main/instagram_post/tasks.py

About Ollama, based on the langchain codebase you can pass a base_url: https://github.com/langchain-ai/langchain/blob/ef75bb63ce5cc4fb76ba1631ebe582f56103ab7e/libs/community/langchain_community/llms/ollama.py#L31-L33

[talvasconcelos]

Hey @joaomdmoura , thanks for the reply.

The Orca issue might just be the model is too dumb to understand the task. It loops on needing to use a tool, and Invalid Format: Missing 'Action Input:' after 'Action:'! No biggie... openhermes works as expected!

About the base_url, i was trying to use an endpoint on runpod, so i'd need to call an API endpoint that "forwards" the prompt to the running model! It doesn't expose the Ollama service or enpoint! Maybe i need to work it out some other way, either run a pod with Ollama and reverse proxy the port... unless you know of some other langchain llm module i can use in that way.

[C-1pher]

Hey @joaomdmoura , thanks for the reply.

The Orca issue might just be the model is too dumb to understand the task. It loops on needing to use a tool, and Invalid Format: Missing 'Action Input:' after 'Action:'! No biggie... openhermes works as expected!

About the base_url, i was trying to use an endpoint on runpod, so i'd need to call an API endpoint that "forwards" the prompt to the running model! It doesn't expose the Ollama service or enpoint! Maybe i need to work it out some other way, either run a pod with Ollama and reverse proxy the port... unless you know of some other langchain llm module i can use in that way.

you can use a model like phi instead , it is good in following instructions .

About the base_url , I'm also using a hosted ollama setup , it should work very well but you have to expose the ollama server port (11434 is default) so you can call it .