[Bug]: Failing to run OpenRouter AND Ollama

[kevin-support-bot]

All-Hands-AI#5310 Issue

---

@BradKML Is there any error mentioned in the logs for the second and third run?

[BradKML]

How do I check for the error in the logs? Like a log dump?

[SmartManoj]

It will show in the terminal itself.

[SmartManoj]

Without Function calling
openrouter/qwen/qwen-2.5-coder-32b-instruct
openrouter/google/gemma-2-27b-it:free
openrouter/meta-llama/llama-3.2-11b-vision-instruct
openrouter/microsoft/phi-3-medium-128k-instruct

[BradKML]

This kept cropping up for Ollama Function 'write_file' not found in available tools: ['execute_bash', 'finish', 'browser', 'execute_ipython_cell', 'str_replace_editor'] with URL set as http://host.docker.internal:11434

For OpenRouter it is even more annoying litellm.BadRequestError: OpenrouterException - when you don't set the URL in the Advanced settings, or when setting https://openrouter.ai/api/v1

[SmartManoj]

In All-Hands-AI#4711, the function_calling agent config was removed. Could you check without function calling using the previous commit or this branch?

---

For OpenRouter, the error may be something different. Without setting the URL, it works. You can check using this script https://github.com/SmartManoj/Kevin/blob/kevin/scripts/llm_checker.py

[BradKML]

@SmartManoj check the messages in the other issues pages, since I am currently using 0.15 not 0.14, so it should be the latest image with Docker. At least that is the expectation when using Ollama and they bug out over some tools being not present.
I need to set the URL such that I have the freedom to choose models (instead of using the defaults, which for testing it is Qwen2.5). I would like to just have the URL as documentation, please.
Tooling works for OpenRouter but string replacement is definitely weak. Also agent state changes and "loops" are also kind of an issue.

[SmartManoj]

Model: openrouter/qwen/qwen-2.5-coder-32b-instruct
Without Function Calling
With Function Calling [Debugging] fixed

[BradKML]

@SmartManoj please try a more complex prompt since the simpler prompt may not be as prone to such issues. use the error logs as reference to design long-form tests to your liking.

[SmartManoj]

44 steps then got a weird API Key error.

[SmartManoj]

Using gemini-exp-1206.

---

try a more complex prompt

how about breaking into smaller tasks first? If all works, then the best approach is to automatically convert a large task into smaller tasks.

[BradKML]

@SmartManoj because the model can sometimes slip when it comes to things file management, which need their own set of prompts to manage... maybe I should shrink the prompt a little but the core procedural prompt has already been broken down into smaller steps, no?
sample.zip

[SmartManoj]

Please create scripts for the following:

First, gather the S&P500 daily OHLC data for the last 10 years using yfinance and cache it.
Then create 5 backtesters using backtesting.py and utilizing pandas-ta
Each backtester should use a different strategy, and browse the internet if necessary
Finally, use tune features from the ray package to do hyperparameter optimization of each of the backtesters
Prioritize on maximizing both Sortino Ratio and Sharpe Ratio as the goal of optimization
Optimization of backtester is done through "sliding window" to prevent overfitting

This can be divided into 5 smaller tasks.

[BradKML]

How do you prompt them such that OpenHands can handle subtasks by itself as a list of things to do?

[SmartManoj]

Currently, it doesn't work like that.

[SmartManoj]

For OpenRouter, the error may be something different.

BerriAI/litellm#7072 will fix that.

[SmartManoj]

How do you prompt them such that OpenHands can handle subtasks by itself as a list of things to do?

How about using CLI mode with task argument?

[SmartManoj]

showme, an O1 alternative.

[BradKML]

@SmartManoj that looks like something that QwQ and R1 need supplementation from ngl, also if it is not FOSS it is harder for people to use at home I think. Vibes like an extension of this discussion #139

P.S. Will do another round of testing in this case to see how the "stuck in the loop" problem happen in the main OpenHands, see if you can recognize the issues

[SmartManoj]

How do you prompt them such that OpenHands can handle subtasks by itself as a list of things to do?

Would you test with PlannerAgent?

Demo

---

After delegate task to CodeActAgent 8d3431f
Demo

[BradKML]

@SmartManoj thanks for the pointers but I can't believe Gemini can be stumped by the versioning vs decimal context issue.

[SmartManoj]

https://x.com/garrethleee/status/1853870656506798454

[BradKML]

We all know tokenization differences for numbers lead to different results (and the "strawberry test" too) but there has to be a remedy through alternative tokenization methods besides triple digits or BPEs. Think "probabilistic tokenization" or other consistency enhancers.

[SmartManoj]

Could you apply that to https://github.com/karpathy/nanoGPT and check how it performs?

[BradKML]

@SmartManoj probably belongs to another issue altogether ngl

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This issue was closed because it has been stalled for over 30 days with no activity.