feat: Configure fallback LLMs for rate limit handling

[malhotra5]

This PR adds support for configuring fallback LLMs that are automatically used when rate limits are hit.

Changes

• Add fallback_llms field to LLMConfig to support a list of fallback LLM configurations
• Implement automatic switching to fallback LLMs when rate limits are hit
• Add automatic reset to primary LLM when rate limit expires
• Add unit tests to verify the functionality

Usage Example

[llm]
model = "claude-3-5-sonnet-20241022"
api_key = "..."

[[llm.fallback_llms]]
model = "gpt-4"
api_key = "..."

[[llm.fallback_llms]]
model = "llama2-70b"
api_key = "..."

The system will automatically switch between these LLMs when rate limits are hit, and return to the primary LLM when the rate limit expires.

Fixes #1263

---

To run this PR locally, use the following command:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:da0d740-nikolaik   --name openhands-app-da0d740   docker.all-hands.dev/all-hands-ai/openhands:da0d740

[enyst]

Just a small thing here, there already is a try / except, it seems we're adding another? Or does github interface fool me 😅

[malhotra5]

Haha yeah there is already a try/except; I was experimenting with OH and wanted to push it beyond its comfort zone to see if it implement weird things 😆

[enyst]

On what LLMs does this work? It really seems like something that litellm would support, it does a lot of string matching on a 100+ LLMs. For example, it does this when it matches various exception strings to a single unified exception that it returns to us (like ContextWindowExceeded across many providers)

Last I checked, they do have fallback LLMs. It doesn't (or didn't? maybe they do now?) do this bit - it doesn't read the response (which may also be in part outside the exception, in the usage field or something, depends on LLM provider)

[malhotra5]

Ah yes they do!

[enyst]

Related issues:

• do fallbacks via litellm Use litellm Router for rate limiting and/or fallback LLMs #4056
• switch automatically [LLM] Support LLM routing through notdiamond #4184
• or do it ourselves (this PR)

[malhotra5]

Closing this PR as implementation is subpar