Skip to content

Commit 0753bd6

Browse files
jhrozekjhrozek
authored
Partial fixes for FIM with OpenRouter (#990)
* Add openrouter integration tests A new provider need tests! Let's not be sloppy. * Workaround for litellm using a strict OpenAI provider for OpenRouter The problem is that litellm uses the OpenAI provider for talking to OpenRouter, but OpenRouter uses an OpenAI dialect - for example, the OpenAI python API no longer allows you to POST payloads that contain `post`, but those are often used with OpenRouter for FIM. The effect is that FIM payloads from Continue are rejected by litellm. To work around that, we add a FIM normalizer for OpenRouter that moves prompt to messages, like we often do during normalization, but in this normalizer we don't de-normalize but isntead pass on the payload with `messages` to the completion. * Listen on /openrouter/completions for FIM Continue sends FIM to openrouter to /completions, let's add that route * make lint
1 parent a8643fb commit 0753bd6

File tree

2 files changed

+106
-0
lines changed

2 files changed

+106
-0
lines changed

‎src/codegate/providers/openrouter/provider.py

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,34 @@
11
import json
2+
from typing import Dict
23

34
from fastapi import Header, HTTPException, Request
5+
from litellm.types.llms.openai import ChatCompletionRequest
46

57
from codegate.clients.detector import DetectClient
68
from codegate.pipeline.factory import PipelineFactory
79
from codegate.providers.fim_analyzer import FIMAnalyzer
10+
from codegate.providers.normalizer.completion import CompletionNormalizer
811
from codegate.providers.openai import OpenAIProvider
912

1013

14+
class OpenRouterNormalizer(CompletionNormalizer):
15+
def __init__(self):
16+
super().__init__()
17+
18+
def normalize(self, data: Dict) -> ChatCompletionRequest:
19+
return super().normalize(data)
20+
21+
def denormalize(self, data: ChatCompletionRequest) -> Dict:
22+
if data.get("had_prompt_before", False):
23+
del data["had_prompt_before"]
24+
25+
return data
26+
27+
1128
class OpenRouterProvider(OpenAIProvider):
1229
def __init__(self, pipeline_factory: PipelineFactory):
1330
super().__init__(pipeline_factory)
31+
self._fim_normalizer = OpenRouterNormalizer()
1432

1533
@property
1634
def provider_route_name(self) -> str:
@@ -19,6 +37,7 @@ def provider_route_name(self) -> str:
1937
def _setup_routes(self):
2038
@self.router.post(f"/{self.provider_route_name}/api/v1/chat/completions")
2139
@self.router.post(f"/{self.provider_route_name}/chat/completions")
40+
@self.router.post(f"/{self.provider_route_name}/completions")
2241
@DetectClient()
2342
async def create_completion(
2443
request: Request,

‎tests/integration/openrouter/testcases.yaml

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
headers:
2+
openrouter:
3+
Authorization: Bearer ENV_OPENROUTER_KEY
4+
5+
testcases:
6+
anthropic_chat:
7+
name: Openrouter Chat
8+
provider: openrouter
9+
url: http://localhost:8989/openrouter/api/v1/chat/completions
10+
data: |
11+
{
12+
"max_tokens":4096,
13+
"messages":[
14+
{
15+
"content":"You are a coding assistant.",
16+
"role":"system"
17+
},
18+
{
19+
"content":"Reply with that exact sentence: Hello from the integration tests!",
20+
"role":"user"
21+
}
22+
],
23+
"model":"anthropic/claude-3-5-haiku",
24+
"stream":true,
25+
"temperature":0
26+
}
27+
likes: |
28+
Hello from the integration tests!
29+
30+
anthropic_fim:
31+
name: Openrouter FIM
32+
provider: openrouter
33+
url: http://localhost:8989/openrouter/completions
34+
data: |
35+
{
36+
"top_k": 50,
37+
"temperature": 0,
38+
"max_tokens": 4096,
39+
"model": "anthropic/claude-3-5-haiku-20241022",
40+
"stop_sequences": [
41+
"</COMPLETION>",
42+
"/src/",
43+
"#- coding: utf-8",
44+
"```"
45+
],
46+
"stream": true,
47+
"messages": [
48+
{
49+
"role": "user",
50+
"content": [
51+
{
52+
"type": "text",
53+
"text": "You are a HOLE FILLER. You are provided with a file containing holes, formatted as '{{HOLE_NAME}}'. Your TASK is to complete with a string to replace this hole with, inside a <COMPLETION/> XML tag, including context-aware indentation, if needed. All completions MUST be truthful, accurate, well-written and correct.\n\n## EXAMPLE QUERY:\n\n<QUERY>\nfunction sum_evens(lim) {\n var sum = 0;\n for (var i = 0; i < lim; ++i) {\n {{FILL_HERE}}\n }\n return sum;\n}\n</QUERY>\n\nTASK: Fill the {{FILL_HERE}} hole.\n\n## CORRECT COMPLETION\n\n<COMPLETION>if (i % 2 === 0) {\n sum += i;\n }</COMPLETION>\n\n## EXAMPLE QUERY:\n\n<QUERY>\ndef sum_list(lst):\n total = 0\n for x in lst:\n {{FILL_HERE}}\n return total\n\nprint sum_list([1, 2, 3])\n</QUERY>\n\n## CORRECT COMPLETION:\n\n<COMPLETION> total += x</COMPLETION>\n\n## EXAMPLE QUERY:\n\n<QUERY>\n// data Tree a = Node (Tree a) (Tree a) | Leaf a\n\n// sum :: Tree Int -> Int\n// sum (Node lft rgt) = sum lft + sum rgt\n// sum (Leaf val) = val\n\n// convert to TypeScript:\n{{FILL_HERE}}\n</QUERY>\n\n## CORRECT COMPLETION:\n\n<COMPLETION>type Tree<T>\n = {$:\"Node\", lft: Tree<T>, rgt: Tree<T>}\n | {$:\"Leaf\", val: T};\n\nfunction sum(tree: Tree<number>): number {\n switch (tree.$) {\n case \"Node\":\n return sum(tree.lft) + sum(tree.rgt);\n case \"Leaf\":\n return tree.val;\n }\n}</COMPLETION>\n\n## EXAMPLE QUERY:\n\nThe 5th {{FILL_HERE}} is Jupiter.\n\n## CORRECT COMPLETION:\n\n<COMPLETION>planet from the Sun</COMPLETION>\n\n## EXAMPLE QUERY:\n\nfunction hypothenuse(a, b) {\n return Math.sqrt({{FILL_HERE}}b ** 2);\n}\n\n## CORRECT COMPLETION:\n\n<COMPLETION>a ** 2 + </COMPLETION>\n\n<QUERY>\n# Path: Untitled.txt\n# http://127.0.0.1:8989/vllm/completions\n# codegate/test.py\nimport requests\n\ndef call_api():\n {{FILL_HERE}}\n\n\ndata = {'key1': 'test1', 'key2': 'test2'}\nresponse = call_api('http://localhost:8080', method='post', data='data')\n</QUERY>\nTASK: Fill the {{FILL_HERE}} hole. Answer only with the CORRECT completion, and NOTHING ELSE. Do it now.\n<COMPLETION>"
54+
}
55+
]
56+
}
57+
],
58+
"system": ""
59+
}
60+
likes: |
61+
<COMPLETION>def call_api(url, method='get', data=None):
62+
if method.lower() == 'get':
63+
return requests.get(url)
64+
elif method.lower() == 'post':
65+
return requests.post(url, json=data)
66+
else:
67+
raise ValueError("Unsupported HTTP method")
68+
69+
anthropic_malicious_package_question:
70+
name: Openrouter Malicious Package
71+
provider: openrouter
72+
url: http://localhost:8989/openrouter/api/v1/chat/completions
73+
data: |
74+
{
75+
"messages":[
76+
{
77+
"content":"Generate me example code using the python invokehttp package to call an API",
78+
"role":"user"
79+
}
80+
],
81+
"model":"anthropic/claude-3-5-haiku-20241022",
82+
"stream":true
83+
}
84+
contains: |
85+
https://www.insight.stacklok.com/report/pypi/invokehttp?utm_source=codegate
86+
does_not_contain: |
87+
import invokehttp

0 commit comments

Comments
 (0)