CodeGate doesn't work with either LiteLLM or OpenRouter

[tan-yong-sheng]

I am trying to use codegate with cline, and connect LLM api via LiteLLM proxy (hosted on cloud)

Environment:

• Windows 11

Here are the steps I did:

1. On laptop, I run a docker container for codegate:

docker run --name codegate -d -p 8989:8989 -p 9090:9090 -e CODEGATE_OPENAI_URL=https://<MY_LITELLM_URL>/v1 --restart unless-stopped ghcr.io/stacklok/codegate

2. Then, I tried to set up the credentials for cline in VS code, as follows:

I thought as I set the environment variable to CODEGATE_OPENAI_URL=https://litellm.tanyongsheng.site/v1, it will change the default environment variable for openai url

However, I still get this error:

And when I tried to switch to gemini 2.0 that I set up in litellm proxy. There is still error coming up:

Hope for help, thanks

[tan-yong-sheng]

I think how it works aldy

As my litellm proxy uses os.environ/GEMINI_API_KEY for the config.yaml file, I need to pass in GEMINI's api key instead of LiteLLM master key: https://docs.litellm.ai/docs/providers/gemini

Extract of sample config.yaml file for my self-hosted litellm proxy

model_list:
  - model_name: gemini-pro
    litellm_params:
      model: gemini/gemini-1.5-pro
      api_key: os.environ/GEMINI_API_KEY 

Here is the amendment I made for CodeGate to work:

Should be working now, and I will close this issue.

[tan-yong-sheng]

But, it seems like leading to another error, which it doesn't trigger when there are secrets inside the file:

Here is the same credentials in my config.ini file

And when I check codegate via docker logs, and I found a lot of errors there:

And not sure if it's related to the errors above, I found there are some 304 errors for my litellm proxy's UI? But, I think I didn't access my litellm proxy's UI that time.... (Not that sure)

I am thinking it's better for me to try to configure it via openai API first, and then transfer to litellm API ...

[jhrozek]

@tan-yong-sheng thanks for filing the issue!

The rate limits were a codegate bug that was fixed in main but not released yet. We'll get a release out shortly.

I've not used the litellm proxy myself, but I assume it works somewhat like openrouter? Sounds like you expected the litellm master key to work? If yes, it might be worth to reopen this issue and investigate.

[tan-yong-sheng]

I see, noted on the info and thanks.

Yes, litellm proxy is kinda similar to openrouter which is to unify the llm endpoints from different providers.

Ya, I previously expect LITELLM_MASTER_KEY to work, but in the end, I found I need to pass the GEMINI_API_KEY here I want to Gemini llm. Also, I could pass in MISTRAL_API_KEY if I want to Mistral llm. It means LITELLM_MASTER_KEY not being used to fill in the image below, but it still works (note: perhaps this is how litellm proxy works and I would add on more info about this when I am available)

As shown in my previous images:

[jhrozek]

ok, reopening for further triage. We need to make sure codegate works well without the workarounds.

Thanks again for finding the bug and filing the issue.

[tan-yong-sheng]

Hi @jhrozek, could I request any docker container (e.g., ghcr.io/stacklok/codegate:test) for testing? I am not that sure if I am doing right to build the docker image for codegate's main repository:

I tried to git clone https://github.com/stacklok/codegate.git and then follow your guide to build docker container on my linux cloud, as shown below:

Yet, when I use cline (even with openai endpoint: https://api.openai.com/v1), it still couldn't trigger and detect credentials. The prompt I used to reproduce the error is read config.ini file

However, for docker logs, no error logs for codegate's docker container as before:

[jhrozek]

Hi @jhrozek, could I request any docker container (e.g., ghcr.io/stacklok/codegate:test) for testing? I am not that sure if I am doing right to build the docker image for codegate's main repository:

I don't think you need to build a local container (unless you want to!) we release several times a month and the current release is just a couple of days old.

I tried to git clone https://github.com/stacklok/codegate.git and then follow your guide to build docker container on my linux cloud, as shown below:

Yet, when I use cline (even with openai endpoint: https://api.openai.com/v1), it still couldn't trigger and detect credentials. The prompt I used to reproduce the error is read config.ini file

I think this is a configuration issue. In order for cline to talk to codegate, you should point it to codegate as the base URL, not to the LLM

However, for docker logs, no error logs for codegate's docker container as before:

I think this would be solved by pointing the extension to codegate.

btw does it still stand that the main issue is not being able to use litellm through the main key?

[jhrozek]

Just a quick note, I think I was able to reproduce your issue (and I think it was the same issue that @danbarr pinged me about some days ago).

Will keep digging to fix this!

[tan-yong-sheng]

I see, thanks a lot. I will test again when the bug (= codegate is not triggered when I ask LLM to read config.ini file to safeguard my credentials) is fixed.

Thanks a lot for your efforts in delivering this product. Tbh, I am impressed with the idea of this project to safeguard credentials and filter risky dependencies when doing AI coding.

[tan-yong-sheng]

btw does it still stand that the main issue is not being able to use litellm through the main key?

Ya, initially I thought we can't use LiteLLM proxy with Cline and Codegate, and it keeps saying authentication error. That's the reason I create this issue.

But, for now, I think it's not that much matter, as I found the workaround to solve this issue. We could use the GEMINI_API_KEY or CODESTRAL_API_KEY for respective LLM models, instead of LITELLM_MASTER_KEY. And it works!

---

Demonstration

Just for your info, here is what I did to use Cline (together with LiteLLM proxy) with and without codegate:

(1) WITHOUT codegate and with LiteLLM proxy

(2) WITH codegate and with LiteLLM proxy

Reply to issue

I hope this explanation is clear. I believe we can maintain the existing configuration, since we have a workaround to enable Cline and Codegate to work together with LiteLLM. Therefore, I suggest closing this issue.

Additional info

(i) the config.yaml for LiteLLM proxy

You could ignore this, but it's for clarify for my setup to self-host LiteLLM proxy on my Linux cloud server. Here is what I setup for config.yaml file for my litellm proxy to access Gemini and Codestral models:

model_list:
  - model_name: gemini/*
    litellm_params:
      # to include all gemini models available
      model: gemini/*
      api_key: <GEMINI_API_KEY_IN_TEXT_FORM>
  
  - model_name: codestral/*
    litellm_params:
      # to include all codestral models available
      model: codestral/*
      api_key: <CODESTRAL_API_KEY_IN_TEXT_FORM>

Reference: https://docs.litellm.ai/docs/proxy/configs

(ii) How to run codegate via docker container

Note: I change CODEGATE_OPENAI_URL from https://api.openai.com/v1 to https://<MY_LITELLM_URL>/v1

docker run --name codegate -d -p 8989:8989 -p 9090:9090 -e CODEGATE_OPENAI_URL=https://<MY_LITELLM_URL>/v1 --mount type=volume,src=codegate_volume,dst=/app/codegate_volume --restart unless-stopped ghcr.io/stacklok/codegate

[lukehinds]

Hey @tan-yong-sheng , great having you here and thanks for the time you took to explain the issues well ❤

Looping back to the secrets sensing, we would need to pattern match those tokens, you will find the matching list here and we currently match the crypto itself, not the variable name.

GITHUB_TOKEN <- we ignore this
ghp_209isokl2kl2k0i0ia... -< we match this

However are just about to rework this system to match both #209 and also look and use of entropy for matching.

Would love to have you try this out when we have something in the way of a prototype?

[tan-yong-sheng]

Looping back to the secrets sensing, we would need to pattern match those tokens, you will find the matching list here and we currently match the crypto itself, not the variable name.

Ah I see, no wonder.

However are just about to rework this system to match both #209 and also look and use of entropy for matching.

Interesting ideas and discussions

Would love to have you try this out when we have something in the way of a prototype?

Yaya, should be no problem. My pleasure.

Thanks as well.

[jhrozek]

@tan-yong-sheng I opened a PR that fixes a very similar issue for OpenRouter. I think fixing your issue for litellm would be along the same lines. Can you suggest an easy way to test litellm proxy to simulate your environment? I found https://docs.litellm.ai/docs/proxy/deploy so I can try that - I'm familiar with using litellm as a library, but not as a proxy

[tan-yong-sheng]

Here is the the way to start an localhost instance for litellm. But I am not that sure if this is working as I never tried localhost version one (note: not available at the moment to test this before writing)..

Reference: https://thinhdanggroup.github.io/litellm-proxy/#setting-up-litellm-proxy-locally

1. Install pypi package

pip install litellm[proxy]

2. Then, on your directory, create config.yaml file

(Note: I get Gemini API key for free in Google ai studio)

model_list:
  - model_name: gemini/*
    litellm_params:
      # to include all gemini models available
      model: gemini/*
      api_key: <GEMINI_API_KEY_IN_TEXT_FORM>
      drop_params: true
  
  - model_name: codestral/*
    litellm_params:
      # to include all codestral models available
      model: codestral/*
      api_key: <CODESTRAL_API_KEY_IN_TEXT_FORM>
      drop_params: true

general_settings: 
  master_key: sk-1234

(Note 1: You could refer https://docs.litellm.ai/docs/proxy/configs for the format of config file, if you need to add more configuration)
(Note 2: your LITELLM_MASTER_KEY is sk-1234 set by config above)

3. start litellm instances

litellm --config=config.yaml

4. At last, you could use either openai or litellm module to call the llm API

https://docs.litellm.ai/docs/proxy/user_keys

For example,

import openai
client = openai.OpenAI(
    api_key="anything", # can try to pass in LITELLM_MASTER_KEY first
    base_url="http://localhost:4000"
)

# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(
    model="gemini/gemini-2.0-flash-exp",
    messages = [
        {
            "role": "user",
            "content": "this is a test request, write a short poem"
        }
    ],
    
)

print(response)

Let me know if you need further help, thanks.

[tan-yong-sheng]

Hi, just wanna check if the method above works to use litellm proxy on localhost (in linux environment)? Let me know if you need any help.

[jhrozek]

Hi, just wanna check if the method above works to use litellm proxy on localhost (in linux environment)? Let me know if you need any help.

Thank you for the pointers. We merged the the openrouter patches last week and the litellm support should be along the same lines. I got sidetracked by other work, but I'll try to get this back on track soon!

If other codegate developers want to take this issue from me, feel free to.

[jhrozek]

@rdimitrov

[rdimitrov]

hey, @tan-yong-sheng 👋

So I've tried an example setup locally and I think I was able to reproduce the issue.

TLDR: The main issue is that our internal dependency on litellm (not the proxy, but the library which we are in the works of removing 🥳 ) re-routes on its own the model provider URL based on the model name. We are expecting to release a fix for this in the next 1 or 2 releases.

The setup that I have is the following:

• Cline: Configured to talk to CodeGate's OpenAI compatible API (note the model name is what LiteLLM Proxy expects it)

• CodeGate: Configured to talk to LiteLLM's Proxy (running locally on localhost:4000) by overwriting the OpenAI URL flag

<codegate serve command> --openai-url=http://localhost:4000

• LiteLLM Proxy: Configured to talk to a local Ollama server (but can be configured to other providers as well)

model_list:
  - model_name: phi4
    litellm_params:
      model: ollama/phi4 # LiteLLM expects the models to be prefixed by their provider
      api_base: http://localhost:11434 # This is where Ollama is serving

general_settings:
  master_key: sk-1234 # That secret is used in the Cline config for API key

The issue:

• This should work, but our internal use of LiteLLM as a library tries to be smart in this case and once it sees the prefixed model name (ollama/phi4) it overwrites the provider URL from what we configured above via --openai-url=http://localhost:4000 to http://localhost:11434 instead essentially bypassing the LiteLLM Proxy.
• It does the same for OpenAI or other providers too, which is why using your LiteLLM key doesn't work, but using the OpenAI key for example works.

Solution:

• The good news is @blkt and @jhrozek are already in the process of replacing our internal use of LiteLLM with our own implementation which will simplify the codebase on one side, but also resolve this and other issues caused by LiteLLM.

I'll ping you once this is released 🙏 The work is almost complete so it should drop in some of the next releases.

[tan-yong-sheng]

Hi, sorry for missing this and being for reply.

I'll ping you once this is released 🙏 The work is almost complete so it should drop in some of the next releases.

Sure, thanks a lot.