[Feature]: Custom Model Endpoint Support

[georgeseifada]

The Feature

Let's say OpenAI comes out with GPT-5 and it has a totally different API. Or Amazon releases a model called Bezos-AI. Users should be able to easily add support for that model's API, to seamlessly integrate it with routing, fallbacks, etc without waiting for it to be officially supported in the library.

Motivation, pitch

The library should be as extensible as possible for users to be able to rely on it long-term.

Twitter / LinkedIn details

No response

[krrishdholakia]

Already supported via openai compatible endpoints support - https://docs.litellm.ai/docs/providers/openai_compatible

Anything additional you require here?

[georgeseifada]

I'm not sure how that would help with an API that's different?

For example, let's say we have a new LLM Embedding API that has:

• A new parameter called amplify (I just made it up for the example)
• Takes in the input like so: {"text": "Hi, my name is Bob.", "metadata": "source": "audio", "language": "english"}

How can I use the new LLM Embedding API with LiteLLM, with all the standard functionality of routing, fallbacks, etc?

[krrishdholakia]

• non-openai params are passed straight through
• if the format is not supported, you could either file an issue to add support, or write your endpoint to fit the openai format (and therefore connect to the proxy as an openai-compatible endpoint like vllm, localai, etc. do).

[georgeseifada]

Got it. This should be sufficient. Thanks for the explanation! 👍