litellm as service

[sfiodorov]

Hello, we are currently searching for an AI abstraction layer that includes features such as user management and rate limiting. We have heard that Litellm is a great solution for our needs. However, our team is not familiar with the Python stack. I was wondering if these same features can be accessed via a Docker container?
If not, if there is a SAAS option?

[krrishdholakia]

Hey @sfiodorov

Does the docker image not work? https://github.com/BerriAI/litellm/blob/main/Dockerfile

Here's the docs: https://docs.litellm.ai/docs/proxy/quick_start#quick-start-docker-image-github-container-registry

If you're looking for a hosted solution, I'd recommend checking out Helicone

[ishaan-jaff]

Hi @sfiodorov we'd love to help, sharing a link to our calendar if you'd like to meet and discuss this: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

[sfiodorov]

Hi.
Thanks for answering.
Im not sure if the docker version contains the same set of features as a library.
For example it seems that there is no client-level budget management and rate limiting.

[krrishdholakia]

@sfiodorov the docker just builds the library -

litellm/Dockerfile

Line 16 in ed07012

COPY . /app

Here's how to track spend/user - https://docs.litellm.ai/docs/proxy/virtual_keys

a call would help us get context

[sfiodorov]

Thanks.
Track spend is not what we are looking for.
We need to limit client rates and budgets. Something like here: https://docs.litellm.ai/docs/budget_manager
But yes, it is better to have a dedicated call. I will check with my team and come back.
Thanks again for quick response

[krrishdholakia]

Hey @sfiodorov we now enable client budget limits - https://docs.litellm.ai/docs/proxy/virtual_keys#set-budgets

Let me know if this is what you were looking for. Also happy to hop on a call to discuss.

Attaching my socials so we can set up a dedicated support channel for your team:

• LinkedIn: https://www.linkedin.com/in/krish-d/
• Discord: Krrish#8748

[sfiodorov]

Hey, that sounds good.
We will evaluate it ASAP.
Thanks for letting us know.