Resolving 'No Module Named vllm' error for use of litellm as proxy for vllm

[yet-another-schizotypic]

I'm encountering an issue integrating litellm with vllm in proxy mode. My objective is to configure litellm to receive OpenAI-compatible requests and then forward them to a local vllm instance.

TL;DR:
["<class 'litellm.exceptions.APIConnectionError'>Status: 500Message: NoneVLLMException - No module named 'vllm'Full exceptionVLLMException - No module named 'vllm'"]

To ensure that the problem isn't with vllm, I've double-checked its functionality.
Run command:

python -O -u -m vllm.entrypoints.openai.api_server --host=127.0.0.1 --port=9000 --model=/workspace/vllm/models/starsnatched-MemGPT/ --served-model-name=vllm/memgpt

vllm logs:

INFO:     Started server process [902]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:9000 (Press CTRL+C to quit)

vllm test query:

curl -X POST http:/127.0.0.1:9000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "vllm/memgpt", "messages": [{"role": "user", "content": "Hello. Tell me a joke, please."}]}'

#The response:
{"id":"cmpl-d04ae59f3b66446bb5b2f0691df75c01","object":"chat.completion","created":19801817,"model":"vllm/memgpt","choices":[{"index":0,"message":{"role":"assistant","content":" Of course! Here's a classic one for you:\n\nWhy don't scientists trust atoms?\n\nBecause they make up everything!\n\nI hope that brought a smile to your face. Do you have any other requests or questions?"},"finish_reason":"stop"}],"usage":{"prompt_tokens":18,"total_tokens":70,"completion_tokens":52}}

vllm logs after the query:

INFO 01-31 22:17:38 async_llm_engine.py:431] Received request cmpl-d04ae59f3b66446bb5b2f0691df75c01: prompt: '<s>[INST] Hello. Tell me a joke, please. [/INST]', prefix_pos: None,sampling params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=1.0, top_k=-1, min_p=0.0, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=32750, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True), prompt token ids: [1, 1, 733, 16289, 28793, 22557, 28723, 15259, 528, 264, 13015, 28725, 4665, 28723, 733, 28748, 16289, 28793], lora_request: None.
INFO 01-31 22:17:38 llm_engine.py:877] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%
INFO 01-31 22:17:39 async_llm_engine.py:110] Finished request cmpl-d04ae59f3b66446bb5b2f0691df75c01.
INFO:     127.0.0.1:48168 - "POST /v1/chat/completions HTTP/1.1" 200 OK

Now, turning back to litellm. I use the following command to run it:

litellm --host 127.0.0.1 --port 8000 --detailed_debug --config /workspace/litellm/config.yaml

My litellm config is very simple:

cat /workspace/litellm/config.yaml

Here is the contents:

model_list:
  - model_name: memgpt # user-facing model alias
    litellm_params: # model-specific parameters
      model: vllm/memgpt # the model's identifier
      api_base: https://127.0.0.1:9000 # the API base URL

litellm_settings: # module level litellm settings
  drop_params: True
  set_verbose: True

It appears that there are no issues when starting litellm, as indicated by its logs:

22:23:54 - LiteLLM Proxy:DEBUG:  setting litellm.drop_params=True
22:23:54 - LiteLLM Proxy:DEBUG:  setting litellm.set_verbose=True
LiteLLM: Proxy initialized with Config, Set models:
    memgpt
22:23:54 - LiteLLM Router:DEBUG:
Initialized Model List [{'model_name': 'memgpt', 'litellm_params': {'model': 'vllm/memgpt', 'api_base': 'https://127.0.0.1:9000'}, 'model_info': {'id': 'e3ef9a16-921d-4813-a038-c2f71cc06f5f'}}]
22:23:54 - LiteLLM Router:DEBUG: Intialized router with Routing strategy: simple-shuffle

22:23:54 - LiteLLM Proxy:DEBUG: INITIALIZING LITELLM CALLBACKS!
LiteLLM Proxy: INITIALIZING LITELLM CALLBACKS!
22:23:54 - LiteLLM:DEBUG: callback: <litellm.proxy.hooks.parallel_request_limiter.MaxParallelRequestsHandler object at 0x7f5fc3c31a80>
callback: <litellm.proxy.hooks.parallel_request_limiter.MaxParallelRequestsHandler object at 0x7f5fc3c31a80>
22:23:54 - LiteLLM:DEBUG: callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x7f5fc3a32260>>
callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x7f5fc3a32260>>
22:23:54 - LiteLLM:DEBUG: callback: <litellm.proxy.hooks.max_budget_limiter.MaxBudgetLimiter object at 0x7f5fc3c31ab0>
callback: <litellm.proxy.hooks.max_budget_limiter.MaxBudgetLimiter object at 0x7f5fc3c31ab0>
22:23:54 - LiteLLM:DEBUG: callback: <litellm.proxy.hooks.cache_control_check.CacheControlCheck object at 0x7f5fc3c31ae0>
callback: <litellm.proxy.hooks.cache_control_check.CacheControlCheck object at 0x7f5fc3c31ae0>
22:23:54 - LiteLLM:DEBUG: callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x7f5fc3a6c5e0>>
callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x7f5fc3a6c5e0>>
22:23:54 - LiteLLM Proxy:DEBUG: prisma client - None
22:23:54 - LiteLLM Proxy:DEBUG: custom_db_client client - None
22:23:54 - LiteLLM Proxy:DEBUG: custom_db_client client None. Master_key: None
/workspace/litellm/litellm-venv/lib/python3.10/site-packages/tzlocal/unix.py:193: UserWarning: Can not find any timezone configuration, defaulting to UTC.
  warnings.warn("Can not find any timezone configuration, defaulting to UTC.")
INFO:     Application startup complete.

However, when I attempt to execute an OpenAI-compatible POST query on litellm, I encounter the following error:

curl -X POST http:/127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "vllm/memgpt", "messages": [{"role": "user", "content": "Hello. Tell me a joke, please."}]}'

#The response:
{"error":{"message":"VLLMException - No module named 'vllm'","type":null,"param":null,"code":500}}

The logs from litellm indicate these specific errors:

22:26:12 - LiteLLM:DEBUG: Request to litellm:
Request to litellm:
22:26:12 - LiteLLM:DEBUG: litellm.acompletion(model='vllm/memgpt', api_base='https://127.0.0.1:9000', messages=[{'role': 'user', 'content': 'Hello. Tell me a joke, please.'}], caching=False, client=None, timeout=None, proxy_server_request={'url': 'http://127.0.0.1:8000/v1/chat/completions', 'method': 'POST', 'headers': {'host': '127.0.0.1:8000', 'user-agent': 'curl/7.81.0', 'accept': '*/*', 'content-type': 'application/json', 'content-length': '101'}, 'body': {'model': 'vllm/memgpt', 'messages': [{'role': 'user', 'content': 'Hello. Tell me a joke, please.'}]}}, metadata={'user_api_key': None, 'user_api_key_user_id': None, 'user_api_key_metadata': {}, 'headers': {'host': '127.0.0.1:8000', 'user-agent': 'curl/7.81.0', 'accept': '*/*', 'content-type': 'application/json', 'content-length': '101'}, 'endpoint': 'http://127.0.0.1:8000/v1/chat/completions', 'model_group': 'vllm/memgpt', 'deployment': 'vllm/memgpt', 'caching_groups': None}, request_timeout=600, model_info={'id': 'e3ef9a16-921d-4813-a038-c2f71cc06f5f'}, max_retries=0)
litellm.acompletion(model='vllm/memgpt', api_base='https://127.0.0.1:9000', messages=[{'role': 'user', 'content': 'Hello. Tell me a joke, please.'}], caching=False, client=None, timeout=None, proxy_server_request={'url': 'http://127.0.0.1:8000/v1/chat/completions', 'method': 'POST', 'headers': {'host': '127.0.0.1:8000', 'user-agent': 'curl/7.81.0', 'accept': '*/*', 'content-type': 'application/json', 'content-length': '101'}, 'body': {'model': 'vllm/memgpt', 'messages': [{'role': 'user', 'content': 'Hello. Tell me a joke, please.'}]}}, metadata={'user_api_key': None, 'user_api_key_user_id': None, 'user_api_key_metadata': {}, 'headers': {'host': '127.0.0.1:8000', 'user-agent': 'curl/7.81.0', 'accept': '*/*', 'content-type': 'application/json', 'content-length': '101'}, 'endpoint': 'http://127.0.0.1:8000/v1/chat/completions', 'model_group': 'vllm/memgpt', 'deployment': 'vllm/memgpt', 'caching_groups': None}, request_timeout=600, model_info={'id': 'e3ef9a16-921d-4813-a038-c2f71cc06f5f'}, max_retries=0)
22:26:12 - LiteLLM:DEBUG:
22:26:12 - LiteLLM:DEBUG: Initialized litellm callbacks, Async Success Callbacks: [<litellm.proxy.hooks.parallel_request_limiter.MaxParallelRequestsHandler object at 0x7f5fc3c31a80>, <litellm.proxy.hooks.max_budget_limiter.MaxBudgetLimiter object at 0x7f5fc3c31ab0>, <litellm.proxy.hooks.cache_control_check.CacheControlCheck object at 0x7f5fc3c31ae0>]
Initialized litellm callbacks, Async Success Callbacks: [<litellm.proxy.hooks.parallel_request_limiter.MaxParallelRequestsHandler object at 0x7f5fc3c31a80>, <litellm.proxy.hooks.max_budget_limiter.MaxBudgetLimiter object at 0x7f5fc3c31ab0>, <litellm.proxy.hooks.cache_control_check.CacheControlCheck object at 0x7f5fc3c31ae0>]
22:26:12 - LiteLLM:DEBUG: callback: <litellm.proxy.hooks.parallel_request_limiter.MaxParallelRequestsHandler object at 0x7f5fc3c31a80>
callback: <litellm.proxy.hooks.parallel_request_limiter.MaxParallelRequestsHandler object at 0x7f5fc3c31a80>
22:26:12 - LiteLLM:DEBUG: callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x7f5fc3a32260>>
callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x7f5fc3a32260>>
22:26:12 - LiteLLM:DEBUG: callback: <litellm.proxy.hooks.max_budget_limiter.MaxBudgetLimiter object at 0x7f5fc3c31ab0>
callback: <litellm.proxy.hooks.max_budget_limiter.MaxBudgetLimiter object at 0x7f5fc3c31ab0>
22:26:12 - LiteLLM:DEBUG: callback: <litellm.proxy.hooks.cache_control_check.CacheControlCheck object at 0x7f5fc3c31ae0>
callback: <litellm.proxy.hooks.cache_control_check.CacheControlCheck object at 0x7f5fc3c31ae0>
22:26:12 - LiteLLM:DEBUG: callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x7f5fc3a6c5e0>>
callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x7f5fc3a6c5e0>>
22:26:12 - LiteLLM:DEBUG: self.optional_params: {}
self.optional_params: {}
22:26:12 - LiteLLM:DEBUG: litellm.cache: None
litellm.cache: None
22:26:12 - LiteLLM:DEBUG: kwargs[caching]: False; litellm.cache: None
kwargs[caching]: False; litellm.cache: None
22:26:12 - LiteLLM:DEBUG: kwargs[caching]: False; litellm.cache: None
kwargs[caching]: False; litellm.cache: None
22:26:12 - LiteLLM:DEBUG:
LiteLLM completion() model= memgpt; provider = vllm
22:26:12 - LiteLLM:DEBUG:
LiteLLM: Params passed to completion() {'functions': None, 'function_call': None, 'temperature': None, 'top_p': None, 'stream': None, 'max_tokens': None, 'presence_penalty': None, 'frequency_penalty': None, 'logit_bias': None, 'user': None, 'response_format': None, 'seed': None, 'tools': None, 'tool_choice': None, 'max_retries': 0, 'logprobs': None, 'top_logprobs': None, 'custom_llm_provider': 'vllm', 'model': 'memgpt', 'n': None, 'stop': None}
22:26:12 - LiteLLM:DEBUG:
LiteLLM: Non-Default params passed to completion() {'max_retries': 0}
22:26:12 - LiteLLM:DEBUG: self.optional_params: {'max_retries': 0}
self.optional_params: {'max_retries': 0}
22:26:12 - LiteLLM:DEBUG: Logging Details: logger_fn - None | callable(logger_fn) - False
Logging Details: logger_fn - None | callable(logger_fn) - False
22:26:12 - LiteLLM:DEBUG: Logging Details LiteLLM-Failure Call
Logging Details LiteLLM-Failure Call
22:26:12 - LiteLLM Router:DEBUG: Attempting to add e3ef9a16-921d-4813-a038-c2f71cc06f5f to cooldown list. updated_fails: 1; self.allowed_fails: 0
22:26:12 - LiteLLM:DEBUG: get cache: cache key: 22-26:cooldown_models; local_only: False
get cache: cache key: 22-26:cooldown_models; local_only: False
22:26:12 - LiteLLM:DEBUG: in_memory_result: None
in_memory_result: None
22:26:12 - LiteLLM:DEBUG: get cache: cache result: None
get cache: cache result: None
22:26:12 - LiteLLM Router:DEBUG: adding e3ef9a16-921d-4813-a038-c2f71cc06f5f to cooldown models
22:26:12 - LiteLLM:DEBUG: set cache: key: 22-26:cooldown_models; value: ['e3ef9a16-921d-4813-a038-c2f71cc06f5f']
set cache: key: 22-26:cooldown_models; value: ['e3ef9a16-921d-4813-a038-c2f71cc06f5f']
22:26:12 - LiteLLM Router:DEBUG:
EXCEPTION FOR DEPLOYMENTS
22:26:12 - LiteLLM Router:DEBUG: {'vllm/memgpt': ["<class 'litellm.exceptions.APIConnectionError'>Status: 500Message: NoneVLLMException - No module named 'vllm'Full exceptionVLLMException - No module named 'vllm'"]}
22:26:12 - LiteLLM Router:DEBUG: Model vllm/memgpt had 1 exception
22:26:12 - LiteLLM:DEBUG: Custom Logger - final response object: None
Custom Logger - final response object: None
22:26:12 - LiteLLM Router:DEBUG: Attempting to add e3ef9a16-921d-4813-a038-c2f71cc06f5f to cooldown list. updated_fails: 1; self.allowed_fails: 0
22:26:12 - LiteLLM:DEBUG: get cache: cache key: 22-26:cooldown_models; local_only: False
get cache: cache key: 22-26:cooldown_models; local_only: False
22:26:12 - LiteLLM:DEBUG: in_memory_result: None
in_memory_result: None
22:26:12 - LiteLLM:DEBUG: get cache: cache result: None
get cache: cache result: None
22:26:12 - LiteLLM Router:DEBUG: adding e3ef9a16-921d-4813-a038-c2f71cc06f5f to cooldown models
22:26:12 - LiteLLM:DEBUG: set cache: key: 22-26:cooldown_models; value: ['e3ef9a16-921d-4813-a038-c2f71cc06f5f']
set cache: key: 22-26:cooldown_models; value: ['e3ef9a16-921d-4813-a038-c2f71cc06f5f']
22:26:12 - LiteLLM Router:DEBUG:
EXCEPTION FOR DEPLOYMENTS
22:26:12 - LiteLLM Router:DEBUG: {'vllm/memgpt': ["<class 'litellm.exceptions.APIConnectionError'>Status: 500Message: NoneVLLMException - No module named 'vllm'Full exceptionVLLMException - No module named 'vllm'"]}

Is it possible to use vllm as a backend for litellm, and if so, how can it be configured?

[ishaan-jaff]

@yet-another-schizotypic thanks for this issue

Is it possible to use vllm as a backend for litellm, and if so, how can it be configured?

Yes it is possible, is your vllm backend OpenAI compatible ?

We can hop on a call and help get you setup with litellm proxy
Sharing my calendly for your convenience: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat